In anticipation of Peer Review Week 2020, and in consideration of the theme ‘Trust in Peer Review’, Delta Think surveyed broadly across a two-week period in August to determine whether COVID-19 has impacted perceptions of preprints. The survey was open to everyone – from Publishers to Librarians to Researchers to the Lay Public with an interest in scientific output.
We entered into the survey with open minds, though a few recurring themes circulating in news outlets were on our minds including: the idea that both traditional journal submissions and articles posted to preprint servers have spiked in the last six months, owing to COVID-19; and the hunch that perhaps increased traffic indicated a correlated spike in trust, or conversely, that challenged findings making their way into mainstream news might have reduced trust. Was the reputation associated with preprints in jeopardy? Or could they play a critical role in speeding up science at a time so critical for the global health community? With these early loose hypotheses and questions, we launched the survey.
We explore the survey results below. In the spirit of transparency, we have posted the underlying data to Zenodo. It’s also important to note that while this was a quick survey, and is not meant to include a representative sample from the participating audiences, it provides interesting top-level findings and points to areas that may be ripe for a further investigation or a deeper dive in future.
Survey Population Demographics
In total, we collected 371 responses to the survey. Of this population:
Of this multi-disciplinary population:
Note: Respondents could select multiple disciplines.
We limited the survey to respondents who were already familiar with preprints by offering an initial question to gauge familiarity. Those who were unfamiliar exited the survey at this point (6% of those starting the survey).
Trust Balances
When it comes to the question of trust in preprints, the average trust level came in almost exactly at the midpoint at 51 out of 100, where 100 indicates the highest trust and 0 indicates no trust. But a few notable variances stand out in the details. Taking a closer look at Authors vs. Non-Authors, we find Authors overall more trusting of preprints, and if we look by role, those who identified as Researchers are the most trusting group of all. Publishers, on the other hand, are the least trusting.
Figure 1: All responses to, “How much do you trust the findings presented in a Preprint manuscript?”
COVID-19 Impact on Behavior & Trust
Next we asked how the COVID pandemic has changed their perspectives on preprints. About 60% of all respondents reported that their trust has remained unchanged despite COVID-19. Still, Non-Authors were more likely than Authors to report a decrease in trust, and Publishers were more likely than Librarians and Researchers to report a decrease in trust.
Looking at specialties, Arts & Humanities and Social Science affiliated respondents were slightly more likely to report a trust increase, while Engineering, Physical Science & Chemistry, and Biological Sciences and Medicine were more likely to report a decrease in trust.
Figure 2: Specialty responses to: “In light of the COVID-19 pandemic, has your trust of Preprints increased, decreased, or remained the same in the last year?”
When asked why their trust increased, those with increased levels primarily reported that speed became more important than pre-publication peer review.
When respondents reported that trust had decreased, overall they pointed to a perceived quality problem and worried that many preprints being published now do not present well-constructed studies or findings. They also noted that traditional pre-publication review is necessary now more than ever.
Overall Benefits of Preprints
The greatest benefits of preprints, according to all respondents, are:
A deeper dive into filters by author status, specialty, role, and region mostly showed speed and openness as the top two benefits. Different groups, however, picked various options as their third most important benefit. Authors wanted increased visibility for their research; Engineering, Physical Sciences & Chemistry chose increased cooperation in research across labs; Biological Sciences and Medicine wanted to facilitate open peer review.
Some interesting findings appeared in the regional breakouts as well. Those from the Rest of World countries selected increased visibility of their research as well as the researcher as the #1 benefit, and East Asia appreciated that it was cost-efficient or free more so than other regions.
Figure 3: Region specific breakdown of top Preprint benefits (respondents could chose up to three)
Conclusion
Nearly seven months into the COVID-19 pandemic, trust in Preprints mostly holds steady for the majority of respondents to this survey, although respondents in Engineering, Physical Science & Chemistry, and Biological Sciences and Medicine had higher levels of doubt, and those in Arts & Humanities and Social Sciences were more likely to express an increase in trust.
And overall, Authors & Researchers were the most positive about preprints of all the groups.
Shoring up preprint review quality, increasing preprint reviews, and increasing awareness of checks already in place on preprint servers, could go a long way toward increasing trust in preprints and growing activity. While the speed and openness of preprints emerge in the data as clear, universally agreed benefits, improving review seems the prudent path forward toward enticing the research community to wholeheartedly embrace these benefits.
Delta Think wishes to thank all those who participated in the survey. For each survey completed, we donated $5 to Project HOPE, up to $1,000.
This article is © 2020 Delta Think, Inc. It is published under a Creative Commons Attribution-NonCommercial 4.0 International License. Please do get in touch if you want to use it in other contexts – we’re usually pretty accommodating.