Early in the COVID-19 pandemic, researchers flooded journals with studies on the then-novel coronavirus. Many publications have streamlined the peer review process for COVID-19 papers, while acceptance rates remained relatively high. The assumption was that policymakers and the public would be able to identify valid and useful research from a very large amount of rapidly disseminated information.
However, in my review of 74 COVID-19 articles published in 2020 in the top 15 generalist public health journals in Google Scholar, I found that many of these studies used poor quality methods. Several other reviews of studies published in medical journals have also found that much early COVID-19 research used poor research methods.
Some of these articles have been cited many times. For example, the most cited public health publication on Google Scholar used data from a sample of 1,120 people, mostly well-educated young women, most of whom were recruited via social media over three days. Findings based on a small, self-selected convenience sample cannot be generalized to a broader population. And since the researchers conducted more than 500 analyzes of the data, many of the statistically significant results are likely chance events. However, this study has been cited more than 11,000 times.
A highly cited article means that many people have mentioned it in their own work. But a high number of citations is not strongly related to the quality of research, because researchers and journals can game and manipulate these statistics. Frequently citing low-quality research increases the likelihood that poor evidence will be used to inform policy, further eroding public trust in science.
Methodology is important
I am a public health researcher with a long-standing interest in research quality and integrity. This interest lies in the belief that science has helped solve important social and public health problems. Unlike the anti-science movement that spreads disinformation about such successful public health measures as vaccines, I believe that rational criticism is fundamental to science.
The quality and integrity of research largely depends on its methods. Each type of research design must have certain characteristics to provide valid and useful information.
For example, researchers have known for decades that studies of the effectiveness of an intervention require a control group to know whether any observed effects can be attributed to the intervention.
Systematic reviews that bring together data from existing studies should describe how investigators identified which studies to include, assessed their quality, extracted the data, and preregistered their protocols. These features are necessary to ensure that the review covers all available evidence and makes it clear to the reader what is worth paying attention to and what is not.
Certain types of studies, such as one-time surveys of convenience samples that are not representative of the target population, collect and analyze data in a way that does not allow researchers to determine whether a variable caused a particular outcome.
All research designs include standards that researchers can refer to. But compliance with standards slows down research. Having a control group doubles the amount of data that needs to be collected, and identifying and thoroughly reviewing each study on a topic takes more time than superficially reviewing some studies. Representative samples are more difficult to generate than convenience samples, and collecting data at two points in time is more work than collecting data all at once.
Studies comparing COVID-19 articles to non-COVID-19 articles published in the same journals found that COVID-19 articles tended to have lower quality methods and were less likely to adhere to reporting standards than non-COVID -19 articles. COVID-19 papers rarely include predetermined hypotheses and plans for how they would report their findings or analyze their data. This meant that there were no safeguards against dredging the data to find ‘statistically significant’ results that could be selectively reported.
Such methodological issues were likely overlooked during the significantly shortened peer review process for COVID-19 papers. One study estimated the average time from submission to acceptance of 686 articles on COVID-19 at 13 days, compared to 110 days in the 539 pre-pandemic articles from the same journals. In my research, I found that two online journals that published a very large number of methodologically weak COVID-19 articles had a peer review process of about three weeks.
Publish or perish culture
These quality control issues were already present before the COVID-19 pandemic. The pandemic has simply sent them into overdrive.
Journals tend to favor positive, “new” findings: that is, results that demonstrate a statistical relationship between variables and supposedly identify something previously unknown. Because the pandemic was in many ways new, it provided an opportunity for some researchers to make bold claims about how COVID-19 would spread, what its effects on mental health would be, how it could be prevented, and how the disease could be treated.