This post is also available in Dutch.
The COVID-19 pandemic has led to an overwhelming flood of (mis)information related to the virus. Here are some tips on how to evaluate the scientific claims you encounter.
The sheer volume of information about the novel coronavirus can be overwhelming and is expanding every day. Scientists and policymakers race to answer important questions: Do recovered people stay immune? Are children likely to transmit the virus? How long until an effective vaccine will be ready? Answers to questions like these will have huge implications for how we move forward as a society, but also for each individual trying to make best decisions to stay safe. Yet, as we search for answers, we are met by a mish-mash of often contradictory information, which can leave us confused, anxious, and possibly vulnerable to misinformation.
The Problem
The COVID-19 pandemic has brought into sharp relief the larger issue of scientific literacy in society. The internet bombards us with scientific claims that are amplified through social media, but there is often no quality filter. Many people understandably feel they don’t have the time or expertise to wade through it all to tease out the truth. At worst, those with nefarious interests can exploit this vulnerability for their own purposes to manipulate public opinion; but even with good intentions we can be led astray by our very human tendencies like confirmation bias and pluralistic ignorance. The result is a disconnect between scientific reality and popular beliefs in society, as in the cases of climate change denial and the anti-vaccine movement.
This problem has become especially stark during the pandemic: Because the virus is novel, scientists are still learning about it and this uncertainty makes it even easier for people to feel confused or even to reject the evidence we do have in favour of their preferred version of facts (see this blog for more information).
So, what can we do? The next section outlines some tips for how you can think critically about scientific claims.
Critical thinking tips
We cannot all be virologists or epidemiologists, but there are some universal principles that can be applied to most scientific claims to help judge their credibility. The following are some questions you can ask yourself.
1. Is the source trustworthy?
When a headline is shared on social media, we may be tempted to take it at face value, especially when it was shared by someone we trust. To better judge the credibility of the information, we need to dig a bit deeper and ask ourselves: Is this a reputable news source? Is the claim based on scientific research or anecdotal evidence? Has the research been peer-reviewed by other scientists? If the claim was based on an anecdote, be very skeptical. The most credible sources are peer-reviewed scientific papers[i]. Popular media reports on scientific studies help communicate these findings to a larger audience, but even these reports can sometimes mislead or exaggerate the strength of the scientific findings. Be wary of sensational language and use #2 and 3 below to help judge whether the claims are justified.
2. Is the claim supported by multiple studies?
Rarely can we make strong conclusions from a single study in science. Replication, the repetition of a given experiment to see if the results remain the same, is critical to the research process. Many factors can affect the results of any single study: the subjects who were included or left out, the researchers performing the study, or even the geographical location. Indeed, sometimes seemingly convincing results can arise just by chance. So, it is important to show that the same results arise over multiple studies[ii].
3. What is the quality of the research?
In an ideal situation, we would have many studies that show the same result; however we do not always have this luxury. Thus, we should also judge the quality of an individual study or claim, to know how much weight to give its findings. Here are three important factors to take into account:
– Sample size: This is the number of people (or animals) who were included. The bigger the sample, the more trust we should put in the study.
– Control Group: If a researcher gives a new drug to 10 patients and they all recover quickly, it may be tempting to conclude that the new drug is effective. However, it’s possible that these patients would have recovered quickly anyways. To show that the drug has an added benefit, the patients who received the drug need to be compared with a control group of patients who receive the usual treatment, but not the new drug. Be skeptical of any study with no control group claiming that a treatment is effective.
– Confounds and alternative explanations: A confound is a potential alternative explanation for a scientific claim. When reading the interpretation of a new finding, try to think of other plausible causes leading to that result (see more about it in this blog)[iii].
4. What are your own motivations?
This question may be the hardest to answer. When evaluating scientific claims, we also need to be critical of ourselves. We are all susceptible to confirmation bias, the tendency to seek out and rely on information that follows our beliefs and matches our identity. If you find yourself paying attention to media that you instinctively agree with and ignoring those that contradict your beliefs, make an effort to diversify your media consumption. While evaluating scientific claims, try to be just as critical of claims that support your beliefs as you are of those that challenge them. Try to be curious and open to the possibility that you could be wrong!
Thinking critically about science doesn’t come naturally to most – it is a skill that needs to be practiced and honed. It is also important to not get discouraged by the lack of clear answers to all of our questions, because good science takes time and repetition. If the coronavirus is a puzzle, then each study being done is a piece snapping into place. And, indeed, the urgency of the situation means that the puzzle is coming together quickly. Yes, sometimes we might lay down a piece in the wrong place, but with more research we will eventually catch our mistakes and get more solid answers to our burning questions. Getting into the habit of thinking critically about science will help us make well-informed decisions and inoculate us against misinformation.
Author: Rebecca Calcott
Buddy: João Guimarães
Editor: Mónica Wagner
Translator: Ellen Lommerse
Translation Editor: Wessel Hieselaar
Photo by Obi Onyeador on Unsplash
[i] In the spirit of openness, many scientists also share their research prior to the peer review process in what is called a preprint. Preprints help speed up the process of sharing research, and should not be discounted, but it is important to be aware that some of their conclusions might change following peer review.
[ii] Ideally, evidence would also come from different kinds of studies. If an animal study, clinical trial, and large-scale epidemiological survey all suggest the same conclusion, we can be more confident in the claim.
[iii] Just because a study doesn’t satisfy all of these criteria, doesn’t make it “bad”. In some situations, it isn’t possible or ethical to run the perfect experiment. However, it does mean that we should take the limitations into account when we weight its conclusions.
