The importance of understanding basic concepts of research design and statistics in everyday life.
We are almost at the end of 2021, and like many of us, I’m sure I’m still processing the culmination of horrors that was 2020 and am in no way prepared to accept that 2022 is at our doorstep. So, let’s do a brief recap of what we saw in 2020. Parts of Australia and the west coast of the United States burned uncontrollably for months. Covid-19 and its rippling effects spread across the world. We had two impeachment charges, an acquittal, and an election. The Pentagon released documents and videos validating the existence of UFOs. Murder hornets became part of our vocabulary. Megan and Harry decided they didn’t want to be royals anymore. An obscene number of political, Hollywood, and YouTube star scandals emerged. It seemed like every day we were, and continue to be, bombarded by some new bit of salacious information that we are forced to analyze on the fly.
During this time, I realized how thankful I am for everything I learned about research design and statistics. I found myself applying things I learned in classes like Psych 144. That knowledge enabled me to think critically about the information presented to me on the news, on social media, and in conversations with peers. I found that I could make informed decisions about the factuality of statements by finding a source and asking myself a few questions.
- What’s the source?
Research-heavy disciplines often rely on peer-reviewed data as the burden of proof for the factuality and reliability of a statement. In general, one may feel better about sources that come from websites that end in .edu, .org, or .gov. However, if the source is anything but one of these, it’s recommended that you take the information with a grain of salt. References often found outside the peer-reviewed arena might discuss opinion rather than scientifically proven statements.
- Who was involved?
This question has two parts. First, we want to know not only who did the research but also what the descriptive statistics were for the research participants. The first part of the question investigates what, if any, bias may exist within the research. For example, if a person from Coca-Cola presents data on Coca-Cola, we may assume that the information being offered might be favorable to Coca-Cola. On the other side, if a person from Pepsi is presenting data on Coca-Cola, bias may also be found. The person or persons analyzing the data should be transparent about any stake they may have in the research. Knowing who is reporting that data can help a researcher decide on the validity that source’s claims.
The second question asks about the sample’s descriptive statistics; sometimes this is reported as demographics. The descriptive statistics tell us how many people participated in the research and usually includes general information like age, race, and gender. However, demographics can also contain more information, including income, education level, and marital status. These descriptive stats are essential because research findings are only as robust as the population they represent. For example, if I were testing a product meant for all young adults aged 8 to 18, but only collected data from a sample consisting of young adults from one socioeconomic class that was homogenous ethnicity-wise and between the ages of 12 to 14, I would not be accurately representing my specified population. Whatever data came from that sample group could not be easily or accurately applied.
Having an idea of the demographics can help us decide how applicable the information is to our family, community, or the United States as a whole. For example, if a piece of research claims that something applies broadly to people, but the descriptive statistics are very narrow, then it’s improbable that the claims are valid. They may be valid within that sample, but it wouldn’t seem likely to apply to the overall population.
- What is the statistical significance of the reported data?
Let’s talk about p-value and statistical significance. P-value describes the probability that the results of a study could be entirely random or wrong. A low p-value is good because it means there is a low chance that the reported results happened because of luck or chance rather than sound science. The threshold set for most studies is p < .05, meaning the p-value is less the 5% or a 1-in-20 chance of the results being wrong or random. P-values less than .05 are considered statistically significant; however, some scientists set their research p-value at a lower threshold of 0.001, meaning that there is a 1-in-1000 chance of being wrong, or that the results were random. Results that fall below this p-value are deemed statistically highly significant.
We can say whatever we want about our research as eloquently or provocatively as we can, but if the p-value is more .05, throw the whole study in the bin. Okay, that’s a bit dramatic, but if we review a study to determine if the hypothesis or research claim is true or false, we need to look no further than the p-value. The research may have provided some interesting talking and jumping-off points for new research, but if it did not reach statistical significance, it did not affirm their hypothesis.
Critical thinking is crucial in our current socio-political medical climate. Not every statement is a verifiable fact. How we evaluate cited research that supports personal or political views can help keep us calm and think rationally in an environment that seems hell-bent on constantly shaking us up. When we hear something salacious, remember to stop and consider these questions: Where did the research come from to support this statement? Who was involved in the research? Did the research achieve statistical significance? If you do that first, you’ll be in a much better position to assess the claim.
By Courtney Hill
fresno state graduate writing studio 