Coffee can cause cancer. Wait, coffee lowers cancer rates.
Eggs are totally fine to eat. No, eggs are bad for you.
At first glance it might appear that scientists cannot make up their minds. However, much of this perceived inconsistency lies in a pervasive misperception of the usefulness of a broad category of science research known as observational studies.
Science research can be broken down into two types of research: Observational data and experimental data. Experimental data is much more robust but more difficult (and expensive) to do. It can be difficult to understand why randomization and equipoise is so important in experimental studies.
Observational data is much easier (and cheaper) to complete, but much less helpful. Observational data, like anecdotes in journalism tend to be more attention grabbing. This at a time the consumption of news has increased even before the COVID-19 pandemic. It is therefore critical to be as informed as possible when faced with a constant deluge of information.
That does not mean anecdotes or observational data is entirely useless. Just as anecdotes can be the first signal of a trend, observational data can further illustrate that trend.
Yet in most cases you need experimental data to determine causation. There are exceptions to this as observational data has led to some major scientific breakthroughs, such as smoking causes lung cancer; lead impairs children’s cognitive function; and fluorinating water prevents cavities. All of these examples have a common thread–the effect is large and widespread.
Yet there are also cases where doing experiments is a waste of time. The benefit could be so obvious that there is no need to do an experiment. An absurd example is the need to wear a parachute for skydiving; there is no need for further doing experimental data, except in parody.
Likewise, thousands of examples of observational studies have initially showed one thing but were eventually proven wrong by experimental data. These original flawed observational studies misled and led to real human suffering.
A great example is hormone replacement which was eventually proven wrong by the Women’s Health Initiative. For example, observational data showed a decrease in coronary heart disease yet the experimental data showed it may increase heart disease. That is not to say hormone therapy should not be used at all, but experimental data showed that nuance is required and improved informed consent for patients.
COVID-19 research is full of such examples. Corticosteroids have been shown to improve mortality in COVID in experimental studies. Thankfully the science ignored the observational data which showed it did not.
Hydroxychloroquine was touted as a miracle drug early on because of a promising observational study. It does not work. It really, really does not work. Thankfully important treatments and vaccines are usually based on experimental studies. The Pfizer and Moderna COVID vaccines are based on robust, large experimental data.
The bedrock of science is experimentation. Yet popular mainstream media more often reports observational scientific data. Hence there is a constant deluge of headlines which grab attention but are based on weak observational data.
Consider these examples: Very hot tea can increase esophageal cancer; phone use is leading to horns growing on young people’s skulls; treatment by female physicians leads to decreased mortality for patients.
Many recommendations and behaviors are based primarily on very weak observational data. They include a vegan diet for heart health, glass of red wine a day and Atkins diet— among many others. That does not mean these recommendations are not true. It just means the justifications are not as strong as many would assert..
To be sure, science is not trying to mislead the public or provide any false promises. Instead it is responding to an insatiable media appetite.
So what is a general approach for the non-scientist consumer or policy maker when trying to interpret all this data?
Increase the level of skepticism of observational data. If an article or media site highlights an observational study, perhaps just ignore it. It is often not worth changing behavior.
Decrease the level of passion about topics based on observational data. There is a very good chance the findings will be proven wrong in the future.
Unless the consumer of these studies is also an expert in the field who plans on doing more research in that topic, little benefit is derived from observational data. Uncertainty is common.
Science will find the truth, but it will take time.
Follow experimental studies. The COVID-19 pandemic has made it even more clear that the need is for less observational data and more experimental data. Gathering experimental data is worth the investment. Demand studies be repeated. Replication crisis rages on in many fields of science including medicine, psychology and economics.
Yes, science can appear messy but this is all a necessary part of the process. Just as Benjamin Franklin said, “Democracy is the worst form of government except for all the others.” Similarly, science is the worst path to the truth–except for all the others.
