How Bad Would It Have to Be?
I would like to humbly suggest that we are in denial. I’m talking about myself and my doctor/scientist friends, and I’m talking about publication bias. Those of us who have done research understand the basic dilemma. Negative findings are not sexy. They don’t sell. They don’t get published in good journals. In a sense, that’s reasonable, because there are a million uninteresting reasons why any experiment might not produce an effect. Whereas the whole idea of a properly controlled experiment is that, if it produces an effect, there are only a few (or ideally, one) possible explanations. Naturally we would be drawn to those experiments. Because we are early in our careers, many of us have had the opportunity to do at least some relatively speculative, hypothesis-generating (rather than hypothesis testing) research. That means we’ve been allowed to pursue the experimental paths that bore fruit.
But what about clinical research? Trials of medications and medical devices must include negative results, or the entire game is rigged. If pharmaceutical companies and manufacturers can selectively publish only the findings that make their products look effective, then we learn essentially nothing from those studies. Let’s say we calculate some basic metrics: the ratio of studies with positive findings that are published, and the ratio of studies with negative findings that are published. Here’s where I think the denial comes in. I want you to take a moment and think those numbers would have to be for you to get really concerned about the basic legitimacy of the research….I’ll wait…
So what are your numbers? How much publication bias are you willing to tolerate in the process that determines the medications you take? Those that you prescribe your patients? Those you give to your children? What about the medical devices implanted in your parents and grandparents?
A study from 2008 in the New England Journal of Medicine used trial registration records and FDA findings to determine these numbers for 12 antidepressant medications. The researchers reviewed 74 studies (representing more than 12,500 patients) and found that all but 1 of the 37 studies viewed by the FDA as supporting the effectiveness of the drug were published. The path to print was more arduous for those studies the FDA deemed to have negative or questionable results. Of the 36 trials, only 3 were published directly, 11 were published in way the authors said conveyed the results as positive, and 22 were not published at all. Meta-analysis suggested that the exclusion of the negative findings inflated effect sizes by 32%.
While this is only one study, and antidepressant trials commonly have relatively thin margins between effects in controls and in treated subjects, other research has suggested that the trials are like cosmic dark matter of biomedical science, a huge mass of missing data most notable in its absence. As Ben Goldacre points out a recent OpEd in the New York Times (and in a new book), the regulatory fixes that were meant to expose these trials have often failed to bring the intended transparency to the process. Furthermore, because many of the medications we use today rely on trials that were conducted before these rule changes, it will take years of strict compliance (so far, elusive) before we can be confident that we aren’t being hoodwinked by the most elementary of statistical shenanigans. For this reason, Goldacre has been championing alltrials.net, a petition campaign to rally support for retroactive registration of the thousands of missing trials. Amazingly, GSK (GlaxoSmithKline) has even committed to supporting the campaign (and to data transparency in general). Perhaps this is a signal of a nascent consensus.
To be clear, I’m not anti-pharma. Some of my best friends are pharmaceuticals. I’m just hoping to get beyond the need for self-delusion.