How Bad Would It Have to Be?

goneFishingI would like to humbly suggest that we are in denial. I’m talking about myself and my doctor/scientist friends, and I’m talking about publication bias. Those of us who have done research understand the basic dilemma. Negative findings are not sexy. They don’t sell. They don’t get published in good journals. In a sense, that’s reasonable, because there are a million uninteresting reasons why any experiment might not produce an effect. Whereas the whole idea of a properly controlled experiment is that, if it produces an effect, there are only a few (or ideally, one) possible explanations. Naturally we would be drawn to those experiments. Because we are early in our careers, many of us have had the opportunity to do at least some relatively speculative, hypothesis-generating (rather than hypothesis testing) research. That means we’ve been allowed to pursue the experimental paths that bore fruit.

But what about clinical research? Trials of medications and medical devices must include negative results, or the entire game is rigged. If pharmaceutical companies and manufacturers can selectively publish only the findings that make their products look effective, then we learn essentially nothing from those studies. Let’s say we calculate some basic metrics: the ratio of studies with positive findings that are published, and the ratio of studies with negative findings that are published. Here’s where I think the denial comes in. I want you to take a moment and think those numbers would have to be for you to get really concerned about the basic legitimacy of the research….I’ll wait…

So what are your numbers? How much publication bias are you willing to tolerate in the process that determines the medications you take? Those that you prescribe your patients? Those you give to your children? What about the medical devices implanted in your parents and grandparents?

A study from 2008 in the New England Journal of Medicine used trial registration records and FDA findings to determine these numbers for 12 antidepressant medications. The researchers reviewed 74 studies (representing more than 12,500 patients) and found that all but 1 of the 37 studies viewed by the FDA as supporting the effectiveness of the drug were published. The path to print was more arduous for those studies the FDA deemed to have negative or questionable results. Of the 36 trials, only 3 were published directly, 11 were published in way the authors said conveyed the results as positive, and 22 were not published at all. Meta-analysis suggested that the exclusion of the negative findings inflated effect sizes by 32%.

While this is only one study, and antidepressant trials commonly have relatively thin margins between effects in controls and in treated subjects, other research has suggested that the trials are like cosmic dark matter of biomedical science, a huge mass of missing data most notable in its absence. As Ben Goldacre points out a recent OpEd in the New York Times (and in a new book), the regulatory fixes that were meant to expose these trials have often failed to bring the intended transparency to the process. Furthermore, because many of the medications we use today rely on trials that were conducted before these rule changes, it will take years of strict compliance (so far, elusive) before we can be confident that we aren’t being hoodwinked by the most elementary of statistical shenanigans. For this reason, Goldacre has been championing, a petition campaign to rally support for retroactive registration of the thousands of missing trials. Amazingly, GSK (GlaxoSmithKline) has even committed to supporting the campaign (and to data transparency in general). Perhaps this is a signal of a nascent consensus.

To be clear, I’m not anti-pharma. Some of my best friends are pharmaceuticals. I’m just hoping to get beyond the need for self-delusion.


~ by nucamb on February 6, 2013.

4 Responses to “How Bad Would It Have to Be?”

  1. […] coin of the realm in experimental science is the p-value (if that’s not news, perhaps this discussion will be more pertinent). To put it perhaps too simply, the p-value is a statistic that tells a […]

  2. […] element of the current science reform movement stems from harsh criticisms that have been leveled against biomedical clinical research. Whether from unintentional bias, or […]

  3. […] journals require such registration for clinical trials (of drugs and medical devices) as a way of improving the quality of clinical research; preregistration helps prevent statistical (and publishing) practices that can bias results. But […]

  4. And now the real questions are bubbling to the surface. Hey ho-which new way to go?
    Trapped within systems ( designed by and for?) to perform or be shunned from the crowd, well that is a problem! One must weigh the conscious choices, especially those hidden under layers of reason and fear. Rational leads to rationalize and saving face is but egos where do we go from here? As an artist, designer with degree in enviro science and communications (Greener link) i see a whole new path emerging and am looking for others willing to explore it with me. The outliers, rebels, those who defy the data..they are pioneers and illuminators. That generic, normal, MOR space -that safe center from which data is spewed forwardi manifests a future based on the past..and that is spiral down of probable choice aka nihilistic. See we really have to jump, take that cosmic leapin imagination and ? After all we have evolution on our side.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: