A misleading piece of statistical rhetoric has appeared in a paper about an experimental antidepressant treatment. The study is
published in the Journal of Affective Disorders
. JAD is a respectable mid-ranked psychiatry journal - yet on this occasion they seem to have dropped the ball badly. The study examined whether the drug armodafinil (Nuvigil) improved mood in people with bipolar disorder who were in a depressive episode. In a double-blind trial, 462 patients were randomized to treatment with either armodafinil 150 mg/day, or placebo, in addition to their regular medication, for 8 weeks.
The results showed no statistically significant difference between the two groups in change on the depression rating scale, the IDS-C30. (p = 0.24 LOCF, p = 0.13 at week 8). So this means that there was no evidence that armodafinil helped treat depression. Oh dear. It's a textbook example of a negative, or null, result. Despite this, however, the paper's abstract concludes on a remarkably upbeat note:
FDA-approved bipolar I depression treatments are limited. Adjunctive armodafinil 150 mg/day reduced depressive symptoms associated with bipolar I disorder to a greater extent than adjunctive placebo, although the difference failed to reach statistical significance. Safety data indicate treatment with adjunctive armodafinil was well-tolerated
The second sentence is quite misleading. The mean reduction in symptoms was indeed slightly higher in the armodafinil group than in the placebo group, but in the absence of statistical significance, the difference of means is meaningless (no pun intended). Even assuming that the drug has no effect, there would be a 50% chance of the drug group having the greater improvement - and a 50% chance of the placebo being greater. One of the values has to be greater, after all. There will always be some mean difference. Which is why we need statistical significance testing. So we simply can't say that the drug "reduced depressive symptoms to a greater extent", not even if we qualify this with the caveat that the effect was not statistically significant. The caveat is so big that it negates the entire statement. Other statements about how the drug group showed "numerically greater" improvement appear in the main text. Had I been a peer reviewer, I would have made the authors remove all such claims. However, this is perhaps a matter of taste. After all, it could be argued that the authors are being transparent, since they are merely stating the facts (there was a difference, that it was non-significant) and allowing readers to draw their own conclusions. The problem with that, though, is that if we're going to treat statistical non-significance as a mere caveat, we ought to do so consistently. Yet the authors don't apply the same logic to the data on the side effects of armodafinil. For instance, one patient in the armodafinil 150 mg group died from a pulmonary embolism during the study. No patients in the placebo group died. So the authors could have written that 'armodafinil was associated with a greater risk of death, although the difference did not reach statistical significance' and put that in the abstract... but they didn't. Similarly, armodafinil 150 mg was associated with a numerically greater risk of mania, hypomania, headache, nausea, insomnia, coronary artery disease, and non-cardiac chest pain - although not necessarily statistically significantly so. Only headache and nausea are noted in the abstract. Reaching the end of the paper, it's no surprise to learn that it was sponsored by and funded by Teva Pharmaceuticals, who sell armodafinil, and that it was written with the 'assistance of' a medical writing firm, and that "Teva provided a full review of the article." Teva clearly have an interest in making their product seem as safe and effective as possible. So it's understandable that they would want this paper to put a positive spin on things. That's the job of Pharma. They succeeded. Academic journals like the Journal of Affective Disorders are supposed to stop such spin from making it into the scientific literature. They failed. P.S Many thanks to Dr. Bernard Carroll for letting me know about this paper.
Ketter TA, Yang R, & Frye MA (2015). Adjunctive armodafinil for major depressive episodes associated with bipolar I disorder. Journal of Affective Disorders, 181, 87-91 PMID: 25933099
