Holding medical journals to account for publishing trials with switched outcomes
Improving the quality of research evidence
In the run-up to Evidence Live 2016, the organisers asked health science students, junior doctors, and early career researchers to write about projects or innovative ideas that they have been part of and that deal with some of the conference’s main themes (see the box at the end of this article). This article features one of the five winners who gain a free place to attend the conference in Oxford between 22 and 24 June 2016.
Aaron Dale, third year graduate entry medical student, University of Oxford
Outcome switching is a major problem in clinical trial reporting that can distort the evidence from which clinical decisions are made. This is when researchers fail to report the outcomes they originally set out to measure and swap them for new outcomes that they didn't initially consider. The best demonstration of why this is a problem is in the webcomic xkcd.
By leaving out the fact that outcome switching has taken place, journals can represent interventions appearing better than they actually are, which misinforms doctors and risks considerable harm to patients.
Outcome switching is highly prevalent in clinical trials; however, prevalence studies alone have not eradicated the problem. This led us to ask how can we, as medical students, make a difference? Our hypothesis was simple: if we could hold individual journals to account for specific cases of outcome switching and engage them in open discussion, we would begin to understand why the problem still persists and, in doing so, help to correct it.
Between October and November 2015 the COMPare team (five medical students in collaboration with three researchers in evidence based medicine) monitored all trials for outcome switching in the top five general medical journals. We compared the prespecified outcomes in the trial protocol or registry entry with those reported in the journal publication. We recorded the number of missing prespecified outcomes and the number of new outcomes silently added, and wrote a letter of correction to journals for each trial containing misreported outcomes. We also published all our results, raw data, and letters to journals on our website.
We have seen change in the way BMJ Open reviews clinical trials for publication: they now require the original trial protocol to be submitted with the corresponding paper to help editors compare pre-specified and reported outcomes. In addition, The BMJ recently published a correction on the REEACT trial based on our letters.  We are also engaged in discussions with the New England Journal of Medicine, Annals of Internal Medicine, the Lancet, and the Journal of the American Medical Association in the hope that we can improve clinical trial reporting. You can read our blogs detailing the responses we have received from these journals on our website.
Evidence Live 2016 conference themes
- Improving the quality of research evidence
- Disentangling the problems of too much and too little medicine
- Transforming the communication of evidence for better health
- Training the next generation of leaders in applied evidence
- Translating evidence into better quality health services
The original version of this article claimed that the COMPare team had prompted The BMJ to change its policy on how it reviews clinical trials submitted to the journal. The submission of protocols along with the manuscript has been a requirement at The BMJ for many years. The BMJ refers to protocols only in cases where editors are uncertain about details of the methods or when they want more information about how or when outcomes were to be measured. The protocols are not used to identify the prespecified outcomes. The BMJ research team rely on the trial registration information for that.
- XKCD. Significant. https://xkcd.com/882/.
- Jones CW, Keil LG, Holland WC, Caughey MC, Platts Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med 2015;13:282 doi:10.1186/s12916-015-0520-3.
- Compare. 2015. compare-trials.org.
- BMJ Open. Requesting Clinical Trial Protocols. 2016. http://blogs.bmj.com/bmjopen/2016/01/19/requesting-clinical-trial-protocols/.
- Published: 06 June 2016
- DOI: 10.1136/sbmj.i2396