Many people who follow the progress of science may be aware of the reproducibility problem in the field, today. This involves reporting the findings of one published study as though they form an unimpeachable fact on their own, without testing its conclusions by recreating it.
The issue is so severe that some putative treatments have risked being fast-tracked through clinical trials far too early in their development and testing. Furthermore, a lack of replication goes against a central tenet of the scientific method.
Accordingly, some initiatives have set out to address the reproducibility crisis.
The RP: CB Crisis
One of these investigations, intending to act on the results of studies in cancer biology, has hit a major snag. Known as the Reproducibility Project: Cancer Biology (RP: CB), the project was set up with the aim of reproducing 50 trials in cancer biology. This was done in response to some concessions from the pharmaceutical industry about the inability to correctly validate the findings of studies in new or improved therapies.
RP: CB was set up by the Center for Open Science in Charlottesville, Virginia in 2013. It was funded by the Laura and John Arnold Foundation to the tune of just over a million dollars. Initially, the project was estimated to take one year per experiment.
Unfortunately, these ambitious intentions may be less than feasible.
Tim Errington, the RP: CB project leader at COS, now admits that his team has had to scale down the original volume of studies to be replicated. At present, they estimated to be able to validate only 36% of the original trial panel.
Errington noted that the RP: CB remit was not as realistic as it could have been. The project’s downsizing could be affected by a number of factors. This included the resources and staff required to complete the replications. Furthermore, these experiments were to be conducted in contract labs, which will add to the cost of the project.
Adding peer review to the RP: CB process has the potential to slow it down considerably. (Source: Kriegeskorte, N., 2012)
In addition, the RP: CB leaders decided that they wanted their replications to be peer-reviewed. This has slowed down the entire process.
Also, the investigating scientists found that they needed more information and background on the studies that they were working on, from the original authors, who were initially sceptical of the project.
Consequently, the RP: CB system began to re-evaluate the number of studies they could take on. In 2015, the team announced that this number was 37; in 2017, it reduced again to 29. This number fell again in early 2018 to the current figure (i.e.,18).
Errington has also admitted that the time-frame per replication played a part in the discontinuation, in some cases. For example, the replicators were not able to figure out from their reference papers exactly what kind of cell-culture technique to use in a certain protocol. As a result, they could be faced with the problem of having to test a range of such methods.
According to Errington, this resulted from absent or withheld data in the methods sections of the original article. This issue was challenging; however, the team talked about it in an interview to highlight the potential value of open-access (or at least database contributions) to modern-day science.
The RP: CB’s scientists may have had to repeat procedures numerous times with minute variations in order to accurately replicate a trial. (Source: Public Domain)
On the other hand, the achievements of RP: CB include 10 fully replicated studies, the results of which are available in the eLife journal. It was revealed that 50% of these studies were validated. However, another 30% did not return solid conclusions, and the remaining 20% exhibited results contrary to those reported in the original papers.
But, the authors of these papers maintain that their findings have, in fact, been reproduced in other, independent labs. The remaining studies did not reach the publication stage at the time of writing.
The RP: CB project is something of an object lesson in reproducibility in science and the difficulties often faced in its achievement.
However, scientists such as Errington would be the first to assert that this process is well worth it, especially for the future of the discipline.
Top Image: RP: CB was set up to test the validity of results reported in cancer biology papers. (Image Source: Myfuture.com @ flickr)
References
Plan to replicate 50 high-impact cancer papers shrinks to just 18, 2018, Science, https://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18 , (accessed 2 August 2018)
Reproducibility Project: Cancer Biology Investigating reproducibility in preclinical cancer research., 2018, elifesciences, https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology , (accessed on 2 August 2018)
Reproducibility Project: Cancer Biology (RP: CB) Reproducibility Review Panel, 2018, COS, https://cos.io/rpcb/ , (accessed 2 August 2018)
No comment