Goal 4: Develop Workforce and Resources

Reproducibility Initiatives in Heart, Lung and Blood Research

Scientists feel tremendous pressure to publish numerous scientific papers in order to receive NIH funding and tenure at academic institutions. Cognitive biases of scientists and publication biases of journals that publish this barrage of papers will likely result in the publication of findings that are probably not reproducible (see "Why Most Published Research Findings Are False" by John P. A. Ioannidis in PLOS Medicine 2005).


Helping the scientific community identify which findings are reproducible and which aren't is critical for the long-term success of the biomedical research enterprise. There are few incentives and even fewer resources currently available to support research that verifies reproducibility.

Tags (Keywords associated with the idea)

Is this idea a Compelling Question (CQ) or Critical Challenge (CC)?: Critical Challenge (CC)

Details on the impact of addressing this CQ or CC:

By distinguishing research findings which are reproducible from those which aren't, researchers will be able to build future research programs on solid scientific foundations.


There are many reasons for why research may not be reproducible, ranging from simple biological variations (cells from one supplier may behave differently than cells from a different supplier) to conscious/unconscious biases or misconduct. No matter what the underlying cause is, irreproducible research findings that are not recognized as such will result in a tremendous waste of time and resources. Graduate students or postdoctoral trainees may waste entire years of their precious training period conducting experiments that are based on published papers which may turn out to be irreproducible.


NHLBI could significantly improve the quality of research by building an infrastructure that supports the assessment of reproducibility and widely shares these findings.

Feasibility and challenges of addressing this CQ or CC:

One of the challenges for assessing reproducibility of published work is that it is considered not very innovative and there is no funding available. NIH grants are awarded to highly innovative proposals which venture into new territories and not proposals which want to confirm the validity of published work. However, the returns of investing into reproducibility testing might be enormous because irreproducible results would be identified and weeded out, thus preventing loss of resources and time.


The NHLBI could develop funding mechanisms specifically designed to support research proposals that will test the reproducibility of high impact findings that have not yet been independently verified. The study sections would review these proposals using novel criteria designed for such studies. The emphasis of the study section review would lie on questions such as "Is this an important enough question that it merits reproducibility testing?" instead of the traditional "Is this a cutting-edge technology that nobody has previously used?"


Challenges would include identifying journals that would published results of these studies and agreeing on what constitutes reproducibility (i.e. is it enough if the major conclusion or effect is reproduced even though the effect size may be very different?).

Name of idea submitter and other team members who worked on this idea: Jalees Rehman


2 net votes
5 up votes
3 down votes
Idea No. 857