The recent article by Davidoff and Batalden and the accompanying commentaries left us with mixed emotions.^1–4 On the one hand we applaud the authors’ efforts to improve the evidentiary base of quality improvement (QI) through publication guidelines. These guidelines will bring structure and rigor to the field of QI, which is a necessity.

However, the author’s plea and support for more rigor was placed in the midst of an argument to “dumb down” the scientific expectation of journals and readers regarding QI research. While we are all frustrated by the slow pace of improvement in quality of care and appreciate the authors’ and commentators’ arguments to relax traditional evidence based medicine standards for QI research and publication, we fear this approach would be detrimental to the field, waste essential and scarce resources, and lead providers and organizations down too many blind alleys.

The authors frame the challenges well. In short, they argue that many reviewers of high impact journals do not fully understand the challenges QI researchers face in conducting quality improvement programs with limited or no funding. Budgetary constraints, coupled with the difficulties of adhering to scientifically acceptable designs in the real world of healthcare delivery, make it impractical to expect that QI studies will traverse the threshold of scientific rigor that is generally expected of more traditional biomedical studies. The authors feel that failing to relax these standards will rob readers of the rich fruits gleaned from those toiling in the trenches.

To be certain, we too have struggled in publishing QI studies. While we believe the authors have correctly identified many of the signs and symptoms, they have misdiagnosed the disease and recommended the wrong therapy. True, QI work is often difficult, underappreciated, and undersupported. It is frequently difficult to determine if a change resulted in an improvement and, if so, whether the intervention was the reason. The ability to generalize from successful experiences at a single site to other settings is tenuous. Because of these and other issues, the publication standards of traditional medical journals frequently result in rejection of well executed QI “studies”. Undoubtedly, some of these studies contain important insights, and their absence from high impact medical journals can be tragic for everyone.

Nevertheless, many of these findings, if pulled from poorly designed or conducted studies, would be wrong, and the choice not to publish them in traditional journals probably improves and protects public health. Reviewers’ concerns about the risk for bias and lack of validity for many QI results are often valid and prevent inappropriate uptake of questionable QI interventions.

Contrary to the view that most QI efforts are rigorous, our experience has revealed that rigorous QI efforts are rare. QI studies commonly lack clarity regarding the study population, interventions and co-interventions, outcome measurements and definitions, and quality assurance procedures for data collection. The amount of missing data is often significant (although generally not reported) and may exceed the data available, and what data are available may be poor in quality. As such, results from one organization that performs well are often cherry picked among others not performing well and presented as the generalizable results. Consequently, many QI studies have significant bias and have a high risk of misinforming readers.

In short, many studies are appropriately rejected for publication by journals despite the hard work performed by staff truly dedicated to improving patient care. We are not saying that disseminating the results of local QI efforts with uncertain methodological rigor is inherently bad. Framed correctly, such anecdotes can generate new hypotheses, spur organizations to consider new approaches (“We’re a pretty similar hospital to that one … perhaps we should try that intervention here …”), and provide needed support and credibility to others in the field.

However, vehicles for disseminating study findings should not confuse readers regarding the scientific rigor of QI projects by bestowing the imprimatur of evidence-based medicine on study findings. Suitable vehicles to disseminate QI experiences and findings are readily available: conferences, newsletters, and websites contain a myriad of interesting anecdotes of improvement projects that can help guide and inspire others working in the field. While some of the findings from these efforts represent the “truth” and some will be generalizable, the absence of rigorous research methods makes it exceedingly difficult—if not impossible—to differentiate the wheat from the chaff.

Indeed, there are many examples illustrating the plausibility of conducting QI studies which are both scientifically sound and feasible. For example, over 120 ICUs in Michigan are participating in a QI collaborative, have collected data using rigorous methods, and more than 10 manuscripts have been submitted or are being prepared to share these findings. There are other examples of well done QI studies that have been published in high impact journals.^5–7 Studies such as these can stand toe to toe with the best in biomedicine on a methodological playing field.

Some might wonder whether it would be harmful if scientific standards for reporting QI studies were relaxed. After all, these QI interventions are not potentially toxic drugs or procedures that could cause harm if applied incorrectly. In fact, we believe that publishing unsound findings can and does cause harm. For example, recent evidence suggests that early observations regarding the value of computerized provider order entry (CPOE) systems may have been less generalizable than previously appreciated and may even, at least in the short term, be harmful.⁸ Even when new practices seem unlikely to cause harm, there are significant opportunity costs involved in adopting them—money and attention could be applied elsewhere.⁹ An example of this issue may be the widespread adoption of rapid response teams (RRTs) which is based on relatively scanty and mostly anecdotal evidence of benefit (the most rigorous clinical trial to date demonstrated no benefit).¹⁰ Without data regarding the benefits and costs, hospitals may inappropriately allocate scarce resources to RRTs at the expense of other interventions such as intensivist or increased nurse staffing, which may have stronger evidence, a larger impact on mortality, and more clear data on estimated costs.^11,12

Moreover, relaxing scientific standards opens the door not only for unintended biases—that is, “researchers” failing to appreciate that their honestly observed result is not correct or generalizable—but other kinds of biases as well. QI efforts can be plagued by conflicts of interest.¹³ Some organizations and individuals who set standards or publish QI results may benefit financially from selling services to healthcare organizations. As such, it is imperative that such recommendations have a strong evidentiary base and potential conflicts of interest remain transparent. In addition to protecting patients, institutional review board processes can help ensure scientific integrity and evaluate potential financial conflicts of interest with minimal burden on QI practitioners.

We do respect the view that, if the bar is set too high, some potentially important findings may not be disseminated. Conversely, lowering the bar will allow dissemination of too much biased information and promulgation of too much unexamined conflicts of interest. All in all, we believe the overall effect will be negative. A better course of action than to “dumb down” the field would be to push for increased support for rigorous QI studies (not only from external funders but from institutions themselves), better training in research methods for individuals working in the field, and more sensitivity on the part of journals and their reviewers regarding the science of QI. If approached in this manner, stakeholders deciding to implement a QI intervention that is informed by the published literature can be confident that the conclusions from the literature they are emulating represent the truth. This will only occur by maintaining scientific integrity. To that end, creating QI specific standards and guidelines for publication (such as those offered by Davidoff and Batalden) represents one giant step forward. Unfortunately, this progress will be lost if we take two steps backward and relax the standards of scientific rigor that have served us so well in clinical medicine and science.