A simple and valid tool distinguished efficacy from effectiveness studies

Abstract

Objective

To propose and test a simple instrument based on seven criteria of study design to distinguish effectiveness (pragmatic) from efficacy (explanatory) trials.

Study Design

Currently no validated definition of effectiveness studies exists. We asked the directors of 12 Evidence-based Practice Centers to select six studies each: four that they considered to be examples of effectiveness trials and two considered efficacy studies. We then applied our proposed criteria to test the construct validity using the selected studies as if they had been identified by a gold standard.

Results

Based on the rationale to identify effectiveness studies reliably with minimal false positives (i.e., a high specificity), a cutoff of six criteria produced the most desirable balance between sensitivity and specificity. This setting produced a specificity of 0.83 and a sensitivity of 0.72.

Conclusion

When applied in a standardized manner, our proposed criteria can provide a valid and simple tool to distinguish effectiveness from efficacy studies. The applicability of systematic reviews can improve when analysts place more emphasis on the generalizability of included studies. In addition, clinicians can also use our criteria to determine the external validity of individual studies, given an appropriate population of interest.

Introduction

Randomized controlled trials (RCTs) are the gold standard in evaluating the effects of treatments. To be clinically meaningful, results must be relevant to specific patient populations in specific settings [1]. Multiple factors determine the external validity (i.e., generalizability or applicability) of RCTs: patient characteristics, condition under investigation, drug regimens, costs, compliance, comorbidities, and concomitant treatments. For practical reasons, trials cannot always take these factors fully into consideration (e.g., costs, poor compliance). Also, certain aspects of study design—eligibility criteria, study duration, mode of intervention, outcomes, adverse events assessment, or type of statistical analysis—greatly influence the degree of generalizability, given an appropriate population of interest.

Clinicians and policymakers often distinguish between the efficacy and the effectiveness of an intervention. Efficacy trials (explanatory trials) determine whether an intervention produces the expected result under ideal circumstances. Effectiveness trials (pragmatic trials) measure the degree of beneficial effect under “real world” clinical settings [2]. Hence, hypotheses and study designs of an effectiveness trial are formulated based on conditions of routine clinical practice and on outcomes essential for clinical decisions.

Efficacy and effectiveness exist on a continuum. Generalizability depends largely on the viewpoint of the observer and the condition under investigation. Baseline patient characteristics (e.g., gender, age, severity of the disease, racial groups) are primary factors in generalizability; thus, depending on the population of interest, generalizability of the same study can range from low to high. Geographic settings (urban vs. rural) and health care systems can also be significant factors [1], although geography may have less influence on generalizability of drug trials than trials of other interventions (e.g., screening programs, behavioral therapy).

Ensuring generalizability may compromise internal validity. Under everyday clinical settings, factors such as patient or doctor preferences [3], [4], or patient–doctor relationships [5], [6] can influence response and compliance. Random allocation, allocation concealment, and blinding negate these factors, thereby increasing internal validity on the one hand and decreasing external validity on the other. Therefore, to some extent, the operational definition of “effectiveness trial” delineates the necessary trade-offs with internal validity. An ideal definition would balance this equilibrium at a point at which satisfactory internal validity accompanies a high degree of generalizability.

Systematic reviews, including meta-analyses, have become an increasingly important source of information for clinical practice. If well conducted, they synthesize large amounts of information and provide estimated effect sizes that have greater precision and generalizability than individual studies [7]. Distinguishing between efficacy and effectiveness contributes an important aspect to analyzing any body of clinical evidence. Furthermore, greater emphasis on effectiveness studies may lead to changes in presentation in systematic reviews and policy initiatives.

In this article, we propose and test seven hallmarks of study design to create a tool that can help researchers and those producing systematic reviews, as well as clinicians who are interested in the generalizability of study results, to distinguish more readily and more consistently between efficacy and effectiveness studies.