Study design and participants

A full description of the study methods appears in our report of the rate of contextual errors.9 Briefly, we obtained data on 111 primary care internal medicine physicians in practice at 14 locations in the Midwest between April 2007 and April 2009. Each physician was visited by up to four different USPs (399 visits), presenting randomly selected counterbalanced combinations of variants of four cases combined in a partial factorial design with the presence or absence of a biomedically complicated or contextually complicated diagnosis. In every variant, USPs included biomedical and contextual red flags in their presentations; variants differed by whether USPs confirmed or denied findings suggested by the red flags if they were probed by the physician. Table 1, reprinted from that study, describes the cases and variants, and the online appendix presents the full details of the cases. For example, in the asthma case, a middle-aged man presents with inadequately controlled symptoms on his current medication regimen, a brand-name steroid inhaler. In the baseline variant, the physician should increase the patient's steroid dosage or add another maintenance drug. In the biomedical variant, the patient's symptoms are the result of gastro-oesophageal reflux disease (GERD), and the physician should instead prescribe treatment for GERD. In the contextual variant, the patient cannot afford his inhaler and is not using it daily, and the physician should instead prescribe a less expensive generic inhaler.

A characteristic of the case design is that the baseline variant of each case represented the diagnostically uncomplicated version of that case, so that the focus of assessment is on the appropriateness of the management plan. Because the diagnosis was not in question, in the baseline variant, a chart review would reveal inappropriate care when evidence-based practices were not followed. For instance, in the asthma case, the physician was assessed based on whether guidelines were followed for stepping up medication therapy, a determination that could be made by reviewing the orders placed in the medical record. In contrast, physician errors made in the biomedical, contextual and bio/contextual variants of cases were not detectable from a review of the medical record as they occurred when physicians failed to recognise that a clinical situation was complicated (eg, the patient's symptoms were due to GERD or inability to afford his inhaler) and treated the patients as if they were presenting with the baseline variant.

Visits were audiorecorded and transcribed. USPs were entered into the scheduling system as actual patients, and copies of the physicians' notes following encounters were forwarded to the project team. Physicians' notes were coded for treatments and tests ordered blind to any knowledge of the case variant. Inter-rater reliability checks were performed for samples of all ratings. The costs of visits were paid by the study. Institutional Review Boards associated with all practice sites approved the protocol.

Cost estimates

We adopted the economic perspective of the patient and their third party payer, if any, with a time horizon of the expected consequences of care during the 30 days following the consultation. We considered only the direct consequences of care associated with diagnosis or misdiagnosis. We did not consider downstream costs beyond the initial recommendations from the consultation, and we did not consider societal costs not incurred by the patient or payer, such as lost productivity. We included only resources related to the immediate diagnostic and therapeutic management of the four conditions (see table 2 for the list of resources we considered for the biomedical and contextual management of each case).

Resources were direct medical costs in the case of unnecessary treatment and foregone direct medical costs in the case of undertreatment. For biomedical variants, the decision of which resources to include was made on the basis of established clinical guidelines for the management of the relevant condition. For contextual variants, clinical experts provided the correct management plans and associated resources during the case development and validation (reported previously elsewhere).12

We estimated resource consumption by eliciting the over or under treatments from the chart. Table 2 lists how much of each resource was measured across the entire cohort. Unit costs were measured in 2009 US dollars, using 2009 Medicare reimbursements by Current Procedural Terminology code or average sales price for medications when available, and local costs (eg, for labour), based on the care plan recommended by the physician. Table 2 gives the costs we assigned for all the missed or unnecessary services associated with our study.

When an unnecessary test or treatment was ordered, we conservatively estimated the cost of error as the cost of the test or treatment procedure itself (eg, we did not consider potential downstream costs associated with false positives or downstream savings associated with a serendipitous identification of additional health problems). When an indicated necessary test or treatment was not ordered, we similarly made the conservative assumption that the cost of the procedure itself was equal to its expected marginal benefit, and used the procedure cost as the cost of error. We did not include costs associated with travel time, lost wages or other indirect costs. As a result, we likely underestimated the true total costs of errors.

For example, when a patient with worsening asthma due to inability to afford an expensive daily medication is mistakenly instructed to increase dosage of that medication rather than prescribed a lower-cost inhaler, we assume that the new expensive prescription will also not be filled, so the direct cost of unnecessary treatment is $0. However, the physician has missed the opportunity to provide the patient with a lower-cost inhaler and the patient loses the benefit associated with that inhaler. We assume the value of this benefit on average is at least the cost of the generic inhaler for 1 month, which was estimated at $163.79 from our Medicare data. That is, the rationale for providing an inhaler to an patient with asthma is that, on average, the inhaler will provide enough benefit to justify its cost, and thus provide at minimum $163.79 of expected benefit per patient over a 30-day horizon; we treat failure to provide this standard of care as incurring a cost of this amount, which serves as a proxy for the expected costs associated with unmanaged asthma over a population of patients (some of whom will experience no actual costs, some of whom will incur the cost of additional clinic visits, some of whom will incur the cost of emergency room visits, etc) Accordingly, the direct medical cost of error in this case is $163.79, which clearly represents a very conservative estimate.

We classified the care in each visit as appropriate (a correct plan of care with no unnecessary care), underuse (failure to order necessary care, but also no unnecessary care), overuse (ordering unnecessary care in addition to necessary care), and misuse (failure to provide necessary care while also providing unnecessary care). By definition, the cost of error for an appropriate plan was $0. Note that it is sometimes possible for overuse to have a cost of $0 when the ordered care is not actually provided (as in the asthma case, when the patient is required to fill a prescription and would likely not do so). Because physicians differed in their rate of planning correct care among case variants, we present the cost of error by variant across all encounters and across only those encounters in which care was not appropriate. Cost of errors in each of the variants was compared using Kruskal–Wallis one-way non-parametric analysis of variance with pairwise comparisons between variants.