Article Text
PDF
Statistics from Altmetric.com
Introduction
Standard admission order sets have become ubiquitous across hospitals to promote adherence to practice guidelines and increase ordering efficiency.1 2 This standardisation arose in part out of a need to minimise waste in healthcare, a phenomenon identified as a major barrier to reducing future healthcare costs.3 However, few studies have systematically evaluated whether these standardised orders can actually promote overordering of investigations. At our academic hospital’s coronary care unit (CCU), a single mandatory generic order set is used regardless of admitting diagnosis and includes optional check boxes for serum thyroid-stimulating hormone (TSH) and brain natriuretic peptide (BNP). We postulated that physicians order investigations differently on admission based on which investigations are included in the admission order set.
Methods
We quasi-randomised a convenience sample of participants in a double-blind fashion to receive either our standard CCU admission order set or a slightly modified version (see online supplementary file). The participants included internal medicine staff physicians, residents and clinical clerks (medical students, year 3 or 4) at our academic centre who were attending grand rounds. After their respective grand rounds, seated participants were provided with a case of a previously healthy 50-year-old man presenting with uncomplicated ST elevation myocardial infarction, now stable postpercutaneous revascularisation. Based on the clinical information provided, ordering TSH and BNP was not clinically indicated. Unaware of the differing versions, volunteer participants received a paper copy of one of the two versions of the admission order set. Researchers distributing the order sets were also not aware of which version of the order set was given to each participant. Volunteer participants were then instructed to write admission orders for the hypothetical patient, after receiving either a CCU admission order set that included TSH and BNP checkboxes (intervention) or the same order set without (control). The order sets were otherwise identical to the casual observer and both included an area to request additional investigations.
Supplementary file 1
The primary outcome was the ordering of TSH and BNP, focusing on comparison between groups. We also assessed whether ordering differed between staff and trainees. Using SAS software, V.9.4 (SAS Institute, Cary, North Carolina, USA), statistical comparisons of ordering proportions in the groups were made by χ2 and two-sided Fisher’s exact tests. Exact CIs were determined for proportions. Accounting for the repeated evaluation of two tests, overall and in four subgroups, Bonferroni α correction was applied (α=0.05/10=0.005). To evaluate group similarity, training levels and general tendencies to order blood work were also compared, using the above statistical tests for categorical assessments and the Wilcoxon rank-sum test for medians.
The study received prior approval from our university research ethics board, and informed consent was obtained from all participants. Disclosure regarding the study objective was provided to all participants after ordering completion, with additional opportunity to opt out. We attest that we have obtained appropriate permissions and paid any required fees for use of copyright protected materials.
Results
Among 145 participants, 3 did not indicate training level and were excluded; thus 142 preprinted order sets were analysed (intervention, n=74; control, n=68). Training levels and blood work ordering tendencies were similar between groups (table 1).
Training levels and general blood work ordering by group
Among intervention subjects, 29 participants (39%, 95% CI 28% to 51%) ordered TSH and 25 participants (34%, 95% CI 23% to 46%) ordered BNP (table 2), compared with no TSH or BNP orders among control subjects (95% CI 0% to 5%; p<0.001 for both comparisons). This difference was similar within all subgroups and most significant among clinical clerks, likely due to the larger sample size of this subgroup (table 2).
TSH and BNP ordering by group, overall and by training level
Discussion
Our experience suggests that the mere presence of a test on an admission order set makes it inherently more likely to be ordered, regardless of clinical utility. Remarkably, when BNP and TSH were omitted from the order set, no participants ordered either test. This difference was observed irrespective of training level. A compelling real world example of this phenomenon was highlighted in table 1 of Leis and Shojania’s4 detailed plan-do-study-act review of a medical directive for urinary catheter removal by nurses on general medical wards.5 On the stroke unit at the hospital involved, 89% of unnecessary catheter insertions were associated with the preprinted admission order set. Similarly, after his participation in our study, a senior resident remarked that preprinted admission order sets ‘make us dumber and we order more tests.’
This crude yet insightful observation underscores the potential ways that order sets can precipitate overordering. First, given that order sets are generally regarded as clinical guideline-consistent,2 blood work on an order set may be misinterpreted as a prompt rather than simply an option. Second, more cognitive effort is involved in stewardship of blood work compared with selecting multiple checkboxes. Furthermore, optional blood work on preprinted order sets may facilitate ‘assurance behavior’, undertaken to instil confidence in management adequacy.6 However, as our study did not explore reasons for ordering decisions, our understanding of the physician perspectives that gave rise to our results is limited.
Interestingly, the intervention group of senior residents did not order BNP even when given the option. At the junior resident level, this trend was also observed, although to a lesser degree. A potential explanation is that internal medicine resident academic days at our centre over the last 2 years have included several discussions about the appropriateness of ordering serum BNP, including its vulnerability to overordering. Alternatively, no formal discussions in this context addressed these same issues with regard to TSH. Although educational interventions tend to be the least effective at enacting healthcare change,7 perhaps the decreased ordering of BNP reflects a cultural change because of this value-based curriculum. This notion is supported by the fact that the students and staff were not exposed to this intervention and were respectively found to be approximately as likely to order BNP as they were to order TSH.
While we acknowledge that our study was a simulation of a single clinical scenario at one institution and was not designed to test a specific intervention in the real world context, it does highlight the possible unintended consequences of standard order sets in promoting low-value testing. The choice to monitor TSH and BNP, blood tests already shown to be vulnerable to overordering among inpatients,8 9 potentially acted synergistically with the order set to drive low-value testing. Indeed, tests that are less vulnerable to overordering may be more immune to this order set effect. In reality, the intervention order set in this study (including BNP and TSH checkboxes) is the current CCU order set at our academic hospital; subsequently some participants were familiar with it in actual application. However, the fact that none of the control participants noticed the absence of the checkboxes and that their removal completely curbed TSH and BNP ordering suggests that familiarity did not play a significant role in overordering. Even so, similar findings in a multicentre, non-simulated, clinical evaluation of this question would improve the reliability and generalisability of our conclusions. A rigorous qualitative evaluation of physicians’ own perceptions as to why they order certain tests when presented would also allow further refinement of order sets as beneficial, cost-effective aids in practice.
Acknowledgments
The authors thank Dr Andrew Lyon for assistance with local hospital statistics and Dr Gary Groot for feedback on study inclusion criteria and design.
Footnotes
Contributors Dr AL for assistance with local hospital statistics. Dr GG for feedback on study inclusion criteria and study design.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.