Article Text
PDF
Abstract
Background Adverse events (AEs) after outpatient orthopaedic surgery are common, but difficult to detect. Electronic health records facilitate abstraction of large quantities of data, and may allow automated identification of ‘triggers’ or clues indicating the possibility of an AE. We evaluated electronic health record-based triggers to detect AEs after outpatient orthopaedic surgery.
Methods The medical records of 1464 patients undergoing outpatient surgical procedures in one of five orthopaedic services at our institution were manually reviewed for the occurrence of 90-day postoperative AEs. We used electronic health records to identify triggers suggestive of an AE. Each trigger was evaluated for positive predictive value (PPV). We constructed a logistic regression model to determine triggers associated with AEs and used the beta coefficients derived from the model to produce a formula for the likelihood of identifying an AE in the medical record.
Results The overall rate of 90-day AEs was 10%, with surgical site infection being the most common (3.3%). Electronic triggers with the highest PPVs for the occurrence of 90-day AEs were antibiotic prescription (75%), emergency department visit (41%), bone/joint or blood culture (41%), repeat surgery (39%) and consult with infectious disease specialist (33%). Using our formula to predict the likelihood of identifying an AE in the medical record, a predicted probability of >0.10 had a specificity of 80% and sensitivity of 53% for actual AE.
Conclusions Electronic health record-based triggers may facilitate quality-improvement efforts to monitor morbidity after outpatient orthopaedic surgery. Further research is needed to understand the optimal use of electronic triggers as surgical quality indicators and as screening tools to flag cases for manual review.
Level of evidence Level III, prognostic study.
- Trigger tools
- Surgery
- Quality improvement
Statistics from Altmetric.com
Introduction
Identification of adverse events (AEs) is central to patient safety and quality improvement.1 AEs are difficult to identify after outpatient surgery because most do not necessitate readmission or return to the operating room. Efforts to monitor, rate and improve the quality of outpatient surgical care are increasingly important. For instance, the Joint Commission requires ongoing professional practice evaluation to monitor clinical competence. Given that orthopaedic surgery is an increasingly outpatient speciality, commonly used hospital-based quality measures are ineffective. There is a need for better outpatient quality measures.
Strategies to detect AEs have traditionally focused on chart review and voluntary reporting of errors.2–4 There is growing recognition that the ‘trigger’ methodology may be a more sensitive and efficient alternative for detecting AEs in hospitalised adult and paediatric patients.5–7 A ‘trigger’ can be defined as an occurrence, prompt or flag found on review of the medical record that triggers further investigation to determine the presence or absence of an AE.5–7 An example of a trigger would be an antibiotic prescription following a low-risk outpatient surgical procedure, which would prompt an in-depth chart review for evidence of an infection. Although the trigger method leads to a more focused and efficient chart review compared with conventional (unfocused) chart review, manually reviewing records to identify triggers can be burdensome and time consuming. Comprehensive electronic health records may result in faster automated ways for estimating AE rates and identifying a selective set of charts for manual review.8–10
The purpose of this study was to develop and validate an electronic health record-based trigger algorithm to identify AEs after outpatient orthopaedic surgery.
Materials and methods
Study design
This retrospective cohort study was conducted at a large urban academic hospital in the USA. Using an institutional review board-approved protocol, we identified patients undergoing outpatient surgical procedures in one of five orthopaedic services at our hospital: (1) hand and upper extremity, (2) shoulder, (3) sports medicine, (4) paediatric orthopaedics and (5) foot and ankle. We considered 300 consecutive procedures for each service. The selected reference date was 31 December 2012, and we went backwards consecutively until reaching 300 procedures for each service.
Our electronic health record data repository, the Partners Research Patient Data Registry (RPDR), was queried to collect data on triggers that might be suggestive of an AE after outpatient orthopaedic surgery. This database contains information on patient demographics, diagnoses and procedures (billing codes), laboratory results and pharmacy prescriptions.11 The RPDR also includes a narrative electronic provider note repository (eg, operative reports, provider notes, radiology reports) that can be queried using free-text search.12 Based on expert consensus, we evaluated the following triggers: (1) antibiotic prescription within 90 days of surgery, (2) opioid prescription after the second postoperative week, (3) >2 postoperative visits within 30 days of surgery, (4) emergency department (ED) visit within 90 days of surgery, (5) repeat surgery within 90 days of the initial surgery, (6) consult with infectious disease specialist within 90 days of surgery, (7) bone/joint or blood culture within 90 days of surgery, (8) complete blood count test within 90 days of surgery, (9) >1 radiograph ordered within 30 days of surgery, (10) radiograph order between 31 and 90 days after surgery, (11) advanced imaging (CT, MRI) ordered within 90 days of surgery, (12) >2 occupational therapy or physical therapy visits per week and (13) total number of physician visits.
Patient characteristics
Of the initial 1500 patients considered (300 per service), we excluded 36 (2.4%) who had missing postoperative data (table 1). The final study cohort consisted of 1464 patients.
Characteristics of the study population
Overall, the mean age of patients undergoing outpatient orthopaedic surgery was 40±21 years, and male patients accounted for 55% of the study sample. The mean patient age ranged from 8.9±5.2 years (paediatric orthopaedics) to 51±16 years (hand surgery). Patients undergoing outpatient procedures in the sports medicine (65%), shoulder (64%) and paediatric orthopaedic (55%) services were more likely to be men; in contrast, procedures in the foot and ankle (59%) and hand (52%) services were more likely to be performed on women.
Outcome measures
The primary outcome of interest was 90-day postoperative AEs. We decided a priori to consider the following AEs: (1) surgical site infection (superficial and deep, including suture abscess), (2) wound separation, (3) haematoma, (4) thromboembolic events (eg, deep venous thrombosis, pulmonary embolism), (5) rash, (6) failure of implant or procedure, (7) disproportionate pain and stiffness, (8) and medical complication. All medical records were manually reviewed by a single chart abstractor who was trained in data collection and monitored on a sample of charts. The chart abstractor and principal investigator were in constant communication throughout the study period to resolve abstraction-related queries about AEs with more subjective definitions (eg, disproportionate pain/stiffness). The overall rate of 90-day AEs was 10% (152 of 1464). The most common AE was surgical site infection (n=48; 42 superficial, 6 deep), followed by disproportionate pain and stiffness (n=23), wound separation (n=17) and haematoma (n=16). The orthopaedic service with the highest AE rate was foot and ankle (15%), and the service with the lowest rate was hand (7.1%).
Statistical analysis
Overall and orthopaedic service-specific rates of AEs were determined at 90 days from surgery (table 1).
Continuous data were presented in terms of the mean and the SD. Categorical data were reported with frequencies and percentages. In order to evaluate the association between each factor to AE occurrence, we performed bivariate analyses using the Pearson χ2 test for categorical variables and independent samples t tests for continuous variables (table 2). Each factor was evaluated for positive predictive value (PPV; table 3).
Healthcare Resource usage in patients with and without adverse events within 90 days of surgery (n=1464)
PPVs by trigger
Multivariable logistic regression modelling was performed to determine which triggers were independently associated with the development of AEs (table 4). The model was age and sex adjusted. All covariates were entered into the model simultaneously, without further selection. Results were reported as ORs with 95% CIs. The beta coefficients derived from the regression model were used to produce a predictive formula for AEs.13 The AE score calculated from the formula was then converted to a percentage with the use of an exponential equation. We calculated actual AE rates across quintiles of predicted probability of AE occurrence. We used the area under the receiver operating characteristic (ROC) curve to assess model discrimination and the Hosmer–Lemeshow test to evaluate model calibration. The predictive performance of this model was compared with that of a base model (only age and sex adjusted) using the area under the ROC curve and the Nagelkerke pseudo R2. The statistical threshold for type I error (alpha) was set at 0.05.
Multivariable regression model of predictors of adverse events after outpatient orthopaedic surgery
The predictive formula for the likelihood of identifying an AE in the medical record was applied to an independent validation cohort of 2279 patients undergoing outpatient orthopaedic surgery in the second half of 2013.
Results
The trigger with the highest PPV for the occurrence of AEs was antibiotic prescription within 90 days of surgery (75%), followed by ED visit within 90 days of surgery (41%), bone/joint or blood culture within 90 days of surgery (41%), repeat surgery within 90 days of the initial surgery (39%) and consult with infectious disease specialist within 90 days of surgery (33%) (table 3).
For surgical site infection––our most common AE––the triggers with the highest PPVs were antibiotic prescription within 90 days of surgery (60%), consult with infectious disease specialist within 90 days of surgery (33%), bone/joint or blood culture within 90 days of surgery (32%), repeat surgery within 90 days of the initial surgery (18%) and ED visit within 90 days of surgery (14%) (table 3).
After controlling for potential confounders in multivariable modelling, we found five triggers to be independently associated with AEs: antibiotic prescription within 90 days of surgery (OR 29, 95% CI 9.9 to 85, p<0.001), ED visit within 90 days of surgery (OR 4.6, 95% CI 2.4 to 8.8, p<0.001), bone/joint or blood culture within 90 days of surgery (OR 3.1, 95% CI 1.3 to 7.3, p=0.011), repeat surgery within 90 days of the initial surgery (OR 2.8, 95% CI 1.2 to 6.3, p=0.013) and opioid prescription after the second postoperative week (OR 2.0, 95% CI 1.2 to 3.6, p=0.014). Our model had acceptable discrimination (area under the ROC curve=0.73, 95% CI 0.69 to 0.78), and was well calibrated (Hosmer–Lemeshow test=0.65) and outperformed the base model without trigger adjustment (area under the ROC curve=0.52, 95% CI 0.47 to 0.57) in predicting AE occurrence. The pseudo R2 was 0.18 for the main model and 0.001 for the base model (table 4).
The predictive formula for the likelihood of an AE was calculated from the beta coefficients of the significant triggers in the regression analysis as follows:
The probability of an AE occurrence is expressed as a percentage as follows: Probability (%)=[ex]×100/(1+ex).
When stratifying patients into quintiles by their predicted probability of AE occurrence, 3.1% of patients in the lowest quintile experienced an AE compared with 26% of patients in the highest quintile (figure 1).
Actual adverse event (AE) rates across quintiles of predicted probability of AE occurrence.
We applied the predictive formula in the validation cohort of 2279 patients to flag cases with a predicted probability >10% (n=211, 9.3%), and found that 72 (34% of flagged records and 3.2% of the screened cohort) had an AE documented in the medical record (PPV=34%).
Discussion
The adoption of electronic health records is increasing,14 ,15 but their use to monitor safety after ambulatory surgery remains unclear. Electronic health records facilitate abstraction of large quantities of data, and may allow automated identification of triggers or clues indicating the possibility of an AE.9 ,10 ,16 We evaluated electronic health record-based triggers to identify AEs following outpatient orthopaedic surgery.
Our analysis should be interpreted cautiously in light of its limitations. First, this study was performed at a single urban academic site, which may limit generalisability to other settings. Because practice patterns differ among institutions, the PPV of some of the factors may be idiosyncratic to our hospital. Second, given that there is no gold standard for AE detection, we could not measure the validity of chart review to identify AEs.17 Third, we relied on what was documented in the electronic health record to perform our AE determinations, which may have resulted in an undercounting of AEs.8 Finally, perhaps the greatest limitation is that although the adoption of electronic health records in US hospitals is improving, only a small number of institutions have a comprehensive electronic health records system.14 ,15 Nonetheless, our findings provide incremental evidence of the use of electronic health records in quality and safety initiatives, and should encourage wider adoption of such technology.
Consistent with recent studies in the neonatal intensive care unit5 and the ED,8 we found that several triggers (eg, antibiotic prescription, ED visit, bone/joint or blood culture, repeat surgery, infectious disease consult) were predictive of the development of AEs after outpatient orthopaedic surgery. These same triggers had the highest PPVs to detect surgical site infection, although the values were lower given the relatively low number of infections. In light of how difficult it is to identify AEs following ambulatory surgery, the use of electronic health records as a screening tool to detect perioperative morbidity seems promising and practical, and merits additional study.
We produced a formula to predict the likelihood of identifying an AE in the medical record. For instance, in a patient who had an ED visit within 90 days of surgery and an opioid prescription after the second postoperative week, the value of X is calculated as follows: X=0.71+1.5−3.1=−0.89; and the percentage probability of detecting an AE in the medical record can be calculated as follows: [e−0.89]×100/(1+e−0.89)=71%. A predicted probability >10% seems to be a reasonable cut-off (specificity: 80%; sensitivity: 53%) for determining whether medical records should be manually reviewed for the presence of AEs. Using the tool in our validation cohort, we were able to limit the number of records reviewed to under 10%, with one in three having an AE.
In conclusion, we found that some electronic health record-based triggers had reasonable predictive values, and could prove useful for initiatives to monitor safety in the perioperative orthopaedic setting. Further research is needed to understand the optimal use of electronic triggers as surgical quality indicators and as screening tools to flag cases for manual chart review. It may be helpful to build on this first step, perhaps by developing outpatient speciality-specific tools (eg, foot, hand, sports, etc), evaluating a larger number of patients and reviewing all charts in the validation cohorts to check the sensitivity and specificity of each tool.
References
Footnotes
Contributors MEM: design, acquisition, analysis and interpretation of the data, and drafting of the manuscript. SJJ: analysis and interpretation of the data, and critical revision of drafted manuscript. DR: design, acquisition and interpretation of the data, and critical revision of drafted manuscript.
Competing interests None declared.
Ethics approval Partners Human Research Committee.
Provenance and peer review Not commissioned; internally peer reviewed.