Article Text
PDF
Abstract
Importance Accurately and routinely identifying factors contributing to inpatient mortality remains challenging.
Objective To describe the development, implementation and performance of a new electronic mortality review method 1 year after implementation.
Methods An analysis of data gathered from an electronic instrument that queries front-line providers on their opinions on quality and safety related issues, including potential preventability, immediately after a patient's death. Comparison was also made with chart reviews and administrative data.
Results In the first 12 months, reviewers responded to 89% of reviews sent (2547 responses from 2869 requests), resulting in at least one review in 99% (1058/1068) of inpatient deaths. Clinicians provided suggestions for improvement in 7.7% (191/2491) of completed reviews, and reported that 4.8% (50/1052) of deaths may have been preventable. Quality and safety issues contributing to potentially preventable inpatient mortality included delays in obtaining or responding to tests (15/50, 30%), communication barriers (10/50, 20%) and healthcare associated infections (9/50, 18%).
Independent, blinded chart review of a sample of clinician reviews detected potential preventability in 10% (2/20) of clinician reported cases as potentially preventable. Comparison with administrative data showed poor agreement on the identification of complications with neither source consistently identifying more complications.
Conclusions Our early experience supports the feasibility and utility of an electronic tool to collect real-time clinical information related to inpatient deaths directly from front-line providers. Caregivers reported information that was complementary to data available from chart review and administrative sources in identifying potentially preventable deaths and informing quality improvement efforts.
- Patient safety
- Quality improvement
- Quality improvement methodologies
- Safety culture
- Significant event analysis, critical incident review
Statistics from Altmetric.com
- Patient safety
- Quality improvement
- Quality improvement methodologies
- Safety culture
- Significant event analysis, critical incident review
Introduction
Amid the efforts to improve quality and safety, organisations continue to seek innovative solutions for many challenges, such as reducing inpatient mortality.1 The number of preventable deaths in hospitals continues to receive attention since the Institute of Medicine (IOM) estimated that as many as 98 000 patients are victims of preventable death in American hospitals.2 ,3 Current trends show that mortality in US hospitals has dropped,4 but it is unclear whether this is due to safer care, scientific advances or other factors such as increased use of outpatient hospice. Recent research has suggested that the amount of error in healthcare may not have changed.5
The imperative to reduce mortality has grown with the greater transparency surrounding inpatient mortality rates and their introduction into governmental initiatives like value based purchasing in the US.6 Yet, hospitals continue to struggle with identifying, quantifying and characterising which inpatient deaths are preventable. Manual chart reviews are expensive, have proven difficult tools for the abstraction of accurate data,7 and can be limited by inter-reviewer variability.8 Another widely accepted practice of reviewing cases in morbidity and mortality conferences is laudable, but can be inadequate for comprehensively identifying system-wide problems.9
To better identify individual and systems issues that may contribute to inpatient deaths, our institution developed a mortality review process that elicits information directly from clinical staff. Our goal was not just to gather information about individual cases, but to identify system-wide issues that could be improved throughout the hospital. We also sought to foster a culture of safety, asking providers to reflect upon quality and safety issues after every inpatient death. Whether an electronic provider based mortality review process was feasible was an open question for us and here we report on our experiences from the 1st year of launch.
The development process
Brigham and Women's Hospital is a 793-bed academic teaching hospital in Boston, Massachusetts. In 2011, the hospital admitted 46 498 patients resulting in 45 659 adult discharges and 7028 newborn discharges. A total of 966 patients died in the hospital, resulting in an observed mortality of 2.1% for the year. Using the University Health System Consortium risk adjustment model, the expected mortality rate was 2.6%. This translated to an observed to expected ratio of 0.82, meaning that after risk adjustment, Brigham and Women's Hospital experienced fewer deaths than would be expected.
Prior to developing the new review tool, we inventoried the mortality review processes already in place. We learnt that most departments were reviewing deaths in some manner, although case selection and depth of review varied. In response to an internal survey, 59% of departments reported reviewing all deaths, 27% of departments reported reviewing some deaths and 14% reported that they did not routinely perform any reviews. Additionally, there was no mechanism to aggregate factors that may have contributed to mortality across departments. These shortcomings highlighted the need for a process that would capture hospital-wide issues.
We began by building consensus around the initiative, determining the best manner to collect mortality-related information, and identifying what information to procure. We sought to capture the opinions and interests of various stakeholders, including attending and trainee physicians, nurses, quality and safety leaders and staff, and risk management (see online supplementary appendix table 1). These discussions led to seven consensus principles around which our electronic mortality review process was constructed (table 1).
Developing the mortality review process—consensus design principles
Principally, we recognised that clinicians feel burdened by administrative processes that compete with clinical duties. Therefore, we knew that the new method should not be burdensome, particularly in high-mortality areas. Second, we appreciated how many clinicians may have apprehensions about the legal ramifications of commenting on the preventability of a patient's death. While the requirements for peer protection vary by state, we designed our process so that their responses would be confidential and peer review protected. Specifically, clinicians are informed: “Your responses may be shared with qualified quality and safety staff, but not with any other members of your care team.”
To capture a wider range of opinions, we designed the process to notify at least two clinicians caring for a patient at the time of every inpatient death: the attending physician and the clinician designated the first responder—typically the intern, resident or physician assistant assigned to the patient. Each death triggers an email asking these clinicians to provide their assessment by completing the mortality review instrument that can be accessed through a link in the message.
We urge the clinicians to complete the questionnaire as soon as possible because we feel fresh knowledge provides more valuable insight. Responders can also confidentially request that other clinicians complete a patient's review. For example, a surgeon can add an anaesthesiologist or an intensive care unit physician can add someone from the patient's floor team prior to transfer (and those added will not know who made the request). Reviews that are not completed within 72 h trigger a reminder. Subsequently, reminders are sent weekly for 6 weeks.
The electronic questionnaire requires no training and takes only 5 min on average to complete. The questions are intended to be broad enough so that the clinician can answer from first-hand knowledge of the case, without needing to refer to a patient's chart. Clinicians are neither discouraged nor prohibited from consulting a chart—it is left to their discretion.
The questionnaire covers four categories of system-level events that may contribute to preventable mortality (see online supplementary appendix table 2). These include:
-
Healthcare associated infections (eg, ventilator associated pneumonia);
-
A range of common hospital associated complications (eg, venous thromboembolism, adverse drug event or surgical complications);
-
Delays in obtaining or responding to tests or procedures (eg, blood work or radiology studies); and
-
Problems in communication between clinical teams (eg, floor team to intensive care unit team, outside hospital transfers arriving without prior notice).
The questionnaire asks the clinician to provide a brief summary of the patient's course, code status and whether it changed during the hospitalisation, and to provide an opinion on the preventability of the patient's death. Preventability is scored on a scale of 1 to 5 with scores of 1 and 2 being non-preventable; a score of 3 representing the presence of a medical error but not thought to be a preventable death; and 4 and 5 being preventable.
The implementation process was iterative and took 15 months. The group began work on the design principles in January 2010 and went through a period of paper-based pilots in the Medical Intensive Care Unit (MICU) from April to June 2010, resulting in major revisions. The team spent the next year building the web-based interface and programming infrastructure. The electronic instrument was piloted in five departments over 3 months in March 2011, covering medical and surgical specialties. The first version of the instrument was officially launched hospital-wide in June 2011.
Since then, based on user feedback, we have continued to make revisions to the questionnaire. One important modification to the instrument was the addition of a feature allowing front-line staff to request contact with the quality and safety department or with our centre for professionalism and peer support to further discuss a case. Screenshots of the instrument may be found in our online supplementary materials (appendix screenshot).
In the year following implementation, we expanded the scope of reviews by creating and launching an emergency department (ED) questionnaire for patients that die in the ED or shortly after admission from the ED. The review instrument is designed to help identify care improvement opportunities in the ED, including triage decisions. We hope to extend the review process to include nurses for a more comprehensive picture on quality and safety related matters.
Evaluation of our process
We evaluated our response data for our 1st year of implementation covering patient deaths between 1 June 2011, and 1 June 2012. In addition to the response rate, we calculated the percentage of patients for which we had received at least one completed review. When we had multiple reviews for a patient, we used the response most critical of the care we provided. For example, a ‘yes’ response to a question of quality concern (presence of an infection, presence of a complication, preventable death) was considered as a yes response across all reviews for a patient-level analysis. The time to complete the questionnaire was measured electronically through the software housing the instrument. Reviews that required less than 10 s or greater than 15 min were excluded from time to complete calculations.
For statistical analysis, all variables were categorical and the results reported as percentages. For proportions calculated using multiple ratings on the same patients (eg, overall percentage of preventability ratings with a score of 1), 95% CIs were calculated using a modified Wilson CI for clustered binary data.10 For patient-level analyses in which one summary dichotomous variable is calculated for each patient (eg, where at least one reviewer reports an infection), Fisher’s exact test was used to compare proportions between groups (eg, preventable and non-preventable cases). When dichotomising cases to preventable or not, deaths that were returned without any preventability score (n=6) were treated as non-preventable. Agreement between a single human reviewer and administrative data were calculated using Cohen’s κ. When considering multiple (two or more) ratings on the same patients (eg, from attending and residents), agreement between a pair of raters is estimated using a pairwise κ coefficient.6 Data were analysed using SAS V.9.3 (SAS Institute, North Carolina, USA). All p values were two-sided, and p values <0.05 were considered statistically significant.
Because there is no established standard in assessing the preventability of inpatient mortality, one investigator performed an independent chart review to compare these findings with those of the front-line reviews performed by clinicians. Using 20 randomly selected cases that were rated 4 or 5 (possibly or likely preventable), and 20 randomly selected cases rated 1, 2 or 3 (not preventable) by clinician reviewers, the investigator (blinded to the clinician ratings) performed a chart review to score the cases on their preventability. The investigator's ratings were then compared with the clinician mortality reviews.
In an effort to ascertain whether the review process revealed additional information not collected in some other manner, we also compared the identification of common inhospital complications implicated in the deaths reviewed with ICD-9 coded data for the same patients.
Results
During the first 12 months of the review process, there were 1068 inpatient deaths. For 99% (1052/1068) of these patients, at least one clinician completed a review of the death. There was an 89% response rate to the electronic notices, with a total of 2491 reviews completed. The median amount of time required for a questionnaire to be returned was 16 h and 38 min, and each took a median time of 3 min and 45 s to complete. A total of 9 respondents requested peer support and 23 requested direct contact with the quality safety department to further discuss the case.
Of the 2,411 reviews given a preventability rating, the death was rated as a ‘1’ (not preventable) in 1759 of them and a ‘4 or 5’ (preventable) in 68 of them (table 2). When reviewer opinions on preventability were dichotomised to not preventable (score of 1, 2, or 3) or preventable (score of 4 or 5), the reviewer agreement was 96.5% and κ was 0.37 (95% CI 0.21 to 0.53). When the reviews were dichotomised to no error (score of 1 or 2) and error in the case (score of 3, 4, or 5), agreement was 94.6% and the κ was 0.34 (95% CI 0.22 to 0.47). Stratifying agreement by level of training (ie, attending physicians compared with house officers or physician assistants), reviewers agreed on preventability of death 96.1% of the time and agreed on the presence of an error 94.6% of the time. Stratification by level of training resulted in κ scores of 0.19 (95% CI 0.06 to 0.32) for preventability and 0.22 (95% CI 0.11 to 0.33) for the presence of error.
Preventability scores based in all reviews and at the patient level
At the patient level, 1002 (95.2%) deaths were rated non-preventable and 50 (4.8%) deaths were rated preventable. In the non-preventable deaths, the most common complication present was a healthcare associated infection (238/1008, 24%) (table 3). Among deaths rated preventable, respondents indicated that a delay in obtaining or responding to tests or procedures was present in 54% of the cases (27/50) and healthcare associated infections in 50% of the cases (25/50).
Factors that contributed to or caused deaths based on preventability of case
Independent, blinded chart review of a sample of 20 cases in which at least one clinician had rated as potentially preventable, detected potential preventability in 10% (2/20) of these cases. Of 20 cases rated as non-preventable by clinicians, the blinded reviewer rated 0% as preventable, resulting in a κ of 0.1. When the data was dichotomised into assessment of ‘error present’ versus ‘no error present,’ the chart reviewer reported an error in 29% of the error cases and in 0% of the no error cases, resulting in a κ of 0.28.
Comparison of conditions present at death as identified in the clinician reviews with those identified via administrative data showed a low level of agreement on a broad range of conditions (table 4). A large amount of disparity was found in the identification of venous thromboembolism, sepsis, technical surgical complication requiring reoperation and pressure ulcers.
Comparison of administrative data (ICD-9 codes) with results from mortality review questionnaires
Discussion
We developed an electronic review process designed to engage healthcare providers in identifying system-wide factors that contribute to inpatient deaths. At the start of our endeavour, whether clinicians would embrace such an approach was unknown. In its 1st year, the new process yielded high rates of completion (89%) that exceeded our expectations. In addition, we found that clinicians provided valuable information on what they thought caused the patient's death, and on potential causes of preventable deaths and suggestions on how care could be improved. As a result of the information provided by clinicians, we have begun implementing changes aimed at improving patient safety (table 5).
Examples of changes implemented as a result of input from caregivers during mortality review process
The high response rate and timeliness with which the reviews were returned demonstrated to us that providers can be willing to take the time to provide information related to quality. Reflecting on the implementation of our process, we have identified aspects of our approach that may have achieved clinician engagement. We started by expending substantial time on gathering stakeholder support. Our team spent several months vetting the process with clinicians, quality leaders and senior leadership. Before launch, we piloted several paper versions of the questionnaire, allowing for modifications aimed at usability.
Another concern for us was whether clinicians would provide candid information. We have learnt that they will. In 7.6% of the deaths, reviewers identified a potential error, and in 7.7%, they provided a suggestion for improvement. Nevertheless, just as under-reporting is a problem with safety reporting systems, we continue to wonder whether clinicians are still not always reporting all concerns through this process. In an effort to increase reporting by showing its value, we have started to disseminate the aggregated results of our reviews as well as stories highlighting improvement efforts initiated in response to these questionnaires.
Since this was a new process, we compared our results against those of traditionally used chart review and administrative data sources. We found poor agreement between the preventability ratings from our blinded retrospective chart review as compared with the clinician reviews. However, in the sample we selected, the clinician reviewers had identified a greater number of preventable cases compared with the independent reviewer. This could support the premise that a third party retrospective chart review may not always be able to capture all the relevant details of a patient's course and that clinicians can have special knowledge of each patient's individual circumstances.11 ,12
Our comparison with administrative data did not allow us to check agreement on preventability ratings (administrative data cannot make a preventability determination), but did allow us check agreement on the presence of quality and safety related complications, and again we found very low level of agreement between the two. Neither the reviewers nor the administrative data consistently identified more complications across all the categories, although there were a number of complications that clinicians identified that administrative data could not. This suggests that clinical reviews could be complementary to administrative data use.
We also found, however, only fair agreement (κ=0.34) when it came to a clinician reporting a possible error. We attribute this disagreement to the often subjective nature of clinical judgment—a phenomenon that has been observed in other work looking at assessing preventable inpatient mortality8 ,13–15—or because different members of the team may have different perspectives or facts based on their role in the team. However, the challenges of using retrospective chart reviews and administrative data to identify certain problems that are not reflected in the chart or in billing codes, such as delays in treatment or communication concerns, further builds the case for directly querying front-line clinicians.
Reducing preventable mortality remains difficult,16 ,17 and we had hoped that querying clinicians would allow for more accurate identification of preventable deaths. Despite our early success in launch, there remain many unanswered questions. Principally, is this tool the best method for assessing preventability? Furthermore, how can we turn the information we have gathered (see online supplementary appendix table 3) into action to improve care, and can these efforts lead to a measureable improvement in inpatient mortality? Can these efforts help foster a stronger awareness of safety? Other institutions with mortality review programmes have seen improvements in mortality rates, and we hope to see such a trend as well.18
Empowered by what our clinicians have reported, as described above, we have created a mortality review committee that reviews cases and provides oversight on intervention. The group reviews deaths for which any respondent reported a potential error, provided any free text responses, or made a suggestion for improvement. Improvement efforts that have resulted include: improved weight-based decision support around anticoagulant use, mechanisms surrounding hand-offs for outside hospital transfers, and enhanced staffing for consults in specific circumstances. We are also exploring how to improve the timing of end of life care for patients, as clinicians frequently report earlier conversations would have been beneficial. While our hospital has been working towards elimination of hospital-associated infections, the findings of these surveys have re-emphasised the importance of these efforts.
Improving quality and safety remains an imperative in US healthcare. One of the largest barriers to quality improvement is the under-reporting of errors and factors contributing to injury, including death. With buy-in from clinical staff and leadership, we have developed a review process that allows clinicians to provide systems-level information on opportunities to improve care. Our results show that a process that engages clinicians and captures all inpatient deaths is indeed feasible as we continue our endeavour to reduce preventable mortality.
Acknowledgments
The authors thank Kaycee Ruprecht, Alexander Shifrin and James Meninno for their exceptional efforts in the development of the electronic review instrument and Troy Tomilonus and Jason Chen for their invaluable assistance with data collection and analysis. The authors also thank Valerie Kong, Li-Hui Ueng, Ryota Konishi, David Rosales, Tracy Hu and Jessica Erickson for their contributions to project development and management.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online Screenshots
- Data supplement 2 - Online Tables
Footnotes
-
Contributors All six authors of this manuscript made substantial contributions to the conception or design of the work; including the acquisition, analysis and interpretation of data. Principally AP was responsible for initial drafts of the work and the remainder of the team worked in concert to revise it critically for important intellectual content. All authors gave final approval of the version to be published. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
-
Funding None.
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.