Article Text
PDF
Abstract
Background Unrecognised changes in a hospitalised patient’s clinical course may lead to a preventable adverse event. Early warning systems (EWS) use patient data, such as vital signs, nursing assessments and laboratory values, to aid in the detection of early clinical deterioration. In 2018, an EWS programme was deployed at an academic hospital that consisted of a commercially available EWS algorithm and a centralised virtual nurse team to monitor alerts. Our objective was to understand the nursing perspective on the use of an EWS programme with centralised monitoring.
Methods We conducted and audio-recorded semistructured focus groups during nurse staff meetings on six inpatient units, stratified by alert frequency (high: >100 alerts/month; medium: 50–100 alerts/month; low: <50 alerts/month). Discussion topics included EWS programme experiences, perception of EWS programme utility and EWS programme implementation. Investigators analysed the focus group transcripts using a grounded theory approach.
Results We conducted 28 focus groups with 227 bedside nurses across all shifts. We identified six principal themes: (1) Alert timeliness, nurses reported being aware of the patient’s deterioration before the EWS alert, (2) Lack of accuracy, nurses perceived most alerts as false positives, (3) Workflow interruptions caused by EWS alerts, (4) Questions of actionability of alerts, nurses were often uncertain about next steps, (5) Concerns around an underappreciation of core nursing skills via reliance on the EWS programme and (6) The opportunity cost of deploying the EWS programme.
Conclusion This qualitative study of nurses demonstrates the importance of earning user trust, ensuring timeliness and outlining actionable next steps when implementing an EWS. Careful attention to user workflow is required to maximise EWS impact on improving hospital quality and patient safety.
- hospital medicine
- information technology
- healthcare quality improvement
- nurses
- qualitative research
Data availability statement
Data are available upon reasonable request.
Statistics from Altmetric.com
Key messages
What is already known on this topic
Hospitals are increasingly implementing more sophisticated early warning system (EWS) algorithms to identify patients at risk of clinical deterioration; their impact on nursing practice has been undetermined.
What this study adds
This qualitative study of nurses identified important EWS implementation factors that may advance or undermine success: the EWS must earn nurses’ trust, be compatible with unit-based workflows and ideally provide nurses with helpful guidance about next steps.
How this study might affect research, practice and/or policy
The findings of this study may aid current or future EWS initiatives to ensure that they add value to front-line nurses and provide desired benefits.
Introduction
Acute care nurses are trained to identify patient deterioration and initiate the appropriate intervention to mitigate complications of illness or care.1–3 Several tools, however, ranging from straightforward4 to computationally complex,5 have been created to assist nurses and care teams in identifying patients with early signs of deterioration. Early warning systems (EWS) are algorithms and associated software tools that use patient data, such as vital signs, laboratory trends or ECG tracings, to pick up on subtle changes in a patient’s condition and detect clinical decline that may be missed by care teams.1 6
There are over 100 EWS documented in the literature.7 8 Most collate vital signs, laboratory test results, cardiac rhythm and other physiological indicators, resulting in a score to indicate the patient’s condition. EWS may also incorporate nursing assessments and documentation into their prediction models, which has been correlated with decreases in in-hospital and postdischarge mortality after 24 hours.9 10 Published accounts of EWS have shown mixed results, likely due to flaws inherent to EWS algorithm technology and deficiencies in the implementation of EWS programmes.11–18
Newer, algorithm-based EWS have increasingly been implemented despite these mixed results and without a clear understanding of how they impact clinical care team functions. This phenomenon has been described both in the professional literature,12 as well as in the popular press.19 20 A deeper understanding of the impact of EWS on care team members, especially nurses, is increasingly warranted. Maximising the benefits of EWS algorithm implementation in tandem with nursing practice may strengthen care quality and improve patient safety. On the contrary, burdens such as alert fatigue and competing priorities may limit the uptake, trust and ultimate success of a newly implemented EWS.
Our hospital implemented an EWS programme that consisted of a commercially available EWS algorithm and a virtual nurse monitoring team. We use the term ‘EWS programme’ to denote the EWS algorithm combined with the centralised monitoring and operational parameters (alert thresholds and workflows) that were implemented. As part of an effort to evaluate and optimise the implementation of the EWS programme, we sought to understand the nursing perspective on the EWS programme’s impact.
Methods
EWS implementation
In June 2018, our hospital implemented an EWS programme, consisting of a commercially available EWS algorithm with centralised virtual nurse monitoring with the goals of reducing preventable in-hospital cardiac arrest, detecting early signs of clinical deterioration and improving the efficiency of care delivery. The EWS algorithm collated data from a patient’s electronic health record (EHR), incorporating 26 EHR data points, including laboratory values, vital signs and nursing assessments.9 It calculated a composite score to summarise a patient’s condition and automatically updated as data were entered into a patient’s EHR.21 Scores were electronically available for all hospitalised patients. The healthcare team could view a patient’s EWS algorithm score, score trend and risk level via the patient’s EHR. The EWS algorithm triggered an alert when a patient’s score trended downward or fell below a threshold score.
In this implementation, EWS algorithm alerts were monitored in real time, around the clock, by a virtual care team (VCT) of 20 critical care nurses. Three VCT nurses worked at once in 12-hour shifts, monitoring all hospital patients across all non-ICU wards (mean 2430 patients monitored/month). The VCT room contained a large, shared monitor that displayed a graph of each patent’s EWS algorithm score trend during their current hospitalisation. The graphs were colour-coded based on the EWS algorithm’s assessment of a patient’s risk of decline (grey for low risk, red for severe risk). When an EWS algorithm alert was triggered, the patient’s information would display on the shared monitor and the VCT nurse’s personal computer monitor. The assigned VCT nurse then followed a standardised protocol to briefly review the patient’s EHR to gather patient context before calling the patient’s acute care nurse to notify them about the EWS algorithm warning. In the initial implementation, VCT nurses called the bedside nurse for every alert. Subsequent changes to the protocol gave VCT nurses discretion to not call on false-positive alerts. VCT nurses did not monitor alerts on patients in the intensive care unit (ICU) or postoperative care unit, and they did not notify floor nurses about EWS alerts on patients with ‘Do Not Resuscitate’ orders. Over the phone, the VCT nurse offered support to the acute care nurse and made case-by-case suggestions for troubleshooting possible patient problems (figure 1).
Flow diagram of the early warning system (EWS) programme.
Acute care nurses were required to take an online training module before go-live. Additionally, the implementation team communicated about the EWS programme through hospital marketing articles, nursing leadership presentations and a guest presentation by the EWS algorithm creator.
Design, setting, participants
This article reports on a qualitative evaluation undertaken as part of the implementation of an EWS programme in an acute care, non-ICU setting, at a 700-bed tertiary care hospital. To assess a range of experiences, six units of varying specialties were sampled based on variation in the frequency of alerts in the 6 months before study initiation. Alert frequency ranged from a mean of 35 calls per month to 111 calls per month (table 1). The focus groups occurred on six units 1 year following the implementation (July 2019 to September 2019).) This study was planned and executed in conjunction with the institution’s nursing special projects office. We received IRB exemption as this work was conducted for quality improvement purposes.
Specialty and alert frequency of sampled units included in this study
Interview guide
With input from the office of nursing research and nursing supervisors, the research team iteratively prepared a semistructured interview guide (online supplemental appendix). Discussion guide topics included: EWS programme use in everyday clinical responsibilities; scenarios where and how the EWS programme was helpful or unhelpful, communication using the EWS, implementation of the EWS programme and ideas for improvement. We made iterative changes to the semistructured interview guide based on incoming data to probe further on topics of emerging themes, such as EWS alert timing and actionability.
Supplemental material
Data collection
Focus groups followed nurse staff meetings on all shifts to maximise nursing input and minimise workflow disruptions. Meetings occurred before or after a nursing work shift. Sessions were conducted by one of three female facilitators from the team who had prior qualitative experience and were not involved in the implementation or ongoing operations EJB, ACP, orEAS). Sessions occurred in a quiet meeting room with the participants and the facilitator sitting circumferentially following staff meetings. Before each focus group, the facilitator described data de-identification and confidentiality, and she discussed the nature of the discussion (eg, to garner nursing perspective for quality improvement purposes). Nurse participation was voluntary and verbal consent was obtained; declines were not formally tracked. Focus groups were audio-recorded and lasted 10–25 min. At the end of each session, participants completed a short demographic questionnaire.
Data analysis
Focus group audio recordings were transcribed verbatim and data were organised using NVivo V.12 (QSR International). We used inductive analysis based in Straussian grounded theory22 to identify themes through constant comparison.23 24 As transcripts became available, two investigators (EJB, BHC) independently immersed themselves in the data to gain a broad sense of participant perspectives. They each completed open coding of transcripts before collectively reviewing data and agreeing on a coding scheme. The codebook was tested for clarity and accuracy by a third independent author (KEF). After thematic saturation was reached, all transcripts were then coded using the final codebook by two independent investigators (EJB, ACP). Codes from two independent investigators were compared and had inter-rater reliability of 98.4%, with a Cohen’s κ of 0.825. Discrepancies were discussed by reviewers until a consensus was reached.
Results
Twenty-eight focus groups were held on six inpatient hospital units. A total of 227 bedside nurses participated, of which 206 (91%) were female, 184 (81%) work full-time and 175 (77%) were bachelor’s-prepared in nursing or beyond (table 2). Focus groups had a mean of eight participants (range: 2–14).
Participant demographics
We identified six principal themes: (1) Timeliness—bedside nurses reported being aware of their patient’s deterioration and having begun or completed the appropriate intervention before the EWS algorithm triggers and the VCT calls; (2) Workflow interruption—nurses reported often being called about an EWS alert while they were busy implementing the appropriate medical interventions; (3) Accuracy—nurses perceived the EWS algorithm to be inaccurate, citing many anecdotes of false positives and false negatives, especially with the EWS’s inclusion of subjective nursing assessment data, such as urine colour and patient mood; (4) Actionability—nurses felt the VCT rarely offered actionable, novel suggestions, however, some nurses appreciated the EWS and VCT as a safety net; (5) Underappreciation of core nursing skills—nurses emphasised the value of in-person assessments and were concerned with the reliance on technology by a hands-on profession; and (6) Opportunity cost—nurses cited the need for additional hands-on support and criticised spending on the EWS programme during a time of hospital financial stress (table 3).
Six principal themes
Timeliness
For the EWS programme to be effective, the care team must be warned about a patient’s deterioration in a timely fashion and be able to initiate interventions promptly. Given that the EWS algorithm depended on data entered by the care team to calculate a patient’s risk score, individuals obtaining patient data and entering it into the EHR were likely aware of abnormalities before the EWS algorithm even had a chance to calculate a composite score. Nurses pointed out this design conflict of the EWS algorithm:
It’s not really helpful because I'm the one that’s entering the data that they're looking at. So then (the virtual care team is) calling me, “Do you know this heart rate’s 140?” Yeah! I just put it in!
In fact, nurses rarely recalled being notified of a patient’s decline by the EWS programme. Instead, they had identified the deterioration and initiated interventions (often emergently) before receiving an EWS alert:
If you're in like (in) an emergent type of situation… or say (the patient) did decline rapidly and we've called a rapid (response), (the virtual care team) will keep calling until they talk to somebody, and it’s like, OK I already know. I'm trying to deal with it. I don't need you to tell me they just declined because I'm aware. I'm in here. It becomes very annoying.
Workflow interruption
One of the unintended consequences of this implementation of the EWS programme was that nurses were often interrupted by the notification of the EWS alert. If an alert happened promptly after the decline occurred, nurses were frustrated to receive a call while trying to initiate interventions, such as calling the patient’s physician or activating the rapid response team.
Sometimes it’s flustering. If you've got a lot going on, and this keeps going off because you've got someone calling you, it’s kind of a bit much. Because you're trying to focus on what you've got in front of you.
In addition, if an EWS alert and associated VCT call were delayed, nurses would be required to step away from their next task to discuss an issue they had resolved.
Nearly all focus groups noted the interruptive nature of the VCT calls; however, some nurses expressed sympathy for the VCT, for example, ‘It’s not the VCT’s fault’, ‘How would they know that I’m busy’. Other nurses reflected on the workflow interruptions with disdain, saying “Seriously! Stop calling!”
Accuracy
Nurses often discussed examples of VCT calling about patients whom they perceived were stable or well i.e., false positive), and the VCT failing to call on patients whom they perceived were declining emergently (i.e., false negatives). This perception undermined the trust in the EWS algorithm.
Most times we get patients that have been really, really sick and I have not heard from (the EWS program). Not a peep. Like tanking (deteriorating), rapiding (requiring rapid response team), and they're never calling me. And I'm charting vitals in real-time and then I chart someone’s mood is off because they're mad at being here and then they call me, like, right away!
Nurses often attributed false positives to the inclusion of subjective nursing assessment documentation, such as urine colour or the patient’s affect, in the EWS algorithm. Inclusion of such data was a novel feature of this EWS algorithm; however, nurses placed little value on alerts that triggered based on such data due to inter-nurse variability.
Actionability
Bedside nurses felt the VCT rarely offered novel or actionable suggestions; however, some nurses appreciated the EWS and VCT as a safety net. Nurses reflected on EWS warnings as go/no-go alerts. In the case of a go alert, the EWS alert may trigger on a patient who is actively deteriorating and requires escalation of care, for example, a rapid response or transfer to the ICU. For alerts without a clear and immediate action (no-go), nurses were often unclear what the next steps needed to be. Nurses reported that the VCT often told them the value that triggered the EWS algorithm alert, but their clinical decision-making was unchanged.
I can't say that I’ve ever acted on something that I've gotten a call on […] as far as being proactive or saying oh gosh, we better call a rapid (response) on that on this person or something because (the EWS program) caught it? No, I can't say that’s happened.
Some nurses appreciated the existence of the EWS programme as a safety net even though their clinical decision-making was rarely, if ever, impacted by the EWS programme.
I don't think it’s ever unhelpful to have more eyes on a patient!
Underappreciation of core nursing skills
Nurses believed that the EWS programme undermined the value of their core nursing skills, and they emphasised the value of their experience. Nurses reported that they felt that patient monitoring was a core component of nursing, and nurses discussed their ability to identify sepsis and initiate early interventions without the EWS programme.
I'm physically seeing my patient. I can see if they're going downhill or not. I feel like we have so many patients that are septic that we're so good at it, we don't need a computer to identify it for us.
Some were also concerned about reliance on technology, emphasising that nursing is a hands-on profession, and such skills cannot be replicated by algorithms.
I think there’s a real danger here in transferring all of our patient care over to graphs and computers and third-party interventions because nursing is a bedside profession and it’s a hands-on caregiving process.
Opportunity cost
Nurses cited a need for additional hands-on support, especially during emergencies. Many criticised EWS programme spending during times of financial stress, reporting that their differential pay was being decreased. When asked for suggestions for improvement, many nurses suggested eliminating the EWS programme and using the money saved to hire additional nurses to increase the size of the existing rapid response team.
We could use these [nurses] on our floor, and throughout the hospital, instead of in front of the computer screen.
However, some nurses appreciated the constant virtual availability of the VCT nurses. EWS algorithm aside, some ward nurses reported they would occasionally reach out to the VCT nurse for advice regarding next steps or to look up care protocols.
Discussion
We identified six themes related to acute care nurses’ perspectives regarding the implementation of a commercially available EWS algorithm and associated virtual nurse monitoring in a large, academic medical centre. The themes reflect barriers encountered by nurses in our study, including concerns about EWS alert timing (timeliness, interruption), low trust that the EWS programme will be helpful (accuracy, actionability), and perceptions of trade-offs between professional experience and algorithmic alerts for predicting deterioration (nursing skills, opportunity costs). While these themes emerged from experiences with one particular EWS algorithm, they are likely to be transferable to others because of their broad nature. Building on the themes, healthcare organisations may wish to consider taking steps to engender trust (prospective validation of algorithms, calibration of alert thresholds, transparency of information), sociotechnical assessments of workflows and their intersections with both data input into algorithms and their downstream alerts, and actionability and guidance around as areas that require solving for EWS programme implementation.
Engendering trust
Systems that aim to predict an outcome must earn the user’s trust to be effective.25 26 Our experience highlights where trust must be earned, and how lack of trust may have undermined the potential value of the EWS programme. If EWS programme users do not trust the system, they may discount important signals. Despite preimplementation education and nurse leadership involvement during implementation, our data showed a lack of trust in the EWS programme by acute care nurses who interacted with it most. Nurses discussed a lack of belief in the EWS algorithm’s ability to effectively identify patients at risk for decline frequently citing perceived false positives and perceived false-negative examples from their practice.
Inaccuracy inherent to the EWS algorithm likely contributed to nurses’ lack of trust, as EWS algorithms suffer from high false-positive rates, sometimes upwards of 60%, which may reduce care-team trust and clinical value of the EWS algorithms.27 While the algorithm had been validated using historical data, rigorous prospective validation with real-life scenarios such as delayed data input or missing data was lacking.10 21
Implementation factors, in addition to algorithm performance, may have also impacted trust. Escobar and colleagues used an iterative approach to the implementation of their EWS programme, which was ultimately found to decrease inpatient mortality.11 Their nuanced implementation included standardised guidelines of roles, responsibilities and next steps that were created after piloting the programme on a smaller scale.28 Like our hospital’s implementation, they used virtual centralised monitoring and allowed for deliberate non-reporting of alerts based on the virtual nurse’s assessment. They reported undertaking efforts to gain acute care nurse trust by including virtual nurses in practice runs to gather and exchange feedback. Focusing on engendering trust with early experiences, data, and better alert calibration may promote buy-in and may ultimately promote success.
Workflow intersections
EWS programmes need to take into consideration the temporality of when patient health data are obtained, entered for computation and reviewed because notable gaps will decrease the effectiveness of the early warning. In this implementation, there were several opportunities for delay throughout the EWS alert and response process. First, delays in charting occurred. When probed, nurses reported a typical charting lag time of 0.5–2.5 hours, citing the need to prioritise time-sensitive patient care tasks. Second, delays in VCT alert processing occurred while VCT nurses triaged EWS algorithm warnings. Third, delays in connecting with the acute care nurse occurred if the VCT could not reach the acute care nurse by phone in a timely fashion. Fourth, delays in closing the loop with appropriate intervention (i.e., fluid resuscitation or a medication change) occurred and may have dampened the utility of the EWS programme.
Ideally, the EWS programme provides the care team with previously unknown information of impending patient decline. However, nurses felt this was rarely the case, possibly due to the lack of timeliness throughout the process. This is consistent with a quantitative study that found medical teams were aware of the patient deterioration before the EWS algorithm triggered in more than 90% of cases.29 In part, this may be due to the workflow mismatch and competing priorities of caregiving, data collection, data input, EWS interpretation and intervention. Watson and colleagues explored the temporality of EWS warnings and found that natural delays in documentation leads to untimely EWS warnings and may minimise the benefits.30
Nurses in our study commonly reported that EWS programme warning calls often interrupted their work. This often caused user frustration and likely contributed to alert fatigue. Literature shows alert fatigue is a common, but detrimental, outcome associated with the implementation of health technology.31 32 Even if initial buy-in is high, compliance often decreases over time.31 33 Bedoya and colleagues implemented an EWS algorithm that fired automatic notifications in a patients’ EHR, which may cause fewer workflow interruptions; staff, however, overwhelmingly ignored the notification, and the EWS programme performed poorly.32 Ultimately, users in our study and others ended up ignoring or discounting the EWS programme.34
Future implementations of EWS should aim to minimise opportunities for delay to maximise EWS programme timeliness with the goal of prompt recognition of clinical deterioration leading to early intervention and prevention of in-hospital mortality. Sociotechnical approaches that review the system participants, their work and how the technology functions in unison are required.35
Actionability and guidance
The large quantity of granular data inputted into a patient’s EHR provides the opportunity for a near-real-time risk assessment by a well-calibrated EWS algorithm. However, EWS alerts may indicate patient decline but do not explain what is wrong or the next course of action, especially when patients did not require immediate stabilisation. In these cases, nurses felt unclear about next steps and would write off the warning as a no-go alert, meaning no intervention was needed or completed. Though these warnings may not require immediate action, they may predict future deterioration. In these cases, nurses may benefit from clearer guidance regarding the next steps. Preimplementation education did not address this specific scenario. Teams may benefit from taking a cognitive time out to bring the patient’s risk of deterioration to the forefront of their minds and evaluate the reasons for a possible decline.
Studies found that nurses had no difficulty intervening and escalating care if an EWS algorithm alert was consistent with the nurse’s assessment of the severity of patient deterioration; however, they hesitated to act if the EWS algorithm did not match their intuition of the patient’s risk.34 36 This hesitancy was compounded by the lack of buy-in from providers36 and fear of being wrong in front of peers or supervisors.37 Future EWS programme implementations should use input from nurse stakeholders, create helpful guidelines that address uncertainty about next steps and provide context about the broader implication of an EWS algorithm alert.
System changes
As a result of lessons learnt during this evaluation, changes were implemented. For example, the VCT stopped calling nurses when they could clearly see actions were being undertaken, such as orders for fluids in response to hypotension. Technology to enable real-time documentation of vital signs and assessments is planned. Guidance around palliative care consultations was created for patients with down-trending or persistently low scores. A prospective outcome evaluation was concurrently undertaken.38
Limitations
It is important to consider the limitations of this qualitative study. First, this study was conducted at a single site which limits the broader applicability of the results. However, we did sample from several units across all shifts and have a large qualitative sample size. This project explores the views of bedside acute care nurses but does not include the views of the VCT nurses or other clinicians who interacted with the EWS programme. Separate discussions with VCT nurses affirmed that they perceived floor nurses to be frustrated by some of the calls, and they were actively adjusting to optimise the utility of EWS alert calls. Lastly, qualitative data were obtained through focus groups, which allowed for rich discussions and a large qualitative sample but introduced the potential for bias and disproportionate speaking time.
Conclusion
In this qualitative study with 227 bedside nurses, we identified six themes related to the use of an EWS programme in the acute care setting. Addressing these themes in implementations of EWS may promote adoption and fidelity. Prospective validation and piloting EWS programmes before large-scale implementation may help engender necessary trust. As EWS algorithms become more ‘black box’, using machine learning or other computational strategies with inner workings that may be opaque to users, steps to engender trust that the algorithm is accurate will likely aid in usage. Timeliness is critical to EWS performance and attention to user workflow is required to maximise EWS impact on improving hospital quality and patient safety. Organisational leaders may use these data when implementing an EWS programme in their institutions.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @bradcrotty
Correction notice Jean M Holt has been updated Jeana M Holt
Contributors All authors qualify for authorship and have reviewed and approved the final manuscript. BHC is responsible for the overall content as the guarantor, accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding This study was funded by Medical College of Wisconsin and Advancing a Healthier Wisconsin Endowment.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.