Use of in situ simulation and human factors engineering to assess and improve emergency department clinical systems for timely telemetry-based detection of life-threatening arrhythmias
- Leo Kobayashi1,
- Ramakrishna Parchuri2,
- Fenwick G Gardiner3,
- Gino A Paolucci3,
- Nicole M Tomaselli3,
- Rakan S Al-Rasheed4,5,
- Karina S Bertsch3,
- Jeffrey Devine3,
- Robert M Boss3,
- Frantz J Gibbs1,
- Eric Goldlust1,
- James E Monti1,
- Brian O'Hearn3,
- David C Portelli1,
- Nathan A Siegel1,
- David Hemendinger6,
- Gregory D Jay1,4,7
- 1Department of Emergency Medicine, Alpert Medical School of Brown University, Providence, Rhode Island, USA
- 2Department of Biomedical Engineering, Rhode Island Hospital, Providence, Rhode Island, USA
- 3Emergency Department, Rhode Island Hospital, Providence, Rhode Island, USA
- 4Rhode Island Hospital Medical Simulation Center, Providence, Rhode Island, USA
- 5King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Kingdom of Saudi Arabia
- 6Information Services, Lifespan, Providence, Rhode Island, USA
- 7School of Engineering, Brown University, Providence, Rhode Island, USA
- Correspondence to Dr Leo Kobayashi, Department of Emergency Medicine, Alpert Medical School of Brown University, Rhode Island Hospital Medical Simulation Center, Suite 106, Coro-West Building, 1 Hoppin Street, Providence, RI 02903, USA;
- Received 28 May 2012
- Revised 19 July 2012
- Accepted 16 August 2012
- Published Online First 11 October 2012
Background and objectives Medical simulation and human factors engineering (HFE) may help investigate and improve clinical telemetry systems. Investigators sought to (1) determine the baseline performance characteristics of an Emergency Department (ED) telemetry system implementation at detecting simulated arrhythmias and (2) improve system performance through HFE-based intervention.
Methods The prospective study was conducted in a regional referral ED over three 2-week periods from 2010 to 2012. Subjects were clinical providers working at the time of unannounced simulation sessions. Three-minute episodes of sinus bradycardia (SB) and of ventricular tachycardia (VT) were simulated. An experimental HFE-based multi-element intervention was developed to (1) improve system accessibility, (2) increase system relevance and utility for ED clinical practice and (3) establish organisational processes for system maintenance and user base cultivation. The primary outcome variable was overall simulated arrhythmia detection. Pre-intervention system characterisation, post-intervention end-user feedback and real-world correlates of system performance were secondary outcome measures.
Results Baseline HFE assessment revealed limited accessibility, suboptimal usability, poor utility and general neglect of the telemetry system; one simulated VT episode (5%) was detected during 20 pre-intervention sessions. Systems testing during intervention implementation recorded detection of 4 out of 10 arrhythmia simulations (p=0.03). Twenty post-intervention sessions revealed more VT detections (8 of 10) than SB detections (3 of 10) for a 55% overall simulated arrhythmia detection rate (p=0.001).
Conclusions Experimental investigations helped reveal and mitigate weaknesses in an ED clinical telemetry system implementation. In situ simulation and HFE methodologies can facilitate the assessment and abatement of patient safety hazards in healthcare environments.
Central and distributed telemetry systems to monitor and display patient vital signs and cardiac rhythms are widely installed in a variety of healthcare environments. Notwithstanding arrhythmia recognition software, ST segment analysis algorithms and other enabling features, these telemetry systems depend on proper configuration, deployment, use and maintenance in order to effectively alert providers to critical changes in patient status. Furthermore, the functionality of telemetry systems can be impaired by the clinical demands and workflow characteristics of specialised acute care settings such as Emergency Departments (ED).
ED telemetry applications have been assessed in select patient cohorts (eg, low-risk chest pain observation unit admissions,1–4 ED patients in transit5–7); however the performance and value of cardiac telemetry and vital signs monitoring for the general, undifferentiated ED population as a whole remains unexamined. Coincident with recent media and regulatory body attention on monitoring-related patient safety,8–13 investigators hypothesised that medical simulation methodologies and human factors engineering (HFE) could be applied to (1) the objective assessment of ED clinical systems performance with respect to detection of life-threatening cardiac arrhythmias and (2) the experimental development and implementation of practical solutions for identified deficiencies in telemetry system functions. This manuscript reports results from the Accessible Real-time clinical Guidance through Updated Signals (ARGUS) programme, intended to apply on-site simulation and HFE investigation to the clinical ED systems tasked with detecting cardiac arrhythmias.
Setting and sample
The study was conducted in two 16-bed units of the adult ED for an academic regional referral hospital (719 beds; annual ED census 103 000 adult visits). Each of the two ED care units featured the StarView telemetry system (Philips Healthcare, Andover, Massachusetts, USA), bedside patient monitors in all treatment rooms (excepting one room on one unit), one centrally-located telemetry display station without dedicated monitoring staff, and one pair of hallway telemetry displays (see online data supplements 1 and 2). The remaining urgent care and behavioural care areas did not feature telemetry monitoring; resuscitation bays and chest pain observation units were excluded from study due to continuous bedside nurse presence for patient monitoring. Personnel from the institution's simulation centre and departments of biomedical engineering and emergency medicine developed and implemented the programme.
The study subject pool included all on-duty clinical personnel (eg, physicians, nurses, technicians, students, ancillary personnel) working in the ED during unannounced study sessions who could be expected to respond to patients exhibiting life-threatening cardiac arrhythmias or vital sign aberrancies. As a study examining the clinical performance of established processes and response systems (ie, not individual responders) during routine ED care, informed consent was not obtained from subjects; confidentiality was maintained, and individual identifiers were not collected. Patients in the live ED environment were not involved. The research protocol was approved by the Institutional Review Board of the study site.
Simulation protocol development and baseline telemetry system performance assessment
On the basis of the American Heart Association recommendation for detection and treatment of non-perfusing ventricular fibrillation/ventricular tachycardia within 3 min of onset, investigators chose a 180-s window for simulation of arrhythmias. Sinus bradycardia at a rate of 20 bpm and ventricular tachycardia at 150 bpm were selected for simulation due to their presumed impact and relevance to ED patient care and safety. A PS/97 (BAPCO, defunct) biomedical equipment testing simulator was configured to generate the appropriate cardiac monitor ECG telemetry signals. An arrhythmia simulation session study tool was developed for observation and recording of system performance data (see online data supplement 3), for example, time between initiation and detection of simulated arrhythmia, detection method (ie, central telemetry display, hallway telemetry display, or bedside monitor (either in-room or broadcast across CareGroup, a broadcasting feature to relay alarms across designated monitors)). Upon finalisation of simulation protocol and checklist (see online data supplement 4), study sessions were scheduled for dates and times that were selected with a random number generator, compatible with research assistant availability and within specified study periods. (Although of potential interest for investigation of the effect of nighttime-associated factors that impact ED clinical operations, study sessions between 23:00 and 7:00 were not able to be conducted due to research personnel scheduling restrictions).
Twenty pre-intervention arrhythmia simulation sessions were conducted to determine baseline ED system performance. After clearance with the ED clinical manager and determination of the absence of study exclusion criteria (eg, surge/disaster conditions, programme personnel on active clinical duty) prior to each scheduled session, investigators temporarily marked an unoccupied ED treatment room as ‘occupied’ on the computerised physician order entry tracking system. An investigator connected the simulator to the in-room bedside monitor to generate an ECG tracing of a study arrhythmia, which was simulated and displayed for 180 s. When available, a second investigator confirmed arrhythmia display on the ED telemetry system, observed for arrhythmia detection by ED personnel and interrupted any potential study-prompted clinical activity that could impact live patients or departmental operations. The simulated arrhythmia was recorded as being detected if and when any ED clinical provider either (1) responded in person to the ED treatment room housing the arrhythmia generator or (2) appropriately interacted with any telemetry system station in response to the arrhythmia or resultant alarm. The first two subjects responding to each arrhythmia simulation received gift certificates as study incentive. No debriefing was completed except for a brief explanation of study protocol.
Development of HFE knowledgebase, determination of telemetry system repair and revision specifications, and implementation of experimental intervention
Study investigators employed a selective, phased HFE approach (consistent with those advocated by experts14–20) to study and improve the existing ED telemetry system implementation. A focused knowledgebase to define pre-intervention system state and HFE objectives was compiled from multiple sources and methods, starting with a literature review, hardware inventory (for type and mechanism of damage and loss), functions diagnostic (eg, device up-time, configuration, communications, networking), real-time clinical use observation and end-user Web survey for needs analysis (querying staff on system knowledge, use and customisation preferences). Findings from the initial assessment activities indicated that the telemetry system (as originally installed in 2005) featured limited accessibility, suboptimal usability, poor utility and general neglect by its anticipated user base. Hardware exhibited signs of moderate physical damage as well as a lack of maintenance, for example, two of the four system-hosting PCs did not boot up; connector pins on ECG cable connectors were bent and precluded signal transmission; keyboard and mouse interfaces were inaccessible or missing. Real-time observation revealed that the central telemetry stations and their alarms were consistently ignored by clinical staff; non-clinical personnel stationed in proximity to the central displays were observed to intermittently mute alarms without reviewing or reporting them. Fifty-three per cent of 18 attending physicians (33% of practice group) and 26% of 21 nurses (approximately 10% of active ED nursing staff) who responded to pre-intervention Web surveys reported that the telemetry system did not impact their clinical practice. Physicians and nurses who indicated definite impact on their clinical practice reported an average of 2±2 (median 1) and 4±4 (2) such instances, respectively, in the 5-year period since system deployment at the study site.
Informal discussions with small user groups comprising registered nurse (RN's), medical doctor (MD's) and ED technicians during shift-change assembly and impromptu on-shift meetings contributed to a better understanding of the relevant hardware, task, process, user, organisational and environmental factors and issues. Institutional expert input and guidance were obtained through discussions with ED clinical practice and administrative leadership councils as well as 5S (seiri (sort), seiton (straighten), seiso (systematic cleaning), seiketsu (standardize), and shitsuke (sustain)), patient safety and simulation workgroups. Given the low incidence of true positive live patient alarm events, direct observation methods were focused on the baseline arrhythmia simulation sessions to generate snapshot task analysis data (and work environment conditions data). RN charting activities were separately observed and web surveyed to characterise relevant workflow needs for optimal application of downstream HFE interventions.
The knowledgebase was then used to determine specifications for ED telemetry system repair and revision within the constraints of fixed monitoring hardware (ie, sensor capabilities), detection software (algorithms) and software usability characteristics (display gestalt, information clustering, alarm message format, content mapping (both cognitive and geographic)). Target specifications for system intervention were defined in select HFE categories (with implementation methods) through a modified Delphi process involving investigators and ED stakeholders: improved alarm audibility (hardware repair and repositioning) and visibility (central display relocation; distributed large-screen display installation at high-impact/high-traffic locations), alarm parameter matching to end-user needs (adjustment of vital sign thresholds and arrhythmia triggers for high-priority ‘red’ alarms; CareGroup monitor pairing) and interface disambiguation (touchpad-based input). Discussions were held iteratively with institutional biomedical engineers, the device manufacturer and end-users to verify the feasibility, processes and value of the repair and revision specifications. Mechanisms for the necessary extension of physical infrastructure, routine system maintenance, personnel orientation and equipment in-servicing (with feedback opportunities) were arranged to address environmental and organisational deficiencies in the existing system.
HFE specifications, methods and mechanisms were integrated to establish an overall system revision design. This resulted in the programme's experimental multi-element intervention, specifically engineered to deliver (1) improved system accessibility , (2) increased system relevance with enhanced signal:noise ratio and system utility for real-world ED practice and (3) organisational processes for system sustainment with a stable user base (see table 1 for details). The intervention was implemented incrementally over 17 months at the study site after completion of baseline simulation-based system performance assessment.
Interim and post-intervention telemetry system performance assessment
An interim analysis was planned and conducted during the intervention phase for system safety monitoring, ongoing assessment for revision of study intervention and to serve as an additional prompt for end-user familiarisation and utilisation of the system (through the intentional co-option of an anticipated Hawthorne effect). Ten interim arrhythmia simulation sessions were completed 3 months into the study.
Upon implementation of all major elements of the study intervention, a set of 20 post-intervention arrhythmia simulation sessions was completed; the study protocol for these observations was adjusted to monitor for simulated arrhythmia detection at the newly-installed nursing station distributed telemetry displays.
Accession and compilation of real-world correlates of telemetry system performance
Live environment alarm log records of all telemetry-monitored beds in the two study ED care units were reviewed daily during the 2-week post-intervention study phase for true positive and false positive ventricular tachycardia red alarms (ventricular rate > 100 bpm for >5 s) and bradycardia red alarms (ventricular rate <40 bpm); patient identifiers and details of their management, disposition and outcomes were not accessed. Unsolicited anecdotal provider reports of system utility to study investigators were collected over the duration of the study.
An informal survey was conducted for up to 100 end-users to gauge the matching of post-intervention system functions with user needs; suggestions and feedback for future improvements were also collected. In order to reformulate and highlight the HFE-modified telemetry system as a new, enabling technology, respondents were additionally asked to indicate where they perceived it to fit on a ‘hype cycle’ framework (see online data supplement 5).22
Descriptive analyses were performed on arrhythmia simulation characteristics (eg, time of day, numbers of patients in live ED, arrhythmia type simulated) for the pre-intervention, interim and post-intervention sessions. Each dataset was examined with two-tailed Fisher's exact tests and Wilcoxon rank-sum tests for the proportion of simulated arrhythmias detected, mean time until arrhythmia detection, clinical role of detector/responder, method of arrhythmia detection and proportion of detected arrhythmias that were correctly identified.
Simulated arrhythmia detection performance data
Twenty pre-intervention, 10 interim and 20 post-intervention arrhythmia simulation sessions were conducted over three separate 2-week periods during the 19 months between October 2010 and April 2012 (see figure 1A,B). Equal numbers of sinus bradycardia and ventricular tachycardia arrhythmias were simulated at randomly pre-selected dates and times (as allowed by study site clinical conditions) across day and evening workshifts on all days of the week; research assistant scheduling precluded overnight shift study sessions. Sessions were split evenly across the two study ED urgent care areas; room locations used were a convenience sample based on departmental clinical activity. Two study sessions were aborted due to protocol violations (data excluded from analysis) and rescheduled.
During pre-intervention sessions, none of the 10 simulated sinus bradycardia episodes were detected; a physician detected 1 of 10 simulated ventricular tachycardia episodes at 70 s through an arrhythmia alarm broadcast across a four-room CareGroup. Overall baseline system performance was 5% detection of simulated arrhythmias.
The 10 interim sessions recorded one sinus bradycardia arrhythmia detection out of five episodes (20%; p=0.50 for Fisher's exact test in comparison with baseline performance) at 80 s and three ventricular tachycardia arrhythmia detections out of five episodes (60%; p=0.29) at 78±54 s, for an overall detection rate of 40% (p=0.03).
The system's overall simulated arrhythmia detection rate during post-intervention sessions was 55% (p=0.001). Three of 10 simulated sinus bradycardia episodes (30% at 152±23 s; not significant (NS) for Fisher's exact test in comparison with baseline performance) and 8 of 10 simulated ventricular tachycardia episodes (80% at 55±49 s; p<0.01; relative risk 2.67, CI (0.98 to 7.22) with respect to sinus bradycardia) were detected (see table 2). Time of day, weekday versus weekend, ED census (total, in-room, waiting) and provider:patient ratios did not exhibit significant correlation with simulated arrhythmia detection in any programme phase. Time to simulated arrhythmia detection did not change across study phases.
Real-world correlation data
Alarm log record review of the live telemetry system for study ED care units after intervention revealed frequent false positive alarms for ventricular tachycardia (124 false out of 127 logged; positive predictive value 0.02) and bradycardia (8 false out of 17 logged; positive predictive value 0.53). Investigators received recurring reports of distinctive instances of system impact on patient management and clinical care (eg, patient admission to higher-acuity setting based on telemetry-based detection of malignant arrhythmia) at a rate of approximately one report every few months (see table 3). Sixteen of 28 respondents to the post-intervention HFE survey perceived the telemetry system as either starting or already helping to empower clinical providers during patient care duties; 11 additional respondents considered the system as having the potential to improve patient care (see online data supplement 5).
Consistent and timely detection of immediately life-threatening conditions, along with expeditious response, is essential in acute care settings. Yet the deterioration of patients from clinical stability into respiratory failure, hypoperfusive states and malignant cardiac arrhythmias may be subtle, rapid, paroxysmal or otherwise unpredictable in such environments.23 As a mechanism to assist busy clinical staff in meeting this challenge, ED telemetry systems offer real-time, automated remote monitoring of their patients. Yet implemented telemetry systems are often unable to meet complex real-world functional demands and fail to overcome complications arising from innate technological limitations. Notwithstanding scientific and medical advances, telemetry systems can exhibit poor network signal:noise characteristics24–26—critical alarms may be overly sensitive and predominantly false, yet significant events may not be sufficiently highlighted; the alarm framework may not even be able to accommodate operational information demands.27 Additional inadequacies may lie in the design of alarms and biomedical interfaces, software algorithms, monitoring utilisation criteria, and a myriad of complex confounding factors (eg, patient movement, ambient noise, provider workload and staffing patterns). With most of these issues beyond their control and ability to correct, frontline clinical end-users may fully disengage from interacting with telemetry systems. Of concern, poor design and functionality in these types of systems have been found to precipitate medical ‘noise pollution’,28–31 ‘cry wolf effect’32–34 or ‘alarm fatigue’,35 ,36 and unintended consequences, even patient harm.37 ,38
Due to ongoing concerns regarding the utility and safety of an existing ED telemetry monitoring system, study investigators conducted the Accessible Real-time clinical Guidance through Updated Signals research programme with the systematic application of HFE methods and in situ medical simulation. Unlike previous reviews of arrhythmia telemetry, primarily set in inpatient cardiac units,39–51 the current study protocol and intervention addressed an ED-care setting designated for patient populations presenting with the full spectrum of illness and injury severity. Objective exploration of the telemetry system in these general purpose ED areas revealed unaddressed weaknesses, unrecognised faults and unfulfilled potential. In confirmation of anecdotal reports, the study found substantial evidence of debilitating system dysfunction resulting from suboptimal configuration, deployment and maintenance—a worrisome demonstration of the misapplication of sophisticated healthcare-enabling technologies.
In response to the identification of these shortcomings, investigators modified and ‘re-booted’ the study site's telemetry monitoring system and existing clinical infrastructure in a recursive and data-driven manner for effective early warning and detection of severe arrhythmias. This was congruent with prior investigations,52 ,53 but with provision for the unique characteristics of ED-practice settings, an emphasis on proactive probing to actively use-test the system,54 ,55 and minimisation of work-disruptive changes in provider responsibilities. Efforts to engage and familiarise ED staff with telemetry system functions also focused on end-user participation in system revision discussions, equipment in-service sessions, and embedding of system use into routine clinical workflow, for example, placement of prominent data-mirroring displays in clear view of nurse workstations for convenient assistance with chart documentation. Experimental data demonstrated how these concrete, real-world utility and usability measures improved system functionality to alleviate alarm fatigue; translational impact at the bedside was also noted through feedback and informal reports. Conversely, a composite analysis of the study site's arrhythmia simulation detection performance (ie, for ‘true positive’ alarms in response to simulated arrhythmia events) and of the live environment telemetry system log records (for ‘true’ and ‘false’ positivities of logged alarms based on retrospective review) still reveals poor overall sensitivity and low positive predictive values, respectively, for significant arrhythmias.
As a clinical informational adjunct, the study site telemetry system is now being used for routine clinical care; studies are planned to assess ongoing operational performance, both simulated and live. As the system is unlikely to ensure faultless detection of all significant monitor alarms, measures to further improve and supplement its functions are being considered at the organisational level, for example, dedicated telemetry monitoring staff,56 ,57 revision of practice guidelines,58 ,59 computerised physician order entry-prompted reduction of unnecessary monitoring and clinical trigger systems.60 ,61 Next-generation sensors38–41 and intelligent monitor display interfaces,66–69 signal filtering28 ,70 and multivariate analyses that aggregate discrete physiological parameter datastreams,71 may help address some of the bedside factors that impact the monitoring and management of increasingly complex and sick patient populations.72 ,73
Distinct from the incremental revisions and upgrading of existing patient monitor and telemetry equipment, innovative systems approaches will be necessary to overcome the persistent challenges involved in effectively detecting, localising and conveying critical patient information as an accurate alarm to appropriate clinical personnel with a minimum of unnecessary disturbance. Potential research and development efforts may feature the integration of smart monitoring instruments, locative devices and intelligent software agents into a provider-embedding, networked and informatics-mediated clinical information space. Akin to evolving ‘augmented reality’ applications, this approach could use commercially available components such as securable, movement-tolerant sensors with wireless connectivity (eg, for fitness monitoring), radiofrequency locator badges, directional visual alert/sound focusing devices, and ‘push’ messaging agents for accurate and selective provider notification. This ‘need-to-know’ data delivery model would bypass, reduce and de-emphasize ambient noise through clear and discrete alarm signals that self-navigate to, and disturb, only specified end-users. Investigators are preparing for collaborative investigations that build on these concepts to better understand the limitations and potential associated with the use of patient-monitoring telemetry systems.
Simulation session numbers were limited by the need to minimise disruptions to patient care and departmental operations; the effect of not having simulated arrhythmia detection data for night shifts on overall telemetry system performance assessment is unknown. The application of simulation to generate select telemetry readings and elicit system-level responses may have suffered from unrecognised limitations that limit the ability to generalise study findings to live clinical settings. Simulations were constrained by ethical issues that would have arisen from fully re-creating the healthcare ‘footprint’ of an actual patient (ie, registration forms, chart work, electronic medical record, bed occupancy, clinical care activities) in a high-occupancy ED. As live environment alarm record data prior to intervention were not obtained, the programme's impact on real-world arrhythmia detection and on causal factors that originally resulted in ED telemetry system underperformance could not be precisely determined. Underlying process issues that pertain to patient entry into the ED-care environment and onto telemetry monitoring systems were not addressed in this study.
Experimental investigation with arrhythmia simulations helped reveal and mitigate weaknesses in an acute care patient monitor telemetry system implementation. Data-driven, participatory HFE with in situ simulation-based assessment methodologies can be applied to the examination and abatement of patient safety hazards in acute care healthcare settings.
The authors would like to acknowledge Mark E. Luke, John A. McDonough RN and Aziz Sultan of the Rhode Island Hospital Biomedical Engineering Department for their technical assistance in assessing, reconfiguring, maintaining and troubleshooting the telemetry system, Jason D Machan PhD (Lifespan statistician) for his assistance in statistical analysis and manuscript preparation.
Contributors All authors contributed substantially to the programme, study and manuscript and are qualified for authorship as specified by ICJME criteria: (1a) Substantial contributions to conception and design: LK, RP, RMB, FJG, EG, JEM, BO, DCP, NAS, GDJ. (1b) Acquisition of data: LK, RP, FGG, GAP, NMT, RSA, KSB, JD, RMB, DCP, DH. (1c) Analysis and interpretation of data: LK, RP, RMB, FJG, EG, JEM, BO, DCP, NAS, DH, GDJ. (2) Drafting the article or revising it critically for important intellectual content: All authors. (3) Final approval of the version to be published: All authors. LK had full access to all of the data in the study and takes responsibility for the integrity of the data, the accuracy of the data analysis and the manuscript as a whole; no additional data are available.
Funding This study was supported by the Department of Emergency Medicine at Alpert Medical School (AMS) of Brown University, University Emergency Medicine Foundation (UEMF), Rhode Island Hospital (RIH), and Rhode Island Hospital Medical Simulation Center (RIHMSC); purchase and installation of the nursing station displays were funded through Information Services, Lifespan. Funding sources did not have any role in the design and conduct of the study; collection, management, analysis and interpretation of the data; and preparation, review or approval of the manuscript. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of AMS, UEMF, RIH, RIHMSC or Lifespan.
Competing interests None.
Ethics approval Lifespan Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.