Background There are only a few studies on handoff quality and adverse events (AEs) rigorously evaluating handoff improvement programmes’ effectiveness. None of them have been conducted in low and middle-income countries. We aimed to evaluate the effect of a handoff programme implementation in reducing AE frequency in paediatric intensive care units (PICUs).
Methods Facility-based, cluster-randomised, stepped-wedge trial in six Argentine PICUs in five hospitals, with >20 admissions per month. The study was conducted from July 2018 to May 2019, and all units at least were involved for 3 months in the control period and 4 months in the intervention period. The intervention comprised a Spanish version of the I-PASS handoff bundle consisting of a written and verbal handoff using mnemonics, an introductory workshop with teamwork training, an advertising campaign, simulation exercises, observation and standardised feedback of handoffs. Medical records (MR) were reviewed using trigger tool methodology to identify AEs (primary outcome). Handoff compliance and duration were evaluated by direct observation.
Results We reviewed 1465 MRs: 767 in the control period and 698 in the intervention period. We did not observe differences in the rates of preventable AE per 1000 days of hospitalisation (control 60.4 (37.5–97.4) vs intervention 60.4 (33.2–109.9), p=0.99, risk ratio: 1.0 (0.74–1.34)), and no changes in the categories or AE types. We evaluated 841 handoffs: 396 in the control period and 445 in the intervention period. Compliance with all items in the verbal and written handoffs was significantly higher in the intervention group. We observed no difference in the handoff time in both periods (control 35.7 min (29.6–41.8) vs intervention 34.7 min (26.5–42.1); difference 1.43 min (95% CI −2.63 to 5.49, p=0.49)). The providers’ perception of improved communication did not change.
Conclusions After the implementation of the I-PASS bundle, compliance with handoff items improved. Nevertheless, no differences were observed in the AEs’ frequency or the perception of enhanced communication.
Trial registration number NCT03924570
- quality improvement
- adverse events
- epidemiology and detection
- critical care
Data availability statement
Data are available in a public, open-access repository: https://osf.io/gn4qu/?view_only=4472676a10a541e0be2f87f0971e7064.
Statistics from Altmetric.com
In clinical settings, communication effectiveness is essential and considered an interactive process.1 Communication errors represent the third most common cause of sentinel events,2 over half of which involve handoff failures.3 Handoff is defined as the exchange of information between health professionals about a patient, accompanied by a change in control or responsibility in their care decisions.4 It is estimated that a typical teaching hospital may experience more than 4000 handoffs per day.5 The Joint Commission has established standardised transmission of information as a patient safety goal and advocates organisations to implement ‘a standardised approach to handoff communications, including an opportunity to ask and respond to questions’.3
Previous studies have shown that effective and standardised communication between caregivers in handoffs is essential for patient safety and anticipating and limiting possible errors.6–11 Different tools have been proposed as models to standardise information transmission during handoffs, many of which are acronyms (to facilitate their use).12 These tools contributed to reducing the handoffs’ length and improved the information quality and the patient’s subsequent care.7 9 13 The Initiative for Innovation in Pediatric Education-Pediatric Research in Inpatient Settings Accelerating Safe Sign-outs (I-PASS) combined rigorous curricular design, traditional healthcare services research, teamwork training and quality improvement efforts to standardise the complex process of resident inpatient handoffs, aiming to improve patient safety.14 The I-PASS acronym referred to both the title and purpose of the study and the mnemonics developed as part of that study, but is mostly known for the mnemonics. I-PASS also combines different strategies to improve handoffs and reduce medical errors, including communication training, the use of mnemonics to standardise handoffs, the restructuring of verbal handoffs by minimising interruptions and involving all team members, and the use of written or computerised tools.12 15–20 The results of a pilot study using an I-PASS precursor and a subsequent multicentred I-PASS analysis have shown reductions in errors and adverse events (AE).7 21 22 A locally adapted version was created for the Argentinean setting.23
Only a few studies on handoff quality and AEs rigorously evaluate handoff improvement programmes’ effectiveness, and none of them have been conducted in low and middle-income countries. I-PASS was developed in tertiary academic medical centres in the USA; therefore, its effectiveness in locations with a different language, availability of resources (eg, lack of electronic medical records (MR)) and complete cultural emersion remain unclear. Stepped-wedge, cluster-randomised controlled designs enable both phased implementation and the use of established statistical approaches to compare control and intervention groups while minimising the potential for bias and confounding.24
We aimed to assess a standardised handoff intervention’s effectiveness in reducing the AE frequency in paediatric intensive care units (PICUs) in a middle-income country using a stepped-wedge, cluster-randomised design.
We conducted a facility-based, cluster-randomised controlled trial with a stepped-wedge design in six PICUs between July 2018 and May 2019 (11 months). All participating units began as control practices without the intervention (3 months). As the trial progressed, units were allocated randomly to receive the intervention in prespecified time periods (1 month per step). This process continued until all of the participating clusters received the intervention. All clusters were exposed to the intervention for at least 4 months (online supplemental appendix II).
The unit of randomisation was the PICU. Sites were assigned to one of five start dates by the study statistician via a computer-generated list of random numbers. Neither site knew the I-PASS bundle until they were randomised to the intervention period. Concealment of the intervention starting date was maintained up to 15 days prior to launching the intervention at each PICU because preparatory activities were needed (eg, setting the introductory workshop date and preparing campaign materials). A flow chart of the allocated sequence and period is presented in online supplemental appendix I.
The study was conducted in six PICUs from five public hospitals in three provinces of Argentina. The PICU eligibility criteria were the absence of a handoff programme and having at least 20 admissions per month.
First, the I-PASS handoff programme’s implementation barriers and facilitators were identified to adapt the intervention at each participating site. The information was gathered from 17 senior healthcare professionals from the participating PICUs through in-depth interviews. Most of the sites, except for one, had an established practice of written handoffs. However, participants complained about the handoff process. They explained that it was lengthy and disorganised, and that participants experienced problems with interruptions and distractions during verbal handoffs and that senior professionals had problems accepting dissent. Regarding the main barriers to intervention implementation, participants mentioned the resistance to change shown by the healthcare team. Other barriers were related to the I-PASS bundle adaptation to local culture and obstacles associated with the way handoffs were conducted. Participants from most of the sites reported that handoffs were conducted with too many people with different experiences and backgrounds and that handoff time was also used for teaching purposes. Only a few participants reported having previous knowledge of I-PASS or other standardised handoff tools. It was helpful that handoffs were already conducted face to face at predefined times and locations, and some previous positive quality improvement experiences were accumulated.
For this study we implemented the Spanish I-PASS bundle used in previous studies in our setting,23 25 consisting of nine elements: (1) the I-PASS mnemonics (Illness severity, Patient summary, Action list, Situation awareness and contingency plans, Synthesis by the receiver) which served as an anchoring component for verbal and written information; (2) an introductory 2-hour workshop with key content about handoff quality and teamwork training; (3) five tools of teamwork training of TeamSTEPPS programme—an evidence-based programme aimed at optimising performance among teams of healthcare professionals: cross-monitoring, brief, debrief, huddle and check-back17; (4) a standardised I-PASS format written handoff template; (5) role-play emphasising the elements of the workshop; (6) faculty development programme; (7) a self-learning module to reinforce the components of the mnemonics; (8) direct observation tools used by the faculty to provide feedback to physicians; (9) an advertising campaign with printed material, posters and stickers with the I-PASS logo and mnemonics for process and culture change.6 7 14 21
The intervention was deployed following predefined guidelines at all sites. During the first week, the I-PASS workshop was given in person with reserved time for questions and answers, and a recorded version was made available for people who could not attend. Four sites used a written handoff template containing the I-PASS acronym, with little variations among them. Two sites used a different written handoff excel template, including the I-PASS acronym (PICUs ‘2’ and ‘3’, online supplemental appendix II). Stickers and posters with the I-PASS acronym reminders were sent to sites before the first week of intervention. There were predefined assigned weeks to reinforce each of the five components of the I-PASS acronym. Role-playing was carried out between the second and the third weeks of implementation; however, two sites could not reach the threshold of attendance (70% of people involved in handoffs) (PICUs ‘1’ and ‘6’, online supplemental appendix II). Each site also maintained an implementation log regularly reviewed to ensure adherence to each handoff programme component. Biweekly meetings were held with each PICU to reinforce implementation strategies according to the presented difficulties and ensure compliance to each handoff programme component.
Measurement of study outcomes
The primary outcome was the rate of preventable AEs per 1000 days of hospitalisation. The Global Assessment of Pediatric Patient Safety (GAPPS) has proved reliable in measuring AE’s rate per 1000 days of hospitalisation.26 27 GAPPS is designed to identify all AEs and not only negative consequences of medical errors. It could also measure whether harm was preventable (online supplemental appendix III) and establish severity ratings.28
We selected a random sample of ≤30 inpatient hospitalisations each month from a list of all inpatient hospitalisations with discharge dates that fell within the month being reviewed. The inclusion criteria were: patients aged <18 years at discharge, with a length of stay ≥24 hours and admitted for acute care. Two independent reviewers were trained in the GAPPS process and assessed AEs following a structured methodology. Primary reviewers (a PICU staff, not necessarily physicians) evaluated the selected MRs using the GAPPS list of 37 possible manual triggers following this sequence: (A) discharge and progress notes, (B) prescriptions, and (C) nursing progress sheets. The triggers were clues that suggested a possible AE. The primary reviewers spent at least 30 min reviewing each hospitalisation. They then presented the suspicions of AE to secondary reviewers (a PICU staff, physician), who independently evaluated whether an AE had occurred and its severity. Following this, all reviewers reached a consensus on every AE with an initial disagreement.28 During the MR review, every AE without an initial trigger was identified and reported in the study. Reviewers were selected from the same site and were not blinded during the study period.
Assessment of written and oral handoffs
The secondary outcome was compliance with adequate verbal and written handoffs and was assessed by direct observation. We took a convenience sample of at least 12 observations per site per month. All physicians were observed at least once per month, presenting or receiving a full-shift handoff. Observers completed an evaluation form for each full-shift handoff using a Likert scale (‘always’, ‘almost always’, ‘sometimes’, ‘almost never’, or ‘never’) to show compliance with the elements of a good-quality handoff (online supplemental appendix III). The observers were blinded to the study period.
Agency for Healthcare Research and Quality Surveys on Patient Safety Culture
Physicians working 4 or more months in each PICU during the study period were surveyed about patient safety culture, emphasising the communication dimension using the Surveys on Patient Safety Culture from the Agency for Healthcare Research and Quality (AHRQ) translated and validated in Spanish.29 The survey was distributed by email to physicians working only in the PICU between June and July 2018 (control period) and between May and June 2019 (intervention period). Two reminder emails were sent to all physicians in both periods after the survey was delivered to delayed responders.
The sample size was estimated to reduce the rate of preventable AE (primary outcome) from 12% to 4.5%.23 Assuming a coefficient of intracluster correlation(ICC) of 0.01, number of steps=5, a cluster size per step=30, a power=80% and an alpha level=5%, the total number of clusters needed was 6.
Data management and statistical analysis
Data collectors were trained in GAPPS tool use and handoff observations using specifically designed data forms. Data collection was performed at sites using the Research Electronic Data Capture (REDCap) system website.30 31 REDCap allowed real-time data entry validation and used branching logic. Data sets were kept inside this system, where they were safe, available for look-up and logged according to Good Clinical Practice and 21 Code of Federal Regulations Part 11.32 After the data were uploaded to the database, discrepancies were checked to ensure completeness, consistency and accuracy. Patient characteristics between the intervention and control periods were compared by considering the correlations of the PICUs. We used a generalised mixed linear model assuming a normal distribution for the continuous variables and a binomial distribution for the categorical ones.
The AHRQ surveys were completed through SurveyMonkey.
Analyses were performed according to the intention-to-treat principle. MR was the unit of analysis. For each MR, we calculated the number of events during the number of hospitalisation days. A generalised mixed linear model assuming a negative binomial distribution was used to estimate the effect of the intervention. The outcome was the number of events, and the offset variable was the number of days of hospitalisation. An indicator group variable (0 for the control, 1 for the intervention) and a variable indicating the month of the study (1–11) were included in the model as a fixed effect. The PICU was included in the model as a random effect. Estimated rates per 1000 days of hospitalisation with their confidence interval (CI) were reported for the control and intervention groups. The impact of the intervention was estimated as the ratio of both rates (intervention/control).
Assessment of written and oral handoffs
The physicians’ observations on the performing handoffs were used as the unit of analysis to evaluate compliance with verbal and written handoffs. The analysis was performed according to the intention-to-treat principle. We first created dichotomous variables from the Likert scale that represented the provider’s compliance for each handoff element. ‘Compliance’ was considered when the provider’s compliance for the handoff element was: ‘almost always’ or ‘always’; ‘Non-compliance’ was considered when it was: ‘sometimes’, ‘almost never’, or ‘never’. A generalised mixed linear model assuming a binomial distribution was used to estimate the effect of the intervention. The outcome was each handoff element. The PICU was entered in these models as a random effect, and the effect sizes were adjusted for time trends by including time in the model as a fixed effect. The estimated proportions during both periods are reported. The OR and 95% CI were used to estimate the effect of the intervention.
AHRQ Surveys on Patient Safety Culture
To analyse the AHRQ Surveys on Patient Safety Culture, the unit of analysis was the physician. We calculated the percentage of positive answers for each item in the survey. A positive response was considered when the physician answered each question: ‘agree’ and ‘strongly agree’, and a negative response when he/she chose ‘strongly disagree’, ‘disagree’, or ‘neither agree nor disagree’. Of the 111 physicians, 72% completed the survey in the control period and 78% in the intervention period. As the survey was anonymous, it was impossible to link the physician’s response during the control and intervention periods, so the analysis was performed assuming that samples were independent of each other. The χ2 test was used to test for differences in proportions. All analyses were performed using R V.126.96.36.199
We reviewed 1465 MRs during the study period with a total of 15 842 patient-days: 767 MRs in the control period and 698 MRs in the intervention period in six different clusters (figure 1). The number of MRs reviewed in each cluster and the period are shown in online supplemental appendix II. The patients’ baseline characteristics were similar between both periods, except that the patients in the control period were slightly younger (table 1).
The rate of preventable AE per 1000 days of hospitalisation was similar in both arms, 60.4 (37.5–97.4) in the control group and 60.4 (33.2–109.9) in the intervention group, p=0.998, with the ratio equal to 1.0 (95% CI 0.74 to 1.34). No differences were observed in the rates of secondary outcomes between the control and intervention groups (table 2). We found variation in the rate of preventable AEs between sites but no differences between periods (intervention/control) in any of the sites (online supplemental appendix II).
Assessment of written and oral handoffs
Finally, we observed 841 full-shift handoffs (396 observations in the control period and 445 in the postintervention period) that yielded 5260 unique patient handoffs for evaluation during the entire study in the six clusters.
Adherence to an adequate verbal and written handoff was measured in both periods (control and intervention), observing increased compliance in all the items after applying the intervention (figure 2). The key elements that improved most were illness severity, action list, and synthesis and the ones that remained most challenging were situations and contingency plans in verbal and written handoffs. The five key elements observed in the 847 verbal and written handoffs are shown in table 3.
Regarding the handoff duration, we observed no difference in the time spent in the full-shift handoff in both periods (control 35.7 min (29.6–41.8) vs intervention 34.7 min (26.5–42.1); difference 1.43 min (95% CI −2.63 to 5.49, p=0.490)). But when we evaluated the time spent with each patient we verified longer duration in the intervention period (intervention 7.29 min (5.77–8.81) vs control 5.96 min (4.69–7.23); difference 1.33 min (95% CI 0.64 to 2.02), p=0.001).
Patient safety culture survey
Eighty-two subjects answered in the control period and 87 in the intervention period. There was no difference in the percentage of positive answers between periods, except for the first question. There were more positive answers in the control period regarding people supporting each other (table 4).
Summary of findings
In this randomised stepped-wedge trial in PICUs in Argentina, we assessed the effect of a standardised handoff intervention to reduce the frequency of AEs and increase the quality of handoffs. The intervention resulted in an overall improvement in the quality of verbal and written handoffs. However, we did not observe changes in the incidence of preventable AEs globally or in any clusters. Improvements were observed in all the items considered needed for a handoff. Still, it was more significant in the handoff’s key elements such as illness severity, action list and synthesis. The majority of the handoff quality items were far below 50% compliance before the intervention, and some of them reached more than 90% adherence after the I-PASS bundle implementation. The intervention was deployed similarly at all sites.
Strengths and limitations
This study had several strengths. We used a rigorous experimental design and achieved similar groups using randomisation. The selected intervention components were previously documented as effective and tailored to address the identified barriers and facilitators from formative research. A trigger tool was used in the MR review, which is generally considered a strength rather than self-reported. Finally, to our knowledge, this is the first trial to evaluate the implementation of the I-PASS bundle and the use of a trigger tool in paediatric patients.
However, this study has some limitations. It is already known that direct observation moves handoffs from ‘backstage’ to ‘front-stage’, and residents perform handoffs differently from their usual practice when they are observed.34 Second, the intervention was evaluated immediately after its implementation. In some PICUs, it was implemented only for 4 months; perhaps more time could be necessary to reduce the AE rate. Third, the year’s periods did not coincide precisely in each cluster, and seasonality cannot be ruled out. Fourth, the cluster sizes reached were lower than calculated due to a decrease in the number of admissions in participating PICUs between November 2018 and January 2019. Finally, the study was only carried out in PICUs in the public subsector of one Latin American middle-income country, which prevents us from extrapolating the results to other populations.
In this trial, the intervention did not significantly affect the rate of AE, the rate of preventable AEs and the severity and categories of AEs. AE rates result from numerous interacting institution structures and processes, and it is possible that variations in the ascertainment of error data or other unmeasured factors were responsible for the lack of improvement in AE rates. We also found substantial interinstitutional variations in the AE rates. Although some sites did not reach the cluster size previously calculated in some months, we reached a sufficient sample size due to the three initial control months and the four final intervention months.
Regarding direct observation of healthcare providers, the majority of the observed staff personnel were physicians with many years of handoff experience, and we believe they were hardly influenced by being observed. Direct observation placed a spotlight on handoffs as a clinical skill, reinforcing the importance of doing it well.
Regarding the time used to carry out the handoff, no differences were observed concerning the total time, and we observed a greater time spent per patient in the intervention stage. This difference could be due to the use of a new tool. Although a differential use of time towards other activities during the handoff cannot be ruled out, the total time did not change in either period. All the participating PICUs had paediatric residents or paediatric intensive care physicians in training, so handoff was also used for teaching activities. These teaching activities often share a mental model by providing the rationale for proposed management, which is an essential feature of quality handoffs.35 36 It has been suggested that the I-PASS has the potential to reinforce an institutional culture that embraces interactive questioning and teaching opportunities to foster shared understanding and optimise patient care.37
The AHRQ survey has been widely used in Spanish-speaking hospitals. None of the participating hospitals had previously used them in their PICUs. No positive changes were observed in the way physicians perceived patient safety related to the communication dimension before and after implementing the intervention. One explanation may be that we only intervened in the handoff process and not in other teamwork communication aspects. The new situational awareness gained with the I-PASS bundle implementation could raise the necessity of more elaborate teamwork training.
Comparison with previous literature
There is a lack of robust evidence on best handoff practices, and the current knowledge on the nature of handoff failures during intershift transfers is scanty. Starmer et al observed that after the implementation of the I-PASS, a similar improvement was observed in the compliance of the items related to the quality of the handoff, while no differences were observed in the load of residents’ work. There were no changes observed in the time used to transfer patients, and not all the sites reported the same level of AE reduction, with heterogeneous compliance of handoff key elements.7 They observed a 23% relative reduction in medical errors and a 30% reduction in preventable AEs. The AE reports were based on direct observations or voluntary reporting. In our study, AE rates were identified in the MR with a tool using pre-established triggers.
Sheth et al demonstrated that a process transfer supported by I-PASS was associated with better efficiency and culture of handoff safety.38 Coffey et al showed residents’ experiences with the implementation of the I-PASS package.39 They promoted other important active factors in this complex intervention, such as the patients’ data automatic import in the electronic transfer document, improvements in the environment transfer, teamwork and communication skills. They also noted that strict adherence might not be necessary to achieve the desired results.39 In our study, the situations and contingency plans in verbal and written handoffs were the most challenging I-PASS features for clinicians. The average compliance of the intervention items was similar to that observed in other studies at the beginning of the I-PASS programme implementation; a significant difference was also observed after the implementation of the quality improvement.15 16 20
In a report of AE in hospital wards of 16 teaching and non-teaching hospitals, it was also not possible to observe an improvement in the number of AEs over time. The AE rate was higher in academic hospitals (26.2 AE per 1000 patient-days, 95% CI 23.7 to 29.0).27 This is the first study to use the GAPPS tool exclusively in PICUs and the first one using the tools in Spanish to the best of our knowledge. However, the AE rate in our study was higher and more severe than that described by Stockwell et al.
We observed an improvement in the quality of handoffs after the implementation of a standardised handoff intervention. No differences were observed in the rate of preventable AEs or the total rate of AEs after using the I-PASS. The perception of improvement in communication also did not change after the I-PASS bundle implementation.
Further research is needed to determine whether this intervention could reduce AE either by different implementation models for a longer duration or by using direct observation or voluntary reporting of AE.
Data availability statement
Data are available in a public, open-access repository: https://osf.io/gn4qu/?view_only=4472676a10a541e0be2f87f0971e7064.
Patient consent for publication
The study was reviewed and approved by the following Institutional Review Boards: Hospital General de Niños Pedro de Elizalde, Comité de Ética en Investigación (DI-2018-303-HGNPE No 7518), Hospital General de Niños Ricardo Gutierrez, Comité de Ética en Investigación (CEI No 18.18), Hospital Nacional de Pediatría ‘J P Garrahan’, Comité de Ética en Investigación, CEIS Hospital de Niños de San Justo (No 037/11), Comité de Docencia Hospital Materno Infantil ‘Dr Héctor Quintana’.
The authors would like to thank specially Amy Starmer, MD, MPH, for her assistance with the adaptation of the I-PASS bundle; Christopher Landrigan, MD, MPH, for his assistance with the GAPPS tool use, to the PICU staff of the Hospital General de Niños Pedro de Elizalde, Hospital General de Niños Ricardo Gutierrez, Units 44 and 45 of the Hospital Nacional de Pediatría ‘J P Garrahan’, Hospital de Niños de San Justo and Hospital Materno Infantil ‘Héctor Quintana’; and thanks to Agustina Mazzzoni, MD, for his thoughtful review of the manuscript. We would like to thank Editage (www.editage.com) for English language editing.
Contributors FJB, EGE, LML and JCV conceived the original idea and study design. FJB and EGE contributed to the design and implementation of the research. RBP, NDM, MP, RG, APR, DV, SA, LG, SL, JO and GR contributed to the implementation of the research. FJB, EGE, ISA, LG and RR contributed to the analysis and interpretation of the results and to the writing of the manuscript. All authors discussed the results and commented on the manuscript.
Funding Salud Investiga ‘Dr Abraam Sonis’ grant, category Multicentric Study, funded by the National Ministry of Health, through the Directorate of Research for Health, Argentine.
Disclaimer The funders of the study had no role in the study design, data collection, data analysis, data interpretation, or writing of the report.
Competing interests FJB, LML, MP, GR, NDM, RBP and ISA report grants from the Ministry of Health of the Nation, through the Directorate of Research for Health, Argentine, during the conduct of the study.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.