Article Text
PDF
Abstract
Background Although many studies of quality improvement (QI) education programmes report improvement in learners’ knowledge and confidence, the impact on learners’ future engagement in QI activities is largely unknown and few studies report project measures beyond completion of the programme.
Method We developed the Academy for Quality and Safety Improvement (AQSI) to prepare individuals, across multiple departments and professions, to lead QI. The 7-month programme consisted of class work and team-based project work. We assessed participants’ knowledge using a multiple choice test and an adapted Quality Improvement Knowledge Assessment Test (QIKAT) before and after the programme. We evaluated participants’ postprogramme QI activity and project status using surveys at 6 and 18 months.
Results Over 5 years, 172 individuals and 32 teams participated. Participants had higher multiple choice test (71.9±12.7 vs 79.4±13.2; p<0.001) and adapted QIKAT scores (55.7±16.3 vs 61.8±14.7; p<0.001) after the programme. The majority of participants at 6 months indicated that they had applied knowledge and skills learnt to improve quality in their clinical area (129/148; 87.2%) and to implement QI interventions (92/148; 62.2%). At 18 months, nearly half (48/101; 47.5%) had led other QI projects and many (41/101; 40.6%) had provided QI mentorship to others. Overall, 14 (43.8%) teams had positive postintervention results at AQSI completion and 20 (62.5%) had positive results at some point (ie, completion, 6 months or 18 months after AQSI).
Conclusions A team-based QI training programme resulted in a high degree of participants’ involvement in QI activities beyond completion of the programme. A majority of team projects showed improvement in project measures, often occurring after completion of the programme.
- quality improvement
- health professions education
- continuing education, continuing professional development
Statistics from Altmetric.com
- quality improvement
- health professions education
- continuing education, continuing professional development
Introduction
Recognising persistent gaps in healthcare quality, payment incentives are now routinely tied to performance.1 2 Moreover, federal agencies and professional organisations have increasingly emphasised the need to train healthcare professionals in quality improvement (QI) methods.3–6 Systematic reviews of published QI education programmes conclude that the most successful programmes include a combination of didactic and project-based experiential learning.7 8 Although many studies of QI education programmes use these strategies and report improvement in learners’ knowledge and confidence, the impact of QI training on learners’ future engagement in QI activities is largely unknown. Moreover, the clinical impact of QI education programmes is not consistently reported and few studies report project measures beyond completion of the programme.8–11 QI projects vary in their complexity and often require more time to complete than the time allotted during the educational programme. Without continuing to monitor the effect of training on the QI activities of past learners and the long-term effect on project measures, QI education leaders have incomplete knowledge of the effects of their programme.
Herein, we report the development, implementation, adaptation and effect of a team-based QI training programme. We report both educational and clinical quality measures and emphasise results on learners and clinical performance beyond the completion of the programme. Furthermore, we describe adaptations made to the programme as a result of our monitoring both learner and project measures.
Methods
Setting and study design
The Academy for Quality and Safety Improvement (AQSI) programme was developed and implemented for healthcare professionals at Northwestern Memorial Hospital and the Northwestern Medical Group, clinical affiliates of the Northwestern University Feinberg School of Medicine. We evaluated AQSI using a before-and-after study design.
Programme development
AQSI began as a collaborative effort among three entities in our academic medical centre: the Department of Medicine, the Center for Education in Health Sciences and the Performance Improvement Department. We first assembled an AQSI Steering Committee consisting of leaders from internal stakeholder groups to provide strategic guidance to the programme. We then conducted a needs assessment consisting of surveys of Division Chiefs and Division QI leaders and a focus group including five physicians who had completed master’s degrees in healthcare quality. Incorporating information from the needs assessment and published studies,7 9 the Steering Committee defined overarching goals and educational strategies. Specifically, we developed a programme for teams of learners who would complete a QI project (ie, experiential learning) and receive coaching and other support to assist their development of QI skills. Additionally, we sought to integrate participants and their projects into health system efforts. We then selected specific topic areas and instructors for classroom sessions and developed mechanisms to coach and support project teams.
Application and selection process
Beginning in 2012, healthcare professionals have been invited to apply for the AQSI programme each summer. Applications are team-based and include a designated leader, a list of team members, a proposed project and a letter of endorsement from clinical leaders. Residency and fellowship directors refer interested trainees to the AQSI programme codirectors so that trainees without a potential project can be matched to developing AQSI team applications. Each team application is independently reviewed by two members of the AQSI Steering Committee prior to group discussion and final selection of teams. Accepted teams begin the programme in October and continue through April.
Programme description
The AQSI programme combines interactive class work and team-based experiential learning through the execution of a team project.
Class work
Programme participants attend 11 90-min classroom sessions, meeting approximately every other week in the early evening. Sessions address core quality topics and the performance improvement method (Six Sigma; Define, Measure, Analyze, Improve, Control (DMAIC)) used in our health system (see table 1). Class sessions are highly interactive and emphasise exercises and discussion to help teams apply lessons to their projects. A flipped classroom model is used for DMAIC classroom sessions to facilitate more time for in-class discussion. Specifically, participants are asked to complete brief (~10–15 min) internet-based modules, developed by our Performance Improvement Department, on DMAIC topics before these sessions. Dinner is provided at classroom sessions and course material is accessible via an internet-based course management system.
AQSI class session topics and instructors
Project work
During the programme, participants work on a QI project of their choosing. Each team receives guidance from a Performance Improvement Coach, with whom they are expected to meet twice a month, and a senior Clinical Mentor, with whom they are expected to meet monthly. Additionally, AQSI teams may use a dedicated analytics consultant to run administrative database queries and have direct access to a Manager of Clinical Informatics to facilitate interventions requiring changes within the electronic health record (EHR). Teams present project updates twice during the programme to the health system’s Improvement Council, which includes senior organisational leaders. Teams also present project updates to one another during the sixth and final classroom sessions.
Reception and certificate
Each team presents a poster at a reception held approximately 1 month after classes finish. Each participant receives an AQSI Certificate at the reception to acknowledge their completion of the programme.
Evolution and adjustment
The programme initially only included participants from the Department of Medicine. As success of the programme grew, other departments participated, including individuals from the Departments of Anesthesiology, Emergency Medicine, Neurology, Psychiatry and Surgery.
During the first 2 years, teams struggled to implement interventions in time to see their effect on project measures by the end of the AQSI programme. With past participant input and Steering Committee guidance, we made several adjustments. First, we added a mid-program classroom session during which teams gave one another a project update and received feedback. Prior to this, teams provided project reports twice during the programme to the health system’s Improvement Council, but only provided a project report to one another during the final classroom session. Second, we reorganised the class schedule, moving DMAIC sessions earlier in the programme. Third, we created specific milestones with expected dates of completion to help teams keep projects on track. Fourth, we set the expectation that teams implement interventions early enough to have some results by the end of the programme. We emphasised that project work should continue after completion of the programme, but felt that presenting results would help participants understand the impact of interventions(s) and that planning adjustments based on early results provided important lessons for learners.
Evaluation
We evaluated AQSI in several ways to assess its effect on learners and clinical quality measures.
Session attendance and evaluations
We took attendance and administered a survey to participants at the end of each classroom session asking them to rate their agreement as to whether learning objectives were met, the instructor presented information effectively and new knowledge was obtained as a result of the session.
Immediate postprogramme survey
Participants completed a survey at the end of the programme to evaluate whether AQSI improved their ability to design, implement and evaluate QI interventions, specific forms of AQSI support (ie, performance improvement coach, clinical mentor, data analytics consultant) and whether the participant would recommend AQSI to others. Items in the evaluation asked respondents to rate their agreement with statements using an ordinal response scale (ie, strongly disagree, disagree, neither agree nor disagree, agree, strongly agree).
Knowledge assessment
We created a 14-question multiple choice test consistent with the US National Board of Medical Examiner Guidelines on writing test questions and administered it to participants before and after the programme.12 Questions assessed learning objectives provided by instructors for core quality topic sessions. We also developed an adapted version of the Quality Improvement Knowledge Assessment Tool (QIKAT), an instrument used to assess knowledge of performance improvement principles13 14 and administered it to participants beginning in year 2. The adapted QIKAT assessed knowledge of DMAIC methodology and consisted of three case scenarios each followed by 2–3 questions (total 8 questions). Short answers were scored, in a blinded fashion, on a 0–3 point scale using a standardised rubric for a total possible 24 points. We tested reliability by having two investigators independently score 20 adapted QIKAT assessments and used concordance correlation coefficients to assess interrater reliability. We found moderate to strong correlations for individual items (r=0.53–0.94) and strong correlation for the total score (r=0.87). We tested the adapted QIKAT construct validity by having 10 performance improvement leaders (experts) and 36 first year medical students (novices) complete QIKAT assessments and found significant differences in mean scores (16.2±0.8 for experts vs 8.7±0.5 for novices; p<0.001). The multiple choice test and adapted QIKAT are provided in an online supplementary appendix.
Supplemental material
Six-month and 18-month postprogramme surveys
We administered surveys to participants at 6 and 18 months after AQSI programme completion to determine their involvement in QI activities and the status of their project. Eligible participants for the 6 month survey included all individuals in the first five cohorts (ie, completing the programme in 2012–2017) of the programme. Eligible participants for the 18 month survey included all individuals in the first four cohorts (ie, completing the programme in 2012–2016). The 6-month survey asked respondents to rate agreement with statements related to their activities and confidence in QI work using an ordinal response scale (ie, strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). The 18-month survey asked respondents to reply yes/no to items asking about their involvement in QI activities.
Analysis
We calculated descriptive statistics of participants’ attendance, demographic characteristics, team characteristics and session evaluations. We report multiple choice test scores as per cent of items answered correctly. Similarly, for ease of interpretation, we report adapted QIKAT scores as the percentage of total possible points achieved. We compared participants’ preprogramme to postprogramme multiple choice test and adapted QIKAT scores using paired t tests and calculated Cohen’s d to estimate the effect size. We calculated descriptive statistics of participants’ involvement in QI activities at 6 and 18 months after AQSI completion. Similarly, we report descriptive statistics for project status at AQSI completion and the number and percentage of projects with postintervention results available and positive postintervention results (ie, improved performance on project measures) at completion of AQSI, at 6 and 18 months. For projects with more than one measure, we considered the result positive if one or more measures showed improvement and no measure declined in performance. To assess the effect of adjustments made to the programme, we compared the percentage of projects with positive results in the first two cohorts (ie, 2013 and 2014) to the final three cohorts (ie, 2015–2017) using χ² tests. All analyses were conducted using Stata V.13.1 (College Station, Texas, USA).
Results
Participant and team characteristics
Overall, 172 individuals and 32 teams participated in AQSI. As shown in table 2, participants included physicians at various levels of training, nurses, pharmacists, advanced practice providers and social workers. Approximately a third of participants (62/172; 36.0%) had prior QI training and/or experience. Teams included a mean 5.4±2.4 participants. Overall, 14 (43.8%) teams included participants from more than one department and 19 (59.4%) included participants from more than one profession.
AQSI participant and team characteristics
Programme attendance and session evaluations
Participants attended a mean 76.8%±15.7% of classroom sessions and rated sessions favourably. In the vast majority of session evaluations collected, participants agreed or strongly agreed that sessions met the stated learning objectives (950/983; 96.6%), that instructors presented information effectively (956/989; 96.7%) and that that they obtained new knowledge from the session (950/991; 95.9%).
Learner outcomes
Immediate postprogramme evaluations
Overall, 172 (100%) participants completed the immediate postprogramme evaluation. The vast majority of participants agreed or strongly agreed that AQSI improved their ability to design (170/172; 98.8%), implement (166/172; 96.5%) and evaluate QI initiatives (168/172; 97.7%) and would recommend AQSI to others interested in QI (166/172; 96.5%).
A majority of participants also agreed that the performance improvement coaches and clinical mentors provided useful guidance (139/167; 83.2% and 131/163; 80.4%) and that the programme provided effective data consultant support (131/166; 78.9%).
Knowledge assessment
The mean multiple choice test score was significantly higher after completion of the AQSI programme (71.9±12.7 vs 79.4±13.2; p<0.001) (see table 3). Similarly, the mean adapted QIKAT score was significantly higher after AQSI completion (55.7±16.3 vs 61.8±14.7; p<0.001). As a reference, the converted mean adapted QIKAT score for the 10 performance improvement leaders (ie, experts) we assessed for reliability was 67.3±11.4. Cohen’s d was 0.58 for the multiple choice test and 0.39 for the adapted QIKAT, suggesting a moderate effect.
Participants’ knowledge of quality improvement before and after the AQSI
Participant outcomes at 6 and 18 months
Overall, 148 of 172 eligible participants (86.0%) completed surveys at 6 months and 101 of 127 eligible participants (79.5%) completed surveys 18 months after completing the AQSI programme. As shown in table 4, the majority of participants at 6 months agreed or strongly agreed that they had applied knowledge and skills learnt to improve quality in their clinical area and to implement QI interventions. Although a majority indicated that they felt confident leading QI changes, less than half indicated they felt confident teaching QI principles. At 18 months, nearly half of the participants had led other QI projects and a majority had participated in QI projects or committees. Relatively few had taught classes on QI topics, but many had provided mentorship in QI.
Effect of AQSI on participants at 6 and 18 months after completion of the programme
Outcomes for participants with and without prior QI training or experience
At 6 months, 50 of 57 (87.7%) participants with prior QI training or experience and 79 of 91 (86.8%) without prior QI training or experience agreed or strongly agreed that they had applied knowledge and skills learnt to improve quality in their clinical area. At 6 months, 39 of 57 (68.4%) participants with prior QI training or experience and 53 of 91 (58.2%) without prior QI training or experience agreed or strongly agreed that they had applied knowledge and skills gained to implement QI interventions. At 18 months, 22 of 39 (56.4%) participants with prior QI training or experience and 26 of 62 participants (41.9%) without prior QI training or experience indicated they had led other QI projects. At 18 months, 32 of 39 (82.1%) participants with prior QI training or experience and 44 of 62 (71.0%) without prior QI training or experience had participated in other QI projects or committees.
Project outcomes
Project teams addressed a range of problems, including those related to safe, effective, patient-centred, timely and efficient care. A summary of team projects is provided in an online supplementary appendix, including details on project focus, year, measure(s), goal(s), domain of quality addressed,15 interventions, DMAIC phase at AQSI completion and results at 6 and 18 months.
Supplemental material
Across all years of the programme, 4 (12.5%) teams were in the Analyze phase, 20 (62.5%) were in the Improve phase and 8 (25.0%) were in the Control phase on AQSI programme completion (see table 5). Fewer teams were in advanced DMAIC stages in the initial two cohorts of participants (ie, 2013 and 2014) compared with the final three cohorts of participants (ie, 2015–2017). Across all years, 14 (43.8%) teams had postintervention results available at AQSI completion with all 14 teams showing positive postintervention results (ie, improved performance on project measures). Overall, 22 (68.8%) of teams had postintervention results at some point (ie, at completion, 6 months or 18 months after AQSI) and 20 (62.5%) had positive results at some point. Significantly fewer teams in the initial two cohorts had positive results at AQSI completion compared with the subsequent three cohorts (8.3% vs 65.0%; p=0.002). Similarly, fewer teams in the initial two cohorts had positive results at some point (ie, at completion, 6 months or 18 months after AQSI) compared with the subsequent three cohorts (33.3% vs 80.0%; p=0.008).
Project status and project measure performance at AQSI completion, 6 and 18 months
Discussion
We found improvements in learner outcomes and clinical quality measures with the implementation and adaptation of AQSI, a programme designed to train professionals in our organisation to lead QI. Our programme included features found to be associated with success in prior studies, including experiential learning through QI projects selected by participants, coaching and integration of the educational programme with the clinical system.8 10 Importantly, we assessed learners’ involvement in QI activities and project measure performance beyond completion of the programme.
Our evaluation found that participants felt the programme improved their ability to design, implement and evaluate QI and that participants had significantly greater knowledge of core quality principles and the DMAIC method after completing the programme. Participants reported a high degree of involvement in, and leadership of, QI initiatives at 6 and 18 months, including participants with no prior QI training or experience. Most prior studies have not assessed participants’ engagement in QI activities after the programme. One exception is a study reporting the long-term experience of the Department of Veterans Affairs (VA) National Quality Scholars Fellowship Programme.16 The study found that 60 of 75 (80%) VA Quality Scholar Fellowship graduates were in clinical or research positions related to the improvement of healthcare. Although not a fellowship, our programme similarly involved a highly motivated group of participants who applied for the programme. Though certainly not conclusive, our results suggest programme participation contributed to high levels of subsequent QI activity. Future research should seek to identify specific factors associated with participants’ long-term engagement in QI.
Although a majority of teams had postintervention results at some point, some did not, and some results did not show improvement. This finding is similar to that seen in prior studies.17 18 After noticing that many AQSI teams in the first 2 years struggled to implement interventions, we made adaptations to better support project work. The adaptations appear to have been successful as a greater percentage of teams in the latter cohorts were in advanced DMAIC stages at AQSI completion and had positive results.
From the beginning of our programme and based on our needs assessment, we provided support to teams through the use of performance improvement coaches, clinical mentors and a data consultant. A majority of participants had favourable perceptions of this support. In addition to helping teams with basic project management strategies, performance improvement coaches helped teams connect with key individuals and groups (eg, committees) within our organisation to facilitate successful implementation of interventions. Many team members informally reported that they had limited prior knowledge of the health system beyond the frontline and how to navigate it to lead change. Our provision of direct access to the data consultant and Manager of Clinical Informatics was similarly helpful as few team members had prior access to data or familiarity with how requests to make EHR changes were made or prioritised.
Our study is also unique in its inclusion of clinical quality measures for projects beyond the completion of the programme. Importantly, assessment of project status beyond completion of the programme allowed us to detect delayed results for teams which had just implemented or were just about to implement interventions at programme completion. Although many projects had sustained positive results, some did not. Others did not assess whether improvements were sustained because repeat data collection would have been labour intensive (eg, medical record review, patient calls).
Interestingly, less than half of participants felt confident teaching QI principles and methods at 6 months and while many had provided mentorship to others at 18 months, relatively few had taught classes on QI. This result may be explained by relatively limited opportunities to formally teach QI to students and residents. Alternately, further training which combines principles of QI and fundamentals of medical education may be needed to prepare individuals to effectively teach and mentor QI.19
We have continued to expand the AQSI programme, offering it to professionals in additional departments. We also recently launched versions of the AQSI programme at two other sites in our healthcare system and plan to evaluate the experience in a similarly rigorous fashion.
Our study has several limitations. First, we used a before-and-after study design. Although use of a control group would have provided a stronger study design, we felt a need to include as many interested learners as possible. Most prior studies evaluating QI curricula have similarly used a before-and-after study design.9–11 Second, though we assessed reliability and construct validity of the adapted QIKAT, we did not perform similar assessment of the multiple choice test. Third, as previously mentioned, our programme involved a highly motivated group of participants. It is possible that a large portion of participants would have had high levels of involvement in QI activity without the training we provided. Related to this point, it is also possible that project outcomes improved over time because of organisational improvements to supporting QI activities. We feel this is unlikely as no major structural or systems changes related to QI occurred during this time. Fourth, our 6-month and 18-month assessments used surveys we created for this study and rely on self-report. Although it is possible that self-reported results could be skewed due to social desirability bias, we feel the results are valid as we have witnessed many past AQSI participants leading QI projects and subcommittees in our health system since completion of the programme. Fifth, our survey questions at 6 and 18 months did not use the same response scale. Using identical questions and response scales may have allowed us to assess patterns of engagement in QI activity over time. Sixth, none of the teams in our programme focused their projects on improving healthcare equity. Given the importance of healthcare disparities, we intend to promote applications in the future which address this domain. Finally, we did not perform statistical testing nor did we require statistical process control methods to evaluate performance for project measures. Though many projects did use run charts and control charts, some project measures were not collected on an ongoing basis (eg, medical record reviews conducted on a sample of patients preintervention and postintervention). Because project results relied on self-report, it is also possible that we may have overestimated the number of projects with positive outcomes. We include an online supplementary appendix with details on project focus, measures and results at 6 and 18 months.
Conclusion
Our programme to train QI leaders was associated with improvements in learner outcomes and clinical quality measures. Importantly, participants reported a high degree of involvement in, and leadership of, QI beyond the programme. Rigorous programme evaluation helped us make adaptations to improve teams’ ability to complete projects and achieve positive results.
Acknowledgments
The authors wish to thank the instructors, coaches, department chairs, health system leaders and Steering Committee members who dedicated their time, effort and expertise to help the program. Most of all, we thank the AQSI participants for dedicating their time and effort to learn new techniques to improve the quality of patient care.
References
Footnotes
Contributors All authors listed on the manuscript have contributed sufficiently to be included as authors.
Funding This project was funded by Northwestern Memorial Hospital and the Northwestern Medical Group.
Competing interests None declared.
Patient consent Not required.
Ethics approval This study was deemed exempt by the Northwestern University Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional unpublished data are not available.