An essential step in achieving a safe culture within healthcare organizations is implementing practices that have been shown to reduce errors. Even cursory evaluation of care in hospitals suggests the need for a great number of these “safe practices”. Responding to this perceived need, the National Quality Forum (NQF) recently developed a list of 30 practices for which an expert panel recommended widespread adoption.¹ The Joint Commission for the Accreditation of Healthcare Organizations (JCAHO) has subsequently mandated implementation of a number of these and other safe practices.²

Because patient safety is a new field, identifying which safe practices are effective has presented a significant challenge. To select practices for its Breakthrough Series Collaboratives, the Institute for Healthcare Improvement (IHI) relied on experts who considered salience of the topic, feasibility, evidence that the practice would improve quality and reduce costs, and demonstration of significant improvement by at least one organization.³

The NQF consensus development process cast a broad net that identified more than 220 candidate safe practices from its steering committee experts, NQF member organizations, a call to more than 100 medical specialty, nursing, pharmacy, and other health professionals, the three “safety leaps” of the Leapfrog Group, and an intensive review of evidence of effectiveness conducted by the UCSF-Stanford Evidence-Based Practice Center.⁴ The final choice of 30 was based on expert consensus regarding specificity, evidence of effectiveness, likely benefit, generalizability, and readiness.

In 1999, prior to the NQF process, the Massachusetts Hospital Association (MHA) and the Massachusetts Coalition for the Prevention of Medical Errors (the Coalition) used a new consensus building model of hospital stakeholders to identify a set of safe medication practices that all Massachusetts hospitals were encouraged to implement. In 2001 we received funding from the Massachusetts Department of Public Health through a cooperative agreement with the Agency for Healthcare Research and Quality (AHRQ) to build upon the 1999 model to implement and evaluate a new process to promote voluntary statewide adoption of two evidence-based safe practices by a “bottom up” process that would identify practices by need, as reflected in the statewide reporting system, and by clinical interest, as determined by a consensus process among stakeholders.

Through the consensus process we selected two practices—reconciling medications and timely and reliable communication of critical test results. Neither practice was included in the list of 30 safe practices developed and released shortly afterwards by the NQF, nor in the initial set of safe practices required for implementation starting in 2003 by the JCAHO. However, the Commission has subsequently adopted both of these practices as mandatory in 2005 and 2006.²

To promote future voluntary collaborative efforts, we describe here the processes we used to (1) select the two new practices to be implemented, (2) develop the operational details of the practices, (3) enlist hospitals to participate in the project, and (4) facilitate implementation of the practices.

METHODS

Selection of safe practices for implementation

Potential topics were identified through solicitation of Coalition members, QI Directors and hospital CEOs, and by review of data from the Massachusetts reporting system on errors, the NQF list of serious reportable events,⁵ and the AHRQ Patient Safety Practices with Greatest Strength of Evidence.⁶ Data were then retrieved from the literature and several data bases on frequency and severity of the problems, the availability of a tested safe practice for each, and evidence for its efficacy. Ten candidate practices were identified (box 1).

The practices to be implemented were selected by a 17 member multi-stakeholder Advisory Committee that included physicians, nurses, pharmacists, QI/risk management/patient safety managers, administrators, regulators, liability insurers, professional associations, and researchers. The committee reviewed the evidence and used five criteria to select the practices: importance of the problem, availability and proven efficacy of the practice, feasibility of implementation, and potential impact on safety. Two practices were selected: (1) reconciling medications and (2) communicating critical test results.

Reconciling medications (RM) is a formal process for creating the most complete and accurate list possible of pre-admission medications for each patient and comparing the physician’s admission, transfer, and discharge orders against that list. Discrepancies are brought to the attention of the physician and, if appropriate, changes are made to the orders. Any resulting changes in orders are documented.⁷

Communicating critical test results (CTR) focuses on timely and reliable communication of critical test results to the clinician who can take action. Critical tests results are any values/interpretations for which delay in reporting can result in a serious adverse outcome for a patient.

Development of operational details

A “safe practice” is in reality a collection of many individual practices—that is, decisions and process changes that need to take place to implement the practice. We used stakeholder groups to develop these subpractices through discussion and iteration with experts.

For each practice we convened a stakeholder group reflecting the members of teams that would be called upon to implement the practices (nurses, physicians, quality specialists, laboratory specialists, and pharmacists) and representing all types of hospitals. The groups were chaired by experts who had previously led implementation of the practices in hospitals. National experts (David Bates for CTR and Roger Resar for RM) provided a review of the literature and their own evidence about the procedure, and explained in detail the problems and challenges in implementation.

Each group defined the decisions and changes that needed to take place to implement the practice. For CTR, we identified nine steps or subpractices required to operationalize the overall practice, for example:

• Define what test results require timely and reliable communication.

• Identify who the results should go to.

• Identify who the results should go to when the ordering provider is not available.

For each of these subpractices we then identified the steps necessary to accomplish it and the requirements and pitfalls to be expected, as well as specific recommendations with explanatory context to facilitate understanding. This process of identifying and specifying all of the steps in each subpractice was laborious and required repeated iterations involving all disciplines that would be affected by the changes being proposed. The recommendations for each practice are described in detail by Rogers et al⁸ and Hanna et al⁹ and on the Coalition’s website (www.macoalition.org).

Enlisting hospital participation

Because our goal was to get as many hospitals in the state as possible to participate in the collaboratives and to implement both initiatives, we conducted an extensive effort to enrol them. An invitation letter was sent to the CEO of each hospital in the state explaining the procedures and how they were selected and inviting them to participate in the collaboratives. The MHA Board of Trustees voted to make implementation of the two practices its “Flagship safety initiatives” for the year, and called on all hospitals to participate in one or both of the initiatives. Hospital CEOs were asked to demonstrate their commitment by signing a “pledge of participation” form. Enrolment was tracked and follow up with hospital CEOs occurred through direct email communication, individual telephone calls, routine staff onsite visits, advisories and bulletins.

Finally, a special program was held for hospital leaders to inform them of the projects. At this briefing the national expert leaders described the practices and their successful experiences, answered questions, and outlined the requirements for participation.

Facilitating implementation of the practices: the collaborative model

A collaborative model was used to implement the practices. Hospitals could participate in both collaboratives. They were asked to form multidisciplinary teams (nurse, physician, quality specialist, and pharmacist or laboratory representative as appropriate), secure support from the CEO, and commit to four collaborative meetings.

Separate meetings were held for each collaborative. At the initial meeting, national experts (DWB and RR) described the details of the practices and their experiences with implementation. Next, process experts from IHI described the “Plan-Do-Study-Act (PDSA)” rapid cycle change methodology for implementing new practices. (The Plan-Do-Study-Act cycle was developed by W Edwards Deming. It has been used by hundreds of healthcare organizations to accelerate change, promoted especially in breakthrough series collaboratives supported by the IHI and Associates in Process Improvement.¹⁰)

Participants were provided with a toolkit containing safe practice recommendations, a change package, and implementation strategies. The “change package” consisted of a detailed explanation of the rationale behind each of the subpractices and recommendations for what was needed to carry them out. It addressed such issues as the roles for each participant (nurse, doctor, pharmacist), development of measures, preliminary data collection, etc. In addition, a set of sample tools (forms, flow charts, policies, etc), implementation tips, and reference materials were provided (for details, see Hanna et al⁹ and Rogers et al⁸).

The importance of engaging leadership and involving all stakeholders, particularly physicians, was emphasized, and teams were given suggestions for enlisting support of clinicians and for dealing with “pushback”. Teams were instructed to collect baseline data and begin their first tests of changes within a few days. They were advised to have frequent meetings and “huddles”, and to file reports with their CEO and with the project staff monthly. A listserv was set up to exchange information and answer questions. In addition, the project director was available for telephone consultation for each practice.

Three additional meetings were held for each collaborative over the next 18 months. Teams presented reports on their progress and shared observations on problems and strategies used to overcome them. Process experts and colleagues from other teams commented on reports and provided advice on changes that could be made to aid progress.

Continuing hospital interest in the collaboratives was actively encouraged by the MHA through publicizing success stories in its communications to hospital leaders. At its 2004 annual meeting, every participating hospital received a status report showing where their team stood in comparison with all participating hospitals.

Evaluation

Based on team reports, success in implementing the practices was rated using an assessment scale similar to the five stage system developed by IHI for rating team progress:³ 1 = planning only, 2 = testing changes, 3 = partial implementation, 4 = fully implemented in some areas, 5 = fully implemented throughout the institution. The objective measure of success for RM was the percentage of medications unreconciled and for CTR was the percentage of critical results communicated within the time limit.

To evaluate the impact of the various factors responsible for hospital team success, surveys of team leaders and CEOs were carried out at the end of the project. Respondents rated (1) factors that contributed to an institution’s decision to participate, (2) organizational and administrative factors that facilitated implementation, including the role of the collaborative, (3) data collection and use of the rapid cycle improvement model, (4) implementation and measurement details for each practice, and (5) barriers to implementation.

RESULTS

Hospital participation

The collaboratives were open to all hospitals in Massachusetts; although several non-acute care organizations participated, the statistical analysis of results focuses on the state’s 66 acute care institutions. Of these, 58 (88%) participated, 50 enrolled teams for RM and 40 enrolled teams for communicating CTR; 32 hospitals sent teams to both.

Interviews with quality leaders in all of the hospitals that did not participate revealed they were small hospitals (seven of the eight had fewer than 50 beds) in small communities or distant from the collaborative meeting sites. Two were geographically isolated. Most were unable to muster the resources and personnel to do the project. Several had been facing significant financial constraints (negative margins) as well as staffing issues such as overburdened quality directors, who also covered for other vacant positions and several cited competing priorities such as upcoming JCAHO surveys.

Response rates to the surveys among participants were high: CEO 76% (44/58 participating hospitals), RM team leaders 84% (42/50 collaborative participants), and CTR team leaders 85% (34/40 collaborative participants).

Post-collaborative surveys showed that a number of factors were perceived by respondents to influence them positively towards participating in the collaboratives (table 1). Factors with highest impact (rated as “some” or “a lot” by 90% or more of CEOs and team leaders) included the sense that the recommendations were likely to work, access to experts, the opportunity to learn from peers, and availability of a set of implementation strategies. Other factors with high salience were evidence provided at the leadership briefing, endorsement of the initiatives by MHA trustees and the Coalition, interest in the topic among clinical leaders, the low cost of participating, and the JCAHO emphasis on patient safety.

Implementation

The teams varied widely in their success at implementing both of the safe practices (fig 1); 10–15% were unable to get beyond the planning stage. However, 50% of RM teams and 65% of CTR teams implemented the practices partially or fully during the collaborative term (stages 3–5). Figure 2 shows the results obtained by some of the successful teams in study units.

Almost all the teams embraced the collaborative method. Elements of the collaboratives that teams reported as especially helpful were hearing about the practices from national leaders, learning from peers, having specific recommendations and a toolkit for implementing each practice, and guidance in developing measurement protocols (table 2).

Surveys showed that teams found implementing the recommended practices difficult (91% of RM and 75% of CTR indicated some or a lot of difficulty). Team leaders perceived a number of barriers that inhibited their progress (table 3). For RM, getting people to change, complexity, staffing time required, and difficulty achieving clinician buy in, and competing priorities were major barriers. For CTR, fewer teams perceived serious barriers: only getting people to change, complexity and competing priorities were cited by a majority. Costs were not cited as a problem by the majority of teams or by the CEOs.

Responses to questions about project features known to facilitate implementation gave some insights into why many teams had problems. One reason was not meeting frequently enough: only 21% met at least biweekly for both RM and CTR. Nor did they convene frequent “huddles”, which were a recommended alternative: only 27% of RM and 18% of CTR used huddles frequently. A minority met at least monthly to analyze data to assess progress (43% of RM and 30% of CTR). About one in five teams did not use PDSA cycles.

Hospital support varied considerably. Hospital leaders did not set aside dedicated staff time for 54% of RM teams or 35% of CTR teams, nor did they assign a senior administrator to support the project in 50% of RM teams or 23% of CTR. Physicians were minimally or not at all involved in 63% of RM teams and 41% of CTR teams.

The effect of these and other factors on reaching implementation stage 3, 4 or 5 were examined through the use of bivariate cross tabulations and their associated χ² values. For CTR, probably because of the small number involved, none of the factors reached statistical significance. For RM, however, better outcomes were associated with active engagement of a senior administrator, engagement of nurses, increased frequency of data collection (all p<0.05) and increased use of small tests of change (frequent PDSA cycles, p<0.001).

The importance of leadership engagement is shown by one of the most successful hospitals. Each month the project team leader met in a “huddle” (15 minutes) with the administrators responsible for the departments affected by the proposed changes, the Vice-President of Medical Affairs and the Vice President and CEO. These huddles were initiated early in the project and occurred consistently throughout. The hospital leaders regularly reviewed project data, assumed responsibility for overcoming challenges and barriers within their areas of accountability, and provided direction and support for the implementation team. They provided critical links to the Patient Safety Committee, the Medical Executive Committee, and the hospital board.

DISCUSSION

This statewide initiative to promote hospital implementation of safe practices had two aims: (1) to motivate all the acute care hospitals in Massachusetts to participate by involving them in the selection process, and (2) to enable all the participants to successfully implement the new practices. To achieve these objectives we used a number of methods culled from prior experience (our own and others) and well known techniques from group change theory.

We used a novel process for identifying and selecting safe practices, building on prior experience with a consensus process among stakeholders. Candidate topics were identified by need, as reflected in the statewide reporting system and solicitation of concerns of hospital CEOs and QI directors.

The two choices had high “resonance” for the following reasons: (1) they addressed clinically important hazards encountered in everyday practice so their solution would make a substantial impact on patient safety; (2) solutions to these problems would require addressing fundamental issues of interdisciplinary communication and teamwork which are recognized as crucial aspects of a culture of safety; (3) impressive success in implementing the practices had already been achieved by a few hospitals; and (4) implementation would not require major new investments and should pay for itself in efficiency gains and reduced adverse events.

The validity of these assessments was confirmed after conclusion of the project by the JCAHO selection in 2005 of both practices as national patient safety goals for hospitals. The salience of the topics is also borne out by the high hospital participation rate. Of the 66 eligible hospitals in the state, 58 (88%) enrolled in at least one collaborative (32 enrolled in both) and sent a team to collaborative meetings. Although we fell short of our goal of 100%, these high rates of participation represent a significant achievement for voluntary participation at a state level in the absence of external reward or coercion.

Surveys of hospital CEOs and team leaders after the end of the study indicated that endorsement by the Coalition and the hospital association and evidence provided at the leadership briefing were important factors in the decision by hospitals to participate, as were interest in the topic among clinical leaders and the intrinsic appeal of the practices. Other factors that we believe played a role were involvement of hospitals in the selection of the practices, ensuring broad stakeholder representation on the selection committee, and support by the leadership of the MHA.

Achieving the second goal—full implementation—was less successful. Nationally, implementation of safe practices has been haphazard and sluggish. Until quite recently it has depended primarily on voluntary initiatives by health care organizations that responded to locally identified problems and, often, developed new practices ad hoc to solve those problems. This effort long antedates the activities of the NQF and JCAHO, and even the landmark Institute of Medicine (IOM) report of 2000 “To Err is Human”.¹¹ This changed in 2003 when the JCAHO began to require hospitals to implement certain safe practices. Preliminary evidence suggests that these regulatory mandates have been more effective in achieving adoption than voluntary efforts.¹² Our study antedates the JCAHO mandates. We examine the causes for success or failure in two categories: the collaborative process and team process.

Collaborative process

The use of collaboratives composed of multidisciplinary teams from many institutions to facilitate change in practice was pioneered by IHI in the mid 1990s. The rationale, methods, advantages, and results have been discussed elsewhere.^3,13 Advantages include economies of scale in terms of instruction in the rapid change model and dissemination of information and materials, the power of learning from peers from other institutions, and the discipline imposed by requirements to report progress regularly and in person. IHI has sponsored over 50 such collaboratives reaching 1000 different organizations,³ and they have been widely used for nationwide initiatives in the USA and elsewhere.

Collaborative initiatives are complex activities that pose significant evaluation challenges.¹⁴ Other activities going on at intervention sites make it very difficult to isolate the impact of the collaborative itself, and designing appropriate evaluation measures is difficult. The collaborative time frame is too short for changes in process of care to translate into measurable results on outcomes.

Evaluations of numerous collaboratives^13,15–17 show striking improvements at individual participating organizations, especially in the adoption of process changes, but varied success for the collaborative as a whole. In an assessment of 40 collaboratives the IHI found that at least half of the participating organizations were able to implement the changes on the target unit and begin spreading to other areas. A follow up survey of past participants showed that 97% of teams at least held the gains from their efforts and 67% were performing better than at the end of the collaborative.¹⁸ The successes can be quite compelling, such as the 22 hospitals participating in Michigan’s Keystone ICU collaborative who were able to reduce their catheter related bloodstream infections to zero.¹⁹

Recent attempts to evaluate collaboratives by comparing results with matched non-participating sites show uneven results: a national collaborative among neonatal ICUs had better results in two process of care measures but no improvement in outcomes;²⁰ an asthma collaborative also showed overall process of asthma care improved significantly in the intervention group but no significant difference in the outcome measures.²¹

Lessons learned

Øvretveit et al²² identify the challenges for running successful collaboratives, including choice of subject, defining objectives, defining roles and expectations, motivating teams, and use of data. Our experience corroborates theirs, with some additional findings.

Positive lessons

1. Using stakeholders to choose subjects. As noted, the subjects had wide appeal because they addressed real world concerns of practising clinicians. They resonated with both the consensus groups and the teams because of their potential to reduce errors and adverse events.

2. Motivating teams. We used national experts who provided convincing evidence of effectiveness of the practices. Teams reported that they found presentations from national experts, learning from peers, the safe practice recommendations, and toolkit materials especially helpful (table 2). Team attendance at collaborative meetings mattered. Although three-quarters of hospitals sent at least one member to three or more of the collaborative meetings, only 32% of RM participators and 38% of CTR participators sent full teams to three or more meetings. Perhaps not surprisingly, successful teams had high attendance: 80% of CTR teams and 50% of RM teams that attended all four meetings scored 4 or 5 on team progress compared with 18% overall.

3. Advance preparation. The identification of the subpractices, development of specific recommendations and strategies for implementing them, and the preparation of tools to be used as templates by participants were crucial. This proved to be a major task, one that has not been sufficiently commented upon or emphasized in the recent movement to get hospitals to implement new safe practices. Even with these aids, our teams found implementation of the practices difficult. Without them, not many would have made much progress. Few hospitals have the financial resources, dedicated time, or expertise to perform these tasks alone.

4. From this experience we conclude that the development of subpractices, recommendations, and strategies should be well worked out before JCAHO or other authority mandates that hospitals implement any guideline. Failure to develop and communicate this “infrastructure” may be a major reason why many hospitals have been slow or have failed in implementing required safe practices. Even with this infrastructure, we found implementation to be challenging.

Negative lessons

Why did only a few teams reach stage 4 or 5 in the implementation process during the collaborative time period? Although ultimate responsibility for implementation lies with the hospitals, we identified five aspects of our management of the collaboratives that may have reduced their effectiveness:

1. Lack of clear expectations for participation. In retrospect it is evident that, in our effort to promote maximum enrolment, we were not clear enough about the expectations for participation, particularly dedicated staff time and active engagement of administrative leadership. Although we strongly recommended physician participation on teams, many did not have them. All of the successful teams did. Finally, we called for and reminded teams to file monthly reports to their CEOs and to the collaborative, but we did not require reporting as a condition of participation.

2. Inadequate engagement of hospital leadership. CEOs signed a pledge of participation to support the initiative but many were not engaged in support of their teams. In retrospect, perhaps the collaborative could have provided more “coaching” for CEOs and involved them more actively in supporting their teams. For example, regular status reports to hospital CEOs and team leaders showing where each hospital team was in comparison with all hospitals might have stimulated improved performance.

3. Insufficient emphasis on use of measures. Before starting implementation of a new practice it is essential to measure the extent of the defect to be corrected (rate of unreconciled medications or number of failed timely communication of critical test results). Not only do these data provide the baseline against which to measure the effectiveness of the practice, they are the evidence of the seriousness of the problem locally that motivates stakeholder involvement. During the project, consistent ongoing measurement is required to tell whether the changes being made are leading to improvement, and to provide motivation for continued action. Thus, another condition of participation should have been to require baseline measurement and to provide ongoing measures in the monthly reports.

4. Difficulty in using rapid cycle change. The concept of “small tests of change” is difficult for most health workers to learn. Many do not know how to collect simple performance data and construct run charts. Although IHI specialists provided a half-day of instruction during the first meeting of the collaboratives, this proved to be insufficient for many. In future an extra day should be added to the first collaborative meeting for instruction in rapid cycle change for those who need it.

5. Insufficient collaborative staffing. Each collaborative had one half-time staff person who organized the meetings, maintained a listserv, collected reports, and provided advice and support for the teams. This proved to be insufficient support for the large groups involved. We were unable to provide regular monthly feedback, conduct more than a few site visits, or provide intensive coaching. We believe at least one full-time person is needed for each collaborative.

Recommendations for collaboratives

The advantages of the collaborative method for changing practice are substantial, but will not be realized unless participants are fully engaged and have full support from hospital leadership. We therefore recommend that adherence to the following should be required as a condition of participation:

• Written commitment from hospital leadership for staffing of the practice implementation.

• High level oversight—that is, a high level administrator responsible to the CEO for making the project work.

• Progress reports by the team at each meeting of the Board.

• Commitment to regular standardized measurement.

• Submission of data on progress to the collaborative monthly, signed by CEO.

• Commitment to use the model for improvement to implement the practices.

Team success

Implementing the practices proved to be tougher than anticipated. These two practices required substantial changes in clinicians’ daily work. To succeed, both required effective teamwork and communication, information transfer, coordination among multiple hospital departments and caregivers, and changes to hospital organizational culture—all major endeavors. Success also required sustained effort on the part of the change team, support by hospital leadership, and serious involvement by all stakeholders. More fundamentally, each required changing the working relationships among professionals. It is possible that a longer implementation period would have resulted in more successes. In fact, later reports at the end of July 2005 showed that a number of teams had made additional progress.

Team success, as measured by the five-level IHI scale, was similar to that achieved in other collaboratives: about half succeeded in implementing the practices to some degree.⁴ Some teams made dramatic improvements in rates of late communication of test results or unreconciled medications (fig 2).

Teams varied considerably in the extent to which they embraced the model for improvement. Survey analysis showed that frequent data collection and use of frequent PDSA cycles were associated with better outcomes. Yet many teams met infrequently, and about one in five did not use PDSA cycles. Conversely, successful teams were those that participated fully in the process of improvement. Team outcomes were better in hospitals in which there was active engagement of a senior administrator.

Recommendations for teams

Our findings suggest that the elements for team success mirror those for the collaborative as a whole. Teams are likely to be more successful in implementing a new practice if they:

• Engage and are supported by a senior level administrator.

• Involve all stakeholders, particularly physicians.

• Actively follow the rapid cycle (PDSA) model.

• Meet frequently.

• Monitor their progress using outcome data analysis at least monthly.

CONCLUSIONS

Experience from two voluntary statewide collaboratives confirms that implementing new safe practices is a difficult and complex task for hospitals. Nonetheless, the collaborative process is a powerful way to motivate and support change. The ideas, enthusiasm, and creative solutions that our clinical teams implemented within their systems were impressive. Small changes fuelled excitement and enthusiasm for larger changes and served as pilots for larger scale projects. Teams learned the value of sharing innovations with others, and to avoid being stalled by the complexity of the projects or to wait for the automated solution. Our experience suggests that, if they receive support from hospital leadership, clinical teams of front line caregivers can develop creative methods for improving teamwork and communication. These skills should enhance patient safety.

Because each practice is in fact a collection of multiple subpractices, these elements need to be carefully developed and separate strategies must be designed for making the multiple changes needed for implementation. Few, if any, hospitals have the resources to do this on their own. Even with extensive advance planning and careful attention to the design of the subpractices and strategies, teams found making changes difficult.

Lack of sufficient development and understanding of subpractices may be one of the reasons many hospitals have found it difficult to implement the new safe practices recently required by the JCAHO. While we applaud and support the move by JCAHO to require hospitals to implement safe practices, our experience suggests that such mandates might better be deferred until after the development of the needed subpractices and recommendations, and after they have been “field tested” through large scale feasibility demonstration projects such as we have described here. This testing could be commissioned by JCAHO or AHRQ (as this project was) and contracted to IHI, QIOs, or regional coalitions.

Sufficient resources need to be made available for these types of efforts to succeed. Even when the best practices and toolkits are available, complex organizational change requires dedicated staff time to test the changes, measure the impact, refine the approach, and spread the changes throughout the organization. Payers and policymakers should seek innovative ways to make funding available for these investments.