Reports of preventable illness due to medication errors are widespread in Canada. However, quantifying the magnitude of the problem has been hampered by a lack of measurement tools. Canadian-specific indicators, or performance measures, of safe medication use do not exist. The objective of this study was to develop a set of Canadian consensus-based indicators for the safe use of medication for both in-patient and outpatient settings.

A panel of 20 national experts was established from a convenience sample of experts representing medicine, nursing, pharmacy, research and decision-makers in hospitals and community settings across Canada. After creating a list of potential indicators from the literature, the final consensus set was chosen by the panel using a Delphi survey process via e-mail.

After three rounds, consensus was obtained on 20 medication-use safety indicators: seven indicators were related to systems of care, five to prescribing/ordering, three to monitoring/assessment, three to medication administration, one to preparation and dispensing and one to purchasing/inventory management. Seventeen of the indicators measure a process of care (in contrast to health outcome); at least 10 have applications outside the in-patient setting.

The resulting 20 medication-use safety indicators are diverse in scope and should be applicable in a variety of practice settings. These indicators may provide clinicians and decision-makers with valuable tools to assess the safety of medication-use systems.

Background

Patient safety is an area of increasing focus in Canada. Within this topic, safe use of medication has consistently been cited as a main area of concern; for example, both the Institute of Medicine report To Err Is Human (Kohn et al. 1999) and "The Canadian Adverse Events Study" (Baker et al. 2004) highlighted problems with adverse drug events. More recently, a special report on medication safety from the Institute of Medicine revealed that approximately 1.5 million preventable adverse drug events occur annually in the United States, resulting in a total cost of US$3.5 billion (Aspden 2006). Patients also recognize problems with the medication-use system. In the 2002 Commonwealth Fund survey, 11% of patients in Canada reported that they had been given the wrong medication at one time or another (Schoen et al. 2003).

An indicator is a quantitative measure of some aspect of patient care that can be used to assess the quality of care being provided. Indicators are not direct measures of quality; they can be viewed as "alerts" to potential problems that require more detailed analysis (Nadzam 1991). Given the importance of medications in modern medicine, the fact that they are the fastest rising healthcare expenditure in Canada and that problems in their use have been widely documented, it is essential to use indicators to assess the medication-use system for both in-patient and outpatient settings.

The medication-use system encompasses the typical course of action related to drug therapy, including ongoing monitoring of patient care and progress (Ackroyd-Stolarz et al. 2005). The stages in the system include prescribing, dispensing, administering and monitoring. Currently, there is no set of indicators for assessing this system that are uniquely fitted to the Canadian environment. As a consequence, performance evaluation and benchmarking of medication safety based on quantitative data are difficult. Furthermore, Canadian medication safety experts should have the opportunity to suggest new indicators. Therefore, the objective of this study was to generate a Canadian set of medication-use safety indicators based on consensus among experts in patient and medication safety.

Methods

The following steps were used to generate a Canadian set of medication-use safety indicators: establishing an expert panel, developing a survey for the indicators and using the Delphi technique to achieve consensus on the indicators. These steps are outlined in detail below.

Expert Panel

The co-investigators and the partnering national organizations - Canadian Council on Health Services Accreditation, Canadian College of Health Service Executives, Canadian Institute for Health Information, Canadian Pharmacists Association, Canadian Society of Hospital Pharmacists and the Institute for Safe Medication Practices Canada - were responsible for identifying nationally recognized experts in the safety of the medication-use system. An attempt was made to ensure that the panel had representation from across the country and from a variety of health professions and backgrounds. The 20 potential panellists who were identified were initially contacted by e-mail; they received a brief description of the study and what would be required of participants. Panellists were selected based on their experience, willingness to participate and availability. All 20 potential panellists asked agreed to participate; the target size of 20 people was governed by resource constraints. The panel consisted of experts in medication safety representing medicine, nursing, pharmacy, research and decision-makers in hospital and community settings across Canada (Table 1).  

[Table 1]

Survey Development

The first step in the development of the survey was to search the literature. A MEDLINE search using the key words patient safety and medication errors produced 494 articles, which were reviewed by the lead author. Unpublished literature, which included information from organizations involved in medication safety, was also reviewed by the lead author. The indicators were developed by the two lead authors based on areas of concerns within the medication-use system discussed in both published and unpublished literature. The indicators were then classified according to stages in the medication-use system outlined by the Wisconsin Patient Safety Institute (2002). Indicators were assigned to one of the following six categories: prescribing/ordering, preparation and dispensing, administration, monitoring/assessment, purchasing/inventory management and systems of care (i.e., continuity of care after in-patient discharge). A draft survey instrument was created and pilot tested by the four co-investigators and five colleagues, who had the opportunity to comment on and suggest additional indicators. The co-investigators and colleagues consisted of experts in medication safety, physicians, nurses, pharmacists and researchers.  

Consensus Building

We used the Delphi technique, which is a multiple iteration survey technique that enables anonymous, systematic refinement of expert opinion, to arrive at a combined or consensual opinion in a short period of time (Goodman 1987). It was first developed to prevent dominant personalities from taking over or intimidating others, as usually happens when groups of experts meet in person. The technique has been used extensively with expert panels to generate consensus on healthcare issues (Campbell and Cantrill 2001; Goodman 1987; Robertson and MacKinnon 2002). The experience of our research team suggests that in similar surveys using the Delphi technique, three or four rounds are typically required to reach consensus. In this study, the survey was distributed by e-mail; therefore, the participants never had the opportunity to meet in person, rendering the consensus process free of strong-personality coercion.

Criteria for Consensus

Surveys were distributed via e-mail. Participants were asked, "Should this indicator be included in a national set of medication safety indicators?" Each panellist ranked the potential indicators on a five-point Likert scale, where 1 = strongly disagree, 2 = disagree, 3 = unsure, 4 = agree and 5 = strongly agree. If the panellists chose 1 or 2, they were asked to explain why they did not think that indicator should be included. In round one, experts had the opportunity to comment on indicators and suggest additional ones, which were then added for the next round. The mean score for each indicator was calculated, and a second survey was prepared, rendering anonymous all comments and suggestions of panellists.  

Consistent with the Delphi technique, surveys were individually tailored so that each respondent received his or her score for each indicator on the previous round as well as the mean score of the group. The cut-offs for rounds one, two and three were >4.0 (agree or strongly agree) for achieving consensus approval. The cut-off for achieving consensus dropped for round one was <2.0, for round two was # 3.5 and for round three was # 4.0. The rationale for the standard for achieving consensus dropped is outlined in the results.

Results

From the literature review and after the pilot test, 53 potential indicators were chosen for inclusion in the first round of the survey. After the responses from round one were collated, 16 indicators had a mean score >4.0 and were thus deemed to be consensus approved. In round two, 70 indicators were presented, including new indicators created based on panellists' comments and new indicators suggested by the panellists. Of these, one indicator had a mean score >4.0 and was thus deemed to be consensus approved. The mean scores of the remaining indicators were between 2.0 and 4.0. Thus, to avoid repeating a review of all 69 indicators, for round three, only the 14 indicators with a mean score between 3.5 and 4.0 were presented to the panel. Of these, only three had a mean score >4.0 and were thus deemed to be consensus approved. The remaining 11 indicators were dropped since most of the indicators had already been in the survey for two rounds. Since consensus was not achieved by that point, the two lead authors felt it would never be achieved. Therefore, 20 indicators in total were consensus approved (mean score >4.0) after three Delphi rounds (Table 2). Consensus was achieved over a two-month period.

[Table 2]

Of the 20 approved indicators, seven are related to systems of care, five to prescribing/ordering, three to monitoring/assessment, three to medication administration, one to preparing and dispensing and one to purchasing/inventory management. Seventeen of the indicators measure a process of care (in contrast to a health outcome); at least 10 have applications outside the in-patient setting (Table 3).

[To view Table 3, please download the PDF.]

Discussion

Although medication safety indicators can be found in the literature (2003/2004 Annual Report. Hospital Pharmacy in Canada: Medication Safety 2004; Agency for Healthcare Research and Quality 2005; American Society of Health-System Pharmacists 2007; Canadian Council on Health Services Accreditation 2006; MacKinnon and McCaffery 2004; Nadzam 1991; Robertson and MacKinnon 2002), there is clearly room for additional indicators, especially those developed specifically for the Canadian healthcare environment. The 20 final medication-use safety indicators on which our panel of experts agreed are quite diverse in scope and should be applicable in a variety of practice settings, including those outside the in-patient environment. The indicators generated in this study reflect the various stages in the medication-use system.

Safety is a concern at all these stages. Leape et al. (1995) found that most adverse drug events (ADEs) occur during the ordering and administration stages (39% and 38%, respectively); 12% occur during the transcription and verification stage and 11% during the pharmacy dispensing stage. Bates et al. (1995) had similar results: of the ADEs that were considered preventable, 49% occurred during the ordering stages, 26% during the administration stage, 11% during the transcription stage and 14% during the dispensing stage.

Our study focused on developing medication-use safety indicators that could be widely used in assessing quality of care in both hospital and community settings once they are validated. At least 10 of the indicators have applications outside the in-patient setting, which is important as evidence shows that the rate of adverse events is four times higher in the community (Gandhi et al. 2003). Forster et al. (2004) found that 23% of patients experienced an adverse event after discharge from hospital; 72% of these were attributable to medications. Nickerson et al. (2005) determined that patients averaged 3.5 drug-related problems at the time of hospital discharge. Thus, the indicators developed in this study reflect areas where problems occur in the delivery of medications and the ongoing monitoring of the patient in the community.  

Limitations

The method used in this study has some limitations. The Delphi technique's ability to achieve consensus results, at least in part, from group interactions that occur anonymously, thus preventing domination by one or a few individuals, role playing or intimidation (Campbell and Cantrill 2001; Goodman 1987). Although this is clearly a strength of the technique, at times, we felt that some in-depth discussion, perhaps via videoconferencing or face-to-face meetings, would have been valuable. For example, consensus was deemed to have been achieved based on mean scores, but, occasionally, some experts' comments contradicted the mean scores (i.e., the majority agreed, but there was not absolute group agreement). Follow-up group interaction to resolve these discrepancies would have been interesting.

Conclusion

This study developed a set of 20 consensus-based medication-use safety indicators. In a second phase, a subset of these indicators will be tested for feasibility, reliability and validity in four health authorities in Atlantic Canada. If these indicators are found to be feasible, reliable and valid, the national partnering organizations will be involved in a comprehensive dissemination and knowledge-transfer strategy to ensure that the indicators are widely used, providing clinicians and decision-makers alike with a valuable tool to assess the safety of the medication-use system in Canada.  

Comments

Be the first to comment on this!