Economic evaluation in patient safety: a literature review of methods
- 1Centre de coordination de l'évaluation clinique et de la qualité en Aquitaine (CCECQA), Hôpital Xavier Arnozan, Avenue du Haut-Lévêque, 33600, Pessac, France
- 2Institut de recherché et documentation en économie de la santé (IRDES), 10, rue Vauvenargues, 75018, Paris, France
- Correspondence to Dr Bruna Alves de Rezende, Centre de coordination de l'évaluation Clinique et de la qualité en Aquitaine (CCECQA), Hôpital Xavier Arnozan, Avenue du Haut Lévêque, 33600, Pessac, France;
Contributors Bruna Alves de rezende is the corresponding author. She did the literature search and article selection and initial analysis. She conceived, structured and filled out the template used for summarising the selected articles. She was responsible for the final redaction of all the versions of the paper. She was also in charge of the redaction of part of the responses to the referees and to the editor's comments. Zeynep Or played an important part in article selection and initial analysis. She revised all of the information in the analysis template used by the authors. She widely contributed to the revision of the first and subsequent versions of the paper and was in charge of writing part of the responses to the referees' and editor's comments. Laure Com-Ruelle contributed to structuring the analysis template and to the analysis of the selected articles. She also contributed to revising the first and subsequent versions of the paper. Philippe Michel contributed to the analysis of the selected articles and revised the first and the subsequent versions of the paper. He widely contributed to rewriting of some sections. He was also in charge of writing part of the responses to the referees' and the editor's comments.
- Accepted 7 December 2011
- Published Online First 6 March 2012
Patient safety practices, targeting organisational changes for improving patient safety, are implemented worldwide but their costs are rarely evaluated. This paper provides a review of the methods used in economic evaluation of such practices. International medical and economics databases were searched for peer-reviewed publications on economic evaluations of patient safety between 2000 and 2010 in English and French. This was complemented by a manual search of the reference lists of relevant papers. Grey literature was excluded. Studies were described using a standardised template and assessed independently by two researchers according to six quality criteria. 33 articles were reviewed that were representative of different patient safety domains, data types and evaluation methods. 18 estimated the economic burden of adverse events, 3 measured the costs of patient safety practices and 12 provided complete economic evaluations. Healthcare-associated infections were the most common subject of evaluation, followed by medication-related errors and all types of adverse events. Of these, 10 were selected that had adequately fulfilled one or several key quality criteria for illustration. This review shows that full cost–benefit/utility evaluations are rarely completed as they are resource intensive and often require unavailable data; some overcome these difficulties by performing stochastic modelling and by using secondary sources. Low methodological transparency can be a problem for building evidence from available economic evaluations. Investing in the economic design and reporting of studies with more emphasis on defining study perspectives, data collection and methodological choices could be helpful for strengthening our knowledge base on practices for improving patient safety.
- Adverse events
- health policy
- cost and cost analysis
- evaluation programs
- epidemiology and detection
- decision making
- evaluation methodology
- healthcare quality improvement
- quality measurement
- pay for performance
- health services research
- incident reporting
- medical error
- human factors
- human error
Since the first publications on system-wide incidence and cost of adverse events (AEs),1–4 interventions for improving patient safety are implemented worldwide by hospitals, insurers and governments.5–8 An AE is defined as a circumstance or event that results in an injury to a patient as a consequence of a medical intervention rather than of the underlying medical condition.9 An AE represents an unintentional harm to a patient arising from any aspect of healthcare management. Most AEs have direct costs for patients and for healthcare providers, organisations and payers. In the April 2011 issue of Health Affairs, Jill Van Den Bos of Milliman's Denver Health Practice reports that AEs in the USA cost about US$17.1 billion in 2008 (0.72% of the US$2.39 trillion spent on healthcare for that year).10
Despite the growing implementation of system-wide patient safety practices,11 economic evaluations are rare. We define system-wide patient safety practices as interventions implemented at national or hospital level, targeted to organisational changes relating to common AEs (infections, medication errors or invasive procedure related events) and showing high sensitivity to context (leadership, culture, or institutional financial status or quality improvement infrastructure).12 Information about costs and effectiveness of these practices is crucial for making evidence-based decisions to allocate limited resources for improving patient safety. Economic evaluations allow for establishing the link between the resources deployed (cost of public interventions) and the results achieved, and are essential for improving accountability and transparency of resource use in the healthcare system. They range from producing cost estimations (how much different AEs and/or safety practices cost) to appraising the costs and the effectiveness of different interventions taking into account alternative uses of resources.
However, economic evaluations may be difficult to identify and appraise. Their relative strengths and weaknesses may not be immediately obvious as economic evaluation of patient safety practices is relatively recent. This narrative literature review provides an overview of the economic evaluations of system-wide patient safety practices and exemplifies some valuable contributions. Our objectives were to identify the types of AEs studied and the methods used in estimating costs of AEs and cost effectiveness of patient safety practices and to highlight some good examples of different types of economic analyses in the literature. We describe the results and discuss the lessons that can be learned for future research.
We searched two medical database (MEDLINE, NHS EED) and one economics database (Econlit) for publications in English and French between 2000 and 2010. The keywords used were adjusted to the dictionary of these bases for identifying relevant literature. Medline and NHS EED were searched with the following MeSH Keywords: (‘adverse events’ OR ‘safety management’ OR ‘cross infections’ OR ‘intraoperative complications’ OR ‘perioperative care’ OR ‘medical practice’ OR ‘equipment and supplies’ OR ‘prescription drugs’) AND (‘health policy’ OR ‘hospital’ OR ‘hospital costs’ OR ‘primary healthcare’) AND (‘evaluation programs’ OR ‘cost and cost analyses). NHS EED was secondarily searched with: (‘adverse events’ OR ‘infection’) AND ‘interventions’ AND (‘evaluation’ OR ‘impact analysis’ OR ‘cost analysis’). Econlit was searched with: ‘patient safety’ OR ‘safety management’ OR ‘adverse events’ OR ‘iatrogenic disease’. This was supplemented with a manual search of references of previous reviews. Grey literature was not taken into consideration.
All articles were first reviewed by a public health medical researcher and by an economist and summarised using a common template: type of adverse event/practice studied, evaluation method, perspective adopted in costing, population/sample studied, cost components included in the analysis, data sources used, cost calculation methods, output/outcome measures used and results obtained. We distinguished three types of economic evaluation: burden studies (cost of AEs), studies examining cost of system-wide patient safety practices and complete evaluations looking into the relationship between costs of resources used for improving patient safety and outcomes obtained. In a second step, we defined six criteria that, in our opinion, are key for a valid (internal validity) and useful (external validity) economic evaluation in patient safety: methodological quality of AE identification, appropriate trade-offs between level of costing details and scale of the project (local or national), methodological quality of cost estimations, appropriate description of cost estimation methods, presentation of cost–benefit ratio, sensitivity analysis. We then conducted an evaluation of all papers according to these criteria. Each paper was independently assessed by a public health medical researcher and by an economist. Each reader quoted if each criterion was fulfilled according to a two-point Likert scale (yes/no, with qualitative appreciation); the discrepancies were discussed by the four members of the research group. The most illustrative papers on one or several criteria are described in the results section.
Our initial research yielded 402 papers of which we examined the abstracts. We excluded editorials and articles that did not carry out original cost evaluations. We retrieved 40 articles of which four reviews were excluded.13–16 We further excluded seven papers looking at the cost (burden) of AEs because the methods used for economic estimations were no different than previously selected articles.17–23 A manual search of the references yielded four more papers. In total, 33 articles were considered representative and reviewed in detail. Only four articles presented data on nationally representative samples.
Selected articles were assigned to three categories (table 1):
Evaluation of the economic burden (incremental costs) of AEs (n=18)24 29–44 50: these studies provide an estimation of the cost of AEs without looking at the cost of interventions for preventing them.
Complete economic evaluations (n=12)27 28 46–49 51–56 which consider both costs and benefits of a given patient safety practice, compared with the status quo situation or at least one alternative strategy.
Overview of studies
Most studies considered only one type of AE, mainly healthcare-associated infections and/or infection control activities, and medication-related AEs, especially in complete evaluations. While this is not a problem in itself for these studies, it means that our understanding of the economic consequences of AEs is quite limited and based on only a few types of AEs. For example, we did not find any economic evaluation on some common preventable AEs, such as falls in hospitals, a priority for the older population in many countries.
Data sources used to establish the incidence of AEs varied: hospital or national surveillance systems, implicit professional assessment, administrative or medical records, laboratory data, published data. Data collection was mostly retrospective, often with case–control designs, and data were collected over a long period (generally 1–5 years), which allowed for avoidance of significant variation in the estimated incidences.
The major characteristics of these studies are summarised in table 1. Tables providing a more detailed description of the content and methods of individual studies (sample characteristics, cost description, effectiveness criteria used, availability of sensitivity analysis, etc), are presented in the online appendices.
Cost information was based on either gross or micro costing techniques. Gross costing is a top-down technique that allocates a total budget to specific services.57 Micro costing precisely estimates various cost components in a bottom-up fashion.57 But the cost components taken into consideration varied. This makes it difficult to compare the estimations from different studies.
For the economic evaluation, various data sources can be used for incidence and cost estimations. What is important is to make an appropriate tradeoff between the data sources used and the scale of the estimation because this will determine the reliability of results and their representativeness on a larger scale. Stochastic modelling, either for estimating the overall cost of AEs or for extrapolating the results to a larger scale, proved to be an effective technique. Stochastic modelling is a tool for predicting outcomes when there is a certain degree of randomness in inputs (explanatory variables). The random variation is modelled based on previously observed fluctuations in inputs for a selected period using time-series techniques and data from primary or secondary sources. Distributions of potential outcomes are derived from a large number of simulations that reflect the random variation in the explanatory variables.
In the next section we describe 10 studies as good examples in building economic evaluations. These studies are selected based on six criteria described in the methods. All of the studies did not match all of the criteria applied (in fact only one of them met all of the criteria) but they all had some solid aspects which could be helpful for building further research. Table 2 summarises their main characteristics according to the criteria used.
Evaluation of the economic burden of AEs
In the past 10 years, a large number of national and international studies have looked at the possibility and pertinence of using routine hospital data for evaluating AEs. Patient safety indicators using the International Classification of Diseases codes and routine hospital databases are increasingly accepted as relevant sources for AE identification and cost evaluation.58
Eber et al provided a good example of using administrative hospital data. They identified the incidence of AEs and estimated cost and mortality due to healthcare-associated sepsis and pneumonia using a nationally representative sample of hospital administrative data.32 Cost estimations are based on tariff-to-cost conversion and the authors provide discounting (over 8-year collection span) and one-way sensitivity analysis. Therefore the results are easily reproducible and allow for good understanding of the margin for intervention.
The other two studies adopted a micro-costing approach in single hospitals.31 41 Cost collection is comprehensive and described in detail in both cases, but sensitivity analysis is lacking. Dietrich et al used a matched comparison for estimating the incremental cost of healthcare-associated pneumonia identified after standardised and validated clinical criteria.31 Prospective and retrospective series are included, which allows for different estimations: in the first case, reliability of AE identification was possibly better, but cost collection stopped at day 30 after admission. In the second case, accuracy of AE identification was not as precise, but costs were followed up for longer. All resources used for diagnosis, treatment, nursing and hospital stay, including drugs, materials and personnel, were taken into account, and interestingly, the authors estimated costs from the healthcare organisation (HCO) and the public insurer's perspective.
Sheng et al used a retrospective matched comparison analysis and prospective data for estimating the incremental cost of healthcare-associated infections in one university hospital.41 The authors provide a detailed description of relevant methods for the identification technique and the quality of micro-costing data collection. All daily expenses were included, distinguishing the different sites of nosocomial infections and complications, and the cost estimation is based on the calculation of the difference between the median hospital costs for cases and controls. While this micro-costing method is time and resource consuming, these studies are useful and necessary for describing costs in details and for understanding accurately major drivers of excess cost linked to AEs. However, the external validity of cost estimations based on small samples remains weak in the absence of sensitivity analysis.
Evaluations of the cost of patient safety practices
To estimate the financial implications of safety regulations in Japan, Fukuda et al identified a bundle of activities for enhancing patient safety (education, staff meetings, reporting, surveillance, audit, etc),26 in line with new safety practices introduced in the early 2000s in teaching hospitals, and measured the costs associated with those activities. The study also provided an estimation of the financial investment required for bringing all hospitals in the country to the mean observed level of patient safety practices. The incremental approach used by concentrating on new safety practices introduced after 1999 provides a valuable example of evaluating the cost of new safety regulations for HCOs. Unfortunately, a sensitivity analysis was lacking.
Stone et al compared the costs of hand hygiene in hospitals with high and low hand hygiene compliance and high and low frequency of alcohol hand rub use.45 They provided a detailed estimation based on a large sample of hospitals: they included all costs associated with developing and distributing information and educational material, cost of staff attending formal educational sessions and use of hand hygiene products. The tradeoff between the level of detail of cost analysis and this national survey was however questionable. The description of the hospital characteristics related to compliance was useful, but did not replace a sensitivity analysis of the validity of the estimations.
Complete economic evaluations
Five studies are described here, all of which used stochastic modelling techniques to some extent.
The study by Brown and Lilford is one of the rare studies evaluating the cost effectiveness of a national policy from the UK National Health System (NHS) perspective.48 It questions the cost-effectiveness of a deep cleaning programme in hospitals. This programme consisted of a manual scrubbing and decontamination process applied to all hospital surfaces and equipment to reduce the incidence and burden of hospital-acquired infections. Since only the programme costs and the opportunity costs of closing wards for deep cleaning were directly available, the authors used stochastic modelling techniques with secondary data from previously published studies and expert opinion for estimating the cost savings and the effectiveness of the patient safety practice, and for deriving utility and monetary values. The authors estimated the incremental cost-effectiveness ratio (ICER) in terms of preventing incidents, improving quality-adjusted life years (QALYs) and avoiding death. They concluded that it is likely that the national deep cleaning programme would not have been adopted if it had been previously evaluated, given its ICERs are largely over the NHS threshold for adopting new health technologies. This study provided a good example of how economic evaluation could support decision making by transparently establishing expected benefits from a safety practice against its predicted cost.
The study by Calugar et al is the only complete economic evaluation that meets all of the criteria we used for analysing the selected papers46; it does not refer to a strict patient safety activity, but rather to a comprehensive safety practice. It describes the costs of a nosocomial pertussis epidemic in a hospital and estimates the potential benefits of imposing vaccination to the hospital staff, both in terms of health and economic outcomes. Multiple data sources were used to obtain detailed cost data and to guarantee good quality of empirical information. Published data related mainly to the potential impact of massive vaccination in terms of reduction of pertussis incidence and the effectiveness of the practice was expressed in these terms. Their model suggested that significant cost savings and benefits could be achieved by vaccinating healthcare workers against pertussis.
Karnon et al studied the potential impact of three different patient safety practices for reducing medication-related AEs at a 400-bed HCO level by using simulation techniques.55 They used a decision tree model that allowed for estimating the potential benefit of each strategy based on a transparent hypothesis. The model was populated with data from multiple sources (usually published data) for estimation purposes. The authors carried out a cost–utility analysis and a cost–benefit analysis, and they compared the results if only service costs were included and when lost health was also accounted for. The expected results shifted from slightly negative or rather neutral valuations (between −£0.154 million and +£0.240 million) to a much greater net benefit (ranging from £13.142 million to £31.504 million depending on the intervention considered) when health benefits were accounted for.
Van Rijen and Kluytmans evaluated the implementation of a methicillin-resistant Staphylococcus aureus (MRSA) national ‘search and destroy’ policy at HCO level using 5-year span retrospective data.49 The authors provided a detailed description of variable costs, but only a gross estimation of fixed costs. The economic estimations were high quality, and despite the lack of sensitivity analysis, it provides a good example of the economic and epidemiological impact of specific safety regulations at HCO level.
Finally, the paper by Wang et al is an unusual study as it provides an evaluation of a strategy aimed at reducing AEs in a primary care setting from the perspective of a private insurer.28 It looked at the impact of the implementation of electronic medical records for controlling potential prescription errors in diagnostic and therapeutic procedures. The authors used stochastic models and performed one-way and five-way sensitivity analyses, on patient panel size, multiple system costs and induced costs because of loss of productivity. The benefit from introducing electronic medical records in the USA was estimated to be around US$85 000 per provider over 5 years (decreased billing errors, improved drug expenditures). However, the benefit was largely conditioned on the way doctors were reimbursed and could range from a net cost of US$2300 to more than US$300 000 net benefit. This paper is a good example of high-quality sensitivity analyses and their potential impact on the results of estimations.
Our review provides an overview of economic studies evaluating system-wide patient safety practices. Most studies presented only the economic burden of AEs with the majority focusing on healthcare-associated infections, and using small samples. Few evaluations based on nationally representative samples provided a full economic evaluation. Yet, economic burden studies (cost evaluations) provide useful answers and can be sufficient for some purposes.
Our review shows that full cost–benefit/utility evaluations are rarely completed as they are resource intensive and often require unavailable data. Cost estimations are often based on average incremental length of stay (LOS). While LOS is positively correlated with hospitalisation costs, this implies excluding events that are not prolonging LOS and those causing immediate death.
Stochastic models using different secondary data sources are common in complete evaluation studies, and could be helpful for assessing the impact of different safety practices.
In this context, explicitly stating the perspective adopted in cost analysis is important, as it should play a role in defining the cost components of the study.59 While the estimation of system-wide costs would not require detailed information on specific procedure costs, this would be useful from the HCO perspective. However, calculating costs always from the hospital perspective could also be restrictive. Clearly, managing AEs involves a wide variety of actors and the costs would be much higher if long-term morbidity and lost productivity were considered.
Sensitivity analysis is also essential for establishing the factors that impact cost–benefit ratios of an intervention because they often depend heavily on the background assumptions. For example, the benefit of introducing electronic medical records in the USA was estimated to be around US$85 000 per provider over 5 years (decreased billing errors, improved drug expenditures). However, the sensitivity analysis showed that the benefit was largely conditioned on the way doctors were reimbursed and could range from a net cost of US$2300 to more than US$300 000 net benefit.28 Nevertheless, most studies do not provide an appropriate sensitivity analysis.
More generally, low methodological transparency can be a problem because the methodology used for estimating costs is not always detailed. This is a problem for building evidence in economic evaluations as it reduces the capacity to reproduce the results and verify the quality criteria.
Economic evaluation is a useful tool for understanding the economic consequences of political or strategic choices and for wisely distributing available resources. Different types of evaluation are used for different purposes. Concerning patient safety practices, most evaluations appear to have a narrow focus whereby evaluators are only concerned with resource consequences of AEs in their healthcare organisation. Therefore, investing in the economic design and reporting of studies with more emphasis on defining study perspectives, data collection and methodological choices could be helpful for strengthening our knowledge base on practices for improving patient safety.
The authors wish to thank the Evaluation and Statistics Department of the French Ministry of Health for the funding of this project. They are also grateful for the comments from the editor and anonymous referees which allowed significant improvements to the paper.
Linked articles 000585.
Funding This research project was partly funded by DREES (Direction de la recherche, des études, de l'évaluation et des statistiques), the Evaluation and Statistics Department of the French Ministry of Health.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.