Article Text
Abstract
Aims To systematically identify and synthesise health service accreditation literature.
Methods A systematic identification and narrative synthesis of health service accreditation literature published prior to 2012 were conducted. The search identified 122 empirical studies that examined either the processes or impacts of accreditation programmes. Study components were recorded, including: dates of publication; research settings; levels of study evidence and quality using established rating frameworks; and key results. A content analysis was conducted to determine the frequency of key themes and subthemes examined in the literature and identify knowledge-gaps requiring research attention.
Results The majority of studies (n=67) were published since 2006, occurred in the USA (n=60) and focused on acute care (n=79). Two thematic categories, that is, ‘organisational impacts’ and ‘relationship to quality measures’, were addressed 60 or more times in the literature. ‘Financial impacts’, ‘consumer or patient satisfaction’ and ‘survey and surveyor issues’ were each examined fewer than 15 times. The literature is limited in terms of the level of evidence and quality of studies, but highlights potential relationships among accreditation programmes, high quality organisational processes and safe clinical care.
Conclusions Due to the limitations of the literature, it is not prudent to make strong claims about the effectiveness of health service accreditation. Nonetheless, several critical issues and knowledge-gaps were identified that may help stimulate and inform discussion among healthcare stakeholders. Ongoing effort is required to build upon the accreditation evidence-base by using high quality experimental study designs to examine the processes, effectiveness and financial value of accreditation programmes and their critical components in different healthcare domains.
- Accreditation
- Health services research
- Quality improvement
- Healthcare quality improvement
Statistics from Altmetric.com
Introduction
Health service accreditation has become ubiquitous in international healthcare as a putative driver of quality and safety.1 The purpose of accreditation programmes is to monitor and promote, via self and external assessment, healthcare organisation performance against predetermined optimal standards.2
Health service accreditation is receiving substantial scrutiny from governments, healthcare professionals and consumers due to the considerable, yet largely unquantified, resources invested in it by governments, health service organisations and accreditation agencies.3 ,4 Yet, the evidence supporting accreditation's capacity to promote high quality and safe organisational and clinical performance is contested.5 This dichotomy has contributed to calls for further accreditation research and syntheses of published evidence to strengthen the evidence-base.6 ,7
The aim of this paper is to examine accreditation's evidence-base by providing a comprehensive, systematic identification and narrative synthesis8 of all empirical research published prior to 2012, thereby examining the processes and effectiveness of health service accreditation programmes. In addition to reporting key features of the literature, critical knowledge-gaps are identified that have important implications for the healthcare industry, policy decision-makers and researchers regarding the effective development and evaluation of accreditation programmes.
We employed narrative rather than statistical methods for three reasons. First, a statistical meta-analysis was not possible due to the limited number of interventional or experimental studies, such as randomised controlled trials (RCTs).9 Second, there is considerable complexity and variety in the accreditation phenomena assessed (ie, the organisational and clinical impacts of different accreditation processes in different healthcare domains and jurisdictional and legislative environments). This heterogeneity means that quantitative comparison of outcomes between studies is problematic. Third, there is substantial diversity within the accreditation literature in terms of the research methods employed (a broad range of qualitative, quantitative and mixed-method studies) and theoretical positions adopted (eg, positivist or social constructionist). Our chosen review method is specifically designed to examine interventions that have been investigated in different ways.8
The paper extends previous reviews2 ,5 ,9–13 in three important ways. First, a larger corpus of research is examined due to the use of a wider publication date range and broader inclusion criteria, facilitating assessment of a range of study types. This paper adopts the position that these disparate studies reveal important aspects of accreditation processes that can be carefully synthesised to produce a more complete understanding of the role and effects of accreditation across the diversity of health service settings globally. Second, the level of evidence and quality of included studies is examined to identify potential deficiencies limiting the strength of the accreditation evidence-base. Third, a content analysis14 is conducted to determine the frequency of prominent themes and subthemes examined in the literature, and identify critical knowledge-gaps.
Methods
A similar multi-method search strategy was employed as in a previous review undertaken by members of the research team.5 First, an interrogation of three electronic bibliographic databases was undertaken between August 2011 and January 2012: Medline from 1950; EMBASE from 1980; and nursing and allied health literature through CINAHL from 1982. Based on the results of previous exploration and testing,5 and their role as MeSH terms in the PubMed research database, ‘accreditation’, ‘Joint Commission on Accreditation of Healthcare Organizations’ and ‘Joint Commission’ were searched as keywords. The keywords were searched separately within each database. Broader keywords, such as ‘quality’ and ‘quality assurance’ produced a large number and wide range of references that were irrelevant for this review. When combined, 58 053 references were identified. Trial searches were undertaken using other research databases (eg, Scopus), but these did not yield additional articles and were excluded from the search.
In addition, 46 grey literature studies were found on the websites of national and jurisdictional government health agencies and healthcare accreditation agencies worldwide, as well as the International Society for Quality in Health Care. In total, 58 099 publications met the initial search criteria (see figure 1).
Within the obtained results, a further filter was applied by searching for references associated with ‘research’ (eg, ‘accreditation AND research’) (see online supplementary table S1). Titles and abstracts were independently reviewed by the first two named authors according to the inclusion and exclusion criteria. Under inclusion criteria, references had to focus on one or more aspect of health service accreditation programmes or processes; that is, mere mention of the term ‘accreditation’ was not sufficient. Additional inclusion criteria were: empirical research (ie, use of one or more research methods); English language; and full text available. Exclusion criteria were: research concerning professional development or medical credentialing programmes; non-systematic literature reviews; and commentaries. These procedures yielded 122 publications.
Key features of references were recorded in a table, including dates of publication, and countries and healthcare domains in which studies were situated. The levels of evidence of included publications was assessed by the first named author using Australian National Health and Medical Research Council (NHMRC) guidelines (see online supplementary table S2).15 Separate classifications were used to assess the level of evidence of studies employing designs capable of producing evidence of causality (intervention group), as compared with studies incapable of producing such evidence (aetiology group). An additional level of evidence category, ‘expert opinion without explicit critical appraisal’ was included in the aetiology group to account for studies employing low level qualitative study designs, which could nonetheless provide valuable insights into important accreditation-related topics.
The quality of studies was determined using an adaptation of the assessment criteria of Cunningham et al16 and NHMRC,17 which were based on published checklists designed to facilitate critical appraisal of a range of study designs (see online supplementary table S3).18 ,19 As assessments of levels of evidence and quality accounted for potential risks of bias at study and outcome levels, these ratings can be used to critically interpret the findings of each study, which are summarised in the online supplementary appendix. To test the validity of these assessments, a random subsample of 25 references were independently examined by two other coauthors. Differences in determinations of levels of evidence and quality were only found for two studies, and these divergences were resolved after discussion among the research team.
As only two RCTs or pseudo-RCTs were identified, we assessed that a formal systematic review such as that undertaken by Flodgren et al 9 would not provide a significant contribution to the field. We pursued an aim that could provide a meaningful contribution and stimulate discussion among interested healthcare stakeholders. That is, we aimed to identify key thematic concerns highlighted in the literature and critical knowledge-gaps requiring attention.
To synthesise the body of literature, a content analysis was performed by the first two named authors to determine the frequency with which eight thematic categories were examined, which were identified in a previous literature review:5 organisational impacts; relationship to quality measures; change mechanisms; programme assessments; financial impacts; professionals’ attitudes to accreditation; consumer views or patient satisfaction; and survey and surveyor issues (see table 1). Several categories were often examined within a single publication. These themes represent the main topics of health service accreditation literature, and each includes a variety of finely-graded subthemes that were distilled by the two lead authors. Due to the range of issues examined within each category, and the varied designs and quality of included studies, we did not aim to reach conclusions regarding the strength of evidence concerning each theme. Instead, by identifying the frequency that key themes and subthemes were explored, the content analysis was used to provide an overall picture of the main topics of investigation within the accreditation literature, including critical knowledge-gaps.
To test the validity of this categorisation of publications, a kappa inter-rater reliability test20 was conducted using a random subsample of 25 references independently coded into thematic categories by the first and second named authors. The authors recorded whether each reference did, or did not, include analysis or discussion regarding each of the eight categories (ie, 25×8=200 separate author assessments that were compared). The kappa inter-rater reliability test result of 0.7 indicates a high level of agreement between the lead authors in their assignment of references to these themes.20
Results
Changes in number of publications over time
There was an increase in publications per biannual interval from 1998 onwards, and in particular, after 2000 and then again 2006. These findings show that over the past decade, accreditation studies are increasing with further recent calls for additional accreditation research.6 ,7
Research settings
The 122 health service accreditation research studies were conducted in 29 countries. Approximately half (n=60) the research was situated in the USA, with Australia generating the next most frequent number of studies (n=16). Six publications assessed accreditation programmes implemented across several European countries.21–26 There were 13 studies that examined programmes in low or middle income countries (LMICs),27–39 classified according to World Bank definitions.40 Hospitals represent the main setting of health service accreditation research (n=79). Other prominent settings include: general practices (n=11); laboratories (n=9); substance abuse clinics (n=7); and mental health facilities (n=7).
Key study features
The levels of evidence and quality ratings of included studies are reported in the online supplementary appendix and summarised in table 2. Intervention or experimental designs capable of producing evidence of causality were employed in 21 studies. A single level 1 study was identified which produced inconclusive results due to limited available evidence.9 Two studies27 ,36 used RCT or pseudo-RCT designs (levels 2 and 3a), and both were based in LMIC settings. In all, 48 studies in the aetiology group were classified as level 4 (ie, cross-sectional or case series studies), and 29 studies solely relied on expert opinion without critical appraisal. While lower level studies highlighted important accreditation issues and themes, their usefulness for evaluating accreditation effectiveness was limited.
Overall, 42 studies met all the assessment criteria relevant for their study design. Only five studies in the intervention group met all relevant criteria. A total of 43 studies met the majority of criteria, and those criteria that were not fulfilled were deemed unlikely to alter the study conclusions. Only some relevant criteria were fulfilled in the remaining studies. Explanations and examples of the main risks of bias within the health service accreditation literature are listed in table 3. Authors of the only RCT concluded that problematic sampling procedures and the types of indicators used for assessment may have biased the results.36
Thematic categories
The content analysis used eight thematic categories identified previously.5 Multiple thematic categories were often explored in individual studies. In all, 21 key subthemes within thematic categories were distilled. Studies that best exemplify key subthemes related to the five most common thematic categories are provided in table 4. To provide greater depth to the content analysis, key issues related to each thematic category are provided below, including indices that have been commonly used for evaluation studies, and examples of the different types of findings produced.
Relationship to quality measures
Quality measures incorporate items defined as indicators of organisational performance rates and patient or healthcare consumer outcomes. Overall, 65 studies examined the relationship between accreditation and different quality measures.6 ,21–23 ,25 ,26 ,29–31 ,36 ,37 ,39 ,42 ,44–46 51–80 ,83 ,84 ,88 ,90 ,95 ,96 ,101–103 ,105 ,106 ,109 ,118–125 Only 28 studies involved comparisons of accredited and non-accredited health services or health service units. Examples of positive findings concerning the relationship between accreditation and organisational performance levels include: a trend between accreditation outcomes and clinical indicator performance in hospitals;6 an association between chest pain centre accreditation and compliance with quality measures regarding acute myocardial infarction;70 and a relationship between accreditation and hospital performance on publicly reported evidence-based processes of care measures.51 Negative findings were also identified, including a study which found that accreditation of health plans was positively associated with some measures of Health Plan and Employer Data Information Set quality, but did not assure a minimal level of performance.118
In contrast with organisational process indicators, quality measures concerning patient outcomes were only examined in nine studies, highlighting a critical knowledge-gap. Examples of patient outcome measures used to examine accreditation impacts include survival rates22 and falls.61 Of the nine studies, six found positive associations between accreditation and patient outcome measures. For example, hospitals with accredited primary stroke centres had lower 30-day risk-standardised patient mortality compared with non-accredited hospitals.78 Other studies produced inconsistent results (ie, associations were found between accreditation and some outcomes but not others) or identified no associations. The varied findings produced using different quality measures for assessment highlight the need for critical examination of the types of healthcare quality information collected, and if it is appropriate to be correlated against accreditation outcomes.78 ,79
Organisational impacts
The impacts of accreditation on organisational processes, policies and environments were examined in 62 studies.3 ,6 ,21 ,23 ,25 ,26 ,28–33 36–39 41–44 ,46 ,49 ,51 ,53–57 59–61 63–66 ,71 ,73 ,75 ,81–99 ,106 ,111 ,112 ,122 ,126 ,127 As listed in table 4, several key subthemes were explored in these studies, including the extent to which accreditation programmes promote: standardisation of care processes; increased compliance with external programmes or guidelines (eg, clinical best-practice); development of organisational cultures conducive to quality and safety; implementation of continuous quality improvement activities; and superior leadership. A total of 21 studies examined organisational impacts by comparing accredited and non-accredited health services or health service units.
The only RCT showed no observed improvement on organisational quality indicators, for example, accessibility and completeness of medical records.36 Other examples of organisational indices used to examine accreditation impacts within the literature include: provision of family counselling;55 leadership and staff involvement;38 staff to patient ratios and waiting times;57 and information management and leadership.30
Accreditation programme assessment
A total of 42 studies examined the development and impacts of accreditation programmes.3 ,6 ,21 ,25 ,26 ,29–31 ,51 ,54 ,55 ,57 ,59–61 ,64 ,66 ,70 ,71 ,73 ,75 ,80–82 ,84 ,86 ,88 ,93 ,96 ,98 ,100–109 ,122 ,128 A combination of positive, negative and neutral impacts were identified (see table 4). Several notable concerns are identified in the literature, including the perceived low quality of some programme standards,29 and discrepancies between accreditation findings and the results of quality or practice audits.26 In addition, it was noted that while certain adverse events, such as infection rates, may be reduced by preventive protocols that are reflected in accreditation standards, other more complex events may require multifaceted strategies that are less easily translatable into standards.101
Seven studies explored the development of accreditation programmes, identifying a number of common barriers (eg, lack of stable funding source) and facilitators (eg, engagement of key stakeholders). Additionally, two studies compared aspects of different international accreditation programmes. In one, Canadian and French accreditation systems were found to be converging towards a model based on similar philosophies.96 In the other, patient survey data were seen to be increasingly integral in programmes worldwide.104 These findings may reflect the increasing global transfer of knowledge in the accreditation field, which may be leading to the homogenisation and standardisation of accreditation programmes.
Change mechanisms
Overall, 41 studies explored how the activity of preparing and undergoing accreditation promotes change in health service organisations.3 ,21–23 ,25 ,28 ,31 ,37–39 41–43 ,46 ,51 ,54 ,56 ,59 ,60 ,65 ,71 ,73 ,81 ,84 ,85 ,87 ,89–91 ,93 ,97 ,99 ,110–113 ,118 ,119 ,121 ,122 ,126 As shown in table 4, four main mechanisms responsible for organisational changes promoted by accreditation programmes were identified: engagement of staff in quality improvement activities, such as self-assessment; promotion of quality systems of care; documentation, collation and use of data for internal and external benchmarking; and implementation of best-practice guidelines. In one study, staff participation in an accreditation process was found to have promoted a quality and safety culture that crossed organisational and professional boundaries.97
Professionals’ attitudes towards accreditation
There were 38 studies that assessed health professionals’ attitudes towards accreditation using multiple methods.21 ,23 ,24 ,27 ,28 ,32–35 ,38 ,39 ,41 ,42 ,46 ,47 ,61 ,66 ,71 ,81 ,84 ,86 ,89 ,92 ,97 ,99 ,100 ,105 ,106 ,109 ,111 ,114–117 129–132 As illustrated in table 4, the literature highlights that health professionals view accreditation as an effective method of promoting high quality organisational processes and patient safety, and are more likely to remain satisfied and employed in accredited organisations. In a pseudo-RCT based on survey data, accredited Egyptian primary healthcare unit providers and their patients believed that accreditation had a positive effect on patient satisfaction and performance.27
Conversely, other studies found that health professionals have concerns regarding the human and financial resources required for organisations to participate successfully in accreditation programmes,99 and that the focus on meeting a large number of accreditation and other regulatory standards may deter more substantial organisational and system-level efforts to fundamentally improve critical problems.66 The two contrasting messages from these papers highlight the need for additional research to examine why some professionals perceive a disconnection between their efforts to address organisational quality and safety problems, and the perceived aims of accreditation programmes.
Financial impacts of accreditation
Fifteen studies examined or included some work on aspects of the financial impacts of accreditation.3 ,4 ,30 ,31 ,47 ,65 ,83 ,84 ,99 ,115 ,116 ,119 ,133–135 However, potential financial benefits were not specifically examined, highlighting a crucial issue requiring additional research. Participation in accreditation programmes was considered to require considerable financial resources, and the return on this investment is questioned.47 The overlap and duplication that can occur among accreditation, regulatory and contractual requirements is identified as a source of financial pressure.4 The costs required to administer accreditation programmes—particularly in LMICs—are described as a threat to their ongoing sustainability.30 ,31 A study that attempted to calculate accreditation costs identified that both health service organisations and accrediting bodies were concerned about the issue.3 While some studies provided detailed costs in relation to specific health services,133 ,135 only one publication attempted to assess accreditation costs from the perspective of a national health system.4 Additional research is required to examine a key issue within the literature: that is, why healthcare managers and practitioners separate accounting for resources they dedicate to their ongoing efforts to improve quality and safety from those resources they invest to demonstrate compliance to standards assessed by an external authority.
Consumer views or patient satisfaction
Despite the increasing role of patients or consumers within contemporary healthcare systems, only 13 studies considered the relationship between accreditation and consumer views or patient satisfaction.6 ,27 ,46 ,84 ,88 ,104 ,106 ,114 ,118 ,122 ,123 ,136 ,137 The literature indicates that accreditation has an undefined impact on the views or satisfaction of consumers or patients. For example, while a quasi-RCT found that accreditation had a positive effect on patient satisfaction and performance,27 accreditation was not linked to measurably better quality of care as perceived by patients and reflected by their recommendation rates of institutions.122 ,123 Similarly, accredited health plans had equivalent or lower performance on patient-reported measures of health plan quality and satisfaction,118 and no relationships were identified between hospital accreditation scores and patient satisfaction ratings.136 These findings suggest that accreditation may target or influence aspects of health service function and delivery that are less visible to patients or consumers.
Surveyors and surveying issues
Only 12 papers explored the topics of surveyor conduct and surveying.30 ,31 ,48–50 ,97 ,100 ,120 ,127 ,129 ,138 ,139 The research regarding surveyors includes: their common features worldwide concerning careers, training, work history and expectations;120 the identification of a typology of surveyor styles;48 and the benefits derived from their surveying activities, including exposure to new methods and innovations.49
Key findings regarding surveying include: visits by independent surveyors are valued by health service staff;100 the introduction of consumer surveyors has been successful, although their role and level of involvement needs clarification;50 and the short notice and patient journey survey methods can complement traditional advanced notification surveys.138 ,139 In addition, accreditation stakeholders report reliability in surveying to be promoted by: the accreditation programme; members’ relationship to the accrediting agency and survey team; accreditation agency personnel; and surveyor workforce management.129 While a key aspect of accreditation programmes, there is relatively little research focused on the topics of surveyor conduct and surveying. As part of examining the validity of accreditation programmes, these issues require further investigation.
Discussion
This paper reviewed the empirical literature investigating the processes and impacts of health service accreditation: a complex organisational intervention used in diverse contexts and researched in multiple ways. Our aim was to identify and condense meaningful research findings with practical implications. In all, 122 empirical studies were located that met the inclusion criteria. The research evidence generally presents health service accreditation as a useful tool to stimulate improvement in health service organisations and promote high quality organisational processes. However, as the quality of studies was moderate and few studies used designs capable of producing strong evidence, a statistical meta-analysis is not feasible and it is not possible to evaluate the effectiveness of accreditation to the level sought in clinical trials. Despite these limitations, our comprehensive synthesis provides an up to date overview of the main themes and subthemes examined in the literature and highlights critical knowledge-gaps that have implications for healthcare stakeholders regarding the effective development and evaluation of accreditation programmes. The use of a broader date range, more detailed synthesis, critical assessment of the quality of included studies and more rigorous inclusion criteria helped extend the findings of a previous review.5
While it appears commentator calls for additional accreditation research are being answered to some extent, the methodological quality and total amount of accreditation research remains modest relative to global investment. In particular, the limited use of clinical outcome measures for evaluation restricts understanding of whether accreditation is associated with improved patient and consumer health outcomes. Therefore, in addition to organisational process and patient and consumer satisfaction measures, greater use of objective clinical outcome indicators would strengthen the evidence-base and provide a greater understanding of the benefits accrued through accreditation. Nonetheless, the possibility of establishing causal links between accreditation and the patient and consumer outcomes emerging from complex, adaptive health service organisations requires careful consideration. The available evidence does not justify a rejection of the validity of accreditation programmes. Absence of evidence is not evidence of absence. While quantitative, outcome-based data can provide a useful summative assessment of the value of health service accreditation processes, exploratory qualitative data can help highlight problematic consequences of accreditation that are difficult to measure objectively, or are infrequently considered in evaluation studies. In this way, qualitative studies can contribute to theoretical developments in this field by uncovering factors which drive, or fail to drive, change in quantitative indicators of performance.
In the context of a postglobal financial crisis with increasing fiscal pressures, notions of value are inextricably linked to considerations of the appropriate role of health service accreditation as a quality and safety strategy. Do the benefits gained represent value for the investment? Without rigorous costing of the resources invested and financial benefits accrued through quality and safety strategies for health service organisations, such as accreditation, evaluations of effectiveness provide limited utility for the purposes of rational health policy decision-making. It is also worth considering whether these costs and economic benefits should be directly attributed to accreditation or should instead be seen as a normal part of providing high quality and safe healthcare.
The consequences of new standards56 or surveying methods138 have been examined only in a limited manner. These are two fundamental components of accreditation programmes worldwide, and the limited research evidence regarding these topics represents a serious deficiency of the literature. The potential role of consumers within accreditation processes is an additional critical issue requiring greater examination. Furthermore, there is a paucity of evidence regarding the relative impact of other accreditation components, such as different forms of organisational self-assessment. In short, the limited evaluation of accreditation elements is likely to impede the evidence-based development of more efficient and effective programmes.
This review was limited by several factors, including that due to the frequently contrasting foci, epistemologies, research contexts, study designs and methods present within the literature, we could not attempt a statistical meta-analysis. Nonetheless, the comprehensive synthesis presented in this review provides an indication of the research evidence concerning accreditation, classified into the most prominent themes and subthemes, and highlights knowledge-gaps. While the exclusion of non-English publications represents another study limitation, our discussions with international colleagues did not identify significant new findings from any non-English publications, which suggests this limitation was unlikely to have biased the results.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Footnotes
Acknowledgements We would like to acknowledge the comments made by the Editor in Chief, BMJ Quality and Safety. We would also like to thank the reviewers of this manuscript for their comments which have been used to assist in the revision of this paper.
Data sharing statement Data extraction and critical appraisal of included studies are available on request from the corresponding author.
Contributors All authors contributed to the study design, data analysis and the presentation of results.
Funding The research was supported under the Australian Research Council's Linkage Projects funding scheme (project number LP100200586) and a National Health and Medical Research Council Programme Grant (project number 568612).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.