Article Text
Abstract
Background: Evidence-based clinical guidelines aim to improve the quality of care. In The Netherlands, considerable time and effort have been invested in the development and implementation of evidence-based guidelines since the 1990s. Thus far, no reviews are available on their effectiveness. The primary aim of this article was to assess the evidence for the effectiveness of Dutch evidence-based clinical guidelines in improving the quality of care.
Methods: A systematic review of studies evaluating the effects of Dutch evidence-based guidelines on both the process and structure of care and patient outcomes was conducted. The electronic databases Medline and Embase (1990–2007) and relevant scientific journals were searched. Studies were only selected if they included a controlled trial, an interrupted time series design or a before and after design.
Results: A total of 20 studies were included. In 17 of 19 studies that measured the effects on the process or structure of care, significant improvements were reported. Thirteen of these studies reported improvement with respect to some of the recommendations studied. In addition, the size of the observed effects varied largely across the recommendations within guidelines. Six of nine studies that measured patient health outcomes showed significant but small improvements as a result of the use of clinical guidelines.
Conclusions: This review demonstrates that Dutch evidence-based clinical guidelines can be effective in improving the process and structure of care. The effects of guidelines on patient health outcomes were studied far less and data are less convincing. The high level of variation in effects across recommendations suggests that implementation strategies tailored to individual recommendations within the guideline are needed to establish relevant improvements in healthcare. Moreover, the results highlight the need for well-designed studies focusing on the level of the recommendations to determine which factors influence guideline utilisation and improved patient outcomes.
Statistics from Altmetric.com
Increasingly, clinical practice guidelines (CPGs) are being developed in all areas of medicine as a means to improve the quality of care. By translating the best available scientific evidence into specific recommendations, guidelines can serve as useful tools to achieve effective and efficient patient care.1 Whereas guidelines initially were based on consensus among experts, guideline development has been gradually formalised and evidence-based guidelines—linking the individual recommendations with their supporting evidence—are becoming standard practice.2 Developing evidence-based guidelines, however, does not guarantee improved quality of care. Effective implementation should ensure guideline adherence in practice and subsequently lead to improved patient outcomes.
Studies measuring the effects of guidelines on quality of care have predominantly focused on effects on clinical practice. Several international reviews showed that the majority of guideline studies resulted in significant improvements with respect to the process of care.345 Fewer studies have focused on the effects of guidelines on patient health outcomes. One review indicated improved patient health outcomes,3 whereas a second review, focusing on primary care, did not find a positive effect.6 However, most of the studies that were included in these reviews used older guidelines that were not developed according to the current standards of evidence-based medicine.7
Guideline utilisation is complex and many factors may influence the impact of guidelines on care. Factors linked to the guideline itself are the strength of the evidence,89 the method of development and transparency of the guideline10 and the perceived adoptability, complexity and triability of the recommendations.1112 Apart from guideline factors, the nature of the implementation strategy can contribute to guideline utilisation. Passive strategies, such as educational material and meetings, generally have a small effect and multifaceted strategies are not necessarily better than single interventions.113 Finally, contextual, organisational and cultural factors may impede or limit guideline adoption regardless of how thoroughly they are implemented.1415
The Netherlands has been a forerunner in evidence-based guideline development and guideline implementation research, compared with other European countries.16 Since 1982, more than 200 guidelines have been developed by the Dutch Institute for Healthcare Improvement (CBO) and the Dutch College of General Practitioners (NHG), the two most prominent guideline organisations in The Netherlands. Historically, the CBO focused on secondary care and the NHG on primary care, since there is a clear-cut distinction between primary and secondary care in the Dutch healthcare system. In the last decade, other organisations have also become active in guideline development. Partly because of the role of the Centre for Quality of Care Research (since June 2008 Scientific Institute for Quality of Healthcare), many implementation studies have been conducted to measure the effectiveness of the Dutch guidelines.16
In spite of a considerable investment in the area of evidence-based clinical guidelines in The Netherlands, thus far, it is unclear to what extent these activities have been successful in improving compliance with guidelines and patient health outcomes. By examining the impact of evidence-based guidelines in a country the size of The Netherlands, which features well-defined organisations responsible for guideline development, unique observations can be made. The primary aim of this study is, therefore, to provide an overview of the effectiveness of Dutch evidence-based guidelines in improving the quality of care. In addition, we want to explore which factors are associated with guideline utilisation and improved patient outcomes.
Methods
Concepts and definitions
In this review, CPGs were defined as “systematically developed statements to assist practitioner decisions about appropriate healthcare for specific clinical circumstances.”17 Guidelines that use the results of systematic literature reviews in formulating the recommendations and that link the individual recommendations with their supporting evidence were regarded as evidence-based CPGs. A recommendation was defined as “any statement that promotes or advocates a particular course of action in clinical care.”18
Implementation was defined as “a planned process and systematic introduction of innovations or changes of proven value; the aim being that these are given a structural place in professional practice, in the functioning of organisations or in the health care structure.”19 Dissemination, on the other hand, is regarded as more passive than implementation and involves strategies such as distributing guidelines or publication of guidelines in scientific journals.
To evaluate effects on quality of care, we used Donabedian’s model, which distinguished the structure, processes and outcomes of care.20 Structure of care refers to “human, physical and financial resources that are needed to provide medical care” (eg, the presence of spirometry in general practice).21 Process of care refers to “the set of activities that go on within and between practitioners and patient” (eg, prescription of medication),21 whereas “the change in a patient’s current and future health status that can be attributed to antecedent health care” (eg, blood pressure) is defined as outcome of care.21
Search strategy
A systematic literature search was conducted in Medline, Embase and relevant Dutch scientific journals. Searches were performed in Medline and Embase of literature published from 1990 to May 2007 using several combinations of key-words (Appendix A). We did not include studies published before 1990, as evidence-based guideline development in The Netherlands started in the early 1990s. To identify Dutch-language publications we performed a sensitive search in Medline (1990–2007) with the free text word “guideline*”, limited to Dutch language. In addition, two relevant Dutch scientific journals, Huisarts & Wetenschap and Nederlands Tijdschrift voor de Geneeskunde, were searched for additional studies.
Two reviewers (ML and JB) independently screened the titles and abstracts of the articles and selected 163 potentially relevant articles. Discrepancies were resolved by discussion and consensus. These articles were further selected according to the following inclusion criteria (fig 1).
The study concerned (a) clinical Dutch guideline(s). Drug formularies, patient guidelines and European guidelines were excluded.
The study addressed the adherence to recommendations related to the process and structure of care and/or the effects of guidelines on patient health outcomes.
The study concerned (an) evidence-based guideline(s). Local or regional protocols and guidelines that were derived from evidence-based national guidelines were also included.
The study included a controlled trial (randomised controlled trial, controlled clinical trial), an interrupted time series or a before and after design. Studies that evaluated the effectiveness of different guideline dissemination and implementation strategies were included as well as studies that measured the effect of a guideline against a non-intervention control group.
In addition, we consulted reference lists from all articles that were retrieved for more detailed information.
Data extraction
Three categories of studies were distinguished based on the target users of the guideline(s): (1) general practitioners; (2) medical specialists and (3) other healthcare providers, such as physiotherapists and midwives. The following data were collected from each study: type of guideline (national or local/regional); clinical area; setting; study design; number of included patients and physicians; type of intervention; process and structure measures; patient outcome measures; effects on process and structure of care; and effects on patient health outcomes.
Methodological quality
The methodological quality of the eligible studies was assessed by one reviewer (ML) and checked by a second reviewer (JB) using the quality criteria of the Cochrane Effective Practice and Organisation of Care Group (EPOC).23 The EPOC quality criteria checklist includes seven criteria for randomised controlled trials, seven criteria for controlled before and after studies and seven criteria for interrupted time series. Although the EPOC criteria were not developed to assess the methodological quality of uncontrolled before and after studies, we used them for these studies as well, since there are no high-quality checklists available to measure the quality of these type of studies. The quality criteria, such as concealment of allocation, follow-up of professionals and follow-up of patients or episodes of care, were scored as “done”, ”not done”, ”not clear” and in some cases as “not applicable.”
Data synthesis
Due to the heterogeneity of the studies, pooling of the results and calculating an overall estimate of the effects were not possible. Instead, we summarised the effectiveness in three categories: mostly effective (if the study demonstrated a significant effect on more than half of the outcome measures), partly effective (if the study reported a significant effect on half or less than half of the outcome measures) and not effective (if no significant effect was demonstrated). If a study evaluated the effectiveness of different guideline dissemination and implementation strategies, effectiveness was determined by the observed significant improvement in either of the study groups, rather than by a significant improvement in the intervention group compared with the control group.
Results
Description of the studies
Setting, study design and type of intervention
Thirty articles referring to 20 different studies fulfilled the inclusion criteria and were included in the selection. Most of the included studies targeted general practitioners (table 1).242526272829303132333435363738394041424344 Three studies were conducted among medical specialists454647 and four studies targeted other healthcare providers.484950515253
The majority of the studies concerned guidelines that had been developed at the national level.242526272829313234353637383940414243444547484950 Most studies focused on diagnosis and treatment,24313233383940434445464748495053 while five studies concerned preventive care25262728293042 and three a combination of both.34353637415152 The most common studied medical conditions were cardiovascular diseases, influenza, diabetes mellitus type II, low-back pain, asthma and COPD.
Most of the studies were cluster randomised controlled trials.2433343536374041434446495053 The studies used predominantly multifaceted intervention strategies to implement the guideline(s).242526272829313233343536373839404142434446474950515253 The intervention strategies most often used as part of multifaceted interventions were educational meetings, distribution of educational material, and audit and feedback.
Methodological quality
Overall, the quality of the included studies was moderate. A priori calculations of sample size were reported in seven studies.24343536374344474950515253 In the majority of the studies that allocated study groups, practices or groups of healthcare providers were the unit of allocation,25263031323334353637404142464950 thereby protecting participants against contamination. In four studies, allocation was by individual provider.243839434453
Five studies reported that more than 20% of providers dropped out.24334243444950 In four studies, data from less than 80% of patients were reported for at least one of the outcome measures or points in time.313233343536373839 Two studies reported an agreement in (some of the) outcomes between raters of less than 90% (or κ<0.8).464950 In most of the studies, however, reliability of outcome measures was not reported at all.242526272829303132343536374041424344454748515253
Effects on quality of care
Four studies evaluated the effectiveness of different dissemination or implementation strategies,304143444950 while 16 studies evaluated the effectiveness of a single implementation strategy (table 2).
Effects on process and structure of care
Of the 19 studies that examined effects of the guideline(s) on the process or structure of care, 17 showed significant improvements (table 2). The majority of these studies reported improvements with respect to some of the recommendations studied.242526282930313234353637404243444546515253 Only four studies showed an effect on all outcomes measuring adherence to the guideline.273839474950 In six studies, improvements were observed in half or less than half of the assessed process or structure of care measures.2431324043444653 Two studies failed to demonstrate any effect on the process or structure of care.3341
Overall, the size of the effects varied largely across the recommendations within a guideline. Significant improvements in adherence to recommendations ranged from 7.2% to 88% in the 17 studies and varied by 76% across recommendations within one study. In studies including a control group, odds ratios ranged from 0.2 (95% CI 0.1 to 0.6) to 27.13 (95% CI 12.86 to 57.24).
Effects on patient health outcomes
Nine studies assessed the effects of guidelines in terms of patient health outcomes. Six of these studies reported significant improvements in at least some of the outcomes studied (table 2).31323435363743444849505152 Wolters et al4344 and De Laat et al5152 reported improvements in all outcome measures, while four studies313234353637484950 demonstrated modest improvements in some of the assessed patient outcome measures. In three studies,38394047 no effect on patient health outcomes was observed.
Characteristics of the studies and effects on quality of care
All studies focusing on preventive care were mostly effective in terms of the process or structure of care (table 3).25262728293042 With respect to type of design, all uncontrolled before and after studies that measured effects on process or structure of care272829455152 or on patient health outcomes485152 were categorised as mostly effective. In contrast, both studies that failed to demonstrate an effect on clinical practice were cluster randomised controlled trials.3341
There were no differences in effects on quality of care with respect to type of healthcare provider, type of guideline or between studies that used multifaceted intervention strategies and studies that used a single intervention.
Discussion
Overall, the results of this study demonstrate that there is evidence for the effectiveness of Dutch evidence-based guidelines on the process and structure of care in The Netherlands. The majority of the studies reported improvement with respect to some of the recommendations studied. In addition, the size of the effects varied largely across recommendations within the guidelines. The effects of guidelines on patient health outcomes were studied far less and data are less convincing. Two-thirds of the studies that measured patient outcomes reported significant improvements. However, the observed changes in patient outcomes were generally modest and only found for some of the outcomes studied.
Findings from our review in terms of the process of care are comparable with those of previous international reviews which demonstrated small to moderate improvements.345 With respect to patient outcomes, results from earlier reviews were inconsistent.36 However, measuring patient outcomes is complex due to many factors such as long delays and confounding of many outcomes.5455 Our review provides some evidence for the effects of guidelines on patient outcomes. It also suggests that guidelines focusing on preventive care are particularly effective in improving the process or structure of care. Nevertheless, the number of studies in our review does not allow us to draw firm conclusions on the effects of guidelines on patient outcomes or on factors that contribute to improved quality of care.
An important finding of our study is that the observed effects varied largely across recommendations. The variation could be explained by barriers related to individual recommendations rather than barriers that apply to the guideline as a whole. For example, although we selected only evidence-based guidelines in our study, the strength of the evidence may vary across recommendations, thereby influencing their impact.89 Also, a recommendation may not be performed because of other factors such as healthcare professional issues (eg, lack of motivation) or environmental factors (eg, lack of resources). Future research should focus on barriers related to both the guideline and its specific recommendations when exploring the association with effects on quality of care.
Furthermore, the fact that the effects of guidelines varied largely across recommendations might suggest that guideline implementation should focus more on individual recommendations rather than the guideline as a whole. Whereas the majority of studies included in our review used multifaceted strategies to implement the guideline, these were generally not tailored to individual recommendations. The nature of the implementation strategy is often the same for all recommendations within a guideline. A more focused approach, based on the results of an analysis of barriers of adhering to individual recommendations could improve the use and effectiveness of guidelines in practice.
One of the strengths of the present study is that it focuses on the effectiveness of evidence-based guidelines on quality of care. Previous reviews also considered guidelines that were not developed according to the standards of evidence-based medicine. In addition, in our study the effects of guidelines within one healthcare system were analysed. The guidelines in the studies included in this review were produced by well-known and credible organisations in The Netherlands and these guidelines generally have acceptable quality scores and are adequate tools for healthcare improvement.5657
As stated before, The Netherlands has been a forerunner in evidence-based guideline development and implementation research in Europe. Based on the current literature, we have no reason to believe that the effectiveness of evidence-based guidelines, in terms of quality of care, is different in other countries or that more robust designs are generally used to assess these effects. Contextual and country-specific factors may, however, influence the effectiveness of evidence-based guidelines in terms of quality of care. In some countries, such as Scotland and New Zealand, guidelines are produced in a similar context as in The Netherlands, making the conclusions applicable to these countries as well.2
Several limitations to the present study can be mentioned. First, despite the long tradition of evidence-based guideline development in The Netherlands, the number of studies measuring the effects of guidelines with a robust design was limited. Therefore, we decided to include uncontrolled before and after studies as well, while taking into account the weaknesses of these designs. However, our results demonstrate that these studies tended to have more positive results than studies using more robust designs. Because of the relatively small number of uncontrolled before and after studies in our review, our conclusions regarding the effectiveness of guidelines were not much affected by it. Nevertheless, this finding highlights the need for well-designed studies measuring the effectiveness of Dutch evidence-based guidelines.
Second, the studies included in our review were very heterogeneous, not allowing pooling of the results. On the other hand, heterogeneity of studies may increase the generalisability of findings as a wider range of different settings, study populations and behaviours are included.58 Third, to determine effectiveness of guidelines on quality of care, we counted the number of measures in each study that showed a significant result. In this assessment, we did not take into account the effect sizes of the individual measures (equal weights are given to improvements of 1 or 70%).58 As an alternative, we categorised the effectiveness of an intervention in mostly, partly and not effective, which may provide more insight than a dichotomy.
In conclusion, there is a huge misbalance between the number of guidelines developed and the number of high-quality studies that assess their effectiveness. Despite this, our review demonstrates that Dutch evidence-based guidelines can be effective in improving the process and structure of care. Evidence on the effectiveness of guidelines on patient outcomes is less convincing. The variation in effects across recommendations suggests that it is useful to focus on recommendations when analysing barriers to guideline adherence and to design implementation strategies tailored to individual recommendations instead of to the guideline as a whole. Further research is needed to determine which factors linked to the guideline and its specific recommendations are important in predicting guideline utilisation and improved patient outcomes.
Acknowledgments
The authors wish to thank Rikie Deurenberg, information specialist at the Dutch Institute for Healthcare Improvement (CBO), for developing and performing the search strategy.
Appendix A: Search strategy
Description: A detailed description of the search strategy used in this review is available online.
REFERENCES
Footnotes
▸ An additional appendix is published online only at http://qshc.bmj.com/content/vol18/issue5
Competing interests None.