Background In this study, we aim to develop a set of quality indicators for optimal perioperative diabetes care throughout the hospital care pathway and to gain insight into the feasibility of the indicator set in daily clinical practice by assessing the clinimetric properties of the indicators in a practice test.
Methods A literature-based modified Delphi method was used to develop a set of quality indicators. To assess clinimetric properties of each indicator (measurability, applicability, reliability, improvement potential and case-mix stability), a practice test was performed in six Dutch hospitals using a sample of 389 major surgery patients with diabetes who underwent abdominal, cardiac or large joint orthopaedic surgery.
Results We developed a set of 36 quality indicators for perioperative diabetes care. The practice test showed that one indicator was inapplicable, and nine indicators were unmeasurable. Interobserver reliability was good (0.61≤k≤0.8) for all indicators except for one with moderate (0.41≤k≤0.6) interobserver reliability. Improvement potential was low (<10%) for five indicators. Twenty-one indicators, including three outcome indicators, nine process indicators and nine structure indicators, could be used to assess the quality of care delivered in our six study hospitals.
Conclusion We developed a face and content valid set of quality indicators for optimal perioperative diabetes care throughout the hospital care pathway, using a rigorous and systematic approach. The results from our practice test show that it is essential to subject indicators to a practice test before applying them for quality improvement purposes.
Trial registration number ClinicalTrials.gov: NCT01610674.
- Diabetes mellitus
- Hospital medicine
- Performance measures
- Quality measurement
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Estimates of diabetes prevalence range from 20% in general surgery patients to nearly 35% in cardiac surgery patients.1 ,2 Surgery for patients with diabetes is associated with longer hospital stay, more use of healthcare resources and greater perioperative morbidity and mortality than surgery for patients without diabetes.2–5 The greater perioperative morbidity and mortality relates, in part, to the higher incidence of coronary heart disease, hypertension, renal insufficiency and postoperative complications associated with diabetes.4 Further, the relationship between perioperative hyperglycaemia and surgical complications is well established for a large range of surgical procedures.6–12
Perioperative care is a complex and multidisciplinary process, involving multiple teams of caregivers and multiple transitions of care. To achieve optimal diabetes care in the perioperative period, all professionals involved should take the specific needs of the person with diabetes into account at all stages of the hospital care pathway that ranges from preoperative evaluation up to hospital discharge.13 Several large professional organisations, such as the American Diabetes Association and more recently the Joint British Diabetes Societies (JBDS), have developed recommendations for optimal diabetes care in the perioperative period, but it is unknown whether this has resulted in optimal or ‘good quality’ care delivery.13 ,14
The development and use of quality indicators have been suggested to help define and assess the quality of the care delivered.15 ,16 Quality indicators are measurable elements of practice performance, for which there is evidence or consensus that they can be used to assess the quality, and hence change in the quality, of care provided.17 Three types of quality indicators can be distinguished. Outcome indicators specify the ultimate goal of the care given, process indicators refer to the actual care that is delivered to patients and structure indicators refer to the organisational structure of a healthcare system.18
This study aims to develop a set of quality indicators for optimal perioperative diabetes care throughout the hospital care pathway and to gain insight into the feasibility of the indicator set in daily clinical practice by assessing the clinimetric properties (quality criteria) of the indicators in a practice test.
Development of quality indicators for perioperative diabetes care
A modified Delphi method was used to develop a set of quality indicators.19–21 Recommendations for optimal perioperative diabetes care were extracted from the literature (international guidelines, websites and PubMed database), translated into potential quality indicators and presented to a multidisciplinary expert panel.
Selection of recommendations
The search for recommendations for optimal perioperative diabetes care was based on the guidelines for Quality Indicator Development of the Dutch Institute for Healthcare Improvement.22 In January 2009, we searched 33 websites of international healthcare organisations, for example, http://www.qualitymeasures.ahrq.gov and http://www.who.int/en/, to identify quality indicators for perioperative diabetes care.22 We continued by searching websites of national and international guideline organisations, and found seven guidelines that contained recommendations on perioperative diabetes care.14 ,23–28 Three reviewers (IH, PJvG and MEJLH) extracted all recommendations for optimal perioperative diabetes care from the American Diabetes Association guidelines. As agreement was very high, one reviewer continued to extract recommendations from the guidelines (IH).
To complement the inventory of recommendations for optimal perioperative diabetes care, we used the PubMed database to review international literature for information about the development of quality indicators and recommendations for perioperative diabetes care. Online supplementary figure S1 depicts the search strategy, and online supplementary table S1 presents the exclusion criteria. Three reviewers checked the abstracts. Potentially relevant publications were checked in full text format.
All recommendations were categorised as ‘patient outcome’, ‘process’ or ‘organisational structure’.18 Recommendations for process were categorised according to the various stages of the hospital care pathway: ‘preoperative evaluation’, ‘hospital admission’ (care that should be initiated on hospital admission), ‘preoperative management’ (care that should be initiated in preparation for the surgery), ‘perioperative management’, ‘postoperative management’ or ‘hospital discharge’.
The rating procedure took place between May 2009 and November 2009. Our expert panel consisted of 35 Dutch experts who had experience with the care for patients with diabetes in the perioperative period (19 endocrinologists/internists with special interest in diabetes care, 6 anaesthesiologists, 5 intensive care specialists and 4 surgeons). The experts worked in 22 Dutch hospitals in the Netherlands. We selected the experts based on their experience with perioperative diabetes care (champions of diabetes care in the Netherlands), while aiming for sufficient diversity in hospital size (small-to-large size, academic and non-academic), the various regions in the Netherlands and disciplines of the panel members.
First questionnaire round
All recommendations for optimal perioperative diabetes care were translated into potential quality indicators by defining ‘who should do what to whom and when’, and were worked into a written questionnaire. The experts rated the quality indicators anonymously. We asked the experts to rate each quality indicator while considering the following criteria:
The recommended care leads to health gain for the patient (eg, fewer complications) or promotes efficiency of care.
The recommended care is generalisable to all patients with diabetes who undergo a major surgical procedure.
There is enough scientific evidence or expert consensus to justify the recommended care.
A Likert scale rating ranging from 1 (not at all relevant for assessing the quality of care delivered) to 9 (very relevant) was added, and the answer category ‘cannot assess’ was made available. Online supplementary figure S2 shows an example of a rating scale for a quality indicator. Experts could also add quality indicators. Quality indicators were selected according to relevancy (median scores 7, 8 or 9) if there was no disagreement (30% or more of ratings in both the 1–3 and 7–9 tertiles). Quality indicators rated ≤6 were discarded.
Expert panel meeting
The analysis of the questionnaires was presented during a panel meeting to discuss the quality indicators with median scores 7, 8 or 9 and disagreement, and to rephrase quality indicators that were not defined specific enough. All panellists who had participated in the first questionnaire round were invited.
Second questionnaire round
Following the meeting, all accepted and added quality indicators, rephrased if necessary, were presented for final remarks and approval to the panel members who had completed the first questionnaire. We added a ranking procedure to single out key recommendations for each stage of the hospital care pathway (‘preoperative evaluation’, ‘hospital admission’, ‘preoperative management’, ‘perioperative management’, ‘postoperative management’ or ‘hospital discharge’). For each stage that contained more than four indicators, we asked the experts to select the top three of most relevant indicators. We awarded an indicator three points for each number-one expert ranking, two points for each number-two ranking and one point for each number-three ranking. Indicators that received >20% of the maximum possible ranking points were included in the final indicator set. The maximum possible ranking points were 57 points (19 experts times 3 points). The experts ranked the quality indicators anonymously.
The application of a systematic and rigorous modified Delphi method for developing quality indicators results in high face and content validity for the indicator set. To gain insight into the feasibility of the final indicator set in daily clinical practice, we performed a practice test, that is, a test to assess the clinimetric properties or quality criteria of indicators in daily clinical practice. We assessed the clinimetric properties of the final indicator set in six Dutch hospitals. There were two academic hospitals (953–1042 beds), two regional teaching hospitals (543–576 beds) and two tertiary teaching hospitals (730–846 beds). All six hospitals performed orthopaedic and abdominal surgery, and three of the six hospitals performed cardiac surgery. In one hospital, the department of abdominal surgery declined participation. The hospitals participated in a study to test the effectiveness of an implementation strategy to improve perioperative diabetes care in daily clinical practice (trial no. NCT01610674). The Regional Review Board for Human Research Arnhem-Nijmegen (CMO, no. 2008/333), approved the study for all hospitals.
Adult patients with diabetes were included if they met our criteria of major surgery:29 (1) the person was under general anaesthesia during abdominal surgery or cardiac surgery, or under spinal or general anaesthesia during large-joint orthopaedic surgery and (2) the minimum operation time was 1 h. Patients who underwent outpatient surgery were excluded. If a participant underwent more than one surgical procedure, we included the first procedure. The surgery took place in the period between March 2009 and March 2010. We retrospectively identified the patients from data in the hospital prescription system, operating room management systems, preoperative screening records and anaesthesiology records. The patients were invited to participate in the study by mail and by phone.30 Informed consent was obtained from all patients. After informed consent had been obtained, information regarding the inclusion criteria was verified with the patient record. We aimed to include a minimum of 30 patients per surgical speciality (abdominal surgery, cardiac surgery and large-joint orthopaedic surgery) in each hospital, that is, 420 patients for a total of 14 departments. We invited 60 patients per surgical speciality if sufficient numbers were available.
The patient data relevant to process and outcome indicators were manually abstracted from the patient records and entered into a computerised database. The data were collected from medical and nursing records, anaesthesiology records, medication records and laboratory reports. Four trained research assistants (EE, MG, ER and AV) and one researcher (IH) searched the patient records for the period between August 2010 and May 2011.
We assessed the quality indicators for the recommended organisational structure of perioperative diabetes care with a questionnaire. A diabetes specialist in each of the six participating hospitals completed the questionnaire.
Clinimetric properties assessed
Outcome indicators and process indicators
Individual patient indicator scores (dichotomous variable with the values 0 and 1) were computed for each patient and for each outcome and process indicator, and the scores described whether or not care was delivered as recommended. We constructed an algorithm for each indicator to calculate the indicator outcomes. Online supplementary figure S3 provides an example of an indicator algorithm. We computed hospital indicator scores (in percentages) by dividing the number of patients who received the recommended care as described by the indicator (numerator) by the number of patients to whom the recommended care applied (denominator). We used these data to assess the following clinimetric properties:
A specific indicator was considered ‘unmeasurable’ if >25% of the patients’ individual indicator scores could not be computed because of missing data.
Indicators that applied to <10% of the patients were considered ‘inapplicable’.
The interobserver reliability of the data collection was expressed in kappa coefficients, that is, the percentage of agreement between two data reviewers at the level of the patient indicator score, corrected for chance. Two independent data reviewers collected a sample of 30 patient records from three hospitals. Values of 0.41≤k≤0.6 were considered ‘moderate’, 0.61≤k≤0.8 were considered ‘good’ and values >0.8 were considered ‘very good’. Values <0.4 were considered ‘poor’.
The improvement potential of a specific indicator was considered ‘low’ if the median potential for hospital improvement was <10% (hospital indicator score ≥90%) in >80% of the hospitals.
For indicators with good clinimetric properties, we explored whether the hospital indicator scores needed correction for case mix. We used multilevel analysis to study the relationship between the indicator results and the following patient characteristics: age, gender, insulin treatment and type of surgery. We corrected for data clustering at the hospital level. Significance for the multilevel analysis was set at p<0.05.
Individual hospital indicator scores (dichotomous variable with the values 0 and 1) were computed for each hospital and for each structure indicator, and the scores described whether or not care was organised as recommended. The number of hospitals that were included in the practice test (six) was too small to determine all the clinimetric properties of the indicators for organisational structure. To get an impression of the ‘improvement potential’ of these indicators, we computed the fraction of hospitals with an organisational structure of perioperative diabetes care as described by the indicator.
Quality indicators for perioperative diabetes care
Selection of recommendations from guidelines and literature
Figure 1 schematically presents the quality indicator development. No existing quality indicators for perioperative diabetes care were found on international websites. In total, 90 recommendations for perioperative diabetes care were derived from seven guidelines.14 ,23–28 The literature review yielded 1318 abstracts for 114 articles that contained recommendations for perioperative diabetes care. From these articles, we derived 22 recommendations that were complementary to the set of recommendations found in the guideline search. Further aggregation of the total of 112 recommendations by topic resulted in a total of 79 recommendations that were translated into potential quality indicators. The evidence level, as provided in the guidelines, was D (ie, a formal combination of expert views or other information, eg, Delphi study, expert opinion or informal consensus) for most quality indicators.35
First questionnaire round
Twenty-seven of 35 experts returned the first questionnaire (15 internists, 5 anaesthesiologists, 4 intensive care specialists and 3 surgeons; response rate 77%). The expert panel rejected 30 quality indicators, accepted 48 indicators as relevant for assessing the quality of perioperative diabetes care, disagreed on one indicator and suggested two new quality indicators.
Expert panel meeting
Six experts attended the panel meeting (three internists, one anaesthesiologist, one surgeon and one intensive care specialist). The panel discussed and rephrased 15 quality indicators, combined 2 quality indicators with similar contents and suggested 2 new quality indicators, resulting in a total of 52 quality indicators.
Second questionnaire round
Nineteen of 27 experts (10 internists, 4 anaesthesiologists, 3 intensive care specialists and 2 surgeons, response rate 70%) completed the second questionnaire. The panel approved all 52 quality indicators, including 4 outcome indicators, 39 process indicators and 9 structure indicators (see online supplementary table S2). The experts prioritised 15 out of the 39 process indicators. The 28 indicators in the prioritised set (4 outcome, 15 process and 9 structure) were systematically operationalised by defining numerators and denominators. Five indicators (see online supplementary table S2, indicator numbers 1, 2, 5, 15 and 27) were operationalised into two or more indicators. Tables 1⇓–3 present the final set of 36 indicators that were tested for feasibility in the study hospitals.
We identified 1100 patients with diabetes who underwent major surgery. We aimed to include 30 patients per surgical speciality in each hospital, but these numbers were not available for the departments of orthopaedic and abdominal surgery in all hospitals. We performed a practice test with the final set of 36 indicators on a sample of 389 patients who underwent abdominal surgery (32%), orthopaedic surgery (39%) or cardiac surgery (29%). Figure 2 shows the patient inclusion. Of these patients, 62% were men and 38% were women. More than half of the patients (57%) had received oral treatment with hypoglycaemic agents only before admission to hospital. Twenty-one per cent of the patients used a combination of oral hypoglycaemic agents and insulin, 18% used insulin only and 4% took dietary measures only.
Clinimetric properties assessed
Outcome indicators and process indicators
Tables 1 and 2 show the results of the practice test. Nine indicators (four outcome and five process) were ‘unmeasurable’ in our study hospitals because the relevant data for computing patient indicator scores were missing at the time of data extraction for >25% of the patients.
As hypoglycaemia rarely occurred, we considered the indicator ‘intravenous glucose ordered for fasting patients with hypoglycaemia’ ‘inapplicable’ in our study hospitals.
The indicator ‘blood glucose measurement every 4–6 h ordered for fasting patients’ had a kappa of 0.51, which indicates moderate interobserver reliability. All other indicators had kappa scores of >0.6.
The potential for improvement was low for four process indicators and one outcome indicator.
Case-mix stability was assessed for 12 indicators that had good clinimetric properties (no. 1, 4, 8, 9, 11, 15, 16, 18, 21–23, 26). All these indicators needed correction for case mix. The most important determinants that influenced several indicator scores were diabetes medication and the type of surgery (cardiac, orthopaedic or abdominal). One indicator ‘blood glucose measured every 2 h during surgery’ was influenced by sex.
Table 3 presents the fractions of hospitals with organisational structures as described by the indicators. Online supplementary table S3 presents the details for these results, which indicate that there is plenty of room for improving most structure indicators. For example, perioperative diabetes care protocols lacked clear descriptions of the tasks and responsibilities of the caregivers in five of the six hospitals. None of the multidisciplinary teams to coordinate perioperative diabetes care included all the professionals who should participate in such a team.
In this study, we developed a set of 36 quality indicators describing optimal perioperative diabetes care throughout the hospital care pathway, based on international literature and a systematic modified Delphi procedure. The application of this systematic and rigorous method for indicator development resulted in indicators with high face and content validity. The results from the practice test showed that 21 indicators, including 3 outcome indicators, 9 process indicators and 9 structure indicators, could be used to assess the quality of care delivered in our six study hospitals.
Our study has several strengths. First, to our knowledge, this is the first study to develop and appraise quality indicators for optimal perioperative diabetes care throughout the entire hospital care pathway using a rigorous and systematic approach. In the UK, the JBDS put audit standards in place as part of their guidelines for perioperative diabetes care.13 There is overlap between our quality indicator set and the audit standards in the JBDS document. However, the selection procedure for the audit standards and how they were appraised is unclear. We used a modified Delphi method to develop a set of quality indicators.18 This method has been used in other studies, and combines scientific evidence with the opinions of experts.15 ,16 ,32 ,36 ,37 Our procedure for quality indicator development follows the common steps of guideline-based quality indicator development that were recommended by Kötter et al.38 Boulkedid et al39 reviewed the use and reporting of the Delphi method for quality indicator selection. They noted considerable variability across studies in the characteristics of the Delphi method, and formulated practical guidelines for using this method for selecting healthcare quality indicators. Our procedure using the modified Delphi method is compatible with these guidelines.
Second, our indicator set includes indicators for patient outcome, process and organisational structure of perioperative diabetes care—all three components of the Donabedian quality of care framework.18 To make inferences about quality, there needs to be an established relationship between these three components. To get an indication of the criterion validity of the quality indicators in our set (ie, were the outcomes more favourable for patients who received care as described with the quality indicators?), we analysed the relationship between the process indicator scores and the length of hospital stay. Preliminary results suggest a trend towards shorter hospital stay for patients whose blood glucose was measured according to the recommended frequency.
A limitation of our study is that the indicator set was developed in a Dutch setting; so, the results do not automatically translate to other healthcare systems. As our set of quality indicators is based on international literature, we are convinced that the indicators are useful for other countries as well. For example, our set could be used as an extension to the requirements for inpatient diabetes care that were formulated by the Joint Commission in the USA, as part of their disease-specific programme on inpatient diabetes care.40 However, we strongly advise to test the clinimetric properties of the quality indicators to determine the feasibility of the indicator set in other settings. A practice test will also facilitate acceptance of the indicators. We provided an example of how to perform such a practice test. A second limitation is that nurses and people with diabetes were not represented in the procedure to develop indicators. We aimed to include experts that had both experience in the care of people with diabetes who have to undergo surgery and up-to-date knowledge of the international literature on perioperative diabetes care. The latter may apply less to nurses and people with diabetes. In a different substudy of a PhD study, we defined optimal perioperative diabetes care from the patients’ view.30
In this study, we developed a set of 36 face and content valid quality indicators. The results from our practice test show that it is essential to subject indicators to a practice test before applying them for quality improvement purposes in a specific setting; only 21 could be used to assess the quality of perioperative diabetes care in our six hospitals. Nine indicators were unmeasurable, which is quite a high number compared with other studies.32 ,36 This was primarily caused by poor availability of medical data and the lack of adequate data resources. Poor availability of medical data and the lack of adequate data resources are common problems in quality assessment efforts.41 As these problems can be solved by changes in administrative policy and data registration, this should not lead to a rejection of these indicators. For example, the introduction of electronic patient records may facilitate the multidisciplinary sharing of relevant patient information and the collection of data to fill the indicators. Similarly, hospital improvement potential was low for five indicators in our six hospitals. These indicators are less sensitive to detecting differences in the quality of care. These indicators are probably not suitable to be used as quality measures in Dutch hospitals. However, again, this should not lead to a rejection of these indicators, as a practice test in a different setting may yield different outcomes. It should be monitored regularly whether adherence continues to be high.
The indicator set consists mainly of indicators regarding the structures and processes of care, and only few regarding patient outcome. This reflects the ratios of these types of recommendations in guidelines. Insurers, policymakers and consumers may be more interested in outcome indicators; however, there are several disadvantages to the use of outcome indicators. Outcome indicators are sensitive to case mix and the methods of data collection.42 ,43 In addition, good outcomes do not necessarily indicate that care delivery is optimal. Indicators regarding the process and structure of care are more valuable in quality improvement initiatives because, unlike outcome measures, they offer a concrete starting point for improvement. The practice test in our six hospitals showed that there was ample room for improvement regarding many indicators. An improvement strategy was developed to address these problem areas.
In conclusion, we developed a face and content valid set of quality indicators that defines optimal patient outcome, professional performance and organisational structure of perioperative diabetes care throughout the hospital care pathway. We showed that it is essential to subject indicators to a practice test before applying them for quality improvement purposes in a specific setting. Future studies must prove the value of the indicator set as a tool to guide and monitor an improvement strategy for perioperative diabetes care.
The authors thank the members of the expert panel for rating the quality indicators, the patients who contributed to this research and the clinical care providers and staff at the participating hospitals. The authors thank Janine Liefers (Radboud University Medical Center, Radboud Institute for Health Sciences, Scientific Institute for Quality of Healthcare) for her contributions to the cleaning and analysis of the data.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Contributors IH designed the tools for collecting data, collected, cleaned and analysed the data and drafted and revised the manuscript. PJvG, HW, CJT and MEJLH contributed to the interpretation of the analysis and drafted and revised the manuscript. MEJLH designed the study, and is the guarantor for the study. All authors had full access to all of the data (including statistical reports and tables) in the study, and can take responsibility for the integrity of the data and the accuracy of the data analysis.
Funding The Netherlands Organization for Health Research and Development (ZonMw) funded this project (Grant No. 80-82315-98-09012).
Competing interests None declared.
Ethics approval Regional Review Board for Human Research (CMO), Arnhem-Nijmegen (CMO no. 2008/333).
Provenance and peer review Not commissioned; externally peer reviewed.