Background No standardised set of quality measures associated with transitioning complex-care patients across the various healthcare settings and home exists. In this context, a structured panel process was used to define quality measures for care transitions involving complex-care patients across healthcare settings.
Methods A modified Delphi consensus technique based on the RAND method was used to develop measures of quality care transitions across the continuum of care. Specific stages included a literature review, individual rating of each measure by each of the panelists (n=11), a face-to-face consensus meeting, and final ranking by the panelists.
Results The literature review produced an initial set of 119 measures. To advance to rounds 1 and 2, an aggregate rating of >75% of the measure was required. This analysis yielded 30/119 measures in round 1 and 11/30 measures in round 2. The final round of scoring yielded the following top five measures: (1) readmission rates within 30 days, (2) primary care visit within 7 days postdischarge for high-risk patients, (3) medication reconciliation completed at admission and prior to discharge, (4) readmission rates within 72 h and (5) time from discharge to homecare nursing visit for high-risk patients.
Conclusions The five measures identified through this research may be useful as indicators of overall care quality related to care transitions involving complex-care patients across different healthcare settings. Further research efforts are called for to explore the applicability and feasibility of using the quality measures to drive quality improvement across the healthcare system.
- Adverse events, epidemiology and detection
- Transitions in care
- Quality measurement
- Performance measures
- Health services research
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
- Adverse events, epidemiology and detection
- Transitions in care
- Quality measurement
- Performance measures
- Health services research
Patients with complex needs associated with their multiple diseases, comorbidities and conditions, frequently require care in multiple settings, and are particularly vulnerable to poorly executed transitions in care.1 ,2 Complex-care patients are managing multiple chronic diseases, taking several medications, may have limited social supports or suffer from dementia, mental illness or addiction. This complex-care patient cohort is predicted to grow substantially as the population ages and chronic diseases continue to be a major expenditure of healthcare costs.3 Poor transitions often result in unnecessary adverse patient outcomes and additional healthcare spending.4
Care transitions refer to a set of actions designed to ensure the coordination and continuation of appropriate healthcare provision as a patient transfers between different levels of care in the same clinical setting or different locations.5 Current literature examining transitions in care has mainly focused on transfers within acute-care organisations and to the home setting.6–8 However, complex patients, at any age, may experience transitions across the continuum of health services from acute care to multiple healthcare settings, such as complex continuing care rehabilitation centres, long-term care, home, or possibly from acute care teaching hospitals to community-based hospitals.
The importance of gaining insight into transfers of complex-care patients across the lifespan and between settings is a critical and timely policy issue. At present, there is no standardised set of quality measures associated with transitioning complex-care patients across the various healthcare settings and discharged and/or readmitted to/from home.9 This is an important gap to address, as quality indicators are increasingly being employed to assess and improve quality of care in many healthcare settings.10 Therefore, the purpose of this study was to define quality measures for care transitions involving complex-care patients across the continuum of care through a structured panel process. Our focus was to develop indicators that would be useful for accountability purposes as well as ones that had the potential for broader quality improvement (QI) efforts.
Overview of modified Delphi panel process
A modified Delphi consensus technique based on the RAND method was used to develop measures of quality care transitions across the continuum of care.11 The panel process was built on a technique that was shown to be effective for developing quality indicator measures in healthcare.12–18 For this study, multiple stages of the modified Delphi technique (literature review, individual rating, face-to-face consensus meeting, and final ranking) were used (see figure 1). Key to this methodology is engaging in multiple rounds of feedback from the expert panel using both anonymous surveys and round table meeting techniques to allow for optimal, unbiased expression of opinions.
Initial work: literature review and selection of initial set of measures
The first step in the Delphi technique is to engage in a structured review of the literature. The intent of this review was to produce a list of measures that had been reported in the literature that would then be presented to a panel of experts to review and rate. This literature search was undertaken in September 2011 using OVID Medline with the following key words with the number of abstracts reported: Quality Assurance, Health Care or Quality of Health Care; Health Transition (4 217 740); Patient Transfer, Patient Discharge, Continuity of Patient Care, Health Services for the Aged, Delivery of Health Care, (714 391) Home Care Services, Chronic Disease, Aged, Long Term Care (2 280 704); Primary Health Care, Delivery of Health Care, Patient Transfer, Continuity of Patient Care (723 017); Health Transition (650); Outcome and Process Assessment (Health Care) and Outcome Assessment (Health Care) (571 586). These search terms were used as they were identified by the research team and key stakeholders as key terms for care transitions.
Other databases beyond OVID Medline were not included as the initial search yielded a significant number of abstracts. The search was limited to those written in the English language (1675) and from 2000 to 2011, as any previous studies would have been included in literature syntheses during this time frame. Other limits included articles that were clinical conferences, lectures and legal cases, newspaper articles, commentaries and bibliographies/biographies (1658). The final limit was to include meta-analysis and reviews only (76). These 76 abstracts were reviewed by three members of the research team to determine which ones reported on outcome measures associated with care transitions. This review yielded 23 articles including two meta-analyses 19 ,20 and 21 literature reviews,21–41 This review was augmented with other literature known to the research team, or were references included in the reference lists of the literature review of primary studies.5 ,41–51
Additionally, a search of targeted journals and grey literature was conducted in early January 2012 that yielded 14 other documents that included evidence for transitions of care outcome measures.9 ,52–64 This second search was undertaken for several reasons. First, some of the documents reviewed referred to other references that were not picked up in the initial literature search.9 ,51–58 61–64 Second, shortly after the literature search was conducted, a report was released on avoidable hospitalisations.52 Third, the research team became aware of additional interventions studies published in early 2012.59 ,60 Of the literature that was identified, all the studies were included in the next phase of the project measure development. Our research team developed an initial survey of potential measures from all measures of quality of transitions of care for complex-care patients listed in the final set of documents reviewed. The list of measures (n=119) were categorised under the following headings: health service usage; patient outcome; patient satisfaction and experience; provider satisfaction. Once the initial set of measures was determined, the research team identified potential panel experts to be engaged in this project.
Panel member selection. The research team developed criteria for panel member selection in partnership with key stakeholders (Canadian Patient Safety Institute, Accreditation Canada and Patients for Patient Safety). Criteria for selecting panel members included having an interest in care transitions, patient safety, QI, patient care experience and related research initiatives. Additionally, key stakeholders (listed above) were asked to provide names of experts and organisations to include as potential panel members. From this, an initial list of panel members was created by the research team which had expertise in care transitions and measure development. There was originally a panel of 16 who committed to being part of the research project, with the final panel of 11 participants representing a broad scope of expertise in care transitions. The withdrawal of the five participants was due to scheduling conflicts. The final panel consisted of clinicians (medicine, pharmacy and nursing) including a general internist with patient safety expertise, researchers with measure development expertise and one who had extensive research background in care transitions. Both a national and provincial level policy maker were part of the panel as were decision makers from a variety of sectors, including acute care, home care, long-term care, complex continuing care/rehabilitation care and primary care. Stakeholder organisations and patient advocacy group were also represented on the final panel.
First round review survey. In the first round of the Delphi approach, the panel was sent the questionnaire electronically 4 weeks prior to the face-to-face panel meeting and asked for input to identify measures worthy of consideration. A reminder email was sent to panelists 2 days before, and on the day of the deadline. We chose to include all relevant measures (n=119) in the survey (see online supplementary file for list). The survey package included an overview of the literature review process, the list of measures with definitions for calculations as described in the literature (where provided), and literature summaries of each of the articles and reports. Additionally, an overview of the rating scale was provided and outlined in table 1⇓.
Specifically, panel members were asked in the survey to evaluate the proposed measures using the Quality Measures Attributes developed by the Agency for Healthcare Research and Quality (AHRQ).65 This framework groups measures according to four broad conceptual areas: (1) importance of the measure; (2) scientific soundness of the measure—clinical logic; (3) scientific soundness of the measure—measure properties; (4) feasibility of the measure. Given the focus of our study was to identify quality measures that may in the future be useful for accountability purposes and to make system improvements for complex-care patient transitions, we also included two additional criteria for evaluation including the usefulness of the measure: (1) to drive QI efforts and (2) for accountability purposes in public reporting (PR). We included the means for the two latter measures, in addition to the overall mean average of the six evaluation criteria, due to our interest in developing measures for accountability purposes and QI efforts. The panel used a nine-point Likert scale which considered scores of 1–3 as strong disagreement, 7–9 as strong agreement and 4–6 as uncertain agreement. An overall total score that averaged all six evaluation criteria was calculated alongside the score for the usefulness of the measure to drive QI efforts and the usefulness of the measure for accountability purposes in PR. Panel members were also provided with the opportunity to suggest additional measures prior to the in-person meeting. To ensure measures developed through the panel are useful and relevant over time, panelists were encouraged to consider the extent to which identified measures can be linked to currently employed quality of care processes as well as what they view as future areas that necessitate attention related to improving transitions of care for complex-care patients across the continuum. Consensus on these surveys was determined to be reached upon a 75% agreement (standard percentage in other studies)16 ,17 among panel members on the three scores (mean total score for all six rating criteria; mean score for usefulness to drive QI; and mean score for usefulness for PR).
Second-round panel meeting
For the second round, panel members met in person for a 1-day facilitated structured discussion of the results compiled from the first round of the questionnaire. At the conclusion of the discussion, members were asked to complete a second round of the questionnaire using the same evaluation criteria and rating scale. After completion of all discussion and questionnaires, panellists were also asked to anonymously comment upon their experience regarding the overall consensus process as a mechanism to ensure the process was fair and unbiased.
The final results were tabulated using the same consensus agreement score (≥75%). The panel was then contacted once more to provide a priority ranking (from 1–11) among the final selected measures. Again, individual responses remained anonymous to all other panel members. The mean frequency of the rankings of the set of measures was calculated to yield the final set of measures. Additionally, panellists were asked for their personal views of where the priority for measure usage and development should be regardless of current data availability and ease in feasibility.
First round ratings
From the initial set of 119 measures, 30 measures received an aggregate rating of >75% (7/9 or more) across one or more of the total score of the six domains, useful to drive QI, and useful for the PR domain (see table 2).
Analysis from the second round of panel member ratings revealed a set of 11 measures (see table 3). This included seven measures that received an aggregate rating of >75% (7/9 or more) in all three scores—total score of the six domains, useful to drive QI and useful for the PR domain; three measures received an aggregate rating of >75% (7/9 or more) total score of the six domains, and useful to drive QI score and between 6 and 6.99 on the useful for PR domain; and one measure received an aggregate rating of >75% (7/9 or more) in the useful to drive QI. Totally, 11 out of the original measures were selected to go to the final round of rating, with wording changes to four of the measures (medication reconciliation, time from discharge to homecare nursing visit for high-risk patients, mortality rate and discharge summary), and readmission rate stratified by three time frames—within 72 h, 30 days and 90 days.
Totally, 21 out of the original 30 measures were eliminated in this round. Analysis of the key discussion points at the panel meeting and hand written notes from the participants revealed that although these measures were important quality measures of care, they were not perceived collectively to be a quality measure of a care transition. For example, several patient clinical outcome measures (eg, nosocomial infections, pressure ulcers, pain, functional status and medical errors) were viewed as not being an outcome of an ineffective care transition. Other key issues identified with those measures that were eliminated included the perception that the measures were unclear or too broad to be an effective indicator of the transition (eg, potentially avoidable hospital admissions and hospital discharge process). Other key discussion points included a focus on patients and family perspectives for what constitutes quality care transitions for complex-care patients across the care continuum. The panel members recommended further work be undertaken to develop more system-level measures that would capture complex-care patient and family members’ satisfaction and experiences along the care continuum.
A postmeeting survey found unanimous agreement from the participants that their opinions were valued, the process was fair, and that no one altered their responses or opinions as a result of feeling intimidated.
Final review ratings
When asked to rank the most important final indicators with respect to overall score, QI and PR purposes, with the exception of readmission rates within 30 days as the number one priority and multiple psychiatric readmissions in 30 days as the last priority, there were mixed responses regarding where the priority focus should be (see table 4). Ranking the measures using the overall total score yielded the following top five measures are readmission rates within 30 days, primary care visit within 7 days postdischarge for high-risk patients, medication reconciliation completed at admission and prior to discharge, readmission rates within 72 h, and time from discharge to homecare nursing visit for high-risk patients.
Through a structured, modified Delphi panel process, a series of quality measures for care transitions involving complex-care patients across the continuum of care were delineated. Our inclusion of health services researchers, decision-making organisations, frontline clinicians, (physicians, nurses and pharmacists) and a patient representative increased the likelihood that the selected quality measures are scientifically sound and have broad applicability and feasibility.10 ,18 Our final list of quality indicators included outcomes from the three domains of focus including nine health services measures (readmission rates within 30 days, primary care visit within 7 days postdischarge for high-risk patients, medication reconciliation completed at admission and prior to discharge, readmission rates within 72 h, time from discharge to homecare nursing visit for high-risk patients, discharge summary all medications and follow-up appointments given to primary healthcare provider, discharge summary all medications and follow-up appointments given to patient, readmission rates within 90 days, multiple psychiatric readmissions in 30 days); one patient outcome measure (mortality/death/cause of death/risk of death within 72 h); and one patient satisfaction and perceptions measure (patient and caregiver satisfaction follow-up during interventions and/or after). These measures were all associated with posthospital discharge and warrant further attention in future research to determine what measures should be included and/or developed to capture measures that hospitals could use to assess the quality of care transitions with complex-care patients.
Implications for practice and policy
To our knowledge, there has been no previous research in the area of quality measures for care transitions involving complex-care patients across the continuum of care. As a next step, further validation of these measures is recommended with other key stakeholders. This work will require: (1) testing of whether these measures actually measure the quality of care (vs differences in case mix); (2) determining if there are sufficient numbers to enable stable estimates when measuring at the provider/hospital level; (3) determining the need for risk adjustment and (4) ascertaining the level of measurement for the indicators. There is potential for these measures to be used as indicators of overall care quality related to care transitions involving complex-care patients across the continuum of care.
There is an increasing focus on moving a patient out of the hospital system to alleviate wait times and in an effort to provide patients with more appropriate care in the right settings, such as home-based care and community-based programmes and services. However, in order to gain the confidence of patients and their caregivers in these movements across settings, it is essential that organisations be accountable for providing and monitoring high-quality transitions. Having measures such as these can allow organisations to ascertain how well they are doing, make improvements where necessary and be accountable to the patients who they are transitioning. We acknowledge that this study has been conducted in the Canadian healthcare system, therefore, generalisablity to other countries and contexts may not apply exactly. However, the Delphi technique described here can be replicated in other countries in order to understand whether these measures are consistent in other contexts.
Moreover, there is broad applicability in the majority of the final quality measures selected as they use data that is already commonly collected by hospitals and other healthcare organisations and could feasibly be collected and reported. Some specific points made by panel members during the discussion merit emphasis. For example, the recommendation to further develop measures that are reflective of complex-care patient and family members’ satisfaction and experiences along the care continuum warrants further attention. This finding is consistent with the view that patients and their families play an integral role in ensuring they receive safe care, as they are the one constant in care transitions processes.5 This work can draw from existing measures such as the Care Transitions Measure used to measure satisfaction with discharge from hospital to home.1 ,5 ,56
Strengths and limitations
Engaging key stakeholders in quality measure development is paramount. Important attributes of high-quality measures are their relevance to the selected problem and field of application, their feasibility and their reliability.10 It is important to note that the inclusion of a patient representative on the panel is important, as a recent systematic review revealed that the participation of patients in quality indicator development is extremely uncommon.10 Our final panel consisted of individuals (researchers, front-line clinicians and a patient) with broad professional representation as well as consultants from stakeholder groups (national accreditation body and provincial quality council). Additionally, our panel included individuals from different healthcare systems and sectors increasing the potential for generalisability of our findings.
The literature review that was included as a summary in the background information and anonymous survey used in our process ensured that all members were entering the discussion with balanced background and content knowledge. We also included open-ended questions in the presurvey to provide an opportunity for panel members to comment on the list of indicators and submit additional indicators. Additionally, an experienced health services researcher facilitated the discussion and maintained equality among all panel members. The anonymous end-of-day survey verified that all panelists were comfortable with the process.
Furthermore, panel discussion was also focused on finding important and relevant measures that could act as quality indicators in addition to the ones initially presented by the research team. This open-ended discussion was presented to the panel prior to the face-to-face meeting to provide opportunity for panelists to review and reflect on the measures presented and potential missing measures reflective of quality care transitions for complex-care patients across the care continuum. Upon review of the measures premeeting and postmeeting, it was evident that the face-to-face discussion was a vital component to our process as the measures and parameters were redefined through the modified Delphi process. This is an important finding that should guide future quality performance measures—research and development.
There are limitations associated with our modified Delphi panel process. First, the review of literature was not a systematic review, rather it was a targeted review that was designed by researchers with expertise in performance measures and care transitions. We drew on the expertise of other researchers, clinicians and researchers that were part of the panel to ensure that key measures were not omitted. Second, there was the potential for selection bias inherent in panel member selection. To minimise selection bias, the research team used a similar process used by one of the researchers17 ,18 and solicited suggestions from unbiased third party individuals, stakeholder groups as well as national patient safety organisations. However, our initial international panels were not able to participate. Thus, some important perspectives may not have been identified. Another measure to reduce the risk of selection bias was the use of predesigned forms containing rating and selection criteria during systematic consensus processes.10 Third, the complex nature around the validity of the measures and whether they are measuring quality of care and the feasibility of collecting data on the identified measures is another limitation. In our study, panelists were not asked to limit their selection to quality measures that could be applied immediately with commonly available data. The use of administrative data for any quality measures always has the limitation of clinical application, as it is only a part of a patient's clinical story. Fourth, our panel had only one patient representative, and although they were a leader of a national patient safety advocacy organisation, their views may not be completely representative of patients and caregivers.
Quality measures are important tools used for accountability and QI in healthcare systems. The modified Delphi consensus panel approach applied in this study enabled us to garner a consensus on measures in an important area—care transitions. Although a number of these measures are not new and are used for the evaluation of other healthcare areas, this study provides evidence and support that these measures may also be effective for evaluating the quality of care transitions. This is the first study focused on care transitions and performance measures, thus, more research and testing of these measures using a rigorous research design are needed to determine their actual usefulness and feasibility. Further work is required to create standardised operational definitions, for example, the patient and caregiver satisfaction follow-up during interventions and/or after measure.
Collectively, future research efforts will produce findings that can be used by healthcare administrators, clinicians and policy makers to monitor care transitions and to work together to improve the quality of transitions of care with complex-care patients. Moreover, individual institutions can evaluate interventions aimed at enhancing care transitions for complex-care patents’ efforts, and provide healthcare systems with a consistent means to compare programme effectiveness across institutions. A longer-term goal is that this work will inform the development of a reporting standard of measures that demonstrate quality outcomes associated with transitions of care with complex-care patients.
This consensus process was funded by a Canadian Institutes of Health Research (CIHR) Meetings, Planning and Dissemination Grant—PA: Health Services and Policy Research 201102 PLH (#248538). We wish to thank the panel members: Dr Brian Hutchison, Senior Advisor for Primary Care—Health Quality Ontario; Carol Kushner, Co-Chair Patients for Patient Safety Canada; Dr Carole Estabrooks, Professor & Canada Research Chair in Knowledge Translation, Faculty of Nursing, University of Alberta; Dr Chris Hayes, Medical Officer, Canadian Patient Safety Institute; Dr Dan Cass, Chair, Patient Safety Review Committee, Office of the Chief Coroner for Ontario, Jennifer Kodis, Clinical Director, Regional Rehabilitation Centre, Complex Continuing Care & Seniors Health, Hamilton Health Sciences; Jonathan Mitchell, Manager, Policy and Research, Accreditation Canada; Dr Karen Chien, Palliative Care, St Michael's Hospital & Staff Physician Bridgepoint Health; Dr Olavo Fernandes, Pharmacy Clinical Site Leader, Toronto General Hospital, UHN; Dr Onil Bhattacharyya, Assistant Professor and Clinician Scientist, University of Toronto, Department of Family and Community Medicine, Li Ka Shing Knowledge Institute of St Michael's Hospital; Dr Walter Wodchis, Associate Professor, University of Toronto, Research Scientist, Toronto Rehabilitation Institute, Adjunct Scientist, Institute for Clinical Evaluative Sciences. The authors would like to thank Stacey Brener for her contribution to the search strategy and literature review.
Contributors The primary authors (LJ, MPL and CB) (1) made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; (2) drafted the article and revised it critically for important intellectual content; and (3) provided final approval of the version to be published. The other three authors (SS, RC and RFL) also contributed to the (1) design and analysis and interpretation of data; (2) revision of the article critically for important intellectual content and (3) provided final approval of the version to be published.
Funding This study was funded by a Canadian Institutes of Health Research (CIHR) Meetings, Planning and Dissemination Grant—PA: Health Services and Policy Research 201102 PLH (#248538).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.