You are here

Article Text

Article menu
Original research
Development of quality of care indicators for patients undergoing total hip or total knee replacement
  1. N F SooHoo,
  2. J R Lieberman,
  3. E Farng,
  4. S Park,
  5. S Jain,
  6. C Y Ko
  1. UCLA School of Medicine, Los Angeles, California, USA
  1. Correspondence to Nelson F SooHoo, UCLA School of Medicine, 10945 Le Conte Avenue, PVUB 3355, Los Angeles, CA 90095, USA; nsoohoo{at}mednet.ucla.edu

Abstract

Objective The objective of this study was to develop evidence-based quality indicators to measure key aspects of care that can be targeted to decrease variations in complication rates between surgeons performing total joint replacement.

Design RAND/University of California, Los Angeles (UCLA) modified-Delphi expert panel method. To accomplish this objective, a proposed set of quality of care indicators was developed through a comprehensive literature search and structured interviews with expert clinicians. An expert panel of orthopaedic surgeons was then convened to rate the validity of these quality indicators using the RAND/UCLA Appropriateness Method. Indicators were classified as valid by the panel based on the median panel rating and the amount of dispersion of panel ratings.

Results There were 101 candidate indicators of quality identified in the six domains of preoperative processes of care, intraoperative processes, postoperative processes, implant selection and the use of new technology, privileging of hospitals and surgeons, and outcomes and comorbidity assessment. A total of 68 of the 101 indicators were rated as valid with statistical agreement.

Conclusions This study identifies measures of structure, process and outcome rated as valid quality indicators for hip and knee replacement. This project provides tools to measure and improve quality of care for patients undergoing total joint replacement.

  • Total knee arthroplasty
  • total hip arthroplasty
  • quality of care
  • quality measures
  • quality indicators
  • quality improvement
  • quality of care

http://dx.doi.org/10.1136/bmjqs.2009.032524

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Total hip and total knee replacement are two of the most commonly performed elective inpatient surgical procedures.1 The Medicare programme alone had claims for 430 726 total knee replacement procedures during the years 1998–2000.2 There is consensus on the excellent outcomes and cost-effectiveness of total joint arthroplasty.3 However, a growing body of evidence has documented variations between providers in the rates of common complications, including infection, thromboembolic disease, revision rates and mortality. This evidence includes previous studies by our research group and others that use administrative databases to document an association between the annual volume of cases performed at a hospital or by a surgeon, and the outcomes of orthopaedic surgical procedures, including total knee and hip arthroplasty.4–10 These differences in outcome persist after adjusting for patient factors, such as age, race and insurance type, which also have statistically significant associations with rates of mortality and complications.5 6 9 10 These variations may be the result of other differences in the case mix between providers, but they raise the concern that they are due to variations in the quality of care being delivered. Given the large number of patients receiving total joint replacement, it is critical to determine if the quality of care at centres with higher complication rates can be improved to decrease these rates.

    This project represents a necessary first step towards improving quality by developing a valid set of quality indicators than can be used to report, measure and improve the quality of care associated with elective total hip and total knee replacement. The quality of healthcare can be assessed on the basis of structure (the attributes of the hospital and provider), process (the actual details of the care provided) or the outcomes of care.11 While many evidence-based process measures exist for general medical care, there has been limited development of quality measures for surgical care.11 This project addresses this relative lack of evidence-based measures by developing, for the first time, a comprehensive set of quality indicators that address the perioperative care of patients undergoing total joint replacement. The value of this study is that it identifies indicators of quality of care rated as valid and appropriate for further empirical testing in a population-based sample to identify which components of high-quality care are prone to variation or are not being delivered optimally.

    Methods

    Overview

    This project used a modification of the RAND/ University of California, Los Angeles (UCLA) Appropriateness Method to develop a set of quality of care indicators using the expertise of practicing orthopaedic surgeons.12 An overview of the process to identify and rate quality indicators is shown in figure 1. This modified Delphi method is iterative with two rounds of independent ratings by a group of expert panelists and a face-to-face group discussion between rounds.12 13 Each panelist has an equal weight in determining the final ratings. This RAND/UCLA method, which also includes an explicit synthesis of findings from the literature, has been shown to produce quality indicators that have face construct and predictive validity.14–18

    Figure 1
    Figure 1

    Overview of the RAND/UCLA Appropriateness Method used in this study to develop and validate quality of care indicators.

    Identification of candidate quality indicators

    The methods for identifying candidate quality indicators for patients undergoing total hip and knee replacement surgery included literature reviews and semistructured interviews with nationally recognised leaders in the field of orthopaedic surgery. During the semistructured interviews, candidate process indicators were identified that were viewed as indicating high quality of care for patients undergoing total hip or knee replacement surgery. These interviews were conducted by one of the co-investigators and asked participants to provide a list of practices that they believed contributed to higher quality of care related to total joint replacement. Interviewees were prompted to provide lists relevant to all perioperative issues, including preoperative care, intraoperative care, postoperative care, the structure of the healthcare delivery system and monitoring of outcomes.

    In addition to the interviews, a literature search was conducted to identify existing evidence-based guidelines relevant to the care of patients undergoing total joint replacement. We divided the individual indicators derived from the interviews and literature search into domains that fell into one of the three perioperative periods: preoperative, intraoperative or postoperative. Additional indicators were identified in the domains of implant selection and use of new technology, privileging of hospitals and surgeons, and outcomes and comorbidity assessment (figure 2).

    Figure 2
    Figure 2

    The six domains of quality indicators for patients undergoing total hip or total knee replacement.

    For each of the identified candidate indicators, an extensive literature search was performed. We looked for previous studies that assessed elements of the structure, process or outcomes of care following total hip or total knee arthroplasty. Using MEDLINE, scientific articles published between 1985 and 2006 were searched for reports of outcomes following total hip or knee arthroplasty. These studies were reviewed and abstracted to record the study population, processes of care, treatment, outcomes, reported comorbid conditions and other relevant variables. Studies were identified as belonging to one of the following categories: randomised clinical trial; nonrandomized controlled trials, cohort or case analysis, multiple time series, or descriptive studies.

    Panel composition

    We contacted the major professional organisations in total joint replacement (American Academy of Orthopaedic Surgeons and American Association of Hip and Knee Surgeons) to enlist their support and request nominations. We selected a group of expert orthopaedic surgeons based on geographical diversity and training. We recruited panelist from each of the four regions designated by the Census Bureau, consisting of the Northeast, Midwest, South and West. The panelists were all fellowship-trained total joint arthroplasty surgeons with a diversity of location of training and years in practice.

    The nine panelists that supplied the final ratings used in our analysis all report being paid consultants for one or more implant manufacturers. One of the co-investigators on this project is a total joint replacement surgeon and also reports being a paid consultant to one or more implant manufacturers. The principal investigator of this protocol does not have any conflicts of interest. The study was fully funded by the Orthopaedic Research and Education Foundation, a private, nonprofit entity.

    RAND/UCLA Appropriateness Method—first-round and second-round ratings of candidate indicators

    An expert panel was convened to evaluate the proposed indicators and covariates developed through the literature review and preliminary interviews with expert orthopaedic surgeons. From the pool of total joint arthroplasty experts, an expert panel of 10 clinicians and researchers was convened. One panelist's ratings were excluded as the panelist attended only a portion of the meeting due to travel difficulties, leaving nine panelists with complete ratings for all potential indicators. The expert panel was conducted at the national headquarters of the American Academy of Orthopaedic Surgeons (AAOS). The medical director of the AAOS was present as an observer. The RAND/UCLA Appropriateness Method was used to evaluate the validity of the proposed indicators.12 17 19 This method is iterative with two rounds of anonymous ratings and a face-to-face group discussion between rounds. This expert panel method has been shown to have a high level of reproducibility and validity.12

    At the panel meeting, participants discussed the evidence that supports or refutes each indicator and the panelists' prior validity rankings. Panelists were given the summary of the first-round rankings and a reminder of their own rankings. Each individual panelist's ratings were kept confidential. Panelists were encouraged to discuss any additional information omitted by the literature reviews as well as to suggest new indicators if they felt that they were relevant based on their clinical judgement. At the completion of discussion, each panelist re-ranked the validity of the indicators using a 1–9 validity scale.

    Statistical analysis

    A median score and standard error were generated from the validity scores of the second-round ratings. Indicators were classified and accepted as valid by the panel based on (1) the median panel rating and (2) the amount of dispersion of panel rating, which is a reflection of agreement. Indicators were accepted as valid with a median rating of 7–9 with agreement on the 9-point scale, uncertain with a panel median of 4–6 or any median with disagreement, and not valid with a panel median of 1–3 with agreement. We used the measure of degree of spread among the panelists to determine the presence of agreement, disagreement or an indeterminate level of agreement or disagreement. To test this, we used a hypothesis that has been standard for other expert panels using the RAND/UCLA Appropriateness Method. Specifically, we first determined if the mean rating fell into either the valid (7–9) or not valid (1–3) scoring range. The indicator was then determined to be valid or not valid with agreement if 90% of the ratings were within the larger region around the mean score (eg, 4–9 for a mean rating of valid or 1–6 for a mean rating as not valid).12 13

    Results

    Candidate quality indicators

    The initial set of candidate indicators were divided into the domains of preoperative care, intraoperative care, implant selection and use of new technology, privileging of hospitals and surgeons, postoperative care, and outcomes and comorbidity assessment. A total of 101 candidate indicators were available for rating with additional subcategories for certain indicators. For each of these domains, literature review and expert interviews were used to propose quality indicators relevant to improving the structure, processes and outcomes of care.

    Quality indicators rated as valid

    All or a portion of 68 out of the 101 candidate indicators were rated as valid with statistical agreement by the expert panel. These included 18 preoperative process indicators, 9 intraoperative process indicators, 10 postoperative process indicators, 8 structural indicators related to implant selection and technology use, 13 structural indicators related to privileging of providers, and 10 outcomes and comorbidity-assessment related indicators. A detailed list of the indicators rated as valid with statistical agreement is included as Appendix A (availabel online).

    Discussion

    This project develops quality indicators for total joint replacement, which measure key aspects of high-quality care that can be targeted to decrease complication rates. The indicators developed by this project fulfill a need for surgical measures of quality care for the common and costly procedures of total hip and total knee replacement. The RAND/UCLA expert-panel method used in this study for developing quality indicators was intended to encourage the support of the orthopaedic community in an effort to monitor and improve quality of care. As a result, this project was able to utilise the expertise of practicing orthopaedic surgeons to create a set of 68 explicit measurement tools (the quality indicators) covering the full range of perioperative care that can be used to monitor and improve the quality of orthopaedic care. These quality indicators provide the tools necessary for further studies designed to measure, compare and improve quality of care for total joint replacement.

    The primary difficulty in attempts to develop quality measure for orthopaedic surgery is that the evidence available to evaluate many of aspects of care consists largely of retrospective studies. The few randomised trials in orthopaedics are often small or have methodological weaknesses.20 The RAND Appropriateness Method was used in this study because it has been validated as an alternative in the situations in which there is not enough level 1 evidence. Specifically in situations where there is no level 1 evidence, the RAND method of combining the best level evidence available with expert opinion is an alternative approach.13 15 18 21–25 It is our intention that publication of the indicators at this stage will stimulate the orthopaedic community to engage in active debate and ongoing study regarding the optimal methods of monitoring and improving the quality of orthopaedic care.

    In general, the findings of our expert panel validated existing guidelines issued by many professional organisations for processes of care. These include guidelines by the American Society of Anesthesiologists (ASA) relevant to preoperative evaluation, Surgical Infection Prevention Project (SIPP) for antibiotics, and American College of Cardiology/American Heart Association for cardiac evaluation. One notable departure from national consensus guidelines was concerning the postoperative processes including a discussion of the widely cited American College of Chest Physicians (ACCP) guidelines for thromboembolic prophylaxis. There was concern among the panelists that the guidelines do not adequately balance the added morbidity of postoperative haematoma following total knee replacement.26 27 There were relatively few intraoperative process indicators judged as valid, and the general discussion centred on the inherent difficulties of standardising complex decisions in the operating room. The panel was able to agree on widely utilised national guidelines, including the National Surgical Infection Prevention Project recommendation for antibiotic prophylaxis and AAOS and Joint Commission on Accreditation of Healthcare Organizations (JCAHO) guidelines for time-out protocols.28 29 The time-out procotols consist of a preoperative checklist to identify the patient identity and surgical site. The endorsement of antibiotic protocols could similarly be incorporated into the preoperative checklists as is being advocated by the World Health Organization.30 Another key standard endorsed was that operative time from incision to completion of closure should not exceed 3 h for routine primary total hip or knee replacement. A study by the Surgical Site Infection Surveillance Service has demonstrated that operative times above 2 h for total hip arthroplasty were linked to a higher risk of surgical site infection.31

    Overall, the value of this project is that it utilised the expertise of practicing orthopaedic surgeons to create explicit and valid measurement tools (the quality indicators) that can be used to monitor and improve the quality of orthopaedic care. The next step will be to field-test the quality indicators in a national sample of institutions to confirm their validity and feasibility. This will allow identification of those indicators that are prone to variation or not being optimally delivered, making them logical targets for quality improvement interventions. This approach has the potential to be a model for translating evidence-based practice into improved outcomes for the large number of patients undergoing total hip and total knee replacement.

    Acknowledgments

    This study was funded by a grant from the Orthopaedic Research and Education Foundation.

    References

    1. ↵
      1. Jha AK,
      2. Fisher ES,
      3. Li Z,
      4. et al
      . Racial trends in the use of major procedures among the elderly. N Engl J Med 2005;353:683–91.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Skinner J,
      2. Weinstein JN,
      3. Sporer SM,
      4. et al
      . Racial, ethnic, and geographic disparities in rates of knee arthroplasty among Medicare patients. N Engl J Med 2003;349:1350–9.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Archibeck MJ,
      2. White RE Jr.
      . What's new in adult reconstructive knee surgery. J Bone Joint Surg Am 2006;88:1677–86.
      OpenUrlCrossRefPubMed
    4. ↵
      1. Hervey SL,
      2. Purves HR,
      3. Guller U,
      4. et al
      . Provider volume of total knee arthroplasties and patient outcomes in the HCUP-nationwide inpatient sample. J Bone Joint Surg Am 2003;85-A(9):1775–83.
      OpenUrlPubMed
    5. ↵
      1. SooHoo NF,
      2. Lieberman JR,
      3. Ko CY,
      4. et al
      . Factors predicting complication rates following total knee replacement. J Bone Joint Surg Am 2006;88:480–5.
      OpenUrlCrossRefPubMed
    6. ↵
      1. Soohoo NF,
      2. Zingmond DS,
      3. Lieberman JR,
      4. et al
      . Primary total knee arthroplasty in California 1991 to 2001: does hospital volume affect outcomes? J Arthroplasty 2006;21:199–205.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Katz JN,
      2. Barrett J,
      3. Mahomed NN,
      4. et al
      . Association between hospital and surgeon procedure volume and the outcomes of total knee replacement. J Bone Joint Surg Am 2004;86-A(9):1909–16.
      OpenUrlPubMed
    8. ↵
      1. Katz JN,
      2. Mahomed NN,
      3. Baron JA,
      4. et al
      . Association of hospital and surgeon procedure volume with patient-centered outcomes of total knee replacement in a population-based cohort of patients age 65 years and older. Arthritis Rheum 2007;56:568–74.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Mahomed NN,
      2. Barrett JA,
      3. Katz JN,
      4. et al
      . Rates and outcomes of primary and revision total hip replacement in the United States Medicare population. J Bone Joint Surg Am 2003;85-A(1):27–32.
      OpenUrlPubMed
    10. ↵
      1. Solomon DH,
      2. Losina E,
      3. Baron JA,
      4. et al
      . Contribution of hospital characteristics to the volume-outcome relationship: dislocation and infection following total hip replacement surgery. Arthritis Rheum 2002;46:2436–44.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Birkmeyer NJ,
      2. Birkmeyer JD
      . Strategies for improving surgical quality—should payers reward excellence or effort? N Engl J Med 2006;354:864–70.
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Fitch K
      . The Rand/UCLA Appropriateness Method user's manual. Santa Monica (CA): RAND, 2001.
    13. ↵
      1. Shekelle P
      . The appropriateness method. Med Decis Making 2004;24:228–31.
      OpenUrlFREE Full Text
    14. ↵
      1. Higashi T,
      2. Shekelle PG,
      3. Adams JL,
      4. et al
      . Quality of care is associated with survival in vulnerable older patients. Ann Intern Med 2005;143:274–81.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. McGlynn EA,
      2. Asch SM,
      3. Adams J,
      4. et al
      . The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635–45.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Shekelle PG,
      2. Chassin MR,
      3. Park RE
      . Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care 1998;14:707–27.
      OpenUrlPubMedWeb of Science
    17. ↵
      1. Liu HH,
      2. Weiss RE,
      3. Jennrich IR,
      4. et al
      . PRESS model selection for repeated measures data. J Computational Statistics & Data Analysis 1999;30:169–84.
      OpenUrlCrossRef
    18. ↵
      1. Mangione-Smith R,
      2. DeCristofaro AH,
      3. Setodji CM,
      4. et al
      . The quality of ambulatory care delivered to children in the United States. N Engl J Med 2007;357:1515–23.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Spencer BA,
      2. Steinberg M,
      3. Malin J,
      4. et al
      . Quality-of-care indicators for early-stage prostate cancer. J Clin Oncol 2003;21:1928–36.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Poolman RW,
      2. Struijs PA,
      3. Krips R,
      4. et al
      . Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Med Res Methodol 2006;6:44.
      OpenUrlCrossRefPubMed
    21. ↵
      1. Kahn KL,
      2. Park RE,
      3. Vennes J,
      4. et al
      . Assigning appropriateness ratings for diagnostic upper gastrointestinal endoscopy using two different approaches. Med Care 1992;30:1016–28.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Nicollier-Fahrni A,
      2. Vader JP,
      3. Froehlich F,
      4. et al
      . Development of appropriateness criteria for colonoscopy: comparison between a standardized expert panel and an evidence-based medicine approach. Int J Qual Health Care 2003;15:15–22.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Shekelle PG
      . Are appropriateness criteria ready for use in clinical practice? N Engl J Med 2001;344:677–8.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Vader JP,
      2. Burnand B,
      3. Froehlich F,
      4. et al
      . Appropriateness of upper gastrointestinal endoscopy: comparison of American and Swiss criteria. Int J Qual Health Care 1997;9:87–92.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Vader JP,
      2. Porchet F,
      3. Larequi-Lauber T,
      4. et al
      . Appropriateness of surgery for sciatica: reliability of guidelines from expert panels. Spine 2000;25:1831–6.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      AAOS. American Academy of Orthopaedic Surgeons clinical guideline on prevention of symptomatic pulmonary embolism in patients undergoing total hip or knee arthroplasty.
    27. ↵
      1. Geerts WH,
      2. Pineo GF,
      3. Heit JA,
      4. et al
      . Prevention of venous thromboembolism: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004;126(3 Suppl):338S–400S.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Bratzler DW,
      2. Houck PM
      . Antimicrobial prophylaxis for surgery: an advisory statement from the National Surgical Infection Prevention Project. Clin Infect Dis 2004;38:1706–15.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Wong DA
      . Spinal surgery and patient safety: a systems approach. J Am Acad Orthop Surg 2006;14:226–32.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      WHO's patient-safety checklist for surgery. Lancet 2008;372:1. doi:10.1016/s0140-6736(08)60964-2.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      National Nosocomial Infections Surveillance System. National Nosocomial Infections Surveillance (NNIS) System Report, data summary from January 1992 through June 2004, issued October 2004. Am J Infect Control 2004;32:470–85.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    Footnotes

    • Funding Orthopaedic Research and Education Foundation, 6300 North River Road, Suite 700, Rosemont, IL 60018-4261.

    • Competing interests None.

    • Ethics approval This study was conducted with the approval of the UCLA School of Medicine.

    • Provenance and peer review Not commissioned; externally peer reviewed.