Article Text

Download PDFPDF

Qualitative complaints and their relation to overall hospital rating using an H-CAHPS-derived instrument
  1. Kyle Kemp1,2,
  2. Sarah Warren1,
  3. Nancy Chan1,
  4. Brandi McCormack1,
  5. Maria Santana2,
  6. Hude Quan2
  1. 1Analytics (DIMR), Alberta Health Services, Calgary, Canada
  2. 2Community Health Sciences, University of Calgary, Calgary, Canada
  1. Correspondence to Kyle Kemp, Analytics (DIMR), Alberta Health Services, 4520 16th Avenue NW, Suite 200, Calgary, Alberta, Canada T3B 0M6; kyle.kemp{at}albertahealthservices.ca

Abstract

Background and objectives Due to the multitude of questions in the Hospital-Consumer Assessment of Healthcare Providers and Systems (H-CAHPS) survey, it may be difficult to decide where quality improvement efforts should be focused. Our organisation has supplemented the survey with a ‘patient complaints’ section. The study objectives were to determine (1) the frequency of qualitative complaints and the demographic/clinical profile of patients lodging them, (2) the most frequent complaint themes and their association with overall experience scores and (3) whether overall experience scores varied based upon the complaint action taken by the patient or the degree of patient satisfaction in the handling of complaints.

Methods From April 2013 to March 2014, 8929 telephone surveys were completed by patients discharged from 93 acute care hospitals in Alberta, Canada. These were successfully linked with the corresponding inpatient record. Open-ended complaints were themed into categories. Mean differences in overall inpatient experience were assessed for each complaint theme, including overall and multiple complaints.

Results 1870 patients (20.9%) reported at least one open-ended complaint. Most frequent complaint themes were nursing (n=491; 5.5% of cohort), medications (n=219; 2.5%) and food (n=193; 2.2%). Increased odds of having a complaint were associated with younger age, being born in Canada and having no documented medical comorbidities. Protective factors were male gender, lower education level, urgent hospital admission, lower resource intensity and length of stay (LOS) <3 days.

Conclusions This is the first investigation of its type using H-CAHPS-based data in a Canadian context. Through replication of this study, other healthcare organisations may determine the association between open-ended complaints and their own overall experience scores.

  • Patient satisfaction
  • Quality improvement
  • Quality improvement methodologies
  • Surveys
  • Health services research
View Full Text

Statistics from Altmetric.com

Introduction

Patient experience surveys capture meaningful, patient-reported accounts of the perceived quality of the healthcare interactions and services that are delivered over the course of the hospital stay. Although many hospitals have captured this type of data for decades, this had been primarily done using ad hoc instruments.1–4 To ensure standardisation and enable valid comparisons of patient experience between hospitals, the Hospital-Consumer Assessment of Healthcare Providers and Systems (H-CAHPS) survey was developed and issued by the Agency for Healthcare Research and Quality in 2006.1 With the introduction of the Affordable Care Act of 2010, H-CAHPS results play a role in the value-based purchasing programme.5 As H-CAHPS results now affect a portion of hospital funding, US-based hospital administrators have a clear incentive to improve the experience of their admitted patients. Additionally, H-CAHPS results are now publicly available.6

Due to the vast amount of data generated, hospital administrators may find it a daunting task to decide where to focus quality improvement efforts.7 One method to overcome this potential challenge would be to examine the relationship between H-CAHPS domains and overall experience scores, as is currently reported.8 From this, an assessment of the specific domains that may provide the greatest gains in overall patient experience is possible.9 ,10 However, it can be argued that these data only provide a piece of the puzzle. Qualitative reports of hospital experience, for example, patient complaints, may provide a richer understanding of where opportunities for improvement may lie.11–13 Unfortunately, many patient experience surveys, including the traditional H-CAHPS one, do not always include a qualitative section for patients to provide their open-ended, qualitative feedback. In our organisation, we have supplemented our inpatient experience survey; one that uses an H-CAHPS-derived instrument, with a patient complaints section. Patients are asked whether they had a complaint about the care that they received, and if so, the nature of this complaint is documented. Follow-up questions then ask the patient to provide their opinion as to how their complaint was received and dealt with. In cases where patients wish to escalate their complaint, the contact information for our Patient Relations department is provided. A critical analysis of the literature showed that an examination of the potential association between patient complaints and the overall experience rating using an H-CAHPS-based tool had not been done.

As such, the present study had three objectives: the first was to determine the frequency of complaints and to determine the demographic/clinical profile of the patients lodging them. The second was to determine the most frequent complaint themes and to assess their association with overall patient experience scores. The third objective was to examine whether overall patient experience scores varied based upon the complaint action taken by the patient or by the degree of patient satisfaction with how the complaint was handled. In examining these three objectives, the present study provides data and a set of methods that may be useful to hospital administrators looking to gain more insight on the types of complaints lodged, as well as which complaint themes are associated with greatest decreases to overall experience (ie, H-CAHPS) scores.

Methods

Study population

In Alberta, Canada, publicly funded healthcare services are provided to the province's four million residents by a single-care provider (Alberta Health Services). From 1 April 2013 to 31 March 2014, 9279 telephone surveys were completed by the organisation, within 42 days of hospital discharge. This number represented 5% of the eligible inpatient population during this time period. At the beginning of the survey, potential participants provided their informed consent. During the informed consent statement, respondents were informed as to why the survey was being conducted, its voluntary nature, that their decision to participate would not affect their medical care and how the data would be used for research and quality improvement purposes.

Inpatient hospital experience was captured using a modified version of the H-CAHPS survey.14 ,15 The survey was comprised of 51 items, including the 32 core H-CAHPS questions. The 19 additional survey questions were developed by our organisation to address additional topics, such as pharmacy care, patient/family involvement in decision-making and patient complaints. Surveys were completed via computer-assisted telephone interview using a standard script and took between 9 and 25 min to complete (median=14). As per the H-CAHPS methodology, the following exclusion criteria were applied: under 18 years of age, less than a 24 h inpatient stay, death during hospital stay (no proxy interviews), psychiatric service code or psychiatric unit inpatient record (H-CAHPS not validated in mental health populations), day surgery or ambulatory procedures.15 For compassionate reasons, our organisation's additional exclusion criteria were applied. These included any dilation and curettage (D & C) procedure, any hospital stay associated with stillbirth or where a mother gave birth and the baby remained in hospital longer than 6 days. Surveys were completed in English only. Proportional sampling was employed to capture no less than 5% of eligible discharges from each of our province's 93 acute care inpatient facilities. The representativeness of our sample of respondents has been previously reported, from a demographic and clinical perspective.16

Study variables and data linkage

Overall inpatient hospital experience was the outcome variable and was scored on a scale from 0 (worst possible score) to 10 (best possible score). This question was read to the patient as follows: “We want to know your overall rating of your stay at <HOSPITAL NAME>. This is the stay that ended around <DATE>. Please do not include any other hospital stays in your answer. Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible…What number would you use to rate this hospital during your stay?”

To capture patient complaints, a series of questions was included near the end of the survey. Patients were asked whether they had a complaint about any of the healthcare services that they received during their stay. Those responding ‘yes’ were then asked five subsequent questions: the first was open-ended, asking the nature of the complaint. Patients were then asked what action they had taken regarding their complaint. Options included “talked directly with a healthcare professional”; “phoned the patient concerns intake line”; “sent an email to the organisation”; “wrote a letter to the organisation”; “completed an online feedback form”; “told a family member or friend”; and “other (specify)”. A question on whether the patient felt their complaint was welcomed (yes or no) was then asked. Then, the patient was asked if they felt their complaint was taken seriously. Options included ‘yes, definitely’; ‘yes, somewhat’; and ‘no’. Finally, the patient was asked to what extent were they satisfied or dissatisfied with how their complaint was handled and addressed. Patients rated this on a five-point scale (1, very dissatisfied; 2, dissatisfied; 3, neither satisfied or dissatisfied; 4, satisfied; and 5, very satisfied).

Data linkage

To allow for an assessment of demographic and clinical characteristics of survey respondents, patient survey data was linked to corresponding inpatient records in the Discharge Abstract Database (DAD). The DAD database is maintained by the Canadian Institute for Health Information,17 ,18 with our organisation retaining a copy of its own data. Exact linkage was performed using each patient's personal health number, the five-digit hospital code and the discharge date from both data sets. An exact match was achieved for 8929 of 9279 survey respondents (96.2% of records). Only these survey respondents who could be linked to their corresponding inpatient record were included for analysis.

The merged data set was then analysed to capture selected demographic and clinical variables. Demographic variables included age group at hospital discharge, sex, marital status, education level and birth location. Patient age groups were classified as 18–39 (years); 40–49; 50–59; 60–69; and 70 and older. Marital status was coded as single (never married); married/common law/living with partner; divorced/separated/widowed. Education level was coded as elementary or junior high, senior high, college/technical school, undergraduate level and postgraduate degree complete. Birth location was classified as ‘in Canada’ versus other. Clinical variables were comprised of admission type, length of hospital stay, discharge disposition, resource intensity weight (RIW) and number of documented medical comorbidities. Admission type was classified as urgent or elective, according to the DAD. Total LOS was classified as <3 days (median in the cohort), 3 days and >3 days. Discharge disposition was classified as discharged home with or without support, versus all other dispositions. RIW is a summary index, provided in the DAD, that incorporates type of diagnosis/procedure, patient age, case complexity and LOS. An RIW >1.0 indicates a greater cost or more resources required than would be typically expected.19 Comorbidity profiles were generated using the Elixhauser Comorbidity Index,20 according to a validated list of International Classification of Diseases, Canadian V.10 codes from each corresponding inpatient record.21 Number of comorbidities was classified as none, versus one or more.

Patient complaints were categorised into themes by a single investigator (SW), according to a list of rules, established a priori. This reviewer performs this thematic analysis on a quarterly basis within our organisation. Patient complaint themes included nurses, doctors, food, medication, hospital cleanliness, discharge, roommates and other (not otherwise classified). In the case of a patient with multiple complaints (eg, nursing staff and food), these were included in all corresponding categories as appropriate. For the purposes of assessing inter-rater agreement, a second rater who was not part of the investigator team categorised the complaints of a subset of respondents (200 respondents, 273 possible complaints). This second rater received the same training and instruction as the standard rater. Agreement between the two raters was achieved in 214 instances (78.4% agreement). To assess the potential additive of effect of multiple complaints, another variable called ‘multiple complaints’ was created and classified as ‘yes’ or ‘no’. The ‘action taken’ and ‘satisfaction with handling of complaint’ questions were recoded for analysis. The ‘action taken’ responses were classified as ‘notified staff’, ‘notified organisation’, and ‘did nothing’. The ‘did nothing’ option included patients who told a family member or friend as it was impossible to determine whether the complaint was ever brought forward to our organisation. The ‘satisfaction with handing of complaint’ question was classified as ‘dissatisfied’, ‘neutral’ and ‘satisfied’.

Analysis

Demographic and clinical characteristics of the sample were assessed using descriptive statistics (frequency, mean, SD), as appropriate. Univariate logistic regression was used to examine complaint presence for each demographic and clinical variable examined. ORs and 95% CIs were generated. Mean differences in overall inpatient experience were assessed for each complaint theme, including overall and multiple complaints using Student's t tests and 95% CIs. To assess for a potential mitigating effect among the ‘action taken’ and ‘satisfaction of handling of complaint’ variables, mean differences in overall experience were assessed for single and multiple complaints using a one-way analysis of variance with post hoc Tukey's comparisons. For reporting purposes, all patient experience scores were converted to a 100-point scale, where 0 was the worst possible and 100 was the best possible score. This process is also performed when calculating the H-CAHPS ‘star ratings’.22 All analyses were performed using SAS Network V.9.3 for Windows (Cary, North Carolina, USA), defining significance as a p value <0.05.

Results

Table 1 outlines the demographic and clinical characteristics of the study cohort. Patients were predominantly 40 years of age and older (70.5%), female (64.1%), married, common-law or living with a partner (69.6%), had pursued postsecondary education (54.6%) and were born in Canada (84.9%). From a clinical perspective, 61.0% of patients were admitted to hospital urgently. The median LOS in the cohort was 3.0 days. The majority (95.1%) of patients was discharged directly home with or without support services, and over half of the sample (54.4%) had no documented medical comorbidities.

Table 1

Cohort demographics and clinical characteristics (n=8929 unless noted)

Overall, responses to the survey were quite positive, with 36.8% of respondents rating their overall experience as 10 out of 10 (best possible). A high proportion of patients (83.3%) rated their care as 8 out of 10 or higher. In all, 1870 patients (20.9%) reported at least one open-ended complaint about the healthcare services they received. The most frequent complaint themes were nurses (n=491; 5.5% of cohort), medications (n=219; 2.5%) and food (n=193; 2.2%). In total, 277 patients (3.1% of cohort) voiced multiple complaints that were categorised into multiple themes. And 830 patients (9.3%) had a complaint that could not be otherwise classified into a theme.

Table 2 displays the results of the univariate logistic regression analyses. Independently, an increased odds of reporting at least one healthcare complaint was associated with younger age (18–69 years old compared with 70 and older), being born in Canada (OR 1.44, 95% CI 1.23 to 1.68, p<0.0001), and having no documented medical comorbidities (OR 1.22, 95% CI 1.10 to 1.35, p=0.0002). Conversely, a decreased odds of reporting at least one complaint was observed for males (OR 0.74, 95% CI 0.66 to 0.83, p<0.0001), lower levels of education (postsecondary or lower vs postgraduate level), having an urgent admission (OR 0.83, 95% CI 0.74 to 0.92, p<0.0001), a lower RIW (OR 0.81, 95% CI 0.73 to 0.90, p=0.0001) and having a length of hospital stay of <3 days (OR 0.87, 95% CI 0.77 to 0.97, p=0.0122).

Table 2

Logistic regression results (odds of having at least one complaint)

The results of the overall experience comparison of patients with a complaint versus those without one are shown in table 3. Under each complaint theme header, we have included examples of what is meant by each complaint theme. Patients with at least one healthcare complaint had a significantly lower overall mean experience score (72.9 vs 89.7; p<0.0001). This directional relationship was seen across all complaint themes, with complaints regarding roommates (eg, loud, mixed gender in same room) having the greatest mean difference between groups (19.8 points: 66.5 for complaint group vs 86.3 for no complaint; p<0.0001). The smallest mean difference observed was for the ‘food’ theme (10.2 points: 76.2 vs 86.4; p<0.0001). Patients with complaints in multiple themes displayed an additive effect upon their experience scores. A mean decrease of 23.4 points was seen among these patients compared with the rest of the cohort (63.5 vs 86.9; p<0.0001).

Table 3

Mean overall inpatient scores by nature of complaint

Table 4a and b shows the results pertaining to the ‘action taken’ and ‘satisfaction with handling of complaint’ questions, respectively. Patients with a single complaint who notified the organisation had a lower mean overall experience score compared with those who spoke directly with their healthcare provider or did nothing (68.0 vs 75.1 and 72.3; p=0.0007 for global comparison, p<0.05 for post hoc comparisons). No difference was seen among patients with multiple complaints based on the action taken. Patients with a single complaint who were satisfied with the handling of their complaint had a higher mean experience score compared with those with a neutral or dissatisfied rating (78.7 vs 73.2 and 67.0; p<0.0001 for global comparison, p<0.05 for post hoc comparisons).

Table 4

Mean overall inpatient scores, by action taken and satisfaction of complaint handling

Discussion

The present study is, to our knowledge, the first to examine the relationship between qualitative complaints and overall patient experience scores using an H-CAHPS-derived instrument. In outlining which complaint themes may be most detrimental to overall experience, our findings present methodology and results that may be useful to hospital administrators seeking to improve the experience of their inpatients. The results of our analyses showed that certain demographics (age, sex, education level, being born in Canada) and clinical characteristics (admission type, LOS, RIW, comorbidity profile) were associated with complaints. With respect to our second study objective, we showed that complaints relative to roommates were associated with the greatest decrease in overall experience scores compared with other complaint themes. Multiple complaints demonstrated an additive effect upon overall experience, with multiple complaints associated with larger decreases in experience scores. Patients with single complaints who notified the organisation had lower overall mean experience scores (compared with those who spoke directly with their healthcare provider or did nothing). Patients who were dissatisfied with the organisation's handling of their complaint also tended to have lower overall scores. These findings, however, were not seen among patients with multiple complaints.

Our findings relative to the demographic and clinical profile of patients who lodged complaints are similar, in part, to previous investigations. In one investigation at four major western Canadian hospitals, Kline et al11 found that females, as well as patients with a higher case complexity (eg, higher RIW), were more likely to have a complaint. The present study, performed nearly a decade later, replicated these findings. Additionally, a previous investigation by our group showed that gender, age group, level of education and clinical factors such as discharge location and LOS were significantly associated with overall experience scores.23

In examining which complaint themes were associated with the greatest decrease in overall experience scores, we were surprised by our findings. Based on our previous experiences with the data, we believed that food complaints and medication complaints (eg, wrong medication given, medication given at wrong time) would feature prominently. This, however, was not the case. Complaints within the ‘roommates’ theme (eg, loud roommates, rooms with mixed genders) were associated with the greatest decreases in overall experience scores. Ward allocation (eg, mixed wards) has been previously raised as a cause for complaints in hospitals,12 ,24 and our findings present clear evidence for improving accommodation arrangements in our own organisation. Complaints regarding hospital cleanliness and nursing were also associated with large decreases in overall experience scores. On a larger scale, communication with nurses has been found to be the domain that is most correlated with overall H-CAHPS scores.8 Although this relationship has not been observed for hospital cleanliness, complaints relating to poor sanitation are common among patients.11–13 ,24 ,25 Included in table 3 are examples of what is encompassed by each complaint theme. We acknowledge that this list is not exhaustive. Had we decided a priori to use a comprehensive taxonomy, such as the ones proposed by Montini et al12 and Reader et al,13 we may have been able to better classify the complaints classified as ‘other’. Upon inspection, a number of complaints classified in this category pertained to hospital access and accommodation, for example, distance needed to walk, parking cost and lack of television/Wi-Fi availability. These have been cited as common complaints in these previous studies.

We observed that patients who addressed their complaint with hospital staff had higher overall experience scores compared with those who notified our organisation or did nothing. This suggests that healthcare providers may provide the best opportunity to remediate patient complaints as this can be done in real time. By providing immediate validation and corrective action in a systemic fashion (ie, not on a case-by-case basis), it may be possible to preserve higher ratings of patient experience. Given that our survey was conducted post discharge, this finding may also suggest that the feeling of closure on the part of the patient may persist long after the concern has been addressed by the provider. This finding may be used to support education of hospital staff around dealing with patient complaints.

There are key strengths to the study. Most notably, as our organisation is the sole provider of in-hospital care in our province, we are able to use administrative data sets to track all patients at multiple locations across the continuum of care (inpatient hospitals, emergency departments, walk-in clinics). This is a significant advantage. Second, we have documented patient complaints in a standard fashion, at the same time as the administration of a valid measure of patient experience (H-CAHPS). In capturing these concurrently, we are also able to forward patients with complaints to our organisation's Patient Relations department, allowing for closure on the part of the patient.

In contrast, our study is not without limitations. First, due to the cross-sectional study design, we are unable to infer any causal link between patient complaints and corresponding patient experience scores. Additionally, the issue of common method variance may be present, given that the dependent and independent variables originated from the same data source. We do not feel, however, that this should detract from our findings. Further study examining patient experience in a longitudinal fashion (eg, over multiple points in time) would allow for a determination of causality.

Second, during the data linkage process, surveys were matched with the corresponding inpatient record from the DAD. Approximately 4% of cases were unable to be matched. Third, despite having standard training and guidance, we acknowledge that categorising complaints into themes based on certain keywords is a subjective process. In the subset of 200 respondents (273 possible complaints), we achieved 78.4% agreement between our two raters. Additionally, we used an ad hoc method for classifying complaints. We acknowledge that our results may have differed if we had used an established taxonomy.12 ,13

An additional limitation is that H-CAHPS, like other experience surveys, are unable to measure patient expectations. This is believed to be more of an issue with elective admissions. For example, in the case of a patient admitted to hospital for elective joint replacement, the given patient has for the most part obtained a preoperative teaching session regarding the procedure, LOS and progression leading to hospital discharge. This, coupled with the fact that many patients have had an opportunity to speak with family members or friends who have had the same procedure, allows the patient to have a preconceived notion as to what to expect during the subsequent hospital stay. Should the patient have a setback in-hospital (eg, postsurgical complication, infection), this may impact the opinion of their experience, despite receiving exemplary care.

As we used a telephone method, our results may not be generalisable to paper-based administration. Previous investigations of H-CAHPS have shown that telephone respondents typically have more favourable inpatient experiences compared with their counterparts who receive a mailed questionnaire.26–29 The presence of social desirability between respondents and telephone interviewers is believed to be the chief contributor to this. To account for this, the Centers for Medicare and Medicaid Services currently applies a survey mode adjustment to their data.30 Lastly, it is reasonable to assume that our methodology did not capture a sample of ‘extreme’ complaints. For example, a patient may be so disenchanted with their hospital stay that they may refuse to answer our survey. Over the study period, the refusal rate was approximately 5%. However, we did not capture the reason for this refusal on the part of potential respondents.

In conclusion, our study results shed light upon the different types of qualitative complaints resulting from inpatient hospital stays in our jurisdiction as well as the characteristics of patients who have lodged them. The corresponding association of each complaint theme with overall experience scores, as documented using a validated survey tool, was also assessed. To our knowledge, this is the first study to do so in a Canadian context—one that employs a universal healthcare system. This rich data set has generated several potential areas for quality improvement within our own organisation. Through replication of our study methodology, other healthcare organisations wishing to determine the association of open-ended complaints with overall experience (eg, H-CAHPS) scores may do so.

Acknowledgments

The authors wish to acknowledge the contributions of the team of Health Research Interviewers from Primary Data Support, Analytics (DIMR); Alberta Health Services. The authors also thank the patients who graciously took the time to participate in the survey.

References

View Abstract

Footnotes

  • Contributors All authors have contributed to the conception of the study, data analysis, writing of the manuscript and have approved the contents of the submitted documents.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.