Patient-reported service quality on a medicine unit

Abstract

Purpose. Service quality, defined as patients’ self-reported experience of care, is used as a metric for evaluating quality. Most studies rely on retrospective consumer surveys rather then more intensive data collection methods, possibly underestimating the incidence of service quality incidents.

Subjects and Methods. The objective of the study was to characterize patient-reported service quality deficiencies on a general medicine unit. We studied a cohort of 228 adult inpatients at a Boston teaching hospital. Investigators reviewed medical records and interviewed patients during the hospitalization and by telephone after discharge. Physician investigators classified patients’ incident reports. We calculated the rate of service incidents, characterized incident types, and used multivariable Poisson and logistic regression models to examine factors associated with patient reporting and overall rating of the hospitalization.

Results. Eighty-eight (38.6%) of 228 patients experienced 157 service quality incidents during the admission, for a rate of 68.9 incidents per 100 admissions. The most common service quality problems involved waits and delays (n = 45), problems with communication between staff and patients (n = 36), and environmental issues and amenities (n = 35). In the multivariable analysis, men (IRR 1.6, 95% CI 1.1–2.2), patients covered by hospitalists (1.5, 1.1–2.2), and patients with more medication allergies (1.1 per allergy, 1.1–1.2) reported more service incidents; patients with Medicaid or free care reported fewer (0.5, 0.3–0.9). Patients with service quality incidents were more likely to describe the hospitalization as other than excellent (adjusted OR 1.8 per incident, 95% CI 1.3–2.5).

Conclusion. Service quality deficiencies are common among medical inpatients, and are strongly associated with patients’ dissatisfaction with the hospitalization

Service quality, defined as patients’ self-reported experience of care, is a useful metric for evaluating health care quality [1,2]. In contrast to technical measures of quality used to assess clinical processes (e.g. beta blocker after myocardial infarction) or outcomes (e.g. surgical site infections), service quality is measured by asking patients about their experience. Consumer surveys characterize global satisfaction and rate specific dimensions of care, such as the perceived adequacy of communication, physical comfort, and family participation in medical decision-making [3–9]. Understanding patients’ experience of care may inform consumers’ selection of hospitals and clinics, health leaders’ assessments of organizational performance, and quality improvement professionals’ choice of improvement targets.

Service quality deficiencies in acute care hospitals are distressingly common, affecting up to 45% of hospitalizations [8]. However, these results may not accurately assess service quality. Surveys are often administered by mail after discharge, in part to allay patients’ concerns that a negative response could affect care. This recall bias could result in an underestimate of service deficiencies. On the other hand, if patients with poor experiences are motivated to reply, then a response bias could overestimate service deficiencies. In addition, use of structured survey instruments may restrict patient responses to categories provided by survey designers.

Accordingly, the objective of this study was to understand the incidence and types of patient-reported service quality deficiencies experienced by a cohort of medical in-patients. We hypothesized that service quality deficiencies would be common, exceeding reported rates. We also sought to analyse factors associated with service quality deficiencies and with patients’ overall rating of the hospitalization to identify improvement targets.

Methods

Study subjects

We studied adult in-patients admitted to a 40-bed medical unit at a Boston teaching hospital from January through April 2003. The unit served general medical patients, a geriatric service, and overflow patients from oncology, surgery, and obstetrics-gynecology units. The hospital’s institutional review board approved the study protocol.

An investigator contacted patients on the first weekday after their admission to explain the study and to request patients’ (or self-identified proxies’) written informed consent. Spanish and Russian interpreters were available as needed, and written materials were provided in English, Spanish, or Russian translation. Patients were excluded if they could not communicate because of cognitive or behavioral problems, if they were unavailable for interview or in isolation, if they had no proxy, or if the patient or proxy spoke a language other than English, Spanish, or Russian. Patients with multiple hospitalizations were enrolled at most once.

We excluded 123 (31.8%) of 387 potentially eligible patients. The most common reasons for exclusion were patient in isolation (n = 45), patient unavailable for interview (n = 2), and cognitive impairment (n = 18). Fewer excluded than eligible patients were non-white (12.1 versus 20.2%; P = 0.04) and excluded patients were less likely to have family or friends present at the time of enrollment (2.6 versus 15.0%; P < 0.001). Excluded and eligible patients were similar in age, gender, ethnicity, need for interpreter, insurance, attending type, length of stay, and number of drug allergies, medications, and diagnoses.

Of the remaining 264 patients, 229 (86.7%) patients or proxies provided written informed consent to participate. Consented and declined patients were distinguished only by a smaller percent of non-English speakers among those who consented (5.7 versus 14.3%; P = 0.06). One consented patient was discharged without subsequent interviews and was excluded from further analyses, yielding a final cohort of 228 patients.

Patient interviews

An investigator conducted brief (5 minutes) interviews with patients or their proxies 2–3 times per week throughout the hospitalization and by telephone 10 days after discharge. She asked respondents three questions [9,10]:

1. Do you believe that there have been any problems with your care during this hospitalization?

2. Do you believe that you were hurt or stayed in the hospital longer than necessary because of problems with your care?

3. Do you believe that anyone made a mistake that affected your care during this hospitalization?

Patients were encouraged to offer additional open-ended comments about their care. In addition, patients were asked during the post-discharge interview to rate the overall quality of their hospitalization on a 5-item scale (from poor to excellent). Verbatim responses were recorded for coding and analysis.

Medical record reviews and data coding

Investigators abstracted demographic and medical history information from patients’ medical records including medication and drug allergy information and medical history.

Two of four board certified internists, blinded to patient identification, independently coded each patient report [10–13]. Reviewers used patient reports to classify four types of events: adverse events (defined as injuries due to medical care), close call errors, medical mistakes without significant risk of harm, and service quality incidents. We defined service quality as patients’ self-reported experience of poor care in dimensions other than technical attributes related to diagnosis and treatment [14–18]. This study addresses service quality incidents; an analysis of patient-reported adverse events and errors was published separately [19].

Reviewers used patient reports to classify the parties involved in or responsible for the reported event. They also classified patients’ verbatim reports by type of service quality problem, using categories informed by the Picker Institute and our previous research [9–11]. The categories included lack of respect for patient preferences, waits and delays, environmental issues and amenities, poor communication, poor coordination of care, and poor interpersonal skills and unprofessional behavior. Reviewers excluded nine reports where the description was confusing or incomplete and reviewers were uncertain that the event occurred as reported.

Differences were resolved by discussion. Interrater reliability of preconsensus coding was assessed with the kappa statistic to correct for chance association; weighted kappa was used for ordinal measures. Interrater agreement was good for reviewers’ certainty that the adverse event, error, or service quality incident had occurred as reported (0.83). Agreement was excellent for type of report (0.97), involved party (0.90), and type of service quality problem (0.99).

Data analysis

We calculated the number, rate, and type of patient-reported and physician reviewer-classified service quality incidents and the distribution of responsible parties. We analysed patient characteristics, stratified by the presence of at least one service quality incident, using the chi-squared statistic for categorical and Wilcoxon rank–sum test for ordinal variables. We created a step-wise multivariable Poisson regression model with forward selection (P < 0.20) to examine the association between the number of service quality incidents and factors that had a statistically significant relationship with service quality in the bivariable analysis. We adjusted the analysis for length of stay to correct for the increased risk of exposure to service incidents with extended hospitalizations. We reported incident rate ratios (IRR) and their 95% confidence intervals (CI).

We also analysed the type and number of service quality incidents, stratified by patients’ overall rating of the admission excellent or (not) using the chi-squared statistic or rank–sum test, as appropriate. We created a step-wise logistic regression model with forward selection (P < 0.20) to identify factors associated with patients’ overall rating of the admission. The model included a binary variable (present or not) for each type of service incident that had a statistically significant association with the overall rating of the admission in the univariable analysis, the total number of service incidents, and the presence of any adverse event or close call error. We reported odds ratios (OR) and CI for this model. All analyses used Stata 7.0 (StataCorp, College Station, TX, USA).

Results

Study cohort

Table 1 summarizes the characteristics of the study cohort. We studied 228 patients with a mean age of 63 years. Twenty-one percent were non-white, 4% Hispanic or Latino, 37% male, and 5% non-English speakers. Most of the patients were Medicare recipients. Patients had an average of 1.2 drug allergies, took 7.1 medications, had 2.4 diagnoses, and had an average length of stay of 4.4 days. A friend or family member was present at the time of enrollment for 16% of patients.

Patient reports

Altogether, 228 patients completed 338 in-hospital and 190 post-discharge (528 total) interviews (mean 2.3 per person; range 1–6) and generated 310 distinct reports (mean 1.4 per person; range 0–8).

Physicians classified 157 reports as service quality incidents. The rate of service quality incidents was 157 incidents per 228 patients at risk or 68.9 per 100 admissions. Eighty-eight (38.6%) of 228 patients experienced at least one service quality incident (range 1–6) and 40 (17.5%) experienced two or more.

Of the remaining reports, physician reviewers classified 75 as favorable assessments of clinical care. Another 62 reports included events that the physicians judged to be adverse events, close calls, and medical mistakes without significant risk; these events are described elsewhere [19]. In the remaining 16 reports, patients were uncertain whether problems or mistakes occurred during the hospitalization.

Table 1 indicates that service quality incidents were more often reported by patients with Medicare and less often among those with Medicaid and free care and among non-English speakers. Service quality incidents were also more common among patients with greater lengths of stay and more prescription medications.

In the multivariable analysis, men (IRR = 1.6; 95% CI = 1.1–2.2), patients with hospitalist coverage (IRR = 1.5; 95% CI = 1.1–2.2), and patients with more medication allergies (IRR = 1.1 for each additional allergy; 95% CI = 1.1–1.2) reported more service incidents. Patients with Medicaid or free care reported fewer incidents (IRR = 0.5; 95% CI = 0.3–0.9).

Types of service quality incidents

Table 2 summarizes the distribution of types of service quality incidents. The most common service incidents involved waits and delays, poor communication between caregivers and patients, and problems with the environment and amenities.

Waits and delays

Patients who reported waits and delays most often described an inadequate response to calls for nursing assistance. They described the nurse call button as ‘useless’ and ‘a waste of my time’. Patients also reported delays related to toileting. For example, ‘My nurse took my diaper off a while ago and said that she would be right back. She never came back and I wet my bed’. They reported delays in the emergency unit and waits of up to 18 hours for transfer to a medical bed.

Poor communication

Patients reported confusion or uncertainty about various aspects of their hospitalization. Many voiced frustration regarding medications that clinicians stopped, started, or changed without explanation. ‘I have been given so many medications in the last 48 hours and I don’t even know what I have been getting’. Another patient reported that ‘They took away my pain medications without telling me.... I don’t care so much that they took it away, I just want to know why’.

Patients reported a lack of information about the hospitalization. According to one patient, ‘I don’t know exactly what they are doing to me’. Another said, ‘I don’t have the foggiest idea of what is going on’. Patients perceived that they were ‘neglected’, that clinicians ‘don’t listen’, and that they were ‘not being heard’. In one incident, information was intentionally withheld from the patient. ‘I had a procedure done today and the doctor kept information from me...I am not a child and I deserve to know’.

Environmental issues and amenities

Patients complained about the environment of care. They reported that no one had cleaned their room or changed their bed or gown for several days and that the facility was dirty: ‘The hospital is filthy. You can catch things there that you would not catch on the street’. They noted unpleasant smells (‘My next door neighbor’s commode was not changed for 30 minutes. The room was a cloud of odor.’) and unsanitary conditions (‘When I was first admitted my sheets had vomit on them. The nurse tried to tell me that it was a stain, but it was fresh vomit and the sheets were all disheveled.’). Food-related complaints were also common. According to one patient, ‘I am a police officer and I have been in and out of prison. The food in prison is better’. Meals were late or never arrived and patients received their neighbors’ meals.

Poor coordination of care

Patients readily recognized inconsistencies with communication and teamwork among clinicians, including imperfect handoffs among nurses at the change of shift. They were confused by the ‘chain of command’ among residents, fellows, and attending physicians. According to one patient, ‘I don’t think that the left hand always knows what the right hand is doing’. Another patient, whose physician was unaware of the admission, asked ‘Who gives orders around here?’

Coordination problems were most apparent when patients received conflicting instructions. In one case, ‘One doctor told me to walk around and the other doctor told me to stay in bed’. In another case, ‘One doctor told me that I was going to be on clear liquids. Another doctor came in and told me that I cannot have clear liquids...They are just not on the same page and do not collaborate’.

Poor interpersonal skills and unprofessional behavior

Patients observed staff members’ lack of respect for patients, arrogance, tactlessness, unhelpfulness, and rude behavior. For example, one patient said ‘I feel like whenever the nurses leave my room, they giggle and they have rolled their eyes at me and at each other’.

Involved parties

Table 3 summarizes the parties that reviewers judged to have been most closely involved in each service quality incident. Clinicians were identified most frequently: physicians accounted for 72 (45.9%) and nurses for 62 (39.5%) of incidents. In 52 (33.1%) incidents, the involved party was not identified.

Service quality and overall assessment of quality

Eighty-two percent (n = 187) of patients answered a question about their overall assessment of hospital quality during the post-discharge telephone survey. Among respondents, 126 (67%) described the quality of care during the last admission as ‘excellent’ and 51 (27%) described quality as ‘good’. Table 4 summarizes the type of service quality incident, stratified by patients’ rating of the hospitalization.

Compared with patients who rated the hospitalization as excellent, those with non-excellent hospitalizations were more likely to report incidents involving waits and delays, poor communication, poor interpersonal skills or unprofessional behavior, and lack of respect for patient needs and preferences. Compared with patients who rated the hospitalization as excellent, those with non-excellent hospitalizations were more than twice as likely to report at least one service incident (26.2 versus 57.4%, P < .001). There was no statistically significant association between hospital rating and sociodemographic, administrative, or clinical variables, or the presence of adverse events and close call errors.

In the multivariable analysis, rating of the admission as other than excellent was statistically associated only with the number of service incidents that occurred during the admission (OR = 1.8 for each additional incident; 95% CI = 1.3–2.5).

Discussion

In this in-patient population, nearly four in 10 patients reported at least one service quality incident during their admission. Taking into account patients with multiple incidents, the overall rate was 69 per 100 admissions. The most commonly reported incidents were waits and delays, poor communication, and problems with the environment and amenities. Reporting a service quality incident was inversely associated with patients’ overall experience of care: reporting even a single incident nearly doubled the odds of rating the admission as non-excellent care.

Despite service problems, about two-thirds of patients rated the overall quality of care as excellent, suggesting in turn that patients judge service quality in the context of their entire hospital experience. The overall assessment is likely informed by factors that we did not measure, such as supportive relationships, positive service experiences, and favorable clinical outcomes. Perhaps we should ask patients how they balance things that go wrong with things that go right [20].

Although the rate of service quality incidents in our study was distressingly high, the prevalence of service problems is compatible with previous research. For example, Cleary reported that up to 44.9% of patients identified problems with at least one aspect of care in a telephone survey of 6455 adults discharged from 62 US hospitals [8]. In a related, mail survey of 62925 adult patients discharged from 301 acute care hospitals in Europe and the United States [9], patients frequently identified problems with continuity and transition of care (28.4–45.1%), information and education (16.7–28.7%), and emotional support (14.7–27.1%).

Although previous investigators identified problems with discharge and transitions of care, our respondents reported waits and delays, poor communication, and problems with the environment and amenities. This discrepancy is likely related to our collection of real-time reports enriched in the day-to-day difficulties that nettled patients. The greater number of problems with transitions in care in retrospective studies suggests that these issues increase in salience after discharge relative to the memory of daily aggravations experienced during hospitalization.

Previous studies found that service problems were more often reported by younger patients, women, non-whites, patients with low educational attainment, and those with poor self-reported health status [6,8,21,22]. We found no relationship between reporting and age or race, that men were more likely to report than women, and that Medicaid or free care insurance (proxies for low socioeconomic status) was inversely related to the number of service incidents. We speculated that it would be easier for older patients, non-whites, and poorer patients to respond, perhaps critically, to in-person interviews than to retrospective written surveys, but we did not observe this effect. Increased reporting by patients with more drug allergies may demonstrate a vulnerability to mishaps, or that these patients are meticulous about reporting all types of events. Patients’ willingness to report service problems when a hospitalist rather than primary care physician served as inpatient attending may reflect the duration of the relationship.

Unfortunately, comparisons across studies are difficult to interpret. Earlier investigators did not aggregate responses into an overall problem score. Our open-ended format may detect fewer problems than a structured instrument that assayed various dimensions of care. Telephone and written surveys are subject to recall and non-response bias, whereas in-person interviews may bias respondents’ assessments of care in accordance with the interviewee’s perception of the interviewer’s expectation. Face-to-face interviews and open-ended responses may also have elicited events that were excluded in a highly structured survey.

This study is subject to several limitations. Its generalizability is limited by its sample size and single study site. The study did not have adequate power to detect an association between sociodemographic and clinical factors and the overall assessment of quality. In addition, reviewers may have misunderstood patients’ verbatim responses because some comments were difficult to interpret. Patients who were excluded because of inability to communicate or because they were isolated may have been at increased risk of service incidents, in fact, underestimating the incidence of service deficiencies. Our 87% participation rate and high rate of post-discharge follow-up (83%) minimize the risk of selection bias. However, if patients with negative assessments were more likely than others to complete the post-discharge survey, we may have overestimated the number of service incidents and underestimated overall quality. Additional studies are needed to corroborate the results reported here, to analyse the relationship between patient-reported quality and ‘hard’ clinical outcomes [23–26], and to examine how patients’ overall assessment of care is informed by factors that we did not measure, such as supportive relationships, positive service experiences, and favorable clinical outcomes.

In conclusion, service quality deficiencies are ubiquitous among medical in-patients. They are so common that they appear to be the norm. Despite the prevalence of these events, patients readily recognize service incidents as problems in the delivery of care. Patients expect more from health care than we reliably deliver. We can, and should, do better.

References

Author notes

1Center for Patient Safety, Dana-Farber Cancer Institute, 2Division of General Medicine and Primary Care, Beth Israel Deaconess Medical Center, Boston, MA, USA, 3Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel, 4Zix Corporation, Dallas, TX, 5Division of General Internal Medicine, Brigham and Women’s Hospital, and 6Harvard Medical School, Boston, MA, USA