Article Text
PDF
Abstract
Objective Patient-centred care has become a priority in many countries. It is unknown whether current tools capture aspects of care patients and their surrogates consider important. We investigated whether online narrative reviews from patients and surrogates reflect domains in the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) and we described additional potential domains.
Design We used thematic analysis to assess online narrative reviews for reference to HCAHPS domains and salient non-HCAHPS domains and compared results by reviewer type (patient vs surrogate).
Setting We identified hospitals for review from the American Hospital Association database using a stratified random sampling approach. This approach ensured inclusion of reviews of a diverse set of hospitals. We searched online in February 2013 for narrative reviews from any source for each hospital.
Participants We included up to two narrative reviews for each hospital. Exclusions: Outpatient or emergency department reviews, reviews from self-identified hospital employees, or reviews of <10 words.
Results 50.0% (n=122) of reviews (N=244) were from patients and 38.1% (n=93) from friends or family members. Only 57.0% (n=139) of reviews mentioned any HCAHPS domain. Additional salient domains were: Financing, including unexpected out-of-pocket costs and difficult interactions with billing departments; system-centred care; and perceptions of safety. These domains were mentioned in 51.2% (n=125) of reviews. Friends and family members commented on perceptions of safety more frequently than patients.
Conclusions A substantial proportion of consumer reviews do not mention HCAHPS domains. Surrogates appear to observe care differently than patients, particularly around safety.
- Patient-centred care
- Qualitative research
- Hospital medicine
- Quality improvement
- Patient safety
Statistics from Altmetric.com
Background
Countries are increasingly including patient experience measures in national public reporting or payment programmes.1–6 In the USA, the Center for Medicaid and Medicare Services (CMS) uses the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) patient experience survey in payment calculations.7 ,8 Hence, health system leaders are increasingly asking clinicians to review and respond to these data.7 ,9 While HCAHPS covers eight domains (eg, nurse and doctor communication, pain control and discharge information), the measure was developed in 2003–2004 and it is unknown whether those domains reflect the aspects of care that are most important to contemporary patients. Since performance measurement tied to public reporting or incentives may direct attention away from other aspects of care,10–12 it is critical that measures of patient experience provide a comprehensive assessment of the aspects of care most salient to patients and their surrogates.
Narrative online hospital reviews provide a relatively popular and novel source of quality information from and for patients,13–18 and represent what patients value and observe about care when not offered structured prompts. Furthermore, the perspectives of friends and family members are increasingly being recognised as important to patient outcomes,19–21 but are not captured in patient-only surveys such as HCAHPS. In addition, prior research suggests that online numerical ratings accompanying narrative reviews are associated with patient outcomes such as mortality, readmissions and infection rates.1 ,22 ,23 While further investigation is warranted, this suggests that the patient and friend and family member reviewers may observe care that is related to these outcomes. Lastly, these online data provide access to assessments of a range of geographically and organisationally diverse hospital types.
Prior analyses have described themes in online reviews of urban outpatient providers,24 ,25 emergency department (ED) visits,26 in hospital-related microblogs on Twitter27 and in preprompted narrative reviews of National Health Service (NHS) hospitals in England.28 There are no published data assessing whether unprompted consumer criteria for hospital quality are similar to domains in structured surveys; nor data assessing narratives from friends and families, separate from patients. Lastly, the original HCAHPS developers chose survey domains based on family and patient focus groups over a decade ago29 ,30 and new domains of importance to patients, friends and families may have emerged with subsequent healthcare system changes. We performed this study to meet the following research objectives: to assess whether the HCAHPS domains are salient to contemporary consumers, whether there are additional domains that might be emerging in a changing healthcare environment, and whether surrogate perspectives may differ from patient perspectives.
Methods
Overview
To achieve the goals of the study we qualitatively analysed a group of online narrative reviews from a diverse sample of US hospitals. Online reviewers numerically score hospitals and provide accompanying narratives. Our underlying premise is that the narrative stories are told for a reason and provide context and justification for their accompanying quantitative score. These narratives therefore provide a data source to understand which domains online reviewers use when judging hospital quality.
In order to analyse the narratives for already-defined HCAHPS domains as well as domains that were not included in HCAHPS but which may be important to consumers, we chose the qualitative method of thematic analysis. This is a flexible approach described as: “searching across a data set—be that a number of interviews or focus groups, or a range of texts—to find repeated patterns of meaning.”31 Key themes are identified through an assessment of prevalence of the themes within the text as well as salience to the research question at hand.31 In addition to our qualitative analysis, we include a quantitative approach to check our assessment of theme prevalence as well as to check our impression that themes differed by reviewer type.
Sample
To create our sample of narrative reviews, we defined a national group of hospitals from the American Hospital Association database, which includes hospital characteristics for all acute care hospitals in the USA (n=6335). We chose to limit our sample to the USA given the heterogeneity of healthcare environments worldwide and because HCAHPS was developed originally in the US context. We stratified the group using the variables the Agency for Healthcare Research and Quality HealthCare Utilization Project uses to create the Nationwide Inpatient Sample, a nationally representative administrative data set: region of the country, urban and rural location, teaching status, and size.32 We randomly sampled hospitals within these strata.
We searched for reviews on the sampled hospitals. In prior work, we found that certain hospital characteristics (high bed number, Western region, and private not-for-profit status) were associated with having more online narrative reviews, with some hospitals with >100 reviews.22 In order to avoid oversampling reviews from hospitals with these characteristics we took two reviews from each hospital we sampled, which we found in prior work to be the median number of reviews per hospital.22 Based on a pilot analysis of 50 reviews33 we anticipated that approximately 250 inpatient reviews would result in thematic saturation.
In order to define our search strategy for online reviews we tested search terms and search engines to optimise our results. For each new hospital search we cleared our prior search history, to prevent this from affecting results for subsequent searches. We used an iterative approach to sample hospital reviews. From among the reviews found within the first five pages of search results, we used the two most recent reviews because people have changed their willingness to review healthcare providers within the last few years.34 ,35 We excluded: ED reviews, because the ED and inpatient setting differ substantially; review sites with structured items and unstructured narrative reviews, since structured items suggest domains to reviewers; and reviews that were less than 10 words long. See online supplementary appendix for additional details on the above processes.
Analysis
As noted above, we used thematic analysis31 and assumed that reviewers wrote with the intention to convey their experience.36 For instance, a review describing a medical error indicates that the reviewer uses safety as a quality criterion, regardless of whether the event was an error according to accepted medical standards. In order to assess for delineated HCAHPS domains as well as additional themes, we read iteratively and openly and created a shared codebook for domains relevant to the research question.31 The thematic analysis approach allows for both the approaches we use here: reading and coding data based on existing literature; as well as reading and coding for themes that emerge from the text that are salient to the research question. This flexible analytical approach focuses on key themes relevant to the research question while still maintaining an openness to new concepts in the data.31
The research team met to discuss the HCAHPS domains, reviewing the survey items that comprise each domain. Two coders (NSB and RAP) created a set of codes and memos representing the HCAHPS domains. We created an additional code ‘General communication’ for global or not person-specific comments about communication (eg, ‘the staff was very comforting in their words’), since these could not be coded as HCAHPS nurse-specific or doctor-specific communication domains. Comments about communication with specific non-clinicians (eg, security guards) were not included in this code. We also developed codes describing reviewer types, diagnostic category, whether paediatric or non-paediatric, and positive or negative valence. See table 1 for coded categories.
Review characteristics (n=244)
Before analysing study data, the primary coders (NSB and RAP) coded several sets of older reviews from an online review site. The coders then met to discuss areas of concurrence and discrepancy to develop a consensus approach before additional analysis. The coders then analysed the primary data set separately by using the HCAHPS domains codes and developing new codes for additional domains. The coders met regularly to review coding and reconcile differences.
Others experienced in qualitative analysis (AL, LEG, GAL, RAD) read all reviews and refined the codebook in an iterative fashion. We maintained methodological rigour using this triangulation, with multiple analysts and an interdisciplinary team.37 Clinical investigators’ backgrounds were obstetric nursing (AL), pulmonary critical care (RAD), internal medicine (LEG and GAL) and paediatrics (NSB). RAP provided a non-clinician, consumer interpretation. Each clinical researcher had been hospitalised personally and/or had at least one close family member hospitalised, for obstetric, medical or surgical admissions and intensive care unit admissions.
We generated descriptive statistics of the consumer reviewers’ characteristics and the proportions of reviews mentioning each HCAHPS domain. We used χ2 analysis, or Fisher's exact test for cell sizes <10, to assess relationships between specific domains and reviewer type.
No identifiable patients were included. We report our findings in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) qualitative research reporting guidelines (see online appendix for COREQ checklist).38 ,39
Results
Review characteristics
Our sampling strategy resulted in a total of 244 reviews within our sampling strata across 193 hospitals (3% of the hospital sampling universe, with balanced groups across strata). In the iterative sampling process, 38 hospitals had no reviews and were replaced with another hospital in the same stratum. Investigators agreed that we reached thematic saturation after reviewing the corpus of reviews. The websites that were the most common sources of reviews were Yelp.com (32.0%, n=78), plus.Google.com (18.9%, n=46) and local.Yahoo.com (16.4%, n=40).
Half of reviews were from patients and 38.1% from friends or family members; in the remainder, we could not determine whether the reviewer was a patient or friend or family member (table 1). The most commonly described diagnoses were in medicine and obstetrics; however, almost half of reviews did not specify the reason for admission. Similar proportions of reviews were negative (48.0%) and positive (45.9%), with few mixed reviews (6.2%). The average star ratings (which were all out of 5 stars) were moderately positive for patients (3.1, SD: 1.8) and friends or family members (3.2, SD: 1.8). Average star ratings differed by service (table 1), as has been noted in prior work on patient experience ratings.40
HCAHPS domains
Fifty-seven per cent of reviews mentioned an HCAHPS domain (table 2). The most frequently mentioned HCAHPS domains were communication with nurses (23.8%) and with doctors (26.6%). Few reviews mentioned pain control (7.4%), discharge information given (7.4%), or communication about medicines before administration (4.9%). Including ‘General communication’ increased the percentage of reviews mentioning any HCAHPS domains by 6.5%, with the remaining 36.5% not mentioning any.
Domains mentioned by reviewer type
Non-HCAHPS domains of patient experience
In contrast, 51.2% of reviews contained themes that are not included in HCAHPS. We report key themes that were salient to the research goal—to contribute to the literature on understanding patient-centred care in order to improve care systems. We defined key themes as those that at least 5% of reviews mentioned (for a minimum definition of prevalence)31 and that were potentially actionable (for additional themes see online supplementary appendix). These were: Financing, including unexpected costs and difficult interactions with billing departments; System-centred care; and Perceptions of safety. We describe the themes below, and provide illustrative quotations. Friends and family members were more likely to comment on perceptions of safety (n=49, 52.7%) than were patients (n=43, 35.0%), without differences in other themes by reviewer type (Financing p=0.17, System-centred care p=0.19).
Financing: unexpected costs and difficult interactions with billing departments
This theme emerged in 11.5% (n=28) of the reviews. Reviewers commented on the variation in pricing for similar services, confusion about billing practices, and the stress of dealing with billing departments and unexpected out-of-pocket costs. A call for greater price transparency stemmed from a desire to avoid unplanned costs.[…] I had a CT Scan and it was $5300. I had one done 6 months later at a hospital in St. Louis and it was $2100…same test, read by the same type of doctor. ?!?! […]
Good service, big bill on specialist. Your insurance may not cover most of it. […] Call the hospital and ask for the cost first before you commit into any specialty sessions […]
I tried to pay on my bill before my insurance made payments and they would not accept them and after insurance paid they were rude and demanding. According to them $50 a month was not enough and they wanted almost $400 a month to keep from turning me into collections […]
System-centred care
Some reviewers experienced the hospital system as disempowering, delivering system-centred, not patient-centred, care. In these comments (7.8%, n=19), reviewers describe experiences resulting from a focus on enforcing system policies, rather than a patient-centred approach.Just recently my son had a seizure. They scheduled us an EEG […] the procedure required my son to be sleep deprived so he only got 3 hours of sleep. I had never been to that hospital and […] check in was at 9:00. I got there at 9:15 […] 5 mins later she calls me up to the desk to tell me the tech performing the procedure has left to do his other appointments. I became very upset my son is only 12 and is at this point exhausted. Then the tech calls there to talk to me and tells me I was supposed to be there at 8:30 […] the procedure was not even until 9:30 and he left 5 min before that”
The quote above illustrates how poor communication and inflexibility in system policies (which expect patients to arrive early for appointments, but give mixed messages as to when, and in which arriving 15 min after check-in evokes a punitive experience—the technician physically leaves the premises, making the situation nearly irreversible) can lead to frustration for patients and families. This quote also illustrates the confusion some patients experience around expected arrival time compared with appointment time, which could convey a message to patients that their time is less important than the overall functioning of the system.
Conversely, patients who experienced the system as patient-centred, rather than system-centred, discussed how all aspects of the system (doctors, nurses, nutrition services, etc) supported the patient's needs, rather than the patient feeling that his or her needs are secondary to the system needs:[…] From the moment I stepped in the door, to the wheelchair ride at the end of the visit, I felt like the most important person there. No waiting for doctors, nurses, meds, meals, periodic checkups […]
Perceptions of safety
This was a theme in 41.4% of the reviews. Subthemes were: trust in specific providers, unsafe practices or outcomes, and observations of missed or incorrect care. In a sensitivity analysis, each of these subthemes was commented upon more frequently by friends and family members than by patients (data not shown).Trust in specific providers: At one point, our family member had inappropriate bleeding from several sites on the body which [a doctor] dismissed as unimportant. Eventually we were able to convince other people in the ICU to take care of the situation.
Unsafe practices or outcomes: My other concern has to do with […] following standard sanitary procedures for infections, masks, gloves, throwing stuff away that is needed and then fetching it out of the trash, not having dressing kits on hand…
The only thing that worries me about this place is that a lot of people seem to come back with c-diff and or infections
Observations of missed or incorrect care: My family member was mostly ignored, […] They brought [her] a tray of food 6 times, despite being told (by us!) that she was unable to eat because she was about to have testing done. […]
While additional reviews in our sample expressed a general distrust of the system, the reviews we included in this category were anchored in specific observations of care around safety or trust. This theme was mentioned more often (52.7%, n=49) in friend and family reviews compared with patient reviews (35.0%, n=43; p<0.01 for comparison).
Discussion
In this study of online narrative reviews from a national sample of US hospitals, we provide new insights regarding consumers’ unprompted perspectives on aspects of hospital quality compared with structured quality assessments. We found that while communication was mentioned often, there were some HCAHPS domains frequently missing in reviews, and that there were additional non-HCAHPS domains raised by patients, friends and family members. We found that family members and friends represented a large proportion of online reviewers. Our findings suggest that these surrogates may differ from patients in which aspects of care they use to define quality.
Strengths of this study include the sampling approach, which used reviews from all online sources, not restricting our sample to one website, and ensured a diverse sample of almost 200 hospitals. Prior studies of online reviews have used geographically limited samples of hospitals and have been restricted to only one or two sources of reviews (eg, Yelp.com, rateMDs, jameda).3 ,24 ,25 ,28 Prior studies have also not analysed themes by reviewer type (patients vs friend or family member). Lastly, most prior studies of online narrative reviews analysed reviews for physicians, not hospitals. Though Lagu et al28 studied online hospital narrative reviews, consumers were prompted first to answer structured domain items and the sample was NHS hospitals in England. Ours is the first study analysing unprompted narrative inpatient reviews in a multipayer environment, in which reviewers may be at risk of financial burden from excess medical costs.
Our findings are important for clinicians and hospital leaders for two reasons. First, a focus on HCAHPS scores, or other patient experience surveys with similar domains, may direct attention to issues that are of lesser importance to consumers, at least as judged by the rarity with which consumers mention them in reviews. Second, since there appear to be topics that are important to consumers but about which data is not collected, clinicians and hospitals risk achieving low patient experience scores without knowing that performance in unqueried domains may need to be improved. In particular, the theme ‘Perceptions of Safety’ indicates that patients and family members are a potential untapped source of information on patient safety.41 Since explanatory, detailed narratives from patients may be more actionable than quantified measures, clinicians and hospital administrators may be interested in using their content to guide quality improvement.42 Because reviews accumulate with time, providers and policy makers may find it worthwhile to monitor narrative reviews for emerging issues, relevant to individual hospitals or to the broader healthcare system.
While HCAHPS is a well validated measure of patient experience,29 ,40 ,43–46 three of its eight domains—pain control, communication about medicines and discharge instruction—appeared in less than 10% of reviews. This echoes findings from the formative HCAHPS focus groups, in which participants, when pressed, preferred information on the other domains compared with pain management, avoiding medication problems, and avoiding problems after discharge.44 However, these domains were still considered important to most focus group participants, and so were included in the HCAHPS survey.44 None of the key domains we describe in this study arose in the original focus groups except hospital-acquired infections, a subtheme within safety. Potential reasons for different themes in the two studies include different ways of eliciting responses (focus group vs online reviews), changes in the US healthcare environment or different populations. It is possible that the broader safety theme may have been an under-recognised theme in the developmental work, at a time when expectations were lower that patients and surrogates could observe aspects of safety. Additional work remains to assess whether the other domains we found, if measured, could better describe patients’ perceptions of their experience, and could be a better gauge of whether a hospital is providing patient-centred care. Including additional domains would increase cognitive burden for respondents and data collection expenses, and so new HCAHPS items would need to be carefully evaluated to assess validity prior to inclusion.
Potential areas of focus for those hoping to make hospitals more patient-centred are suggested in the non-HCAHPS domains we describe.
Financing
There is increasing awareness of differences in prices across hospitals and EDs,47 and a desire for price transparency, as reflected in the reviews. This topic has become more salient in the USA now than in 2003–2004 when the HCAHPS focus groups convened,48 with more US insurance plans incurring high out-of-pocket consumer costs,48 ,49 but is relevant to patients with private insurance worldwide.50 This study provides data documenting that frustrations with price variation and opaque billing processes frequently lead to negative experiences and negative public postings. Clinicians and hospital administrators could consider monitoring and improving this experience for patients. Further research could also assess the relationship between health outcomes and the financial sequelae from the index hospitalisation, given known associations between financial well-being and better health.51
System-centred care
Consumers can encounter administrative barriers to care and experience bureaucratic disempowerment in interactions with the hospital system. This domain is conceptually similar to a set of items Rothman et al52 added to HCAHPS, which they found were more predictive of patient willingness to recommend a hospital than many of the original HCAHPS domains. It is possible that the sense of disempowerment may also contribute to decreased patient activation,53 which has been associated with worse outcomes, including increased readmissions and higher costs.54–58 Gathering data on patient experiences in navigating the hospital systems of care may help hospitals improve their overall patient-centredness, and the resulting impact on patient activation and patient outcomes should be evaluated.
Perceptions of safety
Safety or loss of trust arose as an important theme in more than 40% of consumers’ narrative reviews. Evidence that patient experience ratings are associated with hospital-level rates of nosocomial infections and inhospital mortality,1 ,22 taken together with our findings, suggests that patients or caregivers may be a useful source of information about patient safety. Their observations may be particularly important given stubbornly high rates of failures in patient safety.41 ,59–62 Gathering patient reports on safety has been suggested in the past,41 but has focused on gathering specific incident reports when they happen, rather than a global, standardised and regular assessment from all patients. While discrete process and outcome measures of patient safety exist, a global assessment of perceptions of safety would overcome one limitation of disease-specific measures, which by design are applicable to only a small percentage of hospitalised patients.
Finally, it is not currently standard to survey family and friends regarding the experience of a patient's hospitalisation. The finding that family and friends’ reviews included safety and trust themes more frequently than patients’ implies that this group could provide useful data in these areas. Next steps include developing methods for gathering data from patients, friends and family members on perceptions of safety, and assessing whether acting on these data can lead to improved health.
Limitations include a potential response bias, if perspectives of online reviewers differ from those of the general population. Internet users have been noted to be younger and less sick compared with others.34 ,63 It is possible that this response bias affects the themes, as it has been shown that quality topics of interest differ by age in a group of online visitors to public reporting websites.64 However, our sampling is more geographically diverse than prior studies, including the formative focus groups in the HCAHPS development;65 and the new topics we found, even if they are more salient to one subpopulation of consumers than another, are nonetheless important on face value and for the groups that respondents do represent. Another potential limitation is that people with extreme feelings may be more likely to write unsolicited comments, resulting in a portrayal of quality themes that may differ compared with the general public. However, this is also a limitation of in person focus group data, which is the other potential source of quality domains in hospital care.65 ,66 In addition, most of our research team has clinical training (medical or nursing) and personal or family experiences in the hospital, with one non-clinician coder. This represents a strength in recognising themes that may be actionable for creating a more patient-centred system. However, clinical training may also influence our analysis towards clinical rather than non-clinical domains. In part to address this limitation, we provide the full list of domains from the analysis in the Appendix. Lastly, we sampled reviews from US hospitals, and so findings may not be generalisable to other countries. For countries with public or mostly public insurance models, the findings on financing, though less applicable, may be useful for policy discussions regarding potentially increased privatisation.
Conclusion
In conclusion, this analysis of a national group of online hospital reviews suggests that a widely used tool for measuring patient experience may not adequately capture some potentially important domains, including perceptions of safety and experiences of difficulties with billing and unexpected out-of-pocket costs. Our findings also suggest that friends and family members’ perspectives on patient safety differ from patient perceptions and may provide additional important information.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Twitter Follow Naomi Bardach at @naomibardach
Contributors NSB conceived of and designed the work, supervised collecting the data, analysed the data, drafted and revised the manuscript. She is the guarantor. AL contributed to design, analysis, and gave critical revisions. RAP collected and analysed the data, and contributed to drafting and critically revising the manuscript. LEG contributed to analysis and gave critical revision. GAL contributed to analysis and gave critical revision. RAD contributed to conception and design, analysis, and critically revised the manuscript. All authors give final approval of the manuscript to be published and agree to be accountable for all aspects of the work. All authors had full access to the data and take responsibility for data integrity and accuracy of the analysis.
Funding California HealthCare Foundation, California HealthCare Foundation (KL2 RR024130), National Institute of Child Health and Human Development (K23HD065836).
Competing interests None declared.
Ethics approval The University of California San Francisco Institutional Review Board deemed this study exempt from review.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data sharing: A full data set and code book are available from the corresponding author.