Article Text
PDF
Abstract
Background Patients have adopted web-based tools to report on the quality of their healthcare experiences. We seek to examine online reviews for US emergency departments (EDs) posted on Yelp, a popular consumer ratings website.
Methods We conducted a qualitative analysis of unstructured, publicly accessible reviews for hospitals available on http://www.yelp.com. We collected all reviews describing experiences of ED care for a stratified random sample of 100 US hospitals. We analysed the content of the reviews using themes derived from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) inpatient care survey. We also used modified grounded theory to iteratively code the text of the reviews, identifying additional themes specific to emergency care. The data were double-coded, and discrepancies were evaluated to ensure consensus.
Results Of the 1736 total reviews, 573 (33%) described patient experiences involving the ED. The reviews contained several themes assessed by the HCAHPS survey, including communication with nurses, communication with doctors, and pain control. The reviews also contained key themes specific to emergency care: waiting and efficiency; decisions to seek care in the ED; and events following discharge, including administrative difficulties.
Conclusions These exploratory findings suggest that online reviews for EDs contain similar themes to survey-based assessments of inpatient hospital care as well as themes specific to emergency care. Consumer rating websites allow patients to provide rapid and public feedback on their experience of medical care. Web-based platforms may offer a novel strategy for assessing patient-centred quality in emergency care.
- Emergency department
- Patient-centred care
- Patient satisfaction
- Qualitative research
- Quality measurement
Statistics from Altmetric.com
Introduction
The preferences and perspectives of patients have become influential to the delivery of healthcare in the USA.1–3 For instance, patient-centred outcomes for hospital quality now factor into Medicare reimbursement.4 ,5 Organisations ranging from the Institute of Medicine to the Patient-Centered Outcomes Research Institute have prioritised the development of quality metrics relevant to patients.6–11 Although patient-centred quality metrics for emergency department (ED) care have not been clearly defined, the Centers for Medicare and Medicaid Services (CMS) has commissioned a survey-based instrument that may become the standard method for assessing patient experience in the ED.12–20
Meanwhile, patients have adopted web-based tools to report on the quality of their healthcare experiences.21 In contrast to standardised survey tools with predetermined questions, patients write narratives about their hospital experience. Through consumer rating websites, prospective patients can read about the experiences of others.6 ,22–24 Previous studies have demonstrated that online reviews tend to be favourable to providers.25–28 In addition, online ratings for hospitals have been correlated with the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, the instrument used by Medicare to assess inpatient experiences.29–31 Yelp ratings were strongly correlated with high scores on the HCAHPS survey, and hospitals with high Yelp ratings tended to have lower readmission rates for pneumonia, heart failure and myocardial infarction.29
Rating websites have affected how consumers choose products and services, and evidence suggests that patients use online reviews to select physicians.24 ,32 ,33 Online reviews have the potential to provide EDs with rapid, candid and public feedback on their perceived quality of care.2 ,6 ,26 These reviews, however, may not represent the opinions of the majority of ED patients. Furthermore, online reviews have been criticised as fraudulent, extreme and discordant with more objective quality assessments.34 It is unknown whether the content of online reviews are consistent with previous surveys of patient experiences with emergency care, and it is unknown whether reviews can provide useful data on the quality of care provided in the ED. To our knowledge, the content of online reviews for EDs has not been explored despite their high visibility on the internet.
In this study, we used qualitative methods to analyse the content of online reviews posted on a popular consumer ratings website, http://www.yelp.com.35 First, we used the HCAHPS survey as a framework for analysing the text of the reviews, to determine whether the reviews addressed similar topics as the validated inpatient survey. Second, we sought to identify key themes in the reviews that are specific to patient experiences in the ED. The objective of this analysis was to characterise the content of online reviews and explore their perspectives on ED care.
Methods
Study design
We conducted a qualitative analysis of reviews posted on the website http://www.yelp.com for a sample of US hospitals. We used qualitative methods to analyse the content of reviews rather than make statistical inferences. We employed the Consolidated Criteria for Reporting Qualitative Research to guide analysis and reporting of the data.36
Study setting
Yelp (Yelp, San Francisco, California, USA) is an online portal where users can rate and review local businesses and services, including hospitals and physicians.37 Among other rating websites, Yelp was chosen due to its previous correlation with patient-centred outcomes and because it is among the most popular and accessible websites for hospital reviews.29 In 2014, the website reported an average of 138 million unique visitors each month.35
Users rate businesses and services on a scale from 1 (low) to 5 (high) stars. Users also post an unstructured written review of their experiences. The hospital webpage displays individual ratings and reviews as well as the aggregate rating from all users. Yelp applies automated software to remove reviews suspected to be fraudulent (eg, multiple reviews about the same business written from the same computer).37 ,38 The reviews that Yelp suspected to be fraudulent were not included in our analysis. We excluded other review websites in order to maintain consistency in the population of reviews as well as the format, filtering process and accessibility of the reviews.
Study population
We evaluated individual Yelp webpages for a comprehensive set of US hospitals. This national list consisted of hospitals included in the 2010 American Hospital Association (AHA) survey that reported valid HCAHPS data to the CMS as of 20 July 2013.39 ,40 Hospitals reporting valid HCAHPS data had adequate sample sizes of patients and no discrepancies in the data collection process. We excluded hospitals that were not classified as general medical and surgical facilities according to the AHA survey. Veterans Affairs and other federally managed hospitals were also excluded. We determined the number of Yelp reviews and aggregate Yelp rating for each hospital. We then excluded hospitals with fewer than five Yelp reviews. From the final list, we selected a random sample of hospitals with five or more reviews, stratified by aggregate Yelp rating because review themes may differ by hospital Yelp rating. We selected 100 hospitals, with 25 hospitals in one of four strata based on their Yelp rating: excellent (4.5–5.0 stars), good (3.5–4.0 stars), fair (2.5–3.0 stars) and poor (1.0–2.0 stars). The process of selection, exclusion and randomisation is further described in the online supplementary figure.
Of note, this sampling scheme occurred at the level of the hospital, but our analysis was performed at the level of the individual review. However, we determined that the characteristics of individual reviews, including star rating, were concordant with that of the study hospitals.
Data collection
Data was collected on 1 August 2013. Each hospital was assigned a unique identifier. For each hospital, an investigator manually extracted all available reviews for the five previous years for that hospital into a blinded document, removing references to hospital name. The text of the reviews was copied into NVivo (V.10.0; QSR, Doncaster, Australia), a software tool for qualitative data management and analysis. We also manually collected data on the overall hospital rating, number of reviews per hospital and characteristics of individual reviewers, including gender, number of reviews previously posted and date of first review. Although these data are publicly available, the quotations presented in this manuscript have been edited by the consensus of two authors (ASK, RJS) to protect confidentiality of hospitals, staff and reviewers.
Data analysis
An iterative coding process was used to identify patterns and key themes in the text of the reviews. We first identified all reviews describing patient experiences in the ED. Then, we used two separate approaches. First, we used an a priori set of codes to identify specific constructs related to eight domains, or topic areas, included in the HCAHPS survey, which have been validated as patient-centred quality metrics for inpatient hospital care.41 This survey was selected over other assessments focused on emergency care, including the Press Ganey survey, due to its scientific validity, public availability and importance to national policy.41 These codes were: communication with nurses; communication with doctors; cleanliness of hospital environment; quietness of hospital environment; responsiveness of hospital staff; pain control; communication about medicines; and discharge information.
Our second approach used modified grounded theory to identify a set of codes that emerged from the data de novo. The study team developed this set of codes through an iterative, line-by-line reading of the entire set of reviews, reaching consensus on a list of codes that corresponded to emerging themes. This modified approach differs from traditional grounded theory in the process of data collection, but uses grounded theory principles to define, test and apply codes.42–46 The de novo codes specifically pertained to aspects of care in the ED. To reduce bias among the study team, the reviews were blinded to hospital identity and characteristics. Three coauthors (ASK, BP, YPH) applied both sets of codes to all transcripts. The entirety of the data was double-coded, and discrepancies in coding were discussed and reviewed to ensure consensus. We summarised the codes and analysed relationships between them to identify key themes.
Results
Characteristics of study subjects
We collected a total of 1736 Yelp reviews for the 100 hospitals in our sample, with a median of 12 reviews per hospital (range 5–21, IQR 7–20). The characteristics of the hospitals and reviews are described in table 1. The distribution of characteristics, such as hospital size, was similar for hospitals and individual reviews. However, the distribution of star ratings for individual reviews was skewed towards excellent and poor reviews, as compared with overall star rating for hospitals. Of the individual reviews, 533 (33%) were related to patient experiences in the ED. The distribution and characteristics of reviews pertaining to ED experiences were similar to the larger set of reviews.
Characteristics of study hospitals and reviews
Analysis of reviews in relation to HCAHPS survey domains
We analysed the reviews using the HCAHPS survey domains as a framework to identify key themes in the reviews. The survey domains reflect topics of known importance and relevance to patient experiences of care. Each HCAHPS domain could be matched to recurring themes in the reviews. These themes are summarised in table 2, accompanied by representative quotations. Three HCAHPS domains, in particular, recurred throughout the reviews and were discussed in depth: communication with nurses, communication with doctors and pain control.
ED Yelp review themes organised by HCAHPS survey domains
Communication with nurses
The online reviews emphasise communication with nurses as a crucial element of patient experience, as does the HCAHPS survey. There are two aspects of nursing communication featured in the reviews. One is the tone of communication. In positive reviews, patients often use words such as friendly and kind to describe nurses, who are often mentioned by name:The nurses are all so nice here! One nurse, I think his name was __, was really sweet. I told him I was afraid of needles, and he tried distracting me…. He was a nice guy that made my uncomfortable ED visit more comfortable.
The other aspect of communication is responsiveness, as frequent interactions are interpreted as attentiveness. Negative experiences related to communication with nurses focus on long periods without interactions and lack of transparency for why patients are waiting. Patients are frustrated with perceived inaction in response to the urgency of their condition:After 3 h of sitting in the emergency room and listening to the nurses gossip about everything, I was finally seen. I was only noticed when I got up from the corner and stood at the counter watching one of the nurses check his email….
An aspect of the online reviews missing from the HCAHPS survey is interdisciplinary care. Reviewers rely on nurses for communication about their care, and they are seen as interpreters for physicians. Furthermore, patients comment on all staff with which they interact:Every person I met, from the man at the front desk to the X-ray tech and nurse were excellent.
Communication with doctors
The HCAHPS survey asks whether physicians listened to and explained aspects of medical care. Similarly, the theme of communication with doctors was notable in the reviews. Patients specifically note the amount of time spent with the physician. Reviewers appreciate physicians that take time to explain decisions, present options and answer questions, particularly when the aetiology of illness is not immediately apparent. For example, one reviewer writes:Dr ___ made sure to tell me which tests he sent and why, what blood tests he ordered, and finally why they needed to admit me. I feel like he genuinely cared about me.
Conversely, physicians that are perceived to not spend enough time with patients are criticised:The doctors never fully explained the problem and spent 3 minutes with my parents before they ran out. It felt like the doc could not wait to get out of the room, and the ED did not even look busy.
Some patients felt that abrupt physicians missed or miscommunicated diagnoses that are discovered after discharge.
Pain control
In many reviews, patients present to the ED with painful conditions. The HCAHPS survey asks patients whether their pain was often controlled during their hospital stay. In contrast, the ED reviews emphasise how quickly pain is acknowledged and treated without dismissal of pain as chronic or untreatable. While the modality of pain control is often opioid analgesics, no reviews in this sample mention specific requests for opioids or arguments over opioids. One patient was effusive in praising an ED physician:[He] showed great empathy for my problem. He told me he treats a lot of people with my nerve condition using a ‘nerve block’ shot. He gave me a shot 15 min later, and he was very skilled. I could not feel the needle, and the pain was gone right away!
Negative reviews describe extended periods of waiting before pain is recognised or controlled. However, many reviews also describe failure in identifying the aetiology of pain, which patients perceive as diagnostic error. The HCAHPS survey asks inpatients whether staff did everything they could to help with pain; the ED reviews suggest that patients want and expect the aetiology of their pain to be diagnosed and communicated. One reviewer writes:My abdominal pain became very severe, and I got to the ER around midnight…Nobody gave me anything for pain. Later, I found out my gall bladder was infected and I had surgery.
Themes specific to ED care
The reviews contained three key themes regarding ED care that are not captured in the HCAHPS survey: waiting and efficiency; decisions to seek care in the ED; and events following discharge (table 3).
Themes specific to ED care
Waiting and efficiency
A major component of the reviews was the subject of time. Positive and negative reviews describe the elapsed time for all stages of the encounter. Patients are pleased when their ED visit occurs quickly:I went to ___ last night because I live close. I thought it would be too crowded and heard all sorts of bad things. I was surprised! They sent me home in 2 h, and everyone was incredibly professional.
As these comments demonstrate, the perception of prompt care is often accompanied by praise for other aspects of care. Conversely, long waits accompany negative impressions of overall quality. An important aspect of this theme is that waiting time is not limited to the waiting room. Patients note that they were evaluated quickly but are left waiting before discharge. One reviewer notes:They took me for X-ray in 30 min. I thought that was awesome. But then I sat in the waiting room for six more hours until someone told me my ankle was sprained.
As in this example, patients report they were not given explanations for periods of waiting.
The reviews reveal that some patients have insight into reasons for lengthy waits. In a few instances, reviewers debate the purpose and role of EDs. In response to several complaints over long wait times, one reviewer writes:Shame on you if you have been posting bad reviews! This ED may seem old but it has to do what it has to do: provide service to the community. Do you think that any of the patients are able to pay, given where the hospital is? But the hospital still does its best to keep its people healthy.
Other reviewers acknowledge that the triage system prioritises ill patients and accept that they must wait.
Decisions to seek care in the emergency department
Patients provide reasons for visiting the ED as well as for choosing one hospital over another. Many reviews describe acute, obvious injuries:My wife tripped and broke her arm Saturday morning—we saw a bone sticking out of the skin. We hopped in the car and drove to the ER and she was treated in about 30 seconds.
However, the reviews also include narratives of illness with less clear aetiologies or more chronic courses. In these reviews, patients often require care when primary physician offices or urgent care centres are closed. In other cases, other physicians send patients to the ED:My wife said her blood pressure was really high. I said that we should go to the ER right away. (Actually, I said she should call her doctor. He told her to go straight to the ER, but that makes me sound less nurturing).
Regardless of disease condition, a common theme in explanations for seeking emergency care is that patients perceive their conditions to have become serious, painful or frightening enough to require immediate attention. Accordingly, patients are dissatisfied when the speed with which they are evaluated falls short of expectations. Patients’ own risk assessment often falls short of their triage assessment:I can only give two stars…I think that if you get to the hospital with chest pain you should be seen immediately to make sure you're not having a heart attack.
When patients express reasons for choosing one ED over another, proximity is most commonly cited while the capabilities and resources of the hospital are rarely mentioned. After negative experiences, many patients state they would prefer to travel further for their next emergency:My daughter had an allergic reaction on Memorial Day. We live just around the corner, so we walked. My first thought: not that bad. But as we sat for hours, I wished we could have gone to __ hospital. We did not have a choice because it was an emergency, but I hope we never have to go there again.
I don't care if my arm gets cut off and I could bleed to death. I would rather drive an hour away than go back to this place.
Many reviewers express the desire for more informed choices, but none state that they will avoid EDs altogether.
Events following discharge
The reviews provide patients with the opportunity to report their experiences following discharge. The key themes emerge primarily from negative reviews featuring perceived diagnostic errors and ED revisits. Patients report perceived diagnostic errors, which resulted in minor and serious morbidity. One reviewer describes taking her mother to the ED for a headache:We told the doctors that her mother had a stroke at 56 years. They gave her baby aspirin, Mylanta, nitroglycerin, morphine, and discharged her. No testing and no explanations—just headache and gas. The next night, she had a stroke with bleeding in her brain. I still don't know why they did not do a CT scan the first time.
Many reviews describe perceived treatment failures upon discharge that require patients to return. Reviewers do not generally acknowledge that poor outcomes may be due to disease progression, miscommunication or other factors.
Another important theme is the difficulty that patients encounter in negotiating payment following their care. The reviews note surprise in the out-of-pocket costs of their ED visit, which they discover after their visit. Hospital administrators and insurance companies are noted to be unresponsive and misleading:They sent us not one, but two bills for the service in the ER! We called them many times and tried to work it out. They said they would call back with a resolution. But instead they sent the bill to collections!
Discussion
We conducted a qualitative analysis of online reviews for EDs. Our goal was to describe the content of the reviews, identifying key themes in the stories, attitudes and insights that reviewers post online. When we used the HCAHPS survey to structure our analysis, we found that the reviews address major domains included in the survey. Salient themes include communication and pain control. Reviews describe the amount of time that physicians and nurses spend with patients, who value frequent interactions and transparency. Reviews also note the time needed for providers to acknowledge painful conditions.
We also analysed the reviews for additional themes that were not included in HCAHPS. A key theme is waiting, not just in the waiting room but throughout the visit. The importance of these themes has been noted in previous survey-based studies that have examined patient satisfaction in the ED.16 ,47 One series identified and validated drivers of patient satisfaction that included not receiving help when needed, poor explanations of medical problems, and lack of transparency over waiting times and test results.17 ,48 A systematic review identified interpersonal interactions with providers as the key element of patient satisfaction, with waiting times and perception of technical skills as major factors.16 CMS is currently developing a patient survey, ED CAHPS, to assess ED experiences of care.20 In a preliminary version, nearly all questions involve communication, pain control and waiting.20 When finalised, the ED CAHPS survey may become critical to measurement of patient-centred outcomes, hospital policies and reimbursement, like its inpatient counterpart.18 ,19
This study also identified key themes that have not been observed in previous studies of ED patient satisfaction. Reviewers write about their decisions to seek care in an ED, citing convenience and perceived urgency as major factors. Reviewers also note proximity as the main reason for choosing one ED over another. Another key theme was the reporting of outcomes after discharge, such as whether diagnoses were correct. Many reviews commented on other events occurring weeks to months after the visit, including billing difficulties. These themes are likely to be outcomes of interest to emergency physicians and hospital administrators, and may require additional investigation as potential domains to be included in patient experience surveys. However, it is important to note that the primary purpose of the reviews is to make a recommendation or warning to other patients. The online review forum affords patients the opportunity to communicate with each other in a way that traditional assessments of quality, such as surveys, do not.6
The presence of these novel themes may be due in part to the advisory nature of reviews. Information about triage decisions and outcomes are essential components of recommendations, but not necessarily postvisit surveys. The differences between online reviews and previous survey-based assessments are not just limited to specific themes, but rather the medium through which these opinions are shared. Online reviews introduce a public voice for previously anonymous opinions constrained to surveys. The reviews provide information on efficiency, capabilities and alternatives that could potentially affect other patients’ choices, even in the setting of acute illness.24
The content of online reviews for EDs has not been described previously. Lopez et al28 examined reviews of primary care physicians on Yelp and ratemds.com, determining that reviews were generally positive and discussed bedside manner, technical competence and systems issues such as access to appointments. While online reviews remain controversial, this study demonstrates that Yelp reviews for EDs, when considered in aggregate, reflect established domains of patient-centred quality, many of which are publicly reported.18–20 The findings of this study suggest that patient reviews, whether through the Yelp platform or another venue, have the potential to contain meaningful and constructive feedback on patient experiences.
This study, therefore, raises questions regarding the future of online reporting of healthcare experiences. An online review-based system may have advantages over traditional approaches. First, reviews can be evaluated in real time, facilitating rapid administrative changes and real-time surveillance. Second, the reviews are immediately available, accessible and comprehensible to potential patients. While the HCAHPS survey is publicly reported, those data require more complex investigation and interpretation than a web search. Third, the unconstrained nature of the reviews allows patients to report on aspects of care that were not determined in advance. In the UK, patients are able to report their experiences through a successful website officially sponsored by the National Health Service.26
Potential disadvantages of reviews are that they are unstructured and essentially unregulated, further stoking controversy over the influence of patient satisfaction scores on medical practice: patient satisfaction goals may not always align with responsible patient care.11 Evidence suggests that concern for patient satisfaction or incentives based on patient satisfaction may lead to increased use of imaging or antibiotic prescribing in the ED.49 ,50 Conversely, recent evidence shows that patient satisfaction metrics are not necessarily correlated with ED prescriptions for opioid analgesics.51 While patient-centred care and patient satisfaction should be considered different entities, the distinction is not always clear and can be further complicated by patient expectations, physician-patient communication and financial incentives.52
Another disadvantage is that Yelp and other rating websites permit reviews from any patient or family member, including those admitted to the hospital or discharged from the ED, complicating the interpretation of these data. Traditional assessments typically target different surveys to different categories of patients.20 In addition, there is the potential that reviews are fraudulent. Despite the use of filtering software, it is possible that reviews can be recruited or purchased by hospitals and their competitors alike.33 ,34
Finally, it is possible that individual reviews may damage well meaning physicians and hospitals. While this study does not seek to confirm the validity of the Yelp platform, it demonstrates the potential utility of online reviews on patient choice. Online reviews cannot be censored and are easily accessed through simple web searches. Patients increasingly use these data to make choices for their medical care, although their use has not been demonstrated for EDs or in the setting of emergent illness.24 ,29 Despite these concerns, online reviews will likely become an increasingly important influence on patients as well as ED practice.6 ,23
Limitations
This study has several limitations. First, this qualitative study is exploratory. The results cannot be quantified or correlated with clinical measures of quality. Instead, this analysis seeks to examine the content of the reviews to identify concepts and ideas that can be tested in future work. The study population of Yelp reviewers may not be generalisable to the larger population of ED patients. Reviewers must have web access, familiarity with Yelp and motivation to provide feedback. The study, therefore, considers the opinions of a limited population of ED patients. However, the intent of this investigation was not to analyse the comments of a representative sample of ED patients. Instead, the goal of this investigation was to examine the data that are available online. Similarly, this study analyses reviews for a small number of hospitals. Although the hospitals were randomly sampled from all US hospitals, this study was not designed to demonstrate variation in comments according to ED characteristics.
This study uses the major domains included in the HCAHPS survey as a framework for analysing online reviews. It is important to note that the HCAHPS survey is designed for inpatient care experiences, which have important differences with ED visits. While a separate, validated survey is widely used to assess patient experiences in the outpatient setting, many domains of this study do not apply to the setting of emergency care and other domains are already included in HCAHPS.
An important limitation pertains to our sampling scheme. We selected a stratified random sample of hospitals based on overall star rating, but our analysis examined individual reviews. However, we demonstrated that the distribution of ratings for individual reviews, as well as the proportion of other characteristics, were similar to that of the hospitals included in the study. The individual ratings in this study were slightly skewed towards either positive or negative ratings, although previous work has demonstrated that online reviews tend to be generally favourable.27–29 A final limitation is that Yelp processes reviews using a filter that selects certain reviews and screens for inappropriate comments.35 There is a lack of transparency into this proprietary filter and the exclusion of certain reviews, as the exact criteria by which reviews are posted are not publicly available.
Conclusions
In summary, we characterised the content of online reviews for EDs. Our findings suggest that reviews contain analogous themes to an existing assessment of inpatient care experiences, particularly with regards to communication with nurses, communication with doctors and pain control. Our analysis also identified themes specific to emergency care: waiting and efficiency, decisions to seek emergency care and events following discharge. Online reviews are highly visible and may have consequences for EDs as increasingly greater numbers of patients write and read reviews. It remains to be seen whether online reviews influence patient decisions to seek care at an ED, whether patient ratings reflect clinical outcomes or whether online reviews provide a viable mechanism for collecting patient feedback. Regardless, web-based applications and social media may provide novel strategies for assessing patient-centred quality in emergency care.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Twitter Follow Austin Kilaru @AustinSKilaru and Raina Merchant @RainaMerchant
Contributors ASK, ZFM and RMM conceived and designed the study. ASK and BP acquired and managed the data. All authors analysed and interpreted the data. ASK, RJS, BLR and RMM drafted the manuscript, and all authors critically revised it for important intellectual content. BP and YPH provided technical support and managed the data. RMM takes responsibility for the paper as a whole.
Competing interests None declared.
Ethics approval University of Pennsylvania.
Provenance and peer review Not commissioned; externally peer reviewed.