Article Text
PDF
Abstract
Objective Unsolicited web-based comments by patients regarding their healthcare are increasing, but controversial. The relationship between such online patient reports and conventional measures of patient experience (obtained via survey) is not known. The authors examined hospital level associations between web-based patient ratings on the National Health Service (NHS) Choices website, introduced in England during 2008, and paper-based survey measures of patient experience. The authors also aimed to compare these two methods of measuring patient experience.
Design The authors performed a cross-sectional observational study of all (n=146) acute general NHS hospital trusts in England using data from 9997 patient web-based ratings posted on the NHS Choices website during 2009/2010. Hospital trust level indicators of patient experience from a paper-based survey (five measures) were compared with web-based patient ratings using Spearman's rank correlation coefficient. The authors compared the strength of associations among clinical outcomes, patient experience survey results and NHS Choices ratings.
Results Web-based ratings of patient experience were associated with ratings derived from a national paper-based patient survey (Spearman ρ=0.31–0.49, p<0.001 for all). Associations with clinical outcomes were at least as strong for online ratings as for traditional survey measures of patient experience.
Conclusions Unsolicited web-based patient ratings of their care, though potentially prone to many biases, are correlated with survey measures of patient experience. They may be useful tools for patients when choosing healthcare providers and for clinicians to improve the quality of their services.
- Patient experience
- internet
- quality
- quality measurement
- quality improvement
- patient satisfaction
- hospital medicine
- health services research
- patient safety
- health policy
Statistics from Altmetric.com
- Patient experience
- internet
- quality
- quality measurement
- quality improvement
- patient satisfaction
- hospital medicine
- health services research
- patient safety
- health policy
Introduction
Over the past decade, there has been an explosion of websites, such as Amazon, Tripadvisor and Yelp, in which customers rate their experiences with goods and services. Unsurprisingly, patients are increasingly rating their experiences with the healthcare system on the internet as well. A review conducted during 2009 identified 33 different physician rating websites in the USA.1 While patient web-based ratings of healthcare may not be well received by clinicians,2 patients themselves may value this type of information when they make healthcare choices.
In the USA, websites hosting patient options and rankings are run by private enterprises. In contrast, patients in England are encouraged to rate and comment on their healthcare providers on a government run website called National Health Service (NHS) Choices, which provides comparative information on healthcare organisation performance. Comments are moderated according to a fixed set of rules;3 comments deemed offensive or which refer to individual clinicians are not posted. Healthcare providers are encouraged to respond to comments, and many do.
Traditionally, measures of patient experiences have been derived from large paper-based or electronic-based surveys of randomly selected patients, often coordinated nationally. While such surveys may appear to have advantages of standardised questions and a randomised representative sample (to all patients or to a random subset of patients), they are costly to administer4 and may suffer from response biases.5
As gathering patient opinions and experiences becomes increasingly important, because we care about them inherently, because they might drive improvement through transparency or because we might want to use them in payment policy, it would be useful to compare patient experience captured via traditional surveys with ratings from these new websites. The amount of agreement between patients' online ratings of hospitals and measures of patient experience via these surveys may be useful in considering how relevant these website are, and which of these two strategies to favour in trying to capture patients' voices of their care.
We have previously demonstrated an association between patients' online ratings of their care in hospitals and certain clinical outcomes, including standardised mortality.6 Studies looking at survey measures of patient experience have also demonstrated associations with outcomes.7 ,8 In this paper, we examine unsolicited ratings posted on NHS Choices for all acute hospitals in the England, describing patterns observed and analysing associations with conventional measures of patient satisfaction obtained through formal surveys. We also compare the strength of associations between NHS Choices ratings and clinical outcomes and associations between patient survey measures of experience and clinical outcomes.
Methods
Online ratings data
We obtained aggregate patient ratings for all NHS hospitals in England posted on NHS Choices from 1 January 2009 to 31 December 2010 from the English Department of Health. We excluded 22 hospitals that provide services only for children or specific specialties (eg, oncology or rheumatology hospitals), leaving a sample of 146 general acute hospitals. The NHS Choices website allows patients to indicate whether they would recommend a hospital to a friend (yes/no) and rate hospitals on four specific domains of quality: the cleanliness of the hospital environment; whether they were treated with dignity and respect during their stay; whether they were involved in decisions about their care; and whether the hospital staff worked well together (see table 1 for scoring system). We examined only the ratings, not the free text comments. The proportion of patients recommending a hospital trust was calculated by dividing the number of ratings that recommended the trust by the total number stating a view on recommendation. For ratings on a scale, the mean rating for each hospital trust was calculated.
Patient rating domains on NHS Choices and scores for 2009/2010
As organisational performance in the NHS is generally measured at the level of a hospital trust, data were aggregated at this level. A trust is a single or small group of hospitals in a defined geographical area operated by the same management team. Data were combined by creating weighted arithmetic mean values for each score at each trust. Weighting was based on the total number of comments for each component hospital.
Conventional patient survey data
Conventional measures of patient experience were taken from the 2009/2010 NHS Inpatient Survey.9 ,10 This is the yearly national paper-based survey of patients in all NHS hospital trusts conducted by Picker Europe for the Care Quality Commission, England's health regulator. The survey is similar to the Hospital Consumer Assessment of Healthcare Providers and Systems in the USA.11 It surveys a random sample of patients who have been treated in NHS hospitals for at least one night. Sixty-eight thousand five hundred and ninety-four patients completed the survey between September 2009 and January 2010. The response rate was 52%.12 We selected questions that were closely related to the domains reported on NHS Choices (table 2).
Hospital trust level associations between web-based patient ratings and paper-based surveys of patient experience
Clinical outcome data
Three outcome measures have previously been demonstrated to be associated with online patient ratings in hospitals: standardised mortality ratio, standardised mortality from high risk conditions and emergency readmission rate within 28 days of hospital discharge. We obtained data on these outcomes from Dr Foster,13 a health intelligence company, and NHS Comparators,14 the NHS's comparative health system performance service, for the year 2009/2010.
Statistical analysis
We performed a descriptive analysis of the quantitative data from NHS Choices, calculating mean and median scores in each category, and measures of dispersion including the inter-quartile range. We compared the total number of ratings left on the NHS website for all hospitals with the number of patient admissions and the number of responses to the national inpatient survey.
We compared patient website ratings with other measures of patient experience using Spearman's rank correlation coefficient. This conservative non-parametric test was used because the data in the patient ratings was not normally distributed according to the Shapiro–Wilk test (p<0.05 for four out of five NHS Choices variables described).
To examine the relationship between patient experience and clinical outcomes, we compared both measures of patient experience (survey and online rating) with the three outcome measures using Spearman's rank correlation coefficient. We then compared the strength of the associations using Fischer's z-transformation. Statistical analysis was conducted in Stata V.11.
Results
Between 1 January 2009 and 31 December 2010, there were 10 274 ratings of acute hospitals posted on NHS Choices. This represents a rating of 0.04% of hospital admissions (the NHS had 29 118 009 hospital admissions over the same period15). Nine thousand nine hundred and ninety-seven ratings related to acute general hospitals. The mean and median number of ratings for these general hospitals were 69 and 46, respectively (range 4–290, figure 1). Of those who offered a view about recommending the hospital (9183 ratings), 67.4% (6120) indicated that they would recommend the hospital to a friend. The remaining 914 ratings did not express a view about recommending the hospital, but rated other aspects of the hospital. A summary of the rating results is displayed in table 1.
Histogram of frequency of web-based ratings per hospital trust.
Comparing website and paper-based survey ratings of patient experience
Patients' website ratings of their experiences on hospital cleanliness, being treated with dignity and respect, staff working well together and being involved with decisions about care were significantly associated with responses to similar questions in the national patient survey (Spearman ρ=0.31–0.49, p<0.001 for all) (table 2). The proportion of people willing to recommend the hospital on NHS Choices was also associated with the overall rating of the hospital in the national inpatient survey (Spearman ρ=0.41, p<0.001).
Comparing different measures of patient experience with clinical outcomes
For those quality indicators for which an association has been demonstrated between online ratings and patient outcomes (ie, standardised mortality ratios and readmission rates), we observed that the strength of association between online ratings and outcomes was at least as strong as that between patient survey experience measures and outcomes (table 3).
Comparison of associations between web-based/paper-based ratings and clinical outcomes
Discussion
Key findings
Our results show that despite fears by providers and hospital administrators that rating sites would offer mostly criticisms, ratings of hospitals in England are largely positive, mirroring the results from privately-managed physician rating sites in the USA.1 ,16 ,17
When the NHS launched its NHS Choices website, it was unclear how much the site would be used, both in terms of numbers of responses and geographical distribution. Our results show that every acute hospital trust in England has been rated. However, at present the number of ratings is only a fraction of the number of responses to conventional paper-based surveys, and an even smaller fraction of healthcare contacts. There is also variability between different hospitals, which may reflect different levels of awareness of the rating site among patients or of levels of promotion by hospitals.
Our results demonstrate that patients' website ratings of hospitals and more conventional measures of patient experience from large random surveys are significantly correlated, which is reassuring since it is likely that they are capturing overlapping but different populations of patients and some different dimensions of hospital quality and performance. We also found that clinical outcomes, such as hospital standardised mortality ratio, are at least as strongly correlated with self-selected patient comments left on a website as with the random survey of patients obtained in the national survey.
While traditional surveys have the advantage of random allocation (although they may still suffer from response bias), patients who offer unsolicited opinions about their care may be providing richer feedback. Unsolicited comments may be more likely to pick up low probability but high impact events affecting patient experience, particularly if people who have experienced extremes of care, whether good or bad, are more likely to respond to online reporting schemes. These results do not prove that one method is better than the other, but they do help us to understand how they compare. A comparison of some of the characteristics of the two methods of capturing the patient voice is displayed in table 4.
Comparison of the characteristics of the NHS Inpatient Survey and ratings on the NHS Choices website
Our results do not upend all of the arguments against the use of patient website reporting, including that such reports may come from a biased selection of patients and that they carry the risk of harming doctor–patient relationships. We also note that hospitals may try to encourage particular patients to rate their care online; this adds an element of management to the self-selecting group of raters and may lead to a more complex selection bias. However, we suggest that unsolicited patient comments may have an important place in national quality and safety measurement systems. Our findings add to the increasingly persuasive literature promoting the notion that one needs to view safety, quality and service delivery through a number of lenses to get an accurate picture.18
Limitations
The data from NHS Choices are from the complete years 2009 and 2010. We compared these with available data on quality during the financial year 2009/10, which is not quite an exact match. However, in each case we are using the most recently available data. We also only had access to rating data at the hospital level, not individual ratings, so there may have been some loss of accuracy when hospitals were aggregated together into hospital trusts for our analysis. Further, we were unable to examine the characteristics of patients who posted a rating on NHS Choices and the extent to which ratings differed among age, gender, ethnic and socioeconomic status groups. The number of ratings was small for some hospital trusts; however, if trusts with <10 ratings are removed from the analysis, all results remain significant.
Further research
More research is needed to understand how web-based patient ratings are used by patients in making choices about healthcare—including how they reconcile them with objective clinical information such as outcomes and processes of care—and how this information could be used by healthcare organisations to understand and improve their own services.19 We only analysed quantitative data available from the NHS Choices website and did not examine the wealth of patient comments on the site regarding their own experiences of care. Further study of these experiences and how they could be used by healthcare providers to improve the quality of care would be useful. As a system that relies on patients volunteering to come forward to rate services, the nature of responses will also depend on how active the process of solicitation is. It would therefore be useful to understand why people choose to come forward, and what effect increasing the promotion of such websites would have on the nature of responses. Furthermore, research is needed to dissect the relationship between bad clinical outcomes and patient satisfaction in the context of appropriate medical care. In healthcare, as opposed to most other services with online rating, you can do all the right things, but still have a bad outcome. Having a clearer understanding of the interaction between clinical outcomes and satisfaction measures would be useful.
NHS Choices allows patients to rate their care at the hospital level. This is different to many websites, which rate individual physicians. It may be more difficult to develop insights into care quality when rating individuals, as the smaller number of ratings may preclude robust estimates of patient experience.
Conclusion
Our study demonstrates that, given the opportunity, many patients will provide ratings across a broad range of providers and geography, and that most ratings of hospitals are positive. We also found associations between patient ratings and patient survey responses.
Although we appreciate that there are inherent risks in the use of ratings from a small group of self-selecting patients, we also see that there are advantages of using this new form of information compared with using the traditionally survey approach, in particular its lower cost and its ability to detect episodes of poor care that a random survey might miss. Although online feedback mechanisms should not replace patient surveys, they may provide information on hospital quality that is relevant, complementary to survey data and potentially useful for patients when making choices about their healthcare. Healthcare providers and purchasers should therefore promote the development of such online reporting schemes and encourage their use.
Acknowledgments
We would like to thank the team at NHS Choices, and John Robinson in particular, for providing access to their data.
References
Footnotes
Role of Sponsors Study sponsors had no role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.
Some contents of this manuscript have been accepted to be presented at the BMJ International Forum on Quality and Safety in Healthcare in Paris, 20 April 2012.
Funding There was no specific funding for this project. CM is funded by the Higher Education Funding Council for England and the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care scheme (CLAHRC) for North West London. Imperial College London is grateful for support from the National Institute for Health Research Biomedical Research Centre Funding scheme, the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care scheme, and the Imperial Centre for Patient Safety and Service Quality. Dr Wachter conducted much of this work while at Imperial College London in 2011 under a US-UK Fulbright scholarship. The funding sources had no role in the design and conduct of the study; collection, management, analysis or interpretation of the data; or preparation, review or approval of the manuscript.
Competing interests Dr Wachter reports having an equity interest and/or serving on paid advisory boards for PatientSafe Solutions and CRISI; receiving support for helping to lead a leadership training program for IPC-The Hospitalist Company; receiving honoraria from the American Board of Internal Medicine for serving on its board of directors and Executive Committee; receiving honoraria for many speeches on patient safety and quality; receiving support from John Wiley and Sons for writing a blog; and receiving funding under a contract from the Agency for Healthcare Research and Quality for editing two patient-safety websites and royalties from publishers from two books on patient safety. Dr Wachter is a 2011 recipient of a US-UK Fulbright Award for study in the UK. Professor Darzi was Parliamentary Under-Secretary of State (Lords) in the United Kingdom Department of Health from 2007 to 2009. Professor Majeed has received funding for research from Dr Foster Intelligence, a commercial health intelligence company. The other authors declare no support from any organisation for the submitted work, no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available on request from the corresponding author.