Health services in England and Wales and the United States now seek to develop “patient centred care”.^1,2 Over the last decade, research on patients’ perspectives on care has moved from asking about “overall satisfaction”—a broad concept producing data that are difficult to interpret—to asking about specific patient experiences.³ Issues emerging as critical components of high quality care include information and education; respect for preferences, coordination and continuity; and transitions in care.⁴ Surveys conducted by the Centers for Medicare & Medicaid Services (CMS) in the US^5,6 and the NHS in England and Wales now assess these issues.^7,8 In both countries, data showing variation in experience across geographical areas, hospitals, and health plans are now routinely published.^5–9

Despite these developments, little is known about why variations in patient experiences persist and whether reporting survey data improves care. Younger patients, those with low income, poor perceived health, and from non-black ethnic groups tend to report worse experiences, but such factors explain only a small proportion of the variation observed.^10–14 In general, the range of scores suggests that institutional characteristics and management are important regardless of the population served.^4,10,15

Observations from Wisconsin,¹⁶ California,¹⁷ and Massachusetts^3,18 in the US suggest that public reporting of survey and clinical data may focus attention on improvement efforts. There is no published evidence that such feedback leads to sustained improvement although the Veterans Health Administration, which surveys patients quarterly, has reported a 15% improvement in overall scores between 1995 and 1999.¹⁹ Rogut and Hudson described the response of 15 New York City hospitals to a patient survey in 1994.^11,20 They found that, although hospitals generally thought the survey identified problems, only a few actually launched patient centred interventions.²⁰ One randomised controlled trial of giving survey results to 55 general practitioners in the Netherlands found this had no effect on patients’ evaluations of their care 1 year later.²¹

This study aimed to develop a framework for understanding the factors which affect the use of patient survey data in quality improvement. We interviewed senior health professionals and managers in teams from eight medical groups in Minnesota, USA who had joined a quality improvement collaborative designed to teach them how to use survey results to improve patient centred care.

METHODS

Setting

The collaborative was organised by the Institute for Clinical Systems Improvement (ICSI) in Minnesota and the Consumer Assessments of Healthcare Providers and Systems (CAHPS) team at Harvard Medical School, Boston. ICSI is a state wide consortium of health plans, medical groups, and hospitals that have worked to develop clinical guidelines and quality improvement collaboratives since 1993. Eight self-selected medical groups providing primary and/or secondary care services to urban and rural populations were included.

Study design and interview development

Semi-structured interviews²² with key informants in each group were conducted to identify difficulties or successes they had experienced when trying to use patient feedback or survey data. Detailed descriptions and interpretations of experiences and processes were sought and multiple perspectives solicited.²² Our interview guide was informed by the New York study²⁰ and suggestions from senior physicians, managers, and experts in patient survey work in England and the US. We discussed and refined questions in the CAHPS team and tested them in a pilot interview (box 1). The Institutional Review Board at Harvard Medical School approved the study.

Recruitment of respondents

Team leaders were contacted in January 2004 while the collaborative was developing aims and measures. All eight leaders agreed to be interviewed and one of the authors (ED) visited 2 weeks later to conduct the interviews. Three of the leaders were medical directors, three were directors of clinical improvement or service quality, one was a group manager, and one was a quality improvement coordinator. They had worked in their medical groups for 2–20 years. Four of the leaders invited other colleagues whom they thought had relevant experience to participate in the interview—one director of customer relations, two clinical improvement coordinators, and three other managers—making a total of 14 respondents. Interviews lasted 60–90 minutes and were conducted in the work setting in all but one case. A transcript was returned to each team leader to review.

Analysis of data

One of the authors (ED) identified 58 comments about working with patient feedback or survey data from the transcripts and sorted them into 25 types of barriers. The barriers were then grouped together into three broader categories²² defining sub-themes within each. ED then reapplied this framework to the 16 initiatives that teams had reported to see if lack of these barriers was related to better use of patient survey data, and re-reviewed the transcripts for contradicting examples. The draft paper was shared with team leaders and discussed with them.

RESULTS

All medical groups reported initiatives for improving patient centred care over 1–6 years (box 2).

Each team identified more examples of barriers than of success and these were grouped into those that were (1) organisational, (2) professional, and (3) data related.

Organisational barriers

Lack of supporting values

Lack of an emphasis on patients’ needs in decision making at all levels of the organisation made it difficult to create a tension for change. Three respondents identified a traditional hierarchical management structure, the influence of personalities, and greater importance of staff needs as influences on group culture. For example:

“So that the first remark may be extremely un-patient friendly and you literally have to say: ‘That has no patient focus. Let’s think about why we’re doing this.’ But I think we have great support at the top in the management team in theory. The question is, do they remember it on a day-to-day basis?” (Team leader, Group 5)

Two medical groups described strategies to change their organisational culture towards patient centredness. One leader described realising that their culture was not supporting the generalisation of success from individual projects. Their response had been to “decide which battles to fight”, ensure they did not over commit, to bring clinicians into projects one by one, and to provide adequate administrative support for these.

Competing priorities

Competing priorities that detracted from an organisational focus on patient centred care were financial goals (three respondents), the number of patients to be seen (two respondents), and major restructuring (two respondents). For example:

“The reimbursement for spending time with people is dramatically less than that given for procedures. This results in a feeling that we don’t get paid for listening or supporting people … Providers are often poorly informed about the reason for the visit and what information has previously been gathered. The approach of encouraging people to ‘come in and then we will figure out what to do’ can leave the impression that the system is disorganised.” (Team leader, Group 6)

Lack of a quality improvement infrastructure

Effective response to patient feedback or survey data appeared to require the prior development of quality improvement structures, capacity, and skills. Without leadership committed to quality improvement it was difficult to integrate activity throughout the organisation to tackle complex areas such as patient centred care. For example:

“We had an executive committee that met every week but everything came to them. And they rarely finished anything because they didn’t know how to fix it and they had no other place to put it. So they’d table it and they’d say, ‘Isn’t there somebody that can work on that?’”

“Smaller medical groups don’t have access to things like statisticians and database creators. In fact there’s a huge lack that resource in health care. And I think that’s why we can’t excel at this as fast as manufacturing companies. So I work harder to develop new ways to get data out of old systems.” (Team leader, Group 1)

Seven groups said that ICSI had provided a focus for initiatives, maintaining momentum, gaining professional “buy in”, and teaching quality improvement skills. As this activity gained prominence, four groups reported having developed more coherent internal forums to make collective decisions, assign responsibility, develop champions, and use data to monitor progress.

Professional barriers

Many respondents reported that using survey data was a new concept which challenged traditional ways of working and thinking about measuring quality of care.

Clinical scepticism

All groups had experienced the sceptical responses of staff to survey results. Five reported that one concern—expressed primarily by clinicians—was the degree to which data should be taken seriously. For example:

“They’ve learned the term ‘statistically significant’. They’ve somehow picked that out and they’ll always ask if they are statistically significant. And we do spend a fair amount of time talking about, you know, it’s a good sample size, let’s talk about what we are using the data for, it is not a research study.” (Team leader, Group 3)

When findings seemed critical of, or inconsistent with, clinical experience, clinicians tended to want larger samples. One leader reported needing to wear “that bullet proof vest” while presenting data, and noted that qualitative case examples could paradoxically sometimes be taken more seriously than survey data.

Defensiveness and resistance to change

Five respondents noted patient survey data were potentially threatening. For example:

“There are often questions—‘My patients are sicker’, ‘My patients are different’, ‘My patients are this or that’. You can come up with any different variation, but I think we’ve probably heard them all! So it’s—you know—I think the standard push back that you would get.” (Manager, Group 2)

Three described the difficulty of changing doctors’ independent behaviour. For example: “Generally speaking, people agree with the data; they just don’t think it applies to them!” (Team leader, Group 2)

Projects attempting to influence clinicians were seen as requiring senior clinical “enthusiasts” to bring along “the majority”.

Lack of staff selection for skills

Two respondents identified the need for clinical staff to be able to elicit another’s point of view. For example:

“A lot of it has to do with … their attitude toward giving information and making sure they explain things. And those are habits and personality issues. Some do it extremely well, naturally, and others don’t do it as well. And it’s the ones who don’t do as well, I think it’s a real struggle because we’re asking them to change lifelong habits and that’s going to be hard.” (Team leader, Group 4)

Staff were noted to be selected for technical rather than “people” skills. For example, one group had found that, given the choice, most reception and administrative employees had opted to move away from patient contact to dealing with paperwork and finance. Managers responded by selecting employees for work matching their personality and to support teams to focus on patient needs. Two other groups had found that training reception staff in isolation was ineffective.

Three leaders reported knowing which staff members struggled with communication but a key issue was to avoid presenting them with an overly negative message that might label them as a “bad person”. In one group a senior physician had fed back comparative patient survey data. Results were not made public, but low and high performing clinicians were paired for mentoring and further training was offered if necessary. Three respondents felt that clinician specific data were essential to target interventions, although another was sceptical of using it for formal accountability.

Data related barriers

Lack of expertise

All but one respondent thought that special expertise was necessary to work with survey data. Two teams reported not having had time to synthesise or disseminate reports.

Lack of timely feedback

Three respondents mentioned the long delay from data collection to analysis and feedback as a major limitation. For example:

“It was old data … and it seems like by the time you get that type of data and by the time you look at reacting to it, it’s very easy for people—staff, physicians, whoever it might be—to say: ‘Well, you know that was a long time ago. We’ve already fixed that or that was when we had that receptionist’.” (Manager, Group 7)

Lack of specificity and discrimination

A significant issue for four teams was the need for data specific to a single clinic or patient group. Data suggesting a general problem or dissatisfaction rather than a specific care process that could be changed were seen as difficult to interpret and unlikely to lead to any action. For example:

“We didn’t really have much success with being able to put our hands around anything and really improve anything based on the information that we were getting.” (Manager, Group 7)

Three teams mentioned the problem of determining whether high scores were due to “halo effects”—where patients might be so grateful for treatment they were unwilling to criticise care. Two others mentioned “ceiling effects” at the top end of the scale that made it difficult to know what to focus on.

Uncertainty over effective interventions and rate of change

Three teams reported sustained high results or gradual improvement over several years and one reported no change, despite several interventions. The time from data collection to feedback, intervention, and further measurement made it difficult to infer what had caused what. Three groups reported success in using complaint data. One had found that the most common complaints related to difficulty in obtaining appointments. After a 2 year system redesign, such complaints became the least frequent category. The group used this success to justify turning attention to the next most common complaint of staff being disrespectful and uncaring.

Lack of cost effectiveness

Three teams mentioned the high cost of data collection. For example:

“The numbers weren’t relevant unless they did they did a huge survey. But when they did a huge survey it cost them too much money. They never did anything with it. The data were old and it was viewed as lost money. Where’s the incentive?” (Team leader, Group 1)

DISCUSSION

Limitations of study and summary of findings

This is a small study of experienced enthusiasts for patient centred care from eight medical groups in a unique quality improvement organisation. Their experience may not be representative of all organisations receiving and responding to survey data. Our interviews may have elicited idealised accounts. For example, we have no way of verifying that the interventions reported were actually successful or knowing if the respondent was a barrier, or of identifying all the barriers in each setting. However, the accounts were detailed and the respondents seemed very forthcoming and honest about the difficulties they had faced. The results might therefore provide some insight into why health professionals and managers find it so difficult to use survey results effectively. We identified three main types of factors—organisational, professional and data related—that had previously affected the use of patient survey data in quality improvement (see box 3 for a summary of the overall framework).

Comparison with other findings

Research on the effectiveness of using patient survey data in quality improvement is limited.²³ However, our findings are consistent with Rogut and Hudson’s conclusion that a structured process for addressing problems and obtaining resources was critical in marshalling energy to tackle the issues raised by surveys, as well as a strong motivating force to produce changes in staff behaviour.²⁰ They also found that some staff were frustrated by relatively small sample sizes, the long time it took to carry out the survey, and the fact that additional information had to be collected and considered before solutions could be targeted and action taken.²⁰ A case study of a single US hospital also identified very similar barriers: organisational (size, structures and strategy), characteristics of individuals (fear, scepticism, awareness, training and physician interest), and data problems (not being user centred or linked directly to care processes).²⁴ One Netherlands study found that, despite strong motivation, general practitioners found it difficult to use patients’ evaluations of care to change their behaviour and became sceptical of their value.²⁵

The barriers we identified are similar to those found in other studies of quality improvement. For example, Kaluzny and McLaughin²⁶ describe the steps in making improvements as awareness of a problem, identification of a solution, decision to implement it, institutionalisation (that is, the extent to which total quality management is integrated into ongoing activities of the organisation), and impact. They point out that “institutionalization is unlikely to take place without some observed positive impact”. Shortell and colleagues²⁷ proposed that the extent to which quality management is institutionalised is a function of the organisation’s structure, culture, and implementation approach.

Implications for practice

Survey data often provide information of which busy health professionals and healthcare systems were previously unaware, so findings may be surprising or uncomfortable. It makes little sense for healthcare systems to seek patients’ views and then to discount their concerns as either unrealistic or inevitable. Similarly, seeing clinicians as “the problem” seems neither helpful nor consistent with quality improvement approaches that seek to move away from individual blame to identifying and fixing system failures. Our results suggest that healthcare organisations need to develop cultures that support patient centred care, quality improvement capacity, professional receptiveness and leadership, and technical expertise with survey data. They also emphasise that surveys themselves do not indicate what needs to be done to improve any situation. Further commitment and ingenuity are needed to understand shortcomings in an organisation and to develop solutions.

Implications for future research and policy

More studies about how to use patient survey data effectively are needed. The characteristics of organisations that perform highly on patient centred care¹⁵ are likely to be different from factors needed to transform a low performing organisation into a high performing one. Retrospective case studies of organisations that have successfully improved their patient experience scores may help to identify successful strategies.²⁸ The use of patient surveys in the US and UK reveals differing strengths and weaknesses. In the US, many survey tools have been developed and used widely by health insurance plans. Recently the Centers for Medicare & Medicaid Services published data from national surveys of Medicare beneficiaries.^5,6,9 In England and Wales the NHS has also collected national data using comparable methods. Large databases covering patients with cancer, heart disease, mental health problems, and those attending hospital outpatients, emergency and primary trust are now available.^7,8 Recent analyses of these data show improvement in some areas linked to national priorities,²⁹ but little change elsewhere and little evidence that improvement is due to feedback. Current drawbacks include lack of local ownership of national data and too infrequent collection for effective monitoring or to keep up the momentum for change. A further weakness for both countries may be the belief that improvements in practice will somehow follow naturally from the fact that results are publicly reported. Evidence from this and other studies suggests that many barriers need to be removed before results will start to improve. In both countries policy makers need to seek ways of providing direction and support, information on effective approaches, and forums for organisations to share knowledge as they develop the kind of care that patients need.