Article Text

PDF

Research on patients' views in the evaluation and improvement of quality of care
  1. M Wensing1,
  2. G Elwyn2
  1. 1Centre for Quality of Care Research, University Medical Centre St Radboud, P O Box 9101, 6500 HB Nijmegen, The Netherlands
  2. 2Department of General Practice, University of Wales College of Medicine, Heath Park, Cardiff CF4 4XN, UK
  1. Correspondence to:
 Dr M Wensing, Centre for Quality of Care Research, University Medical Centre St Radboud, P O Box 9101, 6500 HB Nijmegen, The Netherlands;
 M.Wensing{at}hsv.kun.nl

Abstract

The identification of methods for assessing the views of patients on health care has only developed over the last decade or so. The use of patients' views to improve healthcare delivery requires valid and reliable measurement methods. Four approaches are recognised: inclusion of patients' views in the information to those seeking health care, identification of patient preferences in episodes of care, patient feedback on delivery of health care, and patients' views in decision making on healthcare systems. Outcome measures for the evaluation of the use of patients' views should reflect the aims in terms of processes or outcomes of care, including possible negative consequences. Rigorous methodologies for the evaluation of methods have yet to be implemented.

  • patients' views
  • patient preference
  • research methods

Statistics from Altmetric.com

Collecting the views of service users has been a key feature of recent developments in society, but it is only over the last decade or so that the healthcare sector has identified methods for assessing the views of patients. A range of methods is available to integrate patients' views on the delivery and improvement of health care, including short questionnaires to assess patients' needs before a consultation with the clinician, focus groups to include patients' views on clinical guidelines, and surveys to provide patient feedback to care providers. If such methods are used for the evaluation and improvement of healthcare systems, they should be studied in terms of effectiveness, efficiency, and maybe even safety.1 This review shows that this research area has yet to implement rigorous approaches to the collection and synthesis of patients' views. Some of the key issues related to the measurement of patients' views and their use in healthcare improvement are considered, together with ways in which the methods themselves may be evaluated.

MEASURES

Patients' views have different dimensions (table 1). This papers focuses on patients' views of health care—specifically on their preferences, evaluations, and reports.

Table 1

Classification of measures of patients' views

Preferences

Preferences are ideas about what should occur in healthcare systems.2 Related concepts are expectations, perceived needs, desires, wants, requests and priorities. Expectations have two distinct meanings: beliefs about what should occur or what people want of care (“normative expectations”), and beliefs about what will actually happen, irrespective of whether this is wanted (“predicted expectations”).2 The term “preferences”, which has its origins in cognitive psychology and economics, is most often used to refer to individual patients' views about their clinical treatment. The term “priorities” is more often used to describe preferences for healthcare services in a population of patients or citizens.3 Qualitative as well as quantitative methods can be used to study preferences.

Qualitative research methods such as individual interviews and focus groups can be used to elicit preferences. These methods often use open ended approaches such as topic lists rather than structured questionnaires. It is often difficult for patients to decide what is important in general terms, given the limited experience of any one individual. Focus groups generate interaction among participants which may lead to shared views that transcend individual experiences. It may be helpful to present realistic but hypothetical situations as a trigger for discussions. Facilitation skills may be needed to test whether the group views are well considered and stable.

Quantitative methods for eliciting preferences include surveys and consensus methods such as the Delphi and nominal group techniques. Different types of data can be collected including scale responses that range from “not important” to “extremely important”; rankings—for example, preferences expressed in paired comparisons of alternatives; choice of alternatives—for example, a vote for the most desirable alternatives. Individuals can be asked to rate, rank, or vote for different care providers (GP or hospital) or attributes of care providers—for example, short waiting list, adequate information delivery. In a study of patient priorities in different countries high correlations were found between different methods of rating, ranking of, and voting for aspects of general practice care.4 A study of different methods for eliciting treatment preferences, however, found significant differences.5 A range of methods has been developed to collect preference data such as the expectancy value model, multi-attribute utility models, and conjoint analysis models (discrete choice experiments).6

A number of methodological issues related to the use of methods for priority setting in health care have been described.3 A first problem is how “options” are generated: limiting the choices will limit the preference frame. Patients should contribute to the development of a preference framework but they usually lack the expertise to generate a model completely on their own. Decisions in prioritisation issues in healthcare systems inevitably involve a wide array of factors, so methods have to be able to incorporate multidimensional influences. The most realistic methods involve presenting constrained choices where trade offs have to be made between different attributes or alternatives. It is important to be explicit about the methods used for the aggregation of individual preferences because different procedures will lead to different results. Researchers should understand that the choice of methods will influence results and that the assessment processes are at least transparent if methods have the potential to over-represent the views of some population sectors over others. An example of the use of patient preferences is given in box 1.

Box 1 Patient preferences for in vitro fertilisation 7

A conjoint analysis model was developed to predict individuals' preference for receiving different components of an in vitro fertilisation service. Six relevant attributes were identified: chance of taking home a baby, follow up support, time on the waiting list, continuity of staff, cost, and attitudes of staff. Note that these attributes include health outcomes, non-health outcomes, and process attributes. Realistic levels for these attributes were chosen; for instance, “chance of taking home a baby” had the levels 5%, 10%, 15%, 25%, and 35%. A selection of 26 scenarios was chosen (from a possible list of 1000) to achieve a “manageable” option listing for respondents. The 26 scenarios were randomly split into two equal groups and within each group 12 pair wise comparisons were formulated for assessment by randomly selected patients. Regression analysis techniques produced a predictive model of patient preferences for an in vitro fertilisation service. The preferred attributes in this model were staff attitudes, continuity of care, follow up, and chance of taking a baby home.

Evaluations

Patient evaluations are “health care recipients' reactions to salient aspects of the context, process, and result of their service experience”.8 Related concepts are “satisfaction”, “unmet needs”, “judgements”, “complaints”, and “comments”. The term “evaluation” suggests a cognitive process in which specific aspects of care are assessed, while “satisfaction” refers to an emotional response to the whole experience in health care. The term “patient satisfaction” is probably most often used in the literature.

Many studies have used written questionnaires that comprise structured questions with some sort of rating scale. The overall satisfaction with the healthcare experience is usually very high and this often masks less positive evaluations when aspects that are more specific are explored. A literature review showed that questionnaires that asked for evaluations in terms of “satisfaction/dissatisfaction” showed less discrimination than questionnaires that used terms such as “good/bad” or “agree/disagree” with very concrete aspects of care.9 Some questionnaires measure both preferences and experiences and derive evaluations from these two factors by calculating difference or ratio scores.10 There is some evidence that patients distinguish between the two concepts,11 but there is no validated framework for deriving evaluations from preferences and experiences.12

Patients have evaluative responses to experiences in health care which are not necessarily translated into satisfaction,13 and qualitative methods can be used to examine these in more depth. An example is shown in box 2. Qualitative approaches are particularly useful for exploring patients' views in areas that have not been fully elaborated. Thorough data analysis of qualitative material is time consuming. Pragmatic approaches such as logging key themes without undertaking full transcription analyses may be used but, as far as we are aware, the reliability and validity of such approaches have not been assessed.

Box 2 Patient evaluations of low back pain management14

Twenty patients who consulted the general practitioner for low back pain were interviewed shortly after their visit. The general practitioners were also interviewed. A topic list of the key components of a low back pain clinical guideline was used. The data were transcribed and analysed qualitatively. The results revealed that patients often had limited expectations of the consultation. They wanted to hear a diagnosis and expected to receive simple advice. All patients said they complied with the most important advice, which is to stay active, although a few had ideas about possible damage to their back after physical exercise. Patients said they would only take medication if it was strictly necessary, although the guideline recommended analgesics at regular intervals independent of pain. Only one patient demanded physical therapy, although many general practitioners perceived that patients wanted this. Although patients and their general practitioners were satisfied with the chosen management, this study provided deeper understanding of the gap between professional advice and patient motivation to change or act on advice.

Reports

Patient reports represent objective observations of organisation or process of care by patients, regardless of their preferences or evaluations.15 Patients' experiences and their perceptions of professional performance are similar concepts. Patients can, for instance, register how long they had to wait in the waiting room, irrespective of whether this was too long or not. Although reports reflect patients' observations, they do not necessarily imply a patient's perspective on the quality of care. Nevertheless, patient reports can be used for quality improvement. In some situations patients' reports are the most accurate observation method if, for instance, the data are required about a patient's pathway through different healthcare institutions.

Validity of instruments

Instruments for assessment of patient views of health care should be validated to ensure that the tools measure what they are supposed to measure. A review of 195 studies of patient satisfaction published in 1994 showed that only 89 (46%) reported some validity or reliability data and only 11 (6% of 181 quantitative studies) reported content validity and criterion validity or construct validity and reliability.16

Ideally, the instrument should be compared with a gold standard or a criterion measure (an instrument with established validity). For instance, patient reports on the care received can be compared with the medical records or clinicians' reports on the care delivered (box 3). This approach is comparable with the validation of a diagnostic test. A criterion measure for preferences or evaluations is not often available, however, so other approaches are needed.

Box 3 Accuracy of patient reports15

Reports of 380 patients obtained through telephone surveys were compared with medical records which were considered to be a gold standard (a disputable perspective as omissions and document loss confound the measures). For chest radiography, mammography, and electrocardiography, patient reports showed high sensitivity and specificity. For serum cholesterol tests, patients proved to be sensitive but not specific reporters. For blood pressure measurements, faecal tests, and rectal examination, false negative rates were below 0.10. They were somewhat higher for breast self-examination instruction and pelvic examination (0.21–0.22). For testicular self-examination instruction patient reports failed to confirm medical record documentation (false negative rate=0.53).

The validity of most instruments for patients' views should be based on conceptual frameworks that describe a specific domain (the relevant aspects of health care) and it is preferable if patients have been consulted regarding the selection and description of the relevant aspects. Qualitative studies are particularly suitable for that purpose. For instance, the Europep instrument for patients' evaluations of general practice care covers medical care, interpersonal relationships, information and support, and organisation of care. The aspects were selected on the basis of literature studies and qualitative and quantitative studies (box 4).

Box 4 Europep instrument 17

The Europep questionnaire elicits patients' evaluations of general practice care and provides feedback to general practitioners. The originators aimed to develop an instrument that reflected patients' priorities regarding the main areas of general practice care. Validation studies focused on an adequate selection of aspects of care and phrasing of items using a series of systematic evaluations. Literature studies and patient surveys in eight countries were performed to determine these priorities. Preliminary questionnaires were tested in qualitative and quantitative pilot studies. The pre-final 44 item version was formally prepared for international use using forward and backward questionnaire translations. Selection of the final 23 item version was based on the following criteria:

  • The questionnaire should cover five main dimensions: interpersonal relationship; medical care; information and support; continuity and cooperation; facilities, availability and accessibility.

  • Specific items were included if these referred to aspects of care which were prioritised by patients, showed high item response and reasonable discrimination across patients in most countries. Quantitative cut off points were defined for these criteria.

  • Items were excluded if a serious ambiguity or translation problem was found.

Sometimes it is possible to verify whether patients' views are associated with other factors or whether the measure meets criteria set by the theory from which the measure has been derived (construct validity). For instance, most patient satisfaction studies have shown that older patients have more positive evaluations of health care than younger patients. It can therefore be predicted that a new measure for patient satisfaction will show similar associations.

Psychometrics

Quantitative instruments should have adequate psychometric features.18 High item response rates will indicate the presence of questions that are more likely to be relevant and understandable. Some instruments, however, are intended to identify rare events such as medical errors (complaint procedures) or side effects of medications (surveys among drug users). High item response rates are not relevant in such cases. Instruments designed to measure aspects of quality should also show good variation across patients (discrimination) and variation between measurements at different points in time (responsiveness). If indicators are supposed to be clustered within dimensions, validity is supported by proven unidimensionality and high internal consistency. However, not all instruments assume clustering between indicators because indicators may not be seen as repeated measurements. This will be the case for many instruments that measure reports of concrete aspects of care. Ideally, instruments will also show good test-retest reliability.

The most often used reliability coefficients refer to the internal consistency of items within a dimension per patient—for example, Cronbach's alpha. In the context of quality improvement, however, aggregated scores per care provider are often needed—that is, aggregation over many individuals. These figures are based on a number of indicators and a number of patients or events. An example is the percentage of patients with positive evaluations of the accessibility of care in a specific hospital based on a survey of 100 patients who answered 10 different questions on accessibility. Generalisability analyses can be used to calculate reliability coefficients for the aggregated scores.19 It appears that an increase in the number of patients often has more influence on the reliability of the aggregated scores than an increase in the number of indicators.

Sampling

The inclusion criteria for the study population determine the generalisability or external validity of a study or audit. For instance, measurements among patients who attend a clinician are not generalisable to the general patient population registered at a practice or a population of Internet users who visit a site on a health problem. Qualitative studies use theoretical sampling to achieve a specific sample which may be heterogeneous or homogenous, depending on the overall purpose of the work. Quantitative studies use many different forms of sampling methods to achieve a representative sample (random, stratified, etc). It is important to achieve high response rates and low dropouts in order to avoid selection bias (except if this was sought). Non-responders are more likely to be represented by those who are ill, less satisfied with care provided, and less frequent users of health care than responders.20,21 Surveys of interview methods need to consider the impact in case these groups are excluded or drop out.

Response rates in surveys among patients vary considerably. A literature review reported a mean of 60% in response rates and a standard deviation of 21%.22 Many factors may influence the response rate of a survey, such as the motivation of the clinician to recruit patients, the attractiveness of the lay out of a questionnaire, the method of administering the questionnaire to patients, the use of monetary incentives, and possibly the use of information technology for administering questionnaires. Insight into which factors are most relevant is limited. A comparison of handing out questionnaires to visitors to the general practice and mailing questionnaires to patients at home gave response rates of 72% and 63%, respectively; the content of the answers of both sample populations was, however, largely similar.23 A randomised trial showed that written reminders could improve the response rates unless the rate was already above 80%.24

USE OF PATIENTS' VIEWS FOR QUALITY IMPROVEMENT

Table 2 outlines the potential use of patients' views in healthcare delivery and quality improvement.25 One approach focuses on those who want to make choices about their utilisation of healthcare services. Health education materials may include information on patients' views based, for instance, on qualitative research of their experiences in health care. Public reports on the performance of different care providers may include information about patients' evaluations of care.26 A comparison with other care providers requires adequate adjustment for case mix, which is difficult because insight into predictors of patient evaluations of care (and most other indicators) is limited.26

Table 2

Use of patients' views for quality improvement

Another approach focuses on patients in episodes of care using, for instance, shared decision making strategies or patient-held records. Identification of patient preferences is part of most of these approaches. For instance, shared decision making implies that the care provider gives information on relevant options, assesses patient preferences regarding these options, and takes a decision with or checks approval of the patient.27

A third approach provides different types of patient feedback on healthcare received derived from surveys, patient groups, or complaint procedures. These views can be used for continuing education and service improvements. Patient views can be compared with ethical or clinical guidelines for good practice but, in many cases, such standards are difficult to define. An exception is a lawsuit where an ethical or legal assessment is explicitly sought. Comparison with other care providers can help to prioritise issues that need attention.

A fourth approach focuses on the involvement of patients and the public in the design and planning of healthcare systems. This requires information on patients' views such as studies of patient priorities or the assessment of local needs for health care. Patient organisations express the views of patients who coalesce around issues or conditions and these may differ from the aggregated views of individuals in wider populations. In these situations, patients' views are only one of a number of inputs into a wider policy making process.

RELEVANT OUTCOMES

Methods to identify and use patients' views for the improvement of health care can be seen as a technology which should be evaluated in terms of effectiveness and efficiency. The choice of relevant outcomes for the use of patients' views for quality improvement requires further attention. It appears logical to derive outcome measures for the evaluation from the underlying objectives of this effort (table 3).

Table 3

Objectives of patient involvement and relevant measures

It is an ethical and legal rule that patients should be informed and involved in their health care, at least to minimal standards. Many patients wish to be involved in the decision processes, at least to some extent.28 In line with this aim, the process of involvement rather than its outcome is crucial and so it is the ethical principles and patient preferences that define the criteria for effectiveness. For instance, shared decision making can be evaluated in terms of information delivered on treatment options, checking of understanding and preferences, and making a shared decision.29

Patient involvement may also result in better processes and outcomes of care. It could, for instance, make clinicians more responsive to patient preferences, contribute to a better implementation of clinical guidelines, and result in better adherence to treatment, health status and satisfaction with care. Patients can be seen as co-producers of health care because their decisions and behaviour influence healthcare provision and its outcomes. Outcome measures should reflect the effects on process or outcomes of care that are expected.

Integration of patients' views may be driven by political and strategic motivations such as protection of a position in a competitive healthcare market, the wish to have democratic control in the healthcare organisation, or the need to do something for underserved populations. Such aims may be difficult to assess, but measurable outcome measures can be found in some cases—for instance, position on the healthcare market can be evaluated in terms of attendance rates and turnover of patients.

Finally, evaluations should consider possible negative consequences such as unrealistic patient expectations of what health care can deliver; defensive behaviour of care providers, resulting in higher numbers of unnecessary clinical procedures; undermining of professional morale; and increased costs. Such consequences are not imaginary. A randomised trial on low back pain showed that 80% would have chosen radiography if available, but that patients who received radiography often had more pain at 3 months than the control group and were nevertheless more satisfied with the care provided.30

Key messages

  • Patients' views include preferences (ideas about what should occur), evaluations (judgements of aspects of care), and reports (observations of organisation or process of care).

  • The validity of measures of patients' views should be based on conceptual frameworks, preferably derived from rigorous qualitative studies.

  • Effective methods for reporting information on patients' views are needed to influence and improve process and outcomes within healthcare systems.

Not only should the effects of specific methods be studied, but also their actual uptake in health care. Clinicians and patients may lack competence or skills to use specific instruments or have negative attitudes regarding specific approaches. Organisational structures may limit the application of specific methods. Such barriers need to be identified and addressed by means of targeted strategies which should be evaluated in terms of success of uptake of the methods.

CONCLUSIONS

A range of approaches is available to integrate patients' views into healthcare delivery systems and their improvement. The methods to measure and use patients' views should be studied in the context of their intended application. Quantitative as well as qualitative approaches can be used to measure patients' views, and the validity and reliability of the methods should be examined. The effectiveness and efficiency of the methods should be studied in terms of their consequences for process and outcomes of health care. Increased patient participation in health care can be seen as desirable in itself, but this should not inhibit evaluation of the methods used to achieve this aim.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.