Article Text


QUOTE-HIV: an instrument for assessing quality of HIV care from the patients’ perspective
  1. C F Hekkink1,
  2. H J Sixma3,
  3. L Wigersma4,
  4. C J Yzermans3,
  5. J T M van der Meer2,
  6. P J E Bindels1,
  7. K Brinkman5,
  8. S A Danner6
  1. 1Division of Clinical Methods & Public Health, Department of General Practice, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
  2. 2Division of Infectious Diseases, Tropical Medicine and AIDS, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
  3. 3Netherlands Institiute for Health Services Research (NIVEL), Utrecht, The Netherlands
  4. 4Royal Dutch Medical Association, Utrecht, The Netherlands
  5. 5Department of Internal Medicine, Onze Lieve Vrouwe Gasthuis, Amsterdam, The Netherlands
  6. 6Department of Internal Medicine, VU Medical Centre, Amsterdam, The Netherlands
  1. Correspondence to:
 C F Hekkink, Academic Medical Center, University of Amsterdam, Department of General Practice, PO Box 22660, 1100 DE Amsterdam, The Netherlands; 


Background: An HIV-specific version of the QUOTE questionnaire was developed to measure the quality of care of patients infected with HIV from the patients’ perspective. The consistency and validity of the questionnaire was assessed.

Methods: Focus group discussions were held to select aspects for inclusion in the questionnaire that are important to patients with HIV. Item and inter-item analysis, factor analysis, and reliability analysis were performed to test the internal consistency and validity of the questionnaire.

Results: Twenty seven items (13 generic and 14 HIV specific) were used in the QUOTE-HIV questionnaire. Separate factor analyses of the generic and HIV specific aspects indicated that each loaded onto a single factor. The internal consistency of the total questionnaire was good (Cronbach’s alpha ≥0.80). Feasibility of the questionnaire was shown by the diversity of importance and performance scores for general practitioners as well as for HIV specialists and AIDS nursing consultants.

Conclusion: The QUOTE-HIV questionnaire is a useful instrument for measuring the quality of care from the perspective of HIV infected patients.

Statistics from

The importance of incorporating the perspective of the patient when evaluating and designing healthcare programmes is now widely recognised.1 There are three points to consider when assessing quality of care from the perspective of specific groups of users of healthcare services such as patients with specific chronic diseases. Firstly, judgements on quality of care are often formulated by managers or healthcare professionals.2 However, patients’ views on the quality of health care differ from the views of healthcare professionals, managers, insurers, and policy makers3 and should therefore be assessed separately. Secondly, patients’ views on the quality of care have often been assessed by means of patient satisfaction questionnaires. However, instruments that assess patient satisfaction often produce highly skewed scores (90% or more of the respondents are satisfied); they produce absolute scores that are related to individual levels of expectations, needs, and wishes and contain items that are not formulated on the basis of systematic patient interviews.4 Finally, most existing instruments focus on the generic quality of care and not on disease related factors that refer to specific categories of patients. Although suitable in general population studies, such generic instruments do not provide for the specific needs and experiences of large subgroups within the general population.5

Some research has been carried out on the quality of care for people infected with HIV—for example, studies of HIV ambulatory care by Stone et al6 and Stein et al7—but these studies were done before the introduction of highly active antiretroviral therapy (HAART). Other studies have focused on inpatient care8,9 or were not done from the perspective of the patient.10,11

QUOTE instruments (Quality of care through the patient’s eyes) have been developed as part of the research project “Quality of Care from the Patients’ Perspective” and already exist for different categories of frequent users of healthcare services such as patients with asthma and/or COPD, rheumatic diseases, those with severe physical limitations, and frail elderly persons.12 The QUOTE framework is based on the following elements:

  1. Quality judgements of patients are based on the sequence: expectations → performance → importance and are expressed as quality improvement scores.

  2. The instruments are derived from a combination of qualitative and quantitative methods, with patients participating in the development process from the very beginning.

  3. The instruments are based on reports rather then the more subjective satisfaction ratings, following suggestions from Cleary and Edgman-Levitan13 that questions asking for reports tend to reflect better the quality of care and are more interpretable and actionable for quality improvement purposes than ratings of satisfaction or excellence.

  4. The QUOTE instruments reflect the multidimensionality of the care giving process and include generic as well as category specific quality aspects.

  5. The instruments can be used within quality assessment studies as well as projects aiming at quality improvement.

  6. The instruments have a modular structure which allows maximum flexibility.

We have developed an HIV specific version of the QUOTE questionnaire for measuring the quality of care from the perspective of HIV infected patients. The development of the questionnaire is described and its consistency and validity are assessed.


Questionnaire development

Focus group discussions or panels are often used for exploring a specific set of issues. The group is “focused” around a collective activity, varying from watching and discussing a movie to talking about a particular set of questions on one specific topic.14 Such a topic could include patients’ views on quality of care. Focus group discussions with experienced patients can result in a broad range of potential quality of care indicators from the patients’ point of view. Frequent users of healthcare services like the chronically ill become experts by experience15; people infected with HIV may be considered as one such group.

To make sure that we were dealing with subjects who were representative of those infected with HIV, participants were members of and were recruited through the Dutch HIV Patient Society at the end of 1998. Letters were sent to their members in which patients were asked to participate in the focus group discussions. Two focus groups were used to determine subjects’ opinions about what constitutes good quality of care from their point of view and the bottlenecks encountered in the current healthcare system. A flow chart detailing the development of the questionnaire is shown in fig 1.

Figure 1

Flow chart showing the development process of the QUOTE-HIV questionnaire.

The first session was an open group discussion of the definition of good quality of care and the problems to be resolved. These sessions were tape recorded and translated into formal statements of care aspects mentioned by either one or both groups. The first focus group session resulted in 56 statements of care aspects. In the second session the statements of care aspects derived from the first session were grouped and labelled by the participants and prioritised into five levels of importance ranging from 1 (relatively unimportant) to 5 (extremely important). This was done according to the concept mapping process to quantify the relative value of quality of care aspects derived from the focus groups. Concept mapping is a type of structured conceptualisation which can be used by groups to develop a conceptual framework to guide evaluation of planning.16

The QUOTE questionnaire contains 10 generic aspects. These aspects were included in the questionnaire regardless of their importance rating. Further selection of the aspects by the researchers was based on the sorting and rating of the aspects derived by the participants in the second focus group session. These were classified by the researchers into the following sub-dimensions of quality of care: continuity of care, treatment, accessibility, autonomy, costs, information, competence, efficiency, privacy, and accommodation. The most important aspect within each sub-dimension was selected for the questionnaire. The list was completed using all aspects with a relatively high importance score (>3.5) as derived from the concept mapping procedure and regardless of their sub-dimension.

Empirical testing

Based on the patients’ ratings, 17 items were selected and used in the questionnaire together with the 10 permanent generic QUOTE items (see table 2 for complete list). Quality aspects varied from items generally used in patient satisfaction scales to items tailored to the wishes of this specific group of patients. The final questionnaire contained 13 generic items, including the 10 generic items of the already existing QUOTE questionnaires, and 14 HIV specific items.

In order to pilot the questionnaire, two general practitioners who were not eligible for participation in the prospective study sent it to the patients with HIV in their practices. No reminder was sent. To verify whether the focus groups were successful in determining all relevant quality of care items, an open question asking about additional aspects was included in the questionnaire.


In the final questionnaire quality of care aspects were formulated as importance and performance statements. These statements refer to three healthcare providers: general practitioner (GP), medical specialist (SP), and AIDS nursing consultant (AC). Respondents were asked to rate all 27 aspects with respect to the three categories of healthcare providers. For the AIDS nursing consultant four items were not relevant and were left out (see table 4). Importance and performance were measured using a 4-point Likert scale. For the importance aspects scores were calculated for the following categories (1=“not important”, 2=“fairly important”, 3=“important”, and 4=“extremely important”) by linear transformation of standardised values (Z-scores) to values between 0 and 10.5 For the performance aspects the response categories (1=“no”, 2=“not really”, 3=“on the whole yes”, and 4=“yes”) were dichotomised into percentages “yes” and “no”. The performance score (P) represents the proportion of respondents who were not satisfied with the care received.

The concept “quality of health care from the patients’ perspective” was operationalised as the product of importance and (perceived) performance according to the formula: Qij = Iij × Pij. The quality improvement score (Q) on a health service (j) by an individual patient (i) is equal to the importance score (I) multiplied by the (perceived) performance score (P). This formula is based on the PES model suggested by Zastowny et al.17 The conceptual framework of the formula is further explained by Sixma et al.5 A high quality improvement score means that better care is recommended.

Statistical analyses

Data analyses included item and inter-item analyses (non-response, skewness, correlations) to test internal consistency, explorative factor analysis (principal component analysis with varimax rotation and Kaiser normalisation), and reliability analysis. Factor analysis was performed separately on the 13 generic and 14 HIV specific aspects. Reliability scores refer to the total scale as well as to the different subscales within the generic and HIV specific items. Reliability and validity of the scales were evaluated primarily by looking at the importance scores. These importance ratings are assumed to be more stable and less situation dependent than performance scores. Feasibility of the new QUOTE-HIV patients’ instrument was established by a comparison of the quality improvements scores within and between the different healthcare services evaluated. The analyses were performed using SPSS 10.0.7 for Windows (SPSS, Chicago, IL, USA).



The two focus panels met at the end of 1998 and comprised three women and 12 men infected with HIV of mean age 49 years (range 30–62) and a mean duration since diagnosis of HIV infection of 9 years (range 1–15). Eleven were using anti-retroviral therapy. Six were diagnosed with AIDS and three others had symptomatic HIV infection.

The questionnaire was sent to 80 people infected with HIV; 44 questionnaires were returned (55%). Responders were predominantly homosexual men educated to higher level with a variable time since HIV diagnosis (table 1), which is a good representation of the HIV infected population in Amsterdam.18

Table 1

Characteristics of the 44 participants who returned the questionnaire

Test characteristics of the questionnaire

With respect to the validity of the conceptual framework of the QUOTE instruments, separate analyses of the generic and HIV specific items indicated that each loaded onto a single factor (table 2).

Table 2

Factor loadings of the QUOTE-HIV (n=44)

For the total questionnaire the internal consistency as represented by Cronbach’s alpha (≥0.80) was good (table 3). The generic and HIV specific parts were found to have sufficiently reliable scales (≥0.70) except for the HIV specific part of the GP importance scale (0.60). Inter-item correlations indicating the discrimination values of the items are also shown in table 3.

Table 3

Reliability of scales for importance and performance components for general practitioners (GP), medical specialists (SP), and AIDS nursing consultants (AC)

To test the feasibility of the QUOTE-HIV instrument the 27 items were applied to the quality of care given by GPs, HIV specialists, and AIDS nursing consultants. Table 4 shows three scores for each care provider: the importance score (I), the performance score (P), and the quality improvement score (Q).

Table 4

Importance (I), performance (P), and quality improvement (Q) scores of 44 HIV patients for different care providers

The mean importance score shows what is important to patients with each care provider together with their expectations. For GPs, patients rated aspect 3 (“to be taken seriously”) of the highest importance (I=8.5), and for specialists and AIDS nursing consultants they rated aspect 16 (“having special knowledge about HIV”) of the highest importance (I-SP=10; I-AC=8.6). Importance scores ranged from 8.5 to 2.7 for GPs, from 10.0 to 3.7 for specialists, and from 8.6 to 4.2 for AIDS nursing consultants on a scale from 0 (“unimportant”) to 10 (“extremely important”).

The performance score shows the percentage of patients who were not satisfied with the delivered care—for example, the performance score of the specialist for aspect 7 (“never keeps me waiting longer than 15 minutes”) is 0.27, which means that 27% of the respondents reported having to wait longer than 15 minutes in the specialist’s waiting room.

The quality improvement score shows the relative score of priorities and performances on quality. The importance of aspect 6 (“can be easily reached by phone”) for the specialist is valued at 6.5 with 31% of patients reporting that they could not reach their specialist by telephone. The quality improvement score of 2.0 is rather high as a result of the product of a relatively high performance score (P=0.31) and a medium importance score (I=6.5). Thus, according to patients, quality improvement is needed on this aspect.

The quality improvement score can also be used to compare within and between professions and institutions on both individual aspects and combinations of aspects. The quality improvement score of one aspect over several types of healthcare services can be considered. The quality improvement score for aspect 4 (“being aware of my situation at home and at work”) differed for the GP (Q=1.1), specialist (Q=2.0), and AIDS nursing consultant (Q=2.7). This indicates that, from the perspective of patients with HIV, quality improvement concerning this aspect is needed for AIDS nursing consultants and (to a somewhat lesser extent) medical specialists.


Focus group discussions resulted in 27 items (13 generic and 14 HIV specific) being used in the QUOTE-HIV questionnaire. Factor analysis showed that the questionnaire could be divided into two parts: a generic part and an HIV specific part. The internal consistency of the total questionnaire was good (Cronbach’s alpha ≥0.80). Feasibility of the questionnaire was shown by the diversity of importance and performance scores for GPs, HIV specialists, and AIDS nursing consultants. The QUOTE-HIV therefore seems to be a useful instrument for measuring the quality of care from the perspective of patients infected with HIV.

Working with focus groups has its limitations. Firstly, the result of the discussions may be biased by the fact that only a small number of patients were involved, all of whom were members of the Dutch HIV Patient Society. To find out whether the focus groups were successful in determining the relevant quality of care items, an open question was included in the questionnaire. Some patients used this question to comment on the list of items but no new items were mentioned, so we can assume that the questionnaire contains the most relevant items of quality of care for people infected with HIV. Secondly, working with focus group panels includes a subjective element by the moderator and observer(s) who are part of the group dynamics. This influence is minimised by using more than one panel and by recording the meeting on audio or videotape.

We did not subdivide each sub-dimension (continuity of care, autonomy, etc) into different scales because the questionnaire has to be easy to handle and can therefore contain only a limited number of aspects.

With respect to the quality of care aspects reported in table 3, in those such as “take enough time to talk with me”, “treated by the same person”, and “is organised in such way I cannot hear conversations at the desk or in the consulting room” it is hard to distinguish parts of the items from those used in many other satisfaction with care studies such as those reported by Hall and Dornan19 and Wensing.20 However, this is only true for the part of the items which looks at their content and at the way they were formulated. Most items are more informative and more practical than those in existing patient satisfaction scales and better reflect the needs of specific categories of patients.8 For instance, items concerning medication can be expected to be important to everyone in general, but for HIV infected persons who have to take a lot of drugs in complicated dosing schedules such items are especially important. The patients in our study mentioned three medication aspects (“give information about possible side effects of drugs”, “inform me about the pros and cons of a treatment”, and “give information about the use of my HIV medication”) among the five most important aspects of specialist care. Other instruments developed from a patient’s viewpoint to measure patients’ satisfaction such as the QPP (Quality from Patients’ Perspective),21 the PSQ (Patient Satisfaction Questionnaire),22 and the PPS (Patient Perspective Survey)23 have been developed for patients in general. These questionnaires do not focus on patients with a specific disease, and this is an important difference from the principles of each QUOTE questionnaire.

The QUOTE-HIV questionnaire differs from traditional patient satisfaction questionnaires in several respects. Firstly, unlike most satisfaction instruments, QUOTE-HIV offers an importance as well as a performance component so the importance component can be added as a weight factor in the judgement of quality of care. For an aspect with a low performance score that is not rated as highly important one can determine whether improvement is needed, unlike those in which an interaction between performance and importance scores gives a high quality improvement score. The QUOTE-HIV questionnaire is therefore exceedingly useful for identifying aspects that really need improvement. Secondly, patients played a crucial role in the development of this instrument, which ensures that the items important to them are included. As a result, an equal number of items encountered specifically by HIV infected patients as well as known patient satisfaction items were generated.

Key messages

  • The QUOTE-HIV is a questionnaire developed from the patients’ perspective to judge the quality of care for patients infected with HIV.

  • Unlike most satisfaction instruments, QUOTE-HIV has an importance as well as a performance component.

  • The QUOTE-HIV instrument allows a quality improvement score to be calculated which indicates where quality improvement might be most useful.

In table 4 a quality improvement score of 3.5 for GPs is shown for the aspect “give information about my HIV medication”. The QUOTE-HIV questionnaire therefore indicates the need for an education course for GPs in which more attention is paid to the use of HIV medication.

To limit the number of items in the questionnaire it could be restricted to a single time rating of the importance of an item in general. We wanted to compare importance scores as well as performance scores for GPs and HIV specialists and therefore asked for an importance rating for each healthcare provider separately.

Further research is needed for specific patient groups. We did not look at different disease categories or medication use; future studies should validate the instrument for specific categories.

We consider that this instrument can be used widely although some adaptation to the local situation in other parts of Europe might be warranted. Healthcare systems in Europe differ between countries and sometimes between regions in the same country, so the instrument has been provided with a method to adapt it to specific characteristics of health care. For such cross-cultural validation, additional focus group discussions with patients will play an important role. The scope of the instrument is not limited to providers but can be used for functions of care such as the provision of care for non-institutionalised patients with HIV.


View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.