Background: Measurement of the quality of health care is essential for quality improvement, and patients are an underused source of data about quality of care. We describe the adaptation of a set of USA quality indicators for use in patient interview surveys in England, to measure the extent to which older patients receive a broad range of effective healthcare interventions in both primary and secondary care.
Method: One hundred and nineteen quality indicators covering 16 clinical areas, based on a set of indicators for the care of vulnerable elderly patients in the USA, were reviewed by a panel of 10 clinical experts in England. A modified version of the RAND/UCLA appropriateness method was used and panel members were supplied with literature reviews summarising the evidence base for each quality indicator. The indicators were sent for comment before the panel meeting to UK charitable organisations for older people.
Results: The panel rated 102 of the 119 indicators (86%) as valid for use in England; 17 (14%) were rejected as invalid. All 58 indicators about treatment or continuity and follow up were rated as valid compared with just over half (13 of 24) of the indicators about screening.
Conclusions: These 102 indicators are suitable for use in patient interview surveys, including the English Longitudinal Study of Ageing (ELSA). The systematic measurement of quality of care at the population level and identification of gaps in quality is essential for quality improvement. There is potential for transfer of quality indicators between countries, at least for the health care of older people.
- quality of care
- quality indicators
- health surveys
- geriatric medicine
Statistics from Altmetric.com
Remarkable variations in the care provided by individual physicians are widespread and well documented, and reflect the extent of clinical uncertainty about treatment.1 Researchers at RAND and UCLA developed a method for combining the best available research evidence with expert opinion to assess the appropriateness of treatment.2,3 Modifications of this method have been widely used to develop quality indicators for health care.
Quality indicators have been used to show that effective health care is delivered only about half as often as it could be in the USA,4,5 and as many or perhaps even more opportunities for effective care may be being missed in British general practice.6,7 However, we still know surprisingly little about how many people with common medical conditions in the UK receive effective interventions when needed. An independent review published at the end of 2003 concluded that we do not have enough data to know whether the NHS is getting better or worse, particularly for conditions not covered by national initiatives.8 Better measurement of quality at health system level is essential for quality improvement.
Quality measures can be classified into three types. Structure measures consider factors such as the availability of facilities and staff, process measures consider whether treatment adheres to agreed good practice, and outcome measures consider the resulting changes in health status. There are advantages in using measures of process: (1) processes are a more efficient measure of quality;9 (2) for most conditions there is insufficient information to adequately adjust outcomes for differences in case mix between providers; and (3) processes of care are amenable to direct action by providers.10 Where the evidence exists, the healthcare processes should be linked to an improved health outcome.
The thoughtful use of the right quality indicators has the potential to improve health outcomes, at least in specific disease areas.11,12 However, concerns about the increased use of quality indicators remain. The desire to quantify may lead to neglect of other areas of health care where quality is particularly difficult to quantify. This is particularly relevant for older people5 who are major recipients of health care, often for several conditions at once,13 delivered across the boundary between primary and secondary care. There have also been problems with the implementation of quality indicators, and in England the Department of Health has been criticised for using indicators that are not based on evidence of effectiveness12 and for using indicators where the NHS lacks capacity to produce valid data.8
The possible neglect of some care may be reduced by more engagement of the public and patients in the implementation of quality indicators. One approach to public involvement is to assess quality through interviews with patients, an underused source of data about healthcare quality which avoids the problems of extracting data from clinical records.14 The risk of excessive focus on a few specific areas may be less if indicators describe a standard which, if not met, would almost certainly identify poor quality care rather than the optimal care described by practice guidelines.
This paper presents an independently developed set of evidence based indicators capable of measuring the quality of health care of older adults at the level of the health system by covering a broad range of conditions simultaneously, across primary and secondary care, and suitable for use in an interview survey of patients. They are intended for use in interview surveys, including the English Longitudinal Study of Ageing (ELSA), a new national cohort study that offers the opportunity to measure quality of care systematically from a population perspective in 2004.15
The development of evidence based quality indicators de novo is resource intensive so we adapted a comprehensive set of evidence based indicators of care quality for older people published by RAND in the USA in October 2001. The Assessing the Care of Vulnerable Elders (ACOVE) project developed 236 quality indicators in 22 clinical areas for the health care of community dwelling people aged over 75 years.16 The ACOVE conditions were chosen according to their importance in older people and the potential for quality improvement. For inclusion in our study, clinical areas were selected on the basis of both prevalence and importance as a cause of disability for people aged 65 years and older in the Health Survey for England 2000. For these clinical areas we selected all 93 ACOVE quality indicators that could be used in an interview survey without the use of clinical records. Cross checks by the ACOVE researchers between interview data and medical notes where feasible had found that the ACOVE interview data were comparable with notes data, and for some indicators it appeared that respondents remembered higher rates of appropriate interventions than had been documented in the notes.
There are many similarities in the evidence base for medical practice in England and the USA, but considerable differences in the way that services are organised and financed. Previous research has suggested that quality indicators cannot simply be transferred directly between countries without an intermediate process to allow for variation in professional culture or clinical practice.17 We therefore recruited a panel of 10 clinical experts to rate the 93 ACOVE indicators for appropriateness for use in England, alongside another 26 indicators suggested by the panel using a modified version of the RAND/UCLA appropriateness method.2 The method used two rounds of anonymous ratings on a scale ranging from 1 to 9, with a face to face discussion between rounds. Panel members were supplied with literature reviews summarising the evidence base for each quality indicator.
The 16 clinical areas covered by the indicators are: continuity of care, dementia, depression, diabetes mellitus, falls and mobility disorders, hearing loss, hypertension, ischaemic heart disease, medication use, osteoarthritis, osteoporosis, pain management, screening and prevention, stroke and atrial fibrillation, urinary incontinence, and vision. The indicators were also classified into five domains of care: continuity and follow up, diagnosis, prevention, screening, and treatment. The domains of prevention and screening include the indicators in the clinical area of screening and prevention as well as those indicators in other clinical areas that refer to either screening or prevention. For example, the indicator “All persons aged 65 or older should be offered a hearing screen at least once” is in the “hearing” clinical area as well as the “screening” domain.
Box 1 Examples of quality indicators rated as valid* by expert clinical panel
Continuity of care
IF a person aged 65 or older is deaf or does not speak English, THEN appropriate assistance for communication should be provided to facilitate communication between the older person and the doctor or nurse, such as hearing aids, an interpreter, or written or translated materials.
IF a person aged 65 or older has mild to moderate Alzheimer’s disease, THEN the treating physician should discuss treatment with a cholinesterase inhibitor with the patient and the primary caregiver (if available).
IF a person aged 65 or older is diagnosed with clinical depression, THEN antidepressant treatment, talking treatment, or electroconvulsive therapy should be offered within 2 weeks after diagnosis unless within that period the patient has improved, or unless the patient has substance abuse or dependence, in which case treatment may wait until 8 weeks after the patient is in a drug- or alcohol-free state.
IF a person aged 65 or older has diabetes, THEN his or her glycosylated haemoglobin or fructosamine level should be measured at least annually.
IF a diabetic person aged 65 or older has one additional cardiac risk factor (i.e. smoker, hypertension, hypercholesterolemia, or renal insufficiency/microalbuminuria), THEN he/she should be offered an angiotensin converting enzyme inhibitor or receptor blocker.
Falls and mobility disorders
IF a person aged 65 or older reported two or more falls in the past year or a single fall with injury requiring treatment, THEN the patient should be offered a multidisciplinary falls assessment.
IF a person aged 65 or older has a problem with hearing, THEN he or she should be offered a formal audiological evaluation within 3 months.
IF a person aged 65 or older remains hypertensive after non-pharmacological intervention, THEN pharmacological antihypertensive treatment should be initiated.
Ischaemic heart disease
IF a person aged 65 or older has had a myocardial infarction, THEN he or she should be offered a β-blocker.
IF a person aged 65 or older is prescribed a new drug, THEN the patient (or, if the patient is incapable, a caregiver) should receive education about the purpose of the drug, how to take it, and expected side effects or important adverse reactions.
IF a person aged 65 or older with severe symptomatic osteoarthritis of the knee or hip has failed to respond to non-pharmacological and pharmacological therapy, THEN the patient should be offered referral to an orthopaedic surgeon to be evaluated for total joint replacement within 6 months unless surgery is contraindicated.
ALL women aged 65 or older should be offered advice at least once regarding intake of dietary calcium and vitamin D and weight bearing exercises.
IF a person aged 65 or older is treated for a chronic painful condition, THEN he or she should be assessed for a response within 6 weeks.
Screening and prevention
IF a person aged 65 or older has no history of anaphylactic hypersensitivity to eggs or to other components of the influenza vaccine, THEN the patient should be offered an annual influenza vaccination.
Stroke and atrial fibrillation
IF a person aged 65 or older is admitted to the hospital with a diagnosis of acute ischaemic or haemorrhagic stroke, THEN he or she should be admitted to a specialised acute or combined acute and rehabilitative stroke unit, or transferred to a specialised stroke unit if such a unit is available in the hospital.
IF a person aged 65 or older has new urinary incontinence that persists for over 1 month or urinary incontinence at the time of a new evaluation, THEN a dipstick urinalysis and/or mid-stream urine sample should be obtained.
IF a person aged 65 or older is diagnosed with a cataract that limits the patient’s ability to carry out needed or desired activities, THEN cataract extraction should be offered.
The panel was recruited through national organisations in order to reach respected clinicians with access to established networks of colleagues. Members of the British Geriatrics Society’s Special Interest Groups, the Royal College of General Practitioner’s clinical network of general practitioners with a special interest in care of the elderly, and the Royal College of Nursing’s Gerontological Nursing Programme were invited to review the set of proposed indicators. Six consultant physicians in geriatric medicine or medicine for the elderly, three general practitioners, and one nurse made up the final panel. The pack containing the indicator set and rating forms was also sent for review to a senior consultant in old age psychiatry, a senior public health doctor, and the policy officers from Help the Aged and Age Concern, UK charitable organisations for older people. Their comments were considered at the meeting.
Box 2 Examples of quality indicators rated as invalid* by expert clinical panel
Falls and mobility disorders
ALL persons aged 65 or older should be asked about or examined for the presence of balance and/or gait disturbances at least once.
ALL persons aged 65 or older should be offered a hearing screen at least once.
Ischaemic heart disease
IF a person aged 65 or older with established coronary heart disease smokes, THEN he or she should be offered counselling for smoking cessation at least annually.
IF a person aged 65 or over is treated with a non-steroidal anti-inflammatory drug, THEN (1) concomitant treatment with either misoprostol or a proton pump inhibitor should be used OR (2) the non-steroidal anti-inflammatory drug that is used should be selective for cyclooxygenase 2.
ALL women aged 65 or older should be offered advice about the pharmacological prevention of osteoporosis at least once.
(The only indicator rejected due to disagreement.)
ALL persons aged 65 or older should be asked about persistent pain every 2 years.
Screening and prevention
ALL persons aged 65 or older should be asked about who would be a surrogate decision maker, or whether they have an advance directive indicating their surrogate decision maker.
ALL persons aged 65 or older should be asked by their doctor or nurse annually about the presence or absence of urinary incontinence.
A total of 119 quality indicators were presented to the clinical panel which accepted 102 (86%) as being valid for use in England; 17 (14%) were rejected as invalid. Seventy nine of the 93 ACOVE indicators (85%) were approved with no or minor amendments, and 23 of the 26 new indicators suggested by panel members (88%) were approved.
Examples from each clinical area of the indicators rated as valid are given in box 1. The full set of quality indicators rated as valid is shown in Appendix 1 (available online at www.qshc.com/supplemental). Examples of indicators not rated as valid are given in box 2 and the full set of rejected indicators is shown in Appendix 2 (available online at www.qshc.com/supplemental). Only one indicator, which recommended advice about the pharmacological prevention of osteoporosis, was rejected due to disagreement.
Table 1 shows the number of indicators proposed and accepted in the 16 clinical areas: the number of accepted indicators varied from 3 (for continuity) to 12 (for medication use). Table 2 shows the number of indicators proposed and accepted in each of the following five domains of care: continuity and follow up, diagnosis, prevention, screening, and treatment. All of the 58 indicators in the treatment or continuity and follow up domains were accepted, but only just over half (13 of 24) of those in the screening domain were accepted.
This research reports on quality indicators developed in the USA for older people that have also been judged as valid for use in England. One hundred and two indicators of the quality of health care for people aged 65 years and older in England were rated as valid by an expert clinical panel and 79 of these are essentially unchanged from the ACOVE indicators from the USA. All 58 indicators about treatment or continuity and follow up were accepted, but only half (13 of 24) of the indicators about screening were accepted. This more conservative approach by the English panel than the USA panel to indicators about screening perhaps reflects the experience of general practitioners in England with the implications of implementing existing screening programmes—for example, breast and cervical screening. The panel discussion reflected concern about a requirement for general practitioners in England to annually seek out all older adults registered with them to ask about, for example, urinary incontinence. In the USA primary care physicians do not generally have responsibility for a population.
Further work is required to operationalise these quality indicators before use. In general, each indicator requires a minimum of two survey questions, at least one to determine whether the respondent has been diagnosed with the relevant condition, and then at least one more question to determine whether the respondent has received the intervention specified by the indicator. When applying quality indicators it is important to consider what aspects of health care are being measured.18 Quality can be measured across the following six generally accepted dimensions: effectiveness, efficiency, patient centredness, access, equity, and safety. The quality indicators presented in this paper are designed primarily to measure effectiveness, although they can contribute to some extent to the measurement of other dimensions of quality. Other approaches to quality assessment should be used as well to better capture other dimensions—for example, economic tools to measure efficiency.
Lending convergent validity to these indicators are the results from a study in the UK in 1997 which used the RAND/UCLA method to develop review criteria for clinical notes in general practice for angina, asthma, and non-insulin dependent diabetes mellitus.19 Our study concerned quality indicators for use in interview surveys of older adults, but the indicators in areas of clinical overlap are very similar to the 1997 study. For angina both panels recommended treatment of raised serum cholesterol levels, use of aspirin or antiplatelet therapy, and advice about smoking cessation. For diabetes both panels recommended annual measurement of glycosylated haemoglobin and of proteinuria, annual examination of the feet and of the fundi through dilated pupils (although this panel recommended that the eye examination be done by a specialist rather than by any doctor), monitoring of blood pressure, treatment of raised serum cholesterol levels (although this panel set a level of 5 mmol/l as opposed to 7.8 in 1997). This panel also recommended the use of angiotensin converting enzyme inhibitors or angiotensin receptor blockers and daily aspirin therapy which were not recommended in 1997. The development of new indicators by similar processes conducted 5 years apart shows that quality indicators need regular updating to allow for advances in medical knowledge.
The adaptation of this US set of indicators for use in the UK has two main advantages. Firstly, since the development of quality indicators is resource intensive, it is desirable for developed indicators to be shared internationally where possible rather than developed separately in different countries. Secondly, use of the same indicators in the US and UK will allow international comparisons of the quality of healthcare processes between the two countries. These indicators are designed to measure the quality of health care at the population level. They are not designed to measure the quality of health care received by an individual patient, or to underpin appraisal, assessment and revalidation of individual professionals. These are important tasks, but they require a different approach to standard setting.20,21
A high proportion of a quality indicator set developed in the USA for the health care of older people were also rated as valid for use in England.
These 102 quality indicators, covering a broad range of conditions in older adults, are now available for use in England.
The quality indicators are intended to be used in interview surveys of patients covering both primary and secondary care, after being operationalised into appropriate survey questions.
Use in interview surveys will facilitate simultaneous assessment of comorbidities and social conditions and will avoid the problems of data extraction from medical records.
All the indicators in the treatment or continuity and follow up domains were rated as valid compared with only about half of the indicators in the screening domain.
Use of these indicators in a population survey to measure health system performance will allow comparisons to be made about the quality of health care received by different population groups, whether defined by socioeconomic status, wealth, health status, geography, age, or sex. Two studies using RAND quality indicators have recently shown large gaps in the quality of care in the US,4,5 but the paucity of data about the quality of health care in the UK is currently hindering quality improvement.21 The use of these indicators will provide strong signals about where quality improvement is needed, and will act as a stimulus to target quality improvement efforts at the areas with the greatest potential for improved patient care.
The authors thank the clinical panel members: Mr J Adams, Senior Lecturer in Health Studies, Gerontological Nursing, Homerton College, Cambridge; Dr G Archard, General Practitioner, Christchurch and Clinical Governance Lead, NHS Alliance; Dr J Close, Consultant Physician, Department of Health Care of the Elderly, King’s College Hospital, London; Dr S Croxson, Consultant in Care of the Elderly, Bristol General Hospital, Bristol; Dr I Donald, Consultant Physician in Elderly Care, Gloucestershire Royal Hospital, Gloucester; Dr D Forsyth, Consultant Geriatrician and Clinical Director, Department of Medicine for the Elderly, Addenbrooke’s NHS Trust, Cambridge; Dr R Gadsby, General Practitioner, Nuneaton and Senior Lecturer in Primary Care, Warwick University; Professor A L Kinmonth, Professor of General Practice, General Practice and Primary Care Research Unit, Cambridge; Dr P Mayer, Consultant Physician in Geriatric Medicine, South Birmingham PCT, Birmingham; and Dr S Mukherjee, Consultant Physician/Geriatrician, Queen Elizabeth Queen Mother Hospital, Margate, Kent.
The authors also thank Dr S Campbell who co-chaired the clinical panel meeting with NS, the RAND researchers who developed the ACOVE quality indicators used in this study, and all who commented on the proposed indicators.
The appendices are available as downloadable PDFs (printer friendly files).
If you do not have Adobe Reader installed on your computer,
you can download this free-of-charge, please Click here
Files in this Data Supplement:
- [View PDF] - Appendix 1 Quality indicators rated as valid* by expert clinical panel (n=102). Indicators proposed by the panel are shown in bold font (n=23)
- [View PDF] - Appendix 2 Quality indicators rated as invalid* by expert clinical panel (n=17). Indicators proposed by the panel are shown in bold font (n=3)
↵* Valid = median of >6 on a scale of 1–9 with no disagreement (disagreement defined as 3 or more ratings in the 1–3 region, together with 3 or more ratings in the 7–9 region).
↵* Invalid = median 6 or less on a scale of 1–9, or median >6 with disagreement (disagreement defined as 3 or more ratings in the 1–3 region, together with 3 or more ratings in the 7–9 region).
NS was supported by The Commonwealth Fund, a New York City based private independent foundation, and by the UK National Coordinating
Centre for Research Capacity Development. The National Health Service Executive Eastern Region provided funding for the clinical panel meeting.
This work was done at the Department of Public Health and Primary Care, Institute of Public Health, University of Cambridge, Forvie Site, Robinson Way, Cambridge CB2 2SR, UK
The views presented here are those of the authors and not necessarily those of The Commonwealth Fund, its director, officers or staff.
See editorial commentary, p 248
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.