METHODS

Process

A two-round postal Delphi survey was carried out between March and July 2000. The study was conducted entirely by post for cost purposes and to include more respondents over a greater geographical area.

Panellists

Eleven single specialty panels were convened. These consisted of eight separate professional panels, one carer panel, one patient panel, and one made up of people from voluntary organisations (box 2). Panellists from professional groups reflected a combination of practitioners and those selected on the basis of their national reputation—that is, publication record, involvement in specialist societies, or as nominees of professional organisations. Similarly, the patient and carer panels contained patients and carers as well as known patient/carer advocates and those involved in key patient/carer organisations. The composition of panels reflected a geographical spread across Great Britain. Each panel was composed of 9–12 representatives. Panellists were sent a letter of invitation to participate. Those who were unable to take part were asked to nominate a possible replacement.

Indicators

A preliminary set of indicators was constructed from a number of sources. Firstly, previously published guideline statements and quality indicators and standards relating to primary mental health care were reviewed, as well as sources of published/grey literature produced by key organisations. Secondly, patient focus groups were convened to identify aspects of care (and thus indicators) considered important from the patient/user perspective. Finally, 22 of the standards set out in the National Service Framework for Mental Health for England were included. A total of 367 indicators were identified which covered a range of aspects of care at practice, primary care group, and health authority levels.

Questionnaire

The list of potential indicators was developed into a Delphi questionnaire which is reproduced in full on the journal website (www.qshc.com). The indicators were categorised into three levels of service which related to those provided by individual practices, primary care groups, and health authorities. Campbell and colleagues¹⁹ have proposed that quality of care is a combination of access (whether users get the care they need) and effectiveness of clinical and interpersonal care (whether the care is effective when they get it). Practice level indicators were categorised according to these two elements. Some of the indicators could have been placed in one or more categories but, to avoid duplication, these indicators were put into only one category.

Subsequent amendments were made following consultation with a six member reference group composed of a health care commissioner, general practitioner, patient, community psychiatric nurse, counsellor, and psychiatrist.

Delphi round 1

During the first round the Delphi panellists were asked to rate the indicators on two continuous integer 9-point scales for validity and clarity (box 3) with 1=lowest and 9=highest rating.

Indicators were clarified and modified after feedback from the first round according to written comments made by panellists. Fifteen indicators receiving an overall median clarity score across panels of ≤6 were rewritten but retained; 38 indicators were discarded on this basis because of perceived duplication and lack of relevance to primary care, while an additional five indicators were included. A maximum number of 334 indicators could therefore potentially have been included within each panel’s round 2 questionnaire. However, not all panels received all 334 indicators because individual median panel scores were created for each indicator. Indicators which did not receive a median validity score of 7–9 with agreement within a panel were not included in that panel’s second round questionnaire. Agreement was defined as 60% or more scores in the top (7–9) tertile (box 3).²⁰ Each panel therefore had a unique second round questionnaire.

Delphi round 2

Each panellist was sent a second round questionnaire which included three types of feedback based on first round validity scores for each indicator (panellist’s own previous score, panellist’s median panel score, and the frequency distribution of scores within the panellist’s own panel). Panellists were then asked to re-rate the indicators specific to their panel for validity.

Proposed analysis

Previous research has shown that use of a higher cut off point (≥8) is associated with greater reproducibility and reliability.²¹ Validity after second round ratings was therefore defined as an overall median for each panel of ≥8. A more stringent level of agreement of 75% or more in the top (7–9) tertile was applied (box 3). Indicators rated with an overall validity score of 8 or 9 with agreement by all panels were considered face valid indicators. Those rated valid by all panels were operationalised into aspects of care by the research team to produce indicators which could be used pragmatically to facilitate quality improvement. χ² tests were used to determine whether there were significant differences between the types of indicators rated valid by different panels—for example, between the different levels of service.

RESULTS

Response rate

A response rate of 90% was achieved in round 1 and 89% in round 2. The number of returned questionnaires on which analysis was based ranged from a minimum of eight (counsellor, community psychiatric nurse, nurse, psychiatrist) to a maximum of 11 (clinical psychologist, voluntary organisations) per panel. All final analyses reported in this paper are based on second round results.

Consensus among panel ratings

Most of the indicators (97%, n=324) were rated valid by at least one panel in round 2; 26% of the indicators were rated valid by all panels. The latter set of indicators represents a consensus among key stakeholder groups in defining quality of mental health care in primary care and, as such, formed the focus of the study. These indicators have been categorised into 21 aspects of care, 11 relating to general practice and 10 relating to health authorities or primary care groups/trusts (boxes 4 and 5).

Differences in panel ratings

The extent of intra-panel disagreement was extremely low with less than 1% disagreement within any one panel. Disagreement was defined as 30% or more scores in both the bottom (1–3) and top (6–9) tertiles.²⁰ However, there were major differences between panel ratings. Table 1 shows the variation in the total number of indicators rated valid across different panels after the first and second rounds of the Delphi process. Overall, GPs rated the lowest number of indicators as valid (41%) and carers rated the highest number valid (91%).

Panels rated different aspects of care as valid—for example, the GP panel did not rate protocols or guidelines for the management of separate mental health problems (including depression and anxiety) as central to providing good quality care, whereas the carer, clinical psychologist, community psychiatric nurse, patient, and voluntary organisation panels did. Moreover, χ² tests showed that GPs were the only stakeholder group who significantly rated more indicators as valid at the practice level than at the level of the primary care group and health authority (p<0.01). For all other groups there was either no significant difference in their ratings between different levels (commissioners, counsellors, patients, psychiatrists and social workers) or they rated significantly more indicators as important at the higher level (p<0.05) (carers, clinical psychologists, community psychiatric nurses, nurses, voluntary organisations). Given that the focus of this paper is the indicators which were rated valid by all panels, a more detailed account of the major differences between panel ratings will be reported separately as part of a follow up study exploring the factors which influence the ratings of stakeholders in a Delphi survey.

DISCUSSION

Our results show that it is possible to identify agreement between groups of stakeholders in identifying a generic set of valid quality indicators for primary care mental health services. All panels agreed that 26% of the indicators were valid measures of quality of care. However, the results of the Delphi survey also show that different stakeholders value different aspects of quality of care. The final set of indicators listed in boxes 4 and 5 provide a starting point for the development of quality improvement tools.

Limitations of the study

Although the Delphi process is a well validated method for assessing opinion,^9–11 this study has some important limitations. Firstly, the indicators rated were based on best available evidence in 2000 and, as such, must be reviewed in the light of major new research findings. The final set of indicators cannot be considered a comprehensive assessment of service level mental health services. Rather, they represent a consensus among key stakeholder groups in defining quality of care within primary care mental health services.

Secondly, appropriate composition of panels in consensus methodologies is a key factor in determining the legitimacy of the findings.^21,22 Considerable care was taken to select panels in order to reflect a range of expertise. This involved combining practitioners and those with a national reputation (based on publication record and membership of national committees/organisations) for the professional groups, and also including known patient and carer advocates/members of key organisations within the patient and carer panels. However, panels could not be said to be representative of their stakeholder perspective, but merely suggestive of their profession/group.

Thirdly, the aim of the study was to identify indicators rated valid by all stakeholders/panels but, because we attempted to identify all relevant stakeholder groups, there was a built-in bias in having a larger number of professional panels than a single patient and carer panel. Even though the rating process was democratic with each panel’s ratings carrying equal weight, because the carer and patient panels rated more of the indicators valid than, for example, the GP or psychiatrist panels, this meant that the final set of indicators included fewer of the aspects of care of value to patients or carers than of value to GPs or psychiatrists. Those panels rating the least number of indicators valid (GPs, psychiatrists, counsellors) therefore had greater influence on the final set of indicators than those rating the highest numbers of indicators valid (carers, patients, voluntary organisations, and community psychiatric nurses). However, this did not detract from the aim of the study to identify a common set of indicators.

Finally, the set of indicators developed required additional editing by the research team in order to operationalise them for use within quality improvement strategies. However, the meaning of the indicators remained intact.

Implications for quality improvement

The fact that only 26% of the quality indicators were agreed by all panels has implications for quality improvement and assurance purposes, given that the relative proportion of statements agreed amongst panels—particularly between professionals and users—differed. Such differences in the ratings of the various panels reflect the natural diversity of opinion within mental health care. The indicators rated valid by each panel also provide opportunities for quality improvement strategies which address the aspects of care rated valid by each different perspective (these are available from the authors). Policy makers in all countries intending to use consensus methods to provide a multidisciplinary approach to quality assessment and improvement within health and social care need to give thought to this issue in order to create a more equal partnership between what consumers value and what providers value in terms of quality care.

Quality indicators are defined as a measurable element of performance for which there is evidence or consensus that they can be used to assess the quality, and hence change in the quality of care provided.²³ While the indicators listed in boxes 4 and 5 represent consensus among the 11 panels, many are professional aspirations and standards rather than measurable indicators. This is perhaps a reflection of the underdeveloped and nascent evidence base from which to develop outcome based indicators for primary mental health care, and the difficulty in collecting such data in the setting of primary care even in a system as complex and well developed as that in the UK. Measurement depends on reliable data systems which, although readily available in the US health maintenance organisations, are only now in development in healthcare settings in other western countries. It is necessary to set up these systems and infrastructure before quality assurance/control can be both feasible and reliable. We have therefore not attempted to distinguish between the conventional (measurable) quality indicators and the more aspirational indicators. While a number of the indicators will be easily measurable, others—such as those included within patient-staff relations—will require more innovative and eclectic approaches to measurement. Although this will provide considerable challenges to primary care organisations in facilitating quality improvement, the state of the literature relating quality of care to outcome in mental health care compared with conditions such as coronary heart disease⁷ necessitates such an approach to quality improvement.

The final set of indicators can be used for both quality improvement (which does not need measurement) and quality assessment (which does). Importantly, these indicators are not intended to provide a comprehensive tool set for measuring quality of care. Rather, they can be used as a starting point for discussion by primary care organisations and practices and as a means of assisting primary care organisations to address the national targets and standards relating to primary care set out in the National Service Framework for Mental Health for England. They also represent a potential for identifying the current baseline of care provision. In this way the indicators raise awareness around important issues of primary mental health care. This is particularly relevant in localities where progress in developing primary care mental health services is limited and a cultural shift to increase the profile of mental health care is required.

When applying quality indicators it is important to ensure that they are done realistically and that they reflect local circumstances.²⁴ Some areas of the UK have a better infrastructure for primary care mental health services than others.⁷ We have not therefore attempted to establish standards for these indicators. Rather, primary care organisations or practices may adopt a standard which is relevant to their local circumstances and needs and initiates debate at a local level.

Other potential uses of the indicators are to act as sign posts in defining quality of care and to indicate areas where additional resources may be required. The indicators could then be used to aid primary care organisations and/or practices to set locally achievable goals and to develop local standards of care. Given that the indicators themselves are not a comprehensive set of quality measurement tools, it may be necessary—depending on the aims of the quality improvement strategy—to supplement these indicators with other key local and national documents within this field.

Conclusion

The quality indicators developed in this study, although not comprehensive, represent a consensus among key stakeholder groups in defining quality of care within primary care mental health services. They can be used as a starting point in quality improvement strategies and as a means for assisting primary care organisations to address the national targets and standards relating to primary care set out in the National Service Framework for Mental Health for England. Although many of the indicators relate to parochial issues in UK service delivery, the methodology used in their development—with its multi-stakeholder input and involvement of patients, carers, and professionals—could be applied in other settings to produce locally relevant indicators.