Article Text

PDF

Guideline adherence rates and interprofessional variation in a vignette study of depression
  1. H Tiemeier1,
  2. W J de Vries1,
  3. M van het Loo2,
  4. J P Kahan2,
  5. N Klazinga3,
  6. R Grol4,
  7. H Rigter1
  1. 1Netherlands Institute of Mental Health and Addiction, Utrecht, The Netherlands
  2. 2RAND Europe, Leiden, The Netherlands
  3. 3Dutch Institute for Health Care Improvement, Utrecht and Department of Social Medicine, University of Amsterdam, The Netherlands
  4. 4Centre for Quality of Care Research, Nijmegen University – Maastricht University, The Netherlands
  1. Correspondence to:
 Dr H Tiemeier, Department of Epidemiology & Biostatistics, Erasmus Medical Centre Rotterdam, P O Box 1738, 3000 DR Rotterdam, The Netherlands;
 tiemeier{at}epib.fgg.eur.nl

Abstract

Objective: To assess the appropriateness of and variation in intention-to-treat decisions in the management of depression in the Netherlands.

Design: Mailed survey with 22 paper cases (vignettes) based on a population study.

Setting: A random sample from four professional groups in the Dutch mental healthcare system.

Subjects: 264 general practitioners, psychiatrists, psychotherapists, and clinical psychologists.

Main outcome measures: Each vignette contained information on a number of patient characteristics taken from three national depression guidelines. The distribution of patient characteristics was based on data from a population study. Respondents were asked to choose the best treatment option and the best treatment setting. For each vignette we examined which of the selected treatments was appropriate according to the recommendations of the three published Dutch clinical guidelines and a panel of experts.

Results: 31% of all intention-to-treat decisions were not consistent with the guidelines. Overall, less severe depression, alcohol abuse, psychotic features, and lack of social resources were related to more inappropriate judgements. There was considerable variation between the professional groups: psychiatrists made more appropriate choices than the other professions although they had the highest rate of overtreatment.

Conclusions: There is sufficient variation in the intentions to treat depression to give it priority in quality assessment and guideline development. Efforts to achieve appropriate care should focus on treatment indications, referral patterns, and overtreatment.

Statistics from Altmetric.com

Many studies have shown substantial variation and possible deficiencies in the management of depressive illness,1 and suboptimal treatment seems to be pervasive across disciplines, service delivery systems, and cultures.2–5 Because depression is a common and distressing illness with high societal costs, policymakers, professional bodies and other organisations have developed clinical practice guidelines to improve the quality of patient care.

In the Netherlands, as in many other countries, different professions are involved in the treatment of depression. General practitioners play an important role due to the comprehensive primary healthcare system. Psychiatrists, psychotherapists, and clinical psychologists provide care in private practice, community mental health centres, or hospital settings. Guideline development in the Netherlands is mainly professionally driven and started as early as 1980. It has an increasing focus on best scientific evidence.6 Three national guidelines for the treatment of depression have been developed to date. In 1994 both the Dutch general practitioners7 and the psychiatrists (in collaboration with the Dutch Institute for Health Care Improvement and other medical professions)8 published guidelines for depression. Three years later the Dutch psychotherapists developed a guideline on depression.9

To support the development of a revised multidisciplinary guideline for depression, an effort that began in early 2000, we have examined which aspects of care should be targeted by guidelines. Little information is available on the appropriateness of treatment choices; most health service research has concentrated on early detection, establishing the diagnosis, and antidepressant dosage in primary health care.10–13

Several methods have been developed to analyse practitioners' decisions, including clinical databases, standardised patients performed by actors, and questionnaires.14–17 In this study we applied a method based on judgements of paper cases (vignettes) which were derived from a population survey. This is an effective way to achieve a realistic case mix.

The study was designed to determine the variation in intention-to-treat decisions and the adherence rate to guidelines by the different professional groups. It aimed to reveal any lack of knowledge that should be addressed in guidelines on the treatment of depression.

METHODS

Construction of vignettes

Twenty two vignettes depicting patients with an episode of major or subthreshold depression at the time of their first contact with a practitioner were constructed. Each contained information on a number of patient characteristics that were taken from the three national depression guidelines. These were supplemented with patient characteristics defined by an expert panel on depression. The 15 members of this panel were nominated by the four professional organisations (general practitioners, psychiatrists, psychotherapists, and clinical psychologists). The resulting patient characteristics included number and severity of symptoms, social functioning, reason for encounter, and stressors.

The vignettes were not constructed in the abstract, but were based on the data of a population study. The distribution of the patient characteristics systematically reflects those of 140 existing subjects with a first episode of depression. Data were used from the Nemesis study of 1998, a prospective study on the prevalence of psychiatric disorders in the Dutch population aged 18–64.18

Selection of professionals

A sample of 150 general practitioners, 100 psychiatrists, 123 psychotherapists, and 100 clinical psychologists was selected from the national registers of the four professional organisations. The different subsample sizes were motivated by an expected overlap between psychotherapists and psychiatrists and to protect against an anticipated low response from general practitioners.

Box 1 Vignette describing a case of subthreshold depression

A 56 year old woman has been suffering from depressive thoughts for 4 weeks and has lost interest in most of her daily activities. She reports a disturbed sleep pattern and recurrent thoughts about dying. She is not suicidal. No other depressive symptoms were encountered now or in earlier consultations. She is a housewife and lives with her chronically ill husband for whom she has been caring for 4 years. One grown up son still lives at home. He gives some support to his mother in her daily work. The woman presents with complaints of a chronic headache. She would like the emphasis of the treatment to be put on her somatic problem.

Standard: Watchful waiting or psychotherapeutic management. Depression in Primary Care published by the Agency for Health Care Policy and Research20 is the best review of the evidence available at the time of the study.

Respondents' choices: 38% watchful waiting, 29% psychotherapeutic management, 2% psychotherapy, 28% psychopharmacotherapy, 3% combined therapy.

Procedure

The initial survey was mailed in July 1998 with a letter of support from the presidents of the professional organisations. A reminder was sent to non-responders 3 months later. For each of the 22 vignettes, respondents were asked to answer “which treatment should this patient get?” and “what is the best place for treatment?” Two examples of vignettes are given in boxes 1 and 2 and the answering categories can be found in the Appendix.

In analysing the data, the treatment options were divided into five categories: combined therapy (psychotherapy and psychopharmacotherapy), psychopharmacotherapy, psychotherapy (long or short term psychotherapy), psychotherapeutic management, and watchful waiting. For every vignette we examined which treatments were appropriate. This “standard” was the “lowest common denominator” of the three Dutch clinical guidelines which recommend one or more specific treatments depending on the clinical features. In cases of conflicting recommendations, our standard gave the practitioner the benefit of the doubt. The 15 panellists scored all five treatment categories for all vignettes using a modified group judgement method (RAND) to assess the appropriateness of treatments.19 This included a round of anonymous ratings on a nine point scale and a face to face discussion. The standard for our vignettes was adapted if the panel judged a treatment to be appropriate for a specific vignette that was not in agreement with the guidelines. We introduced this procedure for two reasons: (1) it ensured that the standard derived from the guidelines did not reflect a questionable interpretation of the available evidence and (2) it prevented uncertainties resulting from the construction of the vignettes. The resulting standard was therefore extremely liberal compared, for example, with the depression guideline of the US Agency for Healthcare Research and Quality.20

Box 2 Vignette describing a case of moderately severe depression

A 52 year old woman has been feeling desperate and depressed for about 6 weeks. She has lost nearly all interest in her daily activities. A detailed history reveals that she feels guilty because she has neglected her household work. She wakes up early and frequently feels tired. She states that she cannot think clearly and has difficulty in concentrating. There is also some alcohol abuse. The woman receives social welfare payments. Because she lives alone and hardly goes out, she has few friends. In the past weeks she has had a conflict with her landlord who threatened to give her notice. She puts her problems down to herself.

Standard: Psychopharmacotherapy or psychotherapy. Depression in Primary Care published by the Agency for Health Care Policy and Research20 is the best review of the evidence available at the time of the study.

Respondents' choices: 3% watchful waiting, 54% psychotherapeutic management, 11% psychotherapy, 27% psychopharmacotherapy, 5% combined therapy. Note that 43% of respondents choosing psychotherapeutic management combined this with a referral to social services.

Inappropriate treatment options were classified either as “overtreatment” or “undertreatment”. Overtreatment was defined as the choice of an inappropriate intervention where risks outweigh the benefits. Undertreatment was defined as the choice of an inappropriate treatment category where the benefit of a more intensive treatment is not gained. No standard for referral of patients to specialist care could be derived as no specific recommendations are made in the guidelines.

Statistical analysis

Data were analysed using SPSS 9.0 for Windows. To determine the predominant treatment we analysed the frequency of treatment choice per profession using the χ2 test. Logistic regression analysis was used to examine the influence of patient characteristics on intention-to-treat decisions and the association with appropriate choices. These analyses were rerun selecting cases by the severity of disease. A p value of <0.01 was considered to be significant, except for interaction effects where a cut off point of 0.05 was chosen.

RESULTS

Response

From the total sample of 473 surveys mailed, 18 were returned without a forwarding address and six were ineligible because the respondents were retired or no longer working in clinical practice, resulting in an effective sample size of 449 health practitioners. In total, 59% (n=264) of all mailed surveys were returned in a form suitable for statistical analysis. The response rates were 75% (n=87) for psychotherapists, 69% (n=67) for clinical psychologists, 56% (n=53) for psychiatrists, and 40% (n=57) for general practitioners. Eight psychotherapists indicated that they were psychiatrists and were subsequently added to this group. Non-response was not related to age, sex, location of practice, and years of experience.

Choice of treatment category

Table 1 presents an overview of the variation of intention-to-treat decisions per professional group. Respondents opted for medication in more than 50% of all cases (psychopharmacotherapy alone or as part of combined therapy). Psychiatrists and, to a lesser extent, general practitioners relied much more on psychopharmacotherapy, whereas clinical psychologists and psychotherapists more often opted for psychotherapy.

Table 1

Choice of treatment category per profession

Table 2 shows which patient characteristics influenced the treatment decisions for psychopharmacotherapy (including combinations with psychotherapy). Most of the variance is accounted for by severity of depression. Additional psychotic or melancholic features did not result in psychopharmacotherapy being chosen more often.

Table 2

Determinants of the choice for psychopharmacotherapy (expressed as odds ratios)

Appropriateness of intention-to-treat decisions

Overall, 31% of all intention-to-treat decisions were not consistent with the guidelines. Table 3 gives the mean percentage of deviations from standard treatment per profession and distinguishes overtreatment from undertreatment. Undertreatment accounted for two thirds of the inappropriate intention-to-treat decisions. Although psychiatrists had the highest rate of overtreatment, they made more appropriate treatment choices than other professions.

Table 3

Comparison of the intention-to-treat decisions with the standard per profession: appropriate versus undertreatment and overtreatment

The rate of appropriate intention-to-treat decisions of general practitioners and psychiatrists was high in cases with more severe depression, with 72% and 84%, respectively, of patients being treated according to the standard. However, in subthreshold and mild depression the rates of appropriateness were only 59%. Vignette 1 gives an example of such a case. Psychotherapists and clinical psychologists performed better in the milder cases with rates of 68% and 71%, respectively, because they preferred watchful waiting or the psychotherapeutic management recommended in all guidelines.

In table 4 the relation between patient and care provider characteristics and the appropriateness of judgements is shown. Less severe depression, alcohol abuse, psychotic features, and no social resources were related to inappropriate judgements. Vignette 2 illustrates this and gives an example where the appropriateness was particularly low. Significant interactions between the profession of the respondent and patient characteristics are included in table 4. Psychiatrists were much more likely to choose an appropriate treatment when psychotic or vital characteristics were present, while psychologists scored better in cases with alcohol abuse.

Table 4

Determinants of appropriate intention-to-treat decisions and their interaction with profession of the respondent (expressed as odds ratios)

Table 5 shows the patient characteristics that determined the referral pattern of general practitioners. On average, general practitioners referred 25% of patients for specialist mental health care. However, in subthreshold depression this was only 4% while in psychotic depression it increased to 66% of cases.

Table 5

Determinants of intended referral by general practitioners to specialised mental health care (expressed as odds ratios)

DISCUSSION

This study applied a vignette method to investigate the variation in intention-to-treat decisions among four professions involved in the management of depression. The results show that there is considerable diversity in the choice of treatment. Almost one third of intention-to-treat decisions were inappropriate. This is remarkable, considering the liberal definition we applied.

Despite widespread use of the vignette method, there is a debate as to whether the results of such studies reflect clinical practice or merely physician competence. Some feel that abstracts from medical records may give a more realistic view of the behaviour of the individual practitioner.21 However, a recent study reported that vignettes are uniquely suited for comparative analyses between groups of providers and are more effective than chart abstracts if controlled for the case mix.17 We tried to achieve a realistic case mix by basing the distribution of patient characteristics on the outcomes of a population survey. To ensure parity among the respondents, we included in the vignettes the criteria deemed relevant in the existing guidelines.

Response rates were excellent for psychologists and psychotherapists but modest for general practitioners and psychiatrists, reflecting the general trend of falling response rates to postal surveys among general practitioners.22 The modest response rate did not affect the power of our study but may have introduced a selection effect. General practitioners and psychiatrists with a less positive attitude to depression and research may not have participated. This self-selection bias may lead to overestimation of the knowledge of the group as a whole.22–24 The relatively high rate of inappropriate intention-to-treat decisions among general practitioners cannot therefore be accounted for by selection bias.

Variation in the choice of treatment between professions is not necessarily undesirable. For example, guidelines recommend both short term psychotherapy and psychopharmacotherapy as options in untreated cases with moderately severe depression. It is not surprising that psychiatrists relatively often chose psychopharmacotherapy and psychologists chose psychotherapy in these cases as this tendency reflects differences in experience with treatment methods.11,25–28 Some of the choices which we found inappropriate may reflect divergence from a debatable standard. To recommend only watchful waiting or psychotherapeutic management policy in subthreshold depression, for example, could be criticised.29 Vignette 1 illustrates this problem, where many practitioners chose psychopharmacotherapy. However, clinical trials that study the effect of medical treatment in subthreshold depression have only recently been reported.30 Our standard reflected the expert opinion of guideline makers and our panel because rigorously conducted research was unavailable. We feel that treatment choices can be considered inappropriate on this basis, but acknowledge that this remains a matter of debate.31

On the other hand, our finding that the choice of psychopharmacotherapy was not increased in cases with psychotic or melancholic features is disquieting because these features are mentioned in the guidelines as indications for psychopharmacotherapy. Further, none of the guidelines supports the notion that a lack of social resources or alcohol abuse are relative contraindications for psychopharmacotherapy.7–9,20 These patient characteristics deserve attention in quality assessment as they are possible explanations for the frequently reported undertreatment of depression.32 Vignette 2 is an example where many practitioners chose psychotherapeutic management, a choice that is not concordant with the guidelines and considered inappropriate. Our panel confirmed this judgement; there is satisfactory evidence to support psychotherapeutic or psychopharmacological treatment in a patient with these characteristics.

In mental health service research little attention has been given to the possible dangers of overuse driven by providers.33 In our study one third of all inappropriate intention-to-treat decisions was due to overtreatment—that is, the risks outweighed the potential benefits. In view of the use of guidelines, not only to assure quality of care but also to control health care costs, more emphasis should be put on this problem.

We also analysed intention-to-refer decisions. Remarkably, in one third of all cases the general practitioners did not intend to refer patients with psychotic features to specialised mental health care. Depression guidelines to date do not provide much assistance in referral decisions. Ideally, guidelines should recommend which type of patient should be referred to specialised facilities. Because of the lack of evidence on the effectiveness of healthcare settings, this would require consensus among professional associations. Our study suggests that more patients might be treated appropriately if practitioners adopted referral criteria that reflect the experience of the involved professions.

In conclusion, our data show that there is considerable variation in intention-to-treat depression and that many treatment choices made by the care providers were inappropriate. It was possible to identify specific patient characteristics—for example, lack of social resources, alcohol abuse and psychotic features—that were associated with a particularly high number of inappropriate judgements. Education of care providers alone certainly is not sufficient but in view of the slow diffusion of scientific knowledge it seems necessary in order to improve outcomes in depression. Basic considerations like the treatment choice must not be overlooked as modern management of depressed patients focuses on case management and follow up.34 Treatment indications for depression should therefore be given a priority in quality assessment and guideline development. In addition, the results suggest that efforts to achieve more appropriate care should focus on referral patterns and overtreatment of depression.

Key messages

  • There is considerable variation in intention-to-treat depression and many choices made by care providers are inappropriate.

  • Certain patient characteristics such as lack of social resources, alcohol abuse, and psychotic features lead to more inappropriate judgements.

  • Established and explicit referral criteria are necessary to achieve better care for patients with depression.

  • Overtreatment of depression—when risks outweigh potential benefits—should receive more attention in quality assessment.

APPENDIX:

ANSWERING CATEGORIES

Acknowledgments

This study was supported by a grant from the Health Research and Development Council of the Netherlands (ZON). The authors are grateful to Ed Faber for support in data management.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Action points
    Tim Albert