METHODS

Construction of vignettes

Twenty two vignettes depicting patients with an episode of major or subthreshold depression at the time of their first contact with a practitioner were constructed. Each contained information on a number of patient characteristics that were taken from the three national depression guidelines. These were supplemented with patient characteristics defined by an expert panel on depression. The 15 members of this panel were nominated by the four professional organisations (general practitioners, psychiatrists, psychotherapists, and clinical psychologists). The resulting patient characteristics included number and severity of symptoms, social functioning, reason for encounter, and stressors.

The vignettes were not constructed in the abstract, but were based on the data of a population study. The distribution of the patient characteristics systematically reflects those of 140 existing subjects with a first episode of depression. Data were used from the Nemesis study of 1998, a prospective study on the prevalence of psychiatric disorders in the Dutch population aged 18–64.¹⁸

Selection of professionals

A sample of 150 general practitioners, 100 psychiatrists, 123 psychotherapists, and 100 clinical psychologists was selected from the national registers of the four professional organisations. The different subsample sizes were motivated by an expected overlap between psychotherapists and psychiatrists and to protect against an anticipated low response from general practitioners.

Procedure

The initial survey was mailed in July 1998 with a letter of support from the presidents of the professional organisations. A reminder was sent to non-responders 3 months later. For each of the 22 vignettes, respondents were asked to answer “which treatment should this patient get?” and “what is the best place for treatment?” Two examples of vignettes are given in boxes 1 and 2 and the answering categories can be found in the Appendix.

In analysing the data, the treatment options were divided into five categories: combined therapy (psychotherapy and psychopharmacotherapy), psychopharmacotherapy, psychotherapy (long or short term psychotherapy), psychotherapeutic management, and watchful waiting. For every vignette we examined which treatments were appropriate. This “standard” was the “lowest common denominator” of the three Dutch clinical guidelines which recommend one or more specific treatments depending on the clinical features. In cases of conflicting recommendations, our standard gave the practitioner the benefit of the doubt. The 15 panellists scored all five treatment categories for all vignettes using a modified group judgement method (RAND) to assess the appropriateness of treatments.¹⁹ This included a round of anonymous ratings on a nine point scale and a face to face discussion. The standard for our vignettes was adapted if the panel judged a treatment to be appropriate for a specific vignette that was not in agreement with the guidelines. We introduced this procedure for two reasons: (1) it ensured that the standard derived from the guidelines did not reflect a questionable interpretation of the available evidence and (2) it prevented uncertainties resulting from the construction of the vignettes. The resulting standard was therefore extremely liberal compared, for example, with the depression guideline of the US Agency for Healthcare Research and Quality.²⁰

Inappropriate treatment options were classified either as “overtreatment” or “undertreatment”. Overtreatment was defined as the choice of an inappropriate intervention where risks outweigh the benefits. Undertreatment was defined as the choice of an inappropriate treatment category where the benefit of a more intensive treatment is not gained. No standard for referral of patients to specialist care could be derived as no specific recommendations are made in the guidelines.

Statistical analysis

Data were analysed using SPSS 9.0 for Windows. To determine the predominant treatment we analysed the frequency of treatment choice per profession using the χ² test. Logistic regression analysis was used to examine the influence of patient characteristics on intention-to-treat decisions and the association with appropriate choices. These analyses were rerun selecting cases by the severity of disease. A p value of <0.01 was considered to be significant, except for interaction effects where a cut off point of 0.05 was chosen.

RESULTS

Response

From the total sample of 473 surveys mailed, 18 were returned without a forwarding address and six were ineligible because the respondents were retired or no longer working in clinical practice, resulting in an effective sample size of 449 health practitioners. In total, 59% (n=264) of all mailed surveys were returned in a form suitable for statistical analysis. The response rates were 75% (n=87) for psychotherapists, 69% (n=67) for clinical psychologists, 56% (n=53) for psychiatrists, and 40% (n=57) for general practitioners. Eight psychotherapists indicated that they were psychiatrists and were subsequently added to this group. Non-response was not related to age, sex, location of practice, and years of experience.

Choice of treatment category

Table 1 presents an overview of the variation of intention-to-treat decisions per professional group. Respondents opted for medication in more than 50% of all cases (psychopharmacotherapy alone or as part of combined therapy). Psychiatrists and, to a lesser extent, general practitioners relied much more on psychopharmacotherapy, whereas clinical psychologists and psychotherapists more often opted for psychotherapy.

Table 2 shows which patient characteristics influenced the treatment decisions for psychopharmacotherapy (including combinations with psychotherapy). Most of the variance is accounted for by severity of depression. Additional psychotic or melancholic features did not result in psychopharmacotherapy being chosen more often.

Appropriateness of intention-to-treat decisions

Overall, 31% of all intention-to-treat decisions were not consistent with the guidelines. Table 3 gives the mean percentage of deviations from standard treatment per profession and distinguishes overtreatment from undertreatment. Undertreatment accounted for two thirds of the inappropriate intention-to-treat decisions. Although psychiatrists had the highest rate of overtreatment, they made more appropriate treatment choices than other professions.

The rate of appropriate intention-to-treat decisions of general practitioners and psychiatrists was high in cases with more severe depression, with 72% and 84%, respectively, of patients being treated according to the standard. However, in subthreshold and mild depression the rates of appropriateness were only 59%. Vignette 1 gives an example of such a case. Psychotherapists and clinical psychologists performed better in the milder cases with rates of 68% and 71%, respectively, because they preferred watchful waiting or the psychotherapeutic management recommended in all guidelines.

In table 4 the relation between patient and care provider characteristics and the appropriateness of judgements is shown. Less severe depression, alcohol abuse, psychotic features, and no social resources were related to inappropriate judgements. Vignette 2 illustrates this and gives an example where the appropriateness was particularly low. Significant interactions between the profession of the respondent and patient characteristics are included in table 4. Psychiatrists were much more likely to choose an appropriate treatment when psychotic or vital characteristics were present, while psychologists scored better in cases with alcohol abuse.

Table 5 shows the patient characteristics that determined the referral pattern of general practitioners. On average, general practitioners referred 25% of patients for specialist mental health care. However, in subthreshold depression this was only 4% while in psychotic depression it increased to 66% of cases.

DISCUSSION

This study applied a vignette method to investigate the variation in intention-to-treat decisions among four professions involved in the management of depression. The results show that there is considerable diversity in the choice of treatment. Almost one third of intention-to-treat decisions were inappropriate. This is remarkable, considering the liberal definition we applied.

Despite widespread use of the vignette method, there is a debate as to whether the results of such studies reflect clinical practice or merely physician competence. Some feel that abstracts from medical records may give a more realistic view of the behaviour of the individual practitioner.²¹ However, a recent study reported that vignettes are uniquely suited for comparative analyses between groups of providers and are more effective than chart abstracts if controlled for the case mix.¹⁷ We tried to achieve a realistic case mix by basing the distribution of patient characteristics on the outcomes of a population survey. To ensure parity among the respondents, we included in the vignettes the criteria deemed relevant in the existing guidelines.

Response rates were excellent for psychologists and psychotherapists but modest for general practitioners and psychiatrists, reflecting the general trend of falling response rates to postal surveys among general practitioners.²² The modest response rate did not affect the power of our study but may have introduced a selection effect. General practitioners and psychiatrists with a less positive attitude to depression and research may not have participated. This self-selection bias may lead to overestimation of the knowledge of the group as a whole.^22–24 The relatively high rate of inappropriate intention-to-treat decisions among general practitioners cannot therefore be accounted for by selection bias.

Variation in the choice of treatment between professions is not necessarily undesirable. For example, guidelines recommend both short term psychotherapy and psychopharmacotherapy as options in untreated cases with moderately severe depression. It is not surprising that psychiatrists relatively often chose psychopharmacotherapy and psychologists chose psychotherapy in these cases as this tendency reflects differences in experience with treatment methods.^11,25–28 Some of the choices which we found inappropriate may reflect divergence from a debatable standard. To recommend only watchful waiting or psychotherapeutic management policy in subthreshold depression, for example, could be criticised.²⁹ Vignette 1 illustrates this problem, where many practitioners chose psychopharmacotherapy. However, clinical trials that study the effect of medical treatment in subthreshold depression have only recently been reported.³⁰ Our standard reflected the expert opinion of guideline makers and our panel because rigorously conducted research was unavailable. We feel that treatment choices can be considered inappropriate on this basis, but acknowledge that this remains a matter of debate.³¹

On the other hand, our finding that the choice of psychopharmacotherapy was not increased in cases with psychotic or melancholic features is disquieting because these features are mentioned in the guidelines as indications for psychopharmacotherapy. Further, none of the guidelines supports the notion that a lack of social resources or alcohol abuse are relative contraindications for psychopharmacotherapy.^7–9,20 These patient characteristics deserve attention in quality assessment as they are possible explanations for the frequently reported undertreatment of depression.³² Vignette 2 is an example where many practitioners chose psychotherapeutic management, a choice that is not concordant with the guidelines and considered inappropriate. Our panel confirmed this judgement; there is satisfactory evidence to support psychotherapeutic or psychopharmacological treatment in a patient with these characteristics.

In mental health service research little attention has been given to the possible dangers of overuse driven by providers.³³ In our study one third of all inappropriate intention-to-treat decisions was due to overtreatment—that is, the risks outweighed the potential benefits. In view of the use of guidelines, not only to assure quality of care but also to control health care costs, more emphasis should be put on this problem.

We also analysed intention-to-refer decisions. Remarkably, in one third of all cases the general practitioners did not intend to refer patients with psychotic features to specialised mental health care. Depression guidelines to date do not provide much assistance in referral decisions. Ideally, guidelines should recommend which type of patient should be referred to specialised facilities. Because of the lack of evidence on the effectiveness of healthcare settings, this would require consensus among professional associations. Our study suggests that more patients might be treated appropriately if practitioners adopted referral criteria that reflect the experience of the involved professions.

In conclusion, our data show that there is considerable variation in intention-to-treat depression and that many treatment choices made by the care providers were inappropriate. It was possible to identify specific patient characteristics—for example, lack of social resources, alcohol abuse and psychotic features—that were associated with a particularly high number of inappropriate judgements. Education of care providers alone certainly is not sufficient but in view of the slow diffusion of scientific knowledge it seems necessary in order to improve outcomes in depression. Basic considerations like the treatment choice must not be overlooked as modern management of depressed patients focuses on case management and follow up.³⁴ Treatment indications for depression should therefore be given a priority in quality assessment and guideline development. In addition, the results suggest that efforts to achieve more appropriate care should focus on referral patterns and overtreatment of depression.