Using IV analysis to evaluate relationships

To understand these findings and place them in context, it is imperative to understand the advantages and potential pitfalls of this type of analysis. IV analysis is able to support inferences regarding the casual effect of one or more explanatory variables on an outcome by using a third variable—that is associated with the explanatory variable but not directly with the outcome—as the ‘instrument’ variable in the analysis. It helps to account for both measured and unmeasured confounding variables, making it an attractive option in a situation where a randomised controlled trial is either unfeasible or unethical, and where it is not possible to measure or include all possible confounders in an analysis. With greater availability and use of large routine datasets, IV analysis is an increasingly popular approach in health research.2

However, the key issue is whether the instrument variable used is in fact a good instrument. There are several important assumptions that the data must meet for IV analysis to be a valid approach, including that the instrument variable must affect the explanatory variable of interest, must not affect the outcome except via its effects on the explanatory variable and that the instrument is not otherwise associated with the outcome via other covariates (either measured or unmeasured).3 If IV analysis does not meet these assumptions or is otherwise used inappropriately, spurious conclusions can and do result.4 It is complex to design and to implement, and assumptions may not be easy to meet or to evidence.5 6 A suitable instrument may be difficult to identify or may not exist at all. To be useful, an instrument must impact on as many levels of the explanatory variable as possible. Furthermore, inclusion of an instrument does not always remove the need to adjust for covariates, nor is it inevitably an improvement over a standard covariate-adjusted regression.7

Returning to our three questions about variation in referrals by GPs, the study by Svedahl and colleagues used an IV to describe the referral threshold or tendency of each practitioner to admit older adults. This threshold was calculated as the proportion of older adults not known to the practitioner who were referred during out-of-hours work. The authors found that there was variation in this threshold between practitioners and that it appeared to be independent of patient factors such as age or comorbidities that would determine the need for referral; that is, it was systematic—attributable to the practitioner rather than the patients they saw.

Our third question was whether low-referring clinicians were making better decisions (ie, selecting patients most likely to benefit from referral and so reducing overall costs without impacting outcomes for patients who were not referred). The authors compared the IV approach with a conventional multivariable analysis. The multivariable approach suggested that patients who were referred had higher short-term and medium-term risk of death (Hazard Ratio (HR) 1.41 for days 0–10, HR 1.55 for days 0–180), which may be unsurprising if the sickest people are the ones being referred. In contrast, the IV analysis revealed a lower risk of death after referral for patients whose GP had greater referral tendency, compared with patients treated by low-referring doctors who had worse survival (HR 0.53 for days 0–10, HR 0.72 for days 0–180). While a difference in short-term mortality could have arisen from low-referring doctors making appropriate shared decisions about managing patients near the end of life at home rather than in the hospital (where more intervention might prolong life by only a month or two), the observed survival difference at 180 days suggests that lower referral rates included some worse decisions—not referring some patients who might have gained sustained benefit.

Both analyses were adjusted for important patient and visit characteristics to improve the strength of causal inferences. Importantly, the study did not just include the most vulnerable patients: it included all patients over 65, excluded those with multiple out-of-hours contacts in recent months as they will be more likely to be known to physicians, and had only 10% of patients with more than one significant comorbidity recorded.

Implications for evaluating patient safety in complex routine data

The markedly different findings between the IV analysis and multivariable regression analysis remind us of the importance of an appropriate analysis and demonstrate how striking the effect can be for our conclusions, particularly when interrogating observational data and in situations where a research question or analysis plan was absent at the stage of data collection. The potential utility of IV analysis is obvious, but care must be taken with its use and interpretation.

In this case, the authors make an extremely thorough attempt to discover possible violations of assumptions, perform sensitivity analyses to examine their choice of instrument and take care to explicate that the conclusions apply only to those patients whose referral is due to the GP referral threshold. However, even in this convincing case, we must still be cautious. One thing we do not have detail on is the overall quality of the data in terms of cleanliness and missingness, which is important to judge the subsequent quality of any analysis.8 Additionally, instrument–outcome confounders beyond those investigated here—with the potential to derail such an analysis—are plausible and do exist, for example, patient health behaviours,9 reminding us that an exhaustive verification of assumptions is at best difficult, may not be possible with certain datasets and yet may have a profound impact on the analysis. It is also not clear how large the group of patients is to whom the conclusions relate, and thus how broadly applicable the instrument is; that is, some patients are unwell enough that all GPs will refer, some well enough that none will, so what proportion of patients fall into the category with unclear referral indications, for whom GP referral threshold is relevant?

Additionally, as with any relatively novel method, results should be compared with an appropriate comparator—whether this has been done here is questionable. The results shown by Svedahl and colleagues are striking because they are in the opposite direction to the comparator analysis (the multivariable regression), but it is not clear in this case that this comparator has been adequately adjusted using appropriate variables. In the context of the present research question, it is surprising that no measure of general health or comorbidity is included in this comparator. The absence of such important confounders in the multivariable regression analysis results in referral being associated with greater risk of mortality when in reality, the opposite is true.10 In standard practice, such variables would almost certainly be included in such an analysis, meaning that the current conclusion drawn from the multivariate analysis would be extremely unlikely. The authors state it may not be possible to account for a sufficient set of covariates to indicate the patient case mix. That is fair, but it would still seem astute to include any available important covariates. This is probably, therefore, not an entirely realistic comparison of analyses, may overstate the relative contribution of IV analysis and underestimate the utility of a simpler analysis that adequately adjusts for the most relevant covariates.

Clinical implications of individual referral thresholds

Attempting to understand variation in clinical decisions around referral has a long history11 and remains a live problem.12 Variation in general practice referral of patients through suspected cancer pathways can be better explained by tendency to refer (or ‘referral thresholds’) than by variation in diagnostic accuracy for referrals.13 Variation in referral tendency or threshold is plausible, but how should we characterise it? The absence of any relationship between tendency to refer and a wide range of patient characteristics suggests that it is not simply bias against a particular group of patients. Rather it may reflect what Kahneman et al has recently characterised as one of the several forms of ‘noise’ in human judgement.14 If that is the case, only in-depth analysis of individuals’ decision making is likely to clarify its nature. Whatever the cause, the scale of the impact implied by Svedahl et al’s analysis1 suggests this is important. Based on the analysis reported here, GPs working for out-of-hours services might benefit from knowing how their referral rate compared with their peers (and nudging it up if it was on the low side). Furthermore, any guidance about more conservative referral policies should be viewed with caution and implemented and evaluated in ways which permit the early detection of signals indicating harm.