Objectives To estimate the impact of altering referral thresholds from out-of-hours services on older patients’ further use of health services and risk of death.
Design Cohort study using patient data from primary and specialised health services and demographic data from Statistics Norway and the Norwegian Cause of Death Registry.
Participants 491 653 patients aged 65 years and older contacting Norwegian out-of-hours services between 2008 and 2016.
Analysis Multivariable adjusted and instrumental variable associations between referrals to hospital from out-of-hours services and further health services use and death for up to 6 months.
Physicians’ proportions of acute referrals of older, unknown patients from out-of-hours work were used as an instrumental variable (‘physician referral preference’) for their threshold of referral for such patients whose clinical presentations were less clear cut.
Results For older patients, whose referrals could be attributed to their physicians’ threshold for referral, mean length of stay in hospital increased 3.30 days (95% CI 3.13 to 3.27) within the first 10 days, compared with non-referred patients. Such referrals also increased 6 months use of outpatient specialist clinics and primary care physicians. Importantly, patients with referrals attributable to their physicians’ threshold had a substantially reduced risk of death the first 10 days (HR 0.53, 95% CI 0.31 to 0.91), an effect sustaining through the 6-month follow-up period (HR 0.72, 95% CI 0.54 to 0.97).
Conclusions Out-of-hours patients whose referrals are affected by physician referral threshold contribute substantially to the use of health services. However, the referral seems protective by reducing the risk of death in the first 6 months after the referral. Thus, raising the threshold for referral to lower pressure on overcrowded emergency departments and hospitals should not be encouraged without ensuring the accuracy of the referral decisions, ideally through high-quality randomised controlled trial evidence.
- decision making
- patient safety
- primary care
- health policy
Data availability statement
Data may be obtained from a third party and are not publicly available.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
WHAT IS ALREADY KNOWN ON THIS TOPIC
Following increasing demands for health services and emergency department crowding, avoiding unnecessary hospital admissions is a major policy objective and may affect physicians’ thresholds for referral.
Variation in physicians’ thresholds for referring their patients to the hospital may affect the health services trajectory and outcome of a substantial number of patients, where the indications for referral are not clear cut.
There is a lack of empirical evidence on which referral threshold is most beneficial for patients and health services, and how altering the referral threshold will affect health services use and patient outcomes.
WHAT THIS STUDY ADDS
This is the first observational study to estimate the impact of different thresholds for referring patients to hospital in cases where the indications for referral are not clear cut.
Our findings suggest that patients who, based on their treating physicians’ threshold for referral for other patients, are more likely to be referred, contribute considerably to the use of health services, however, they also have a substantially reduced risk of death.
Ageing populations are increasing pressure on health services; consequently, avoiding unnecessary hospitalisations is a major policy objective.1 2 Physicians working in primary care regularly face dilemmas over referring or not referring patients to hospital. Such gatekeeping may have a crucial influence on service use, but consequences for patient outcomes are poorly understood.3 4 In cases where the symptoms and findings are diffuse but still may be associated with severe illness, some physicians will refer and possibly contribute to increasing emergency department crowding and other health service use. Others will choose a wait-and-see approach or initiate other non-hospital care and possibly delay access to vital specialised care. Consequently, in practice, many patients will have their decision of referral determined by the personal threshold of the physician assessing them. Emergency referrals from out-of-hours services are of particular interest since the physician often has scarce or no knowledge about the patient and limited time, resources, and access to medical records, in contrast to the regular general practitioners (GPs) working normal hours.
The consequences for patient safety and health service use from altering referral thresholds are mainly unknown and complex to study.5 Ideally, changes to clinical practice with possible impact on systematically delaying or avoiding hospitalisation should be investigated using randomised controlled trials (RCTs). However, this would most likely not be tolerated by physicians or patients, as RCTs for this purpose would be both ethically and practically challenging. Observational data could help describe the outcomes of patients who are referred or not. However, patients who are referred to hospital are likely to be sicker than those not referred, and it is unclear whether it is possible to account for a sufficient set of covariates to control for all differences between individuals.
Instrumental variable analysis is a design that can, under some assumptions, estimate the causal effects of interventions in the presence of unmeasured confounding of the exposure-outcome relationship by using observational data.6 Physician prescribing preferences have been widely used to estimate comparative effectiveness and safety of different treatments.7–9 Physicians’ preferences for referring, as indicated by their referral decisions about their previous patients, are known to predict their referral decisions regarding their future patients.10 These physician referral preferences, or referral thresholds, are unlikely to be related to the presenting symptoms of their current patient, especially for out-of-hours physicians seeing a particular patient for the first time. Thus, instead of studying whether patients are actually referred or not, we compare patients with a substantial difference in probability of referral based on their physician’s threshold for referral. Instrumental variable analysis can provide valid estimates of causal effects of referrals, even if there are differences in the case mix of referred and non-referred patients.
Older patients are more challenging to assess and highly affected by a referral decision, as they are often frailer and more prone to adverse events if hospitalised.11 ,12 ,13 They are also more vulnerable to overdiagnosis, because of incidental findings in clinical, laboratory and radiological investigations,14 where clinical guidelines may suggest follow-up. Furthermore, if admitted to the hospital, it may take time to arrange proper primary care to allow for discharge, demanding capacity. It is, therefore, most important to avoid unnecessary referrals for these patients.
We investigated health service use and deaths that occurred up to 6 months after the initial contact with the out-of-hours primary care services for older patients (aged >64 years) previously unknown to the physician and who had few previous out-of-hours visits.
We used the referral preferences of out-of-hours primary care physicians as an instrumental variable to estimate the impact of altering referral thresholds on older patients’ future use of health services and risk of death.
Materials and methods
The Norwegian healthcare system is based on universal access, with only limited patient co-payments for GP services and outpatient specialist clinics and no co-payment for hospital admissions.15 Out-of-hours primary care in Norway is mainly organised through GP cooperatives, the dominant model in Europe.16 These handle most acute illnesses that occur outside office hours, and, as with normally scheduled GP care, provides gatekeeping services for secondary care. When primary care physicians decide to immediately refer a patient to the hospital, a hospital physician will assess the patient again, and decide whether an admission is appropriate. If an admission is considered unnecessary, this assessment will be registered as an acute outpatient specialist contact. About one-fourth of out-of-hours contacts with patients aged 70 years and older in Norway result in an immediate unplanned hospital referral.10 In most cases, patients have no influence on the choice of physician in the out-of-hours setting. Participating in the out-of-hours services is mandatory for GPs, but other physicians also staff the service and manage about half of the contacts.17
Study population and design
The study population included all patients 65 years and older, assessed by an out-of-hours physician in Norway, either face-to-face at the out-of-hours service station or by telephone during the study period from 1 January 2008 to 31 December 2016. We included all contacts between 16:00 and 08:00 hours on weekdays and whole Saturday, Sunday and public holidays. Using a unique identification number, we linked information on specialised health service use (psychiatric care was not included),18 primary care physicians use,19 demographic information from Statistics Norway20 and date of death from the Norwegian Cause of Death Registry.21 Information from the Norwegian General Practitioner Register was linked to each patient contact by a unique physician ID.22 Please see online supplemental material for a detailed description of data sources and study design.
In figure 1, we list the criteria for study inclusion. To reduce the risk of including contacts where the physician was familiar with the patient, we excluded contacts where patients had been assessed by the same physician previously, where the patient was assessed by their regular GP and from patients without a registered GP. Frequent attenders to out-of-hours services are more likely to be known to physicians.23 Hence, we excluded all contacts from frequent attenders (>4 consultations per year). Patients who were registered with illogical combinations, like the date of death occurring before the date of out-of-hours contact, were also excluded.
Referral to the hospital for admission was defined as a registered unplanned visit to hospital (both inpatient and outpatient visits) within 10 hours of an index contact with the out-of-hours services.
We measured health service use as the number of (1) hospital inpatient days (where day care treatment was counted as 0.5 days), (2) days with contacts with primary care physicians and (3) days with outpatient specialist contacts in hospitals (both elective and acute). Risk of death was measured as time to death for any reason. All outcomes were measured during four time periods after the index consultation: 0–10 days, 0–30 days, 0–90 days and 0–180 days.
Instrumental variable analyses
We used a measure of the physician’s preference for referral of as a candidate instrumental variable in order to estimate the causal effect of hospital referral at various tendency levels. There are key assumptions for instrumental variable estimation, as shown in figure 2. First, the instrumental variable must be strongly associated with the exposure (relevance). It is not possible to set a distinct cut-off for a relevance test. Although a cut-off of F-statistics >10 have been used,24 this is only a rule of thumb, and does not guarantee that any specific estimate will be unbiased or sufficiently well powered for the hypothesis of interest. Second, there should be no confounders of the instrumental variable-outcome relationship (independence). Third, the instrument must affect the outcome only through the exposure (exclusion restriction). Fourth, when assessing the effect of referrals attributable to referral threshold, patients’ likelihood of being referred should not be reduced if they attend a high referring physician versus a low referring physician (monotonicity).25 We can test the first assumption by estimating the association between physicians’ referral behaviour for patients other than the index patient and the referral decision for the index patient. The second and third assumptions are not directly testable, but they can be falsified, by estimating the association of the physician’s overall referral preference and the characteristics of their case-mix. The fourth assumption is not testable in our case, but it is likely that patients who were referred by a low referring physician would be at least as likely to be referred by a high referring physician. The estimate will be valid for patients whose referrals can be attributed to their physician’s referral threshold, and therefore not for patients who would always be referred, or those that would never be referred.26
Our instrumental variable ‘physician referral preference’ was calculated as the physicians’ proportion of contacts with previously unknown patients 65 years and older, in out-of-hours work (from 2008 to 2016) leading to an immediate unplanned hospital visit within 10 hours, presumably through an acute referral to hospital. To increase the strength and precision of the instrument, we chose to use information about the physicians’ activity through the whole study period. However, to avoid having activity from the index contact affect the instrument, we used the physician’s referral proportion of female patients as an instrument for referring male patients and vice versa. Mean physician referral preference for male and female patients, respectively are presented in table 1. To handle potential systematic patient differences across physicians, particularly between geographical areas and time, we matched patients into groups defined by combining information on patients of the same sex, visiting the same out-of-hours station in the same 8-hour time unit during the day, within the same year. For example, a female patient visiting a service in the afternoon in 2015 was only compared with other female patients visiting the same service the other afternoons (16:00–23:59 hours) in 2015. By analysing only within-group variability, we effectively controlled for all confounding that was constant within each group. To avoid the effect of possible patient selection in situations where two or more physicians were on-call at the same time, we used the weighted average of physician referral preferences within each 8-hour time unit in each service. Details of the study design are presented in online supplemental material.
Statistical analyses and analytical design
We compared health service use among patients with and without referral to hospital following out-of-hours consultations, by using both multivariable adjusted analyses and instrumental variable analyses. Follow-up began on the day of each index contact with out-of-hours services and lasted for the entire follow-up period (10, 30, 90 and 180 days). We excluded contacts from the last months in 2016, where the patients could not be observed through the defined follow-up time (from July to December in the analyses for 180 days, October to December for 90 days, December for 30 and 10 days).
For the multivariable adjusted analyses, we used linear regression models to investigate the associations between referral and further health service use. In these analyses, we adjusted for patient age and age squared, sex, year, month, weekday and hour of the contact. Furthermore, we used Cox regression with time from index contact as the time axis to investigate the association between referral and mortality. Precision was evaluated with 95% CIs with robust SEs.
For the instrumental variable analysis, we used a within-group estimator for instrumental variable regression (ivreghdfe in STATA) to study the effects on further health service use.27 A challenge for our data was that the observations were clustered within physicians as well as patients, and it was essential that our SEs allowed for this clustering. We adjusted for month, weekday, patient age and age squared, in addition to the within-group estimation (time unit, year of contact and patient sex). Mortality was analysed with a within-group estimator using stratified Cox analyses, and a two-sample instrumental variable estimator (the delta method), corrected for clustering on the physician level (as patients could only die once).28 Precision was evaluated with 95% CIs with robust SEs clustering on physician and patient ID.27
Assumptions and additional analyses
We assumed that our instrument was unrelated to the health status and comorbidities of the physicians’ current patients. To support this independence assumption, we analysed associations between our instrument and possible confounding patient characteristics such as age, sex, immigration status (yes/no), education (completed <13 years or >13 years), health service contacts 30 days before the index contact (ie, GP visits, planned and unplanned hospital admissions and outpatient visits), discharge diagnoses from previous hospital stays divided in main chapters (International Classification of Diseases, Tenth Revision, Chapter IX—Diseases of the circulatory system, and Chapter II—Neoplasms) and Charlson Comorbidity Index.29 Results of this test are presented in online supplemental table S2. We also analysed the strength and the variability of the instrument, and this is presented in online supplemental table S3.
For sensitivity analysis for the instrumental variable, we used an alternative definition of the instrument for physician referral preferences. This definition was the physicians’ proportion of patient contacts that led to a referral for all patients, excluding contacts from the same year and 1 year before and after the index contact. This exclusion made the instrument less prone to effects from cases where certain physicians work certain shifts close to the time of the index contact, thus seeing patients with systematically different morbidities. We also performed sensitivity analyses where we adjusted the instrumental variable for referral diagnosis group (main chapter in International Classification of Primary Care-2), both for the multivariable adjusted regression analysis of health services use (reghdfe in STATA) and for Poisson regression analyses (ppmlhdfe in STATA) of mortality. The results are presented in online supplemental table S3.
Patient and public involvement
We used data from nationwide health registries and had no patient or public involvement. Thus, no patients were involved in setting the research question or the outcome measures, nor in designing or conducting the study. No patients were asked to advise on interpretation or writing up of results. Our project is affiliated with several patient organisations, and we will use involvement from their representatives in the communication of results.
During the study period, there were 1 798 482 contacts with the out-of-hours services made by 605 509 unique patients (figure 1). Excluding contacts not fulfilling the eligibility criteria, left 944 512 contacts from 491 653 unique patients. Twenty-six per cent of the contacts resulted in an acute referral. The mean patient age was 77.4 years, and 43% of the contacts were made by male patients (table 1). There were no substantial deviations between the included study population and the whole population, except for their previous healthcare use, which corresponds well with our selection criteria of patients previously unknown to the physician and the system (online supplemental table S1). There was a strong association between our instrumental variable ‘physician referral preference’ and the probability of referral for the index patient (F-value≈1200, online supplemental table S3). One SD increase in the physician referral preference (instrument) was associated with a risk difference of about four percentage points for index referral to the hospital (online supplemental figure S2).
Use of health services
The instrumental variable analysis suggests an increase of 3.30 (95% CI 3.13 to 3.47) days in hospital following an acute referral from an out-of-hours physician 0–10 days after the index contact (figure 3). This estimated effect was lower than the estimate for the multivariable adjusted analysis of 4.12 days (95% CI 4.11 to 4.14), with a p value for difference <0.001. Patients who were referred had a substantial increase in health service use, present up to 6 months after the index consultation. There was a slight increase in days with outpatient visits, with similar estimates for the instrumental variable and multivariable adjusted estimates. With 3 and 6 months follow-up, there was also an increase in days with primary care physician contacts.
In figure 4, we present the HRs for death within the defined follow-up periods of 10, 30, 90 and 180 days, showing larger differences between the two methodological approaches. While the multivariable adjusted estimates suggested a substantial increase in the risk of dying during first 10 days after a hospital referral (HR 1.41, 95% CI 1.37 to 1.45), the instrumental variable estimate suggested a substantial protective effect of referral. For the first 10 days, the risk of dying was almost halved (HR 0.53, 95% CI 0.31 to 0.91), and this effect seemed to sustain, although steadily weakening through the follow-up period.
Analyses of confounder balance related to the instrument showed weak or no associations, supporting our assumptions of independence between the confounders and the instrument (online supplemental table S2). The confounders were also balanced across different values of the instruments (online supplemental figure S1).
The estimates did not substantially differ when using a different specification of the preference instrument or when adjusting for referral diagnosis in the analyses (online supplemental table S3).
In this national study with nearly 1 million out-of-hours contacts from patients aged 65 years or older, we found substantial effects of being referred to the hospital for patients whose referral was attributable to their physicians’ threshold for referral. Using instrumental variable analyses, we found that patients had substantially more days in the hospital, higher use of outpatient specialist clinics and primary care physicians the following 6 months if they were referred. Furthermore, while multivariable adjusted regression analyses estimated a substantially increased risk of death among all referred patients, the instrumental variable regression analyses suggested a substantially reduced risk of death for patients whose referrals were attributable to their physicians’ referral threshold. Collectively, these findings suggest that raising the threshold for referral may substantially lower the use of specialised services, weakly reduce GP workload, but have a decisive impact on patient safety through increased mortality for the affected patients.
Comparison with existing literature
To our knowledge, this is the first study assessing the effect of an acute referral to hospital from an out-of-hours service for patients with possibly unclear indications for referral. Previous studies have found substantial differences between out-of-hours physician referral desicions,30 31 suggesting a varying threshold for referring across physicians.32 33 An essential aspect of variation in referral thresholds between physicians is for which patients this variation will apply. Most patients in primary care would not be affected by the physicians’ varying thresholds, as their referral needs are quite obvious. However, for some patients, the referral threshold of their treating physician will be decisive for their further care. It is also reasonable to believe that these patients are the ones most likely to be affected by general requests for primary care physicians to lower their referral rates through increasing their referral threshold. Budget pressures for health services across the world34–36 and challenges from crowded emergency departments37 have led to initiatives to reduce patient inflow to costly and limited specialised services. The incongruent views on referral practice within the health services serve as a base for conflict.38 39 Restricting referral opportunities and using GP referral rates as quality indicators have been debated.40–42 GPs with high referral rates, and assumably low referral thresholds, have been targeted to change practice to reduce pressure on specialised care, whereas there have been less, if any, such systematic efforts towards GPs with low referral rates to change their practice. Although reducing the pressure on services may be necessary, the impact on the patients whose entrance to specialised services is delayed or even denied by such changes in referral practice is challenging to study. Observational studies using multiple variable adjustment or propensity scores are likely to suffer from residual confounding by indication: patients who are referred will in general be sicker than those who are not referred. Referred patients are likely to be sicker on both observable factors, and factors that are extremely challenging to measure and control. As a result of this, multivariable and propensity score estimates of the effects of referral are likely to be biased. Our results suggest that patients with a referral attributable to their physicians’ referral threshold are likely to lead to substantially higher hospital use and slightly higher use of outpatient specialist clinics in the time after referral. These patients are also more likely to have future GP service use, hence such referrals do not relieve GPs from future workload.
Our results suggest also a substantially reduced mortality, sustaining through the first 6 months after a referral attributable to physician’s referral threshold. If the assumptions from our instrumental variable analyses hold, our results suggest that the potential impact from altering referral thresholds should be carefully assessed before being implemented, ideally via a large well-powered RCT. Our results also have implications for cases where, when a hospital’s capacity limits are reached, thresholds for referral are automatically raised. Our findings indicate that these situations may have detrimental effects on the affected patients. This is also in line with the findings of increased mortality for hip fracture patients who are discharged due to increased pressure/strain on the hospital.43
Strengths and limitations
Our study is based on population-based comprehensive registers with complete information and exact censoring, collected for reimbursement purposes, minimising the risk of differential misclassification. We used an instrumental variable design in an attempt to address confounding by indication. Hence, by using the physician referral preference as an instrument to predict the patient’s chance of referral, we could estimate the effects of referrals attributable to the physicians’ referral threshold, and not to the patients’ underlying health conditions. To give valid results, our instrument should satisfy a set of assumptions. Although we had a large study, instrumental variable analyses are less statistically powerful (less precise) than other approaches, and the CIs were relatively wide. Our preference instrument showed weak or no associations with possible confounding variables, including across different values and definitions of the instrument, a result in favour of our independence assumption. There should be no effect of the physician referral preference on our outcomes other than via hospital referral for the index patient. This assumption could be violated if the physician referral preference was associated with specific treatment actions unrelated to the referral decision. However, if this violation were to be substantial, our findings would still support our conclusion about not mainly targeting these high-referring physicians to improve their decisions or treatment actions.
Implications for future research
Referral thresholds have been thoroughly studied with the aim of identifying unnecessary hospital referrals and admissions. Our findings suggest that efforts to reduce referral rates may have the unintended side effect of increased mortality. Our study supports the contention that referrals attributable to physicians’ referral thresholds impose a high use of hospitals, possibly avoidable by raising referral thresholds, thus representing a target to reduce pressure on specialised services. However, the finding of substantially reduced mortality for these patients imply that at least for some of them, such referrals have crucial value, and that simply asking primary care physicians to raise their referral threshold could have detrimental consequences. Our findings prompt a need for further research to identify and evaluate more specific measures other than simply raising the thresholds for referral to reduce the increasing inflow of patients to specialised care, particularly considering the ageing and increasing population. Improving opportunities to observe patients over time in out-of-hours services may improve the appropriateness of referrals. Increasing the capacity in emergency departments for such observation could also be helpful, as prolonged observation is shown to lead to lower admission rates from the emergency departments.44 However, our findings support the need for thorough evaluation of any measures taken to reduce referrals from primary care.
Out-of-hours patients whose referrals to hospital are attributable to their physicians’ referral threshold contribute substantially to the use of health services. However, such referrals seem protective by reducing the risk of death during the first 6 months after the referral. These findings imply that raising the thresholds for referral as a measure to lower pressure on overcrowded emergency departments and hospitals should not be encouraged without further study with high-quality studies, such as cluster RCTs.
Data availability statement
Data may be obtained from a third party and are not publicly available.
Patient consent for publication
This study was approved by the Regional Committees for Medical and Health Research Ethics (2016/2158/REK midt). Participant consent was not required, as we used only routinely collected health data, and there were no possibilities of identification of participants.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Contributors I, the Submitting Author (ERS) has the right to grant and does grant on behalf of all authors of the Work (as defined in the below author licence), an exclusive licence and/or a non-exclusive licence for contributions from authors who are: (i) UK Crown employees; (ii) where BMJ has agreed a CC-BY licence shall apply and/or (iii) in accordance with the terms applicable for US Federal Government officers or employees acting as part of their official duties; on a worldwide, perpetual, irrevocable, royalty-free basis to BMJ Publishing Group Ltd (“BMJ”) its licensees and where the relevant Journal is co-owned by BMJ to the co-owners of the Journal, to publish the Work in this journal and any other BMJ products and to exploit all rights, as set out in our licence. The Submitting Author accepts and understands that any supply made under these terms is made by BMJ to the Submitting Author unless you are acting as an employee on behalf of your employer or a postgraduate student of an affiliated institution which is paying any applicable article publishing charge (“APC”) for Open Access articles. Where the Submitting Author wishes to make the Work available on an Open Access basis (and intends to pay the relevant APC), the terms of reuse of such Open Access shall be governed by a Creative Commons licence—details of these licences and which Creative Commons licence will apply to this Work are set out in our licence referred to above. Other than as permitted in any relevant BMJ Author’s Self Archiving Policies, I confirm this Work has not been accepted for publication elsewhere, is not being considered for publication elsewhere and does not duplicate material already published. I confirm all authors consent to publication of this Work and authorise the granting of this licence. I accept full responsibility for the work and/or the conduct of the study, I had access to the data, and controlled the decision to publish.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.