Article Text
PDF
Abstract
The ever increasing pressures to ensure the most efficient and effective use of limited health service resources will, over time, encourage policy makers to turn to system modelling solutions. Such techniques have been available for decades, but despite ample research which demonstrates potential, their application in health services to date is limited. This article surveys the breadth of approaches available to support delivery and design across many areas and levels of healthcare planning. A case study in emergency stroke care is presented as an exemplar of an impactful application of health system modelling. This is followed by a discussion of the key issues surrounding the application of these methods in health, what barriers need to be overcome to ensure more effective implementation, as well as likely developments in the future.
- Healthcare quality improvement
- Quality improvement methodologies
- Quality improvement
Statistics from Altmetric.com
Introduction
Internationally, the ever rising demands, costs and expectations in health services coupled with restricted or even reducing budgets has led inevitably to increasing pressures on health policy makers to ensure efficient and effective use of resources. One response has increasingly been to turn to systems modelling and simulation to assist in the decision making process. Such techniques hold the promise to improve outcomes and cut costs through an evidence-informed analysis of service design and delivery alternatives.
Systems modelling and simulation (also referred to as operational or operations research) in healthcare has a history dating back over half a century,1 however the vast majority of research in this field has occurred in the last 20 years. Its rapid evolution has been enabled largely by the increasing availability and accessibility of computer technology. There is now a growing interest in the use of these techniques to identify potential service improvements and provide an improved evidence base for proposed changes in delivery.2 This has particular resonance in the context of healthcare quality and safety where potential cost savings need to be assessed against risk.
Despite this growing interest, serious and widespread use of systems modelling and simulation in healthcare remains limited. Although there is undoubtedly some history of applying these techniques in healthcare management3 ,4 ,5 healthcare lags behind other industries where there is a long and proven track record in applying modelling approaches.6 Few leading car manufacturers or call centre managers, for instance, would think of establishing new operations or make key system changes without running a computer simulation to test differing configurations and minimise the risks associated with full implementation. These approaches, however, are not routine in the management of healthcare. Despite chief executives, operations managers, clinicians and others seeking tools to improve service organisation, use of modelling and especially the dissemination of modelling products and the application of modelling results in healthcare, is at best patchy.
This lack of implementation in healthcare contrasts markedly with levels of research in healthcare modelling and simulation. The following excerpts from reviews of the research literature clearly illustrate the ‘implementation gap’ which exists between research and serious application in the field:Despite the increasing numbers of quality papers published in medical or health services research journals we were unable to reach any conclusion on the value of modelling in healthcare because the evidence of implementation was so scant.7
Despite the wealth of contributions, relatively few academic papers on health issues in [operational research or management science journals] address issues of outcome, implementation or the use of the work reported8
Although more than 90% of the publications provided some discussion of the utility of simulation for analysing changes in the delivery of surgical care, only half reported on simulation models that were constructed to address the needs of managers or policy-makers, and only a quarter reported some involvement of health system managers and policy-makers in the simulation study.9
In this paper, we outline the wide array of modelling and simulation techniques that are available and the range of areas in health where these can be applied. Through a selected case study we demonstrate the potential value and impact of these. We go on to discuss some of the barriers to wider uptake of systems modelling and simulation, despite the broad scope for potential applications, and highlight areas for future developments.
Outline of approaches
Systems modelling and simulation encompasses a wide diversity of approaches.10 ,11 These range across a spectrum from so-called ‘soft’ techniques at one end to ‘hard’ techniques at the other. Soft methods include problem structuring and conceptual modelling techniques such as Soft Systems Methodology,12 Strategic Options and Decision Analysis (SODA) and Strategic Choice Approach,13 These approaches are used to tackle complex and unstructured problems with multiple stakeholders and typically aim to help improve group understanding of the aims of a system, to ask questions of it, and to facilitate team consensus. Such techniques generally employ qualitative methods such as cognitive mapping and extensive interaction with stakeholders through facilitated workshops. Examples of applications of soft methods in health include a study on improving the organisation of multidisciplinary team meetings for colorectal cancer14 and addressing the gap between patients’ and providers’ expectations in NHS hospital outpatients’ department.15
At the other end of the spectrum, hard systems modelling makes extensive use of mathematical and computer simulation methods to provide quantitative analysis and insights to problems that have a clearer structure, metrics and quantitative outputs. Mathematical methods such optimisation,16 data envelopment analysis17 and queuing theory18 use analytical formulations to develop models that fit a problem description under a range of restrictive assumptions. In comparison, computer simulation methods such as system dynamics,19 Monte Carlo simulation,20 discrete event simulation 21 and agent based simulation 22 often allow for fewer assumptions to be used to capture details of the care system at the expense of more complex and time-consuming experimentation procedures and some loss of clarity and repeatability in the model. Importantly these techniques can also often provide clear tools for process visualisation which can be instrumental in facilitating understanding among decision makers (table 1).
Outline of methods commonly used in healthcare system modelling and simulation
For a given project, the choice of approach between soft and hard methods is influenced by a range of factors, including the objectives of the study and type of problem being addressed. For example, if the objective is to bring into consideration the different opinions from a variety of stakeholders, to deal with high uncertainty or to compare strategic options, then a soft method would tend to be more appropriate. If nature of the study and the type of the decision problem calls for forecasts or insights that are quantitative in nature then one of the hard modelling and simulation techniques would be more suitable.
Choosing an appropriate hard modelling method for a particular problem can be challenging and depends on a number of factors such as the organisational decision support requirements or the research questions for applied health research projects; the features of the problem and nature of the trade-offs being considered; the presence, coverage and quality of clinical and operational data; the modelling requirements and specifications; the time and resources available; the availability of skills and expertise in the modelling team including the availability of appropriate software tools if required.
For the purposes of illustration, assume that the choice is between an analytical method, system dynamics and discrete event simulation. The problem being tackled is the organisation of stroke prevention services at the regional level and the objective is to evaluate the likely impact of different options of service organisation on patient outcomes (eg, life-years saved) and levels of resources required to meet anticipated demand for services.
With analytical modelling, only a few of the essential components of the whole system and the flows of patients between those components could be captured. For example, it would be difficult to include many (or any) stochastic elements such as random effects and uncertainties especially at the individual patient level (eg, time of stroke onset). Nevertheless, such an approach could potentially provide some initial insights about some of the system trade-offs as well as help in scoping scope the problem and identifying data needs.
With system dynamics modelling, it would be possible to capture aggregate flows of patients, any feedback effect in the care system (which occurs when outputs of part of the system are ‘fed back’ as inputs to another part) and the effects of time delays and non-linear relationships between these flows. Again, the system dynamics methodology does not allow for random effects or information at individual patient level to be captured. However, it would allow for the analysis of the dynamic interactions between the system components and variables and how these ‘play out’ over time.
Discrete event simulation would give the capacity of capturing individual patients and their unique trajectories as they flow through the entire care system. It would allow the inclusion of random effects and a large number of different patient attributes such as age, gender, CHADS2-VASc score, etc. Experimentation with the models could take place over extended time horizons where patients move through the modelled system as they experience events at discrete points in simulated time. Discrete event simulation would finally provide the flexibility to incorporate capacity and resource constraints explicitly and to capture the ‘competition’ between modelled entities (eg, patients) for access to limited resources (eg, appointments in clinic). There are however drawbacks such as the need for more and finer grained data to estimate the values of input parameters, longer model implementation times and increased computational costs associated with running experiments.
In this example, discrete event simulation would be the modelling method of choice. It provides the capacity to track individual patient journeys through the care system, to capture complex interactions of patients from diagnosis through to various forms of treatment (as informed by the disease progression of each simulated patient), and the ability to model notions of limited availability of resources (and their associated costs). System dynamics would be more appropriate if there was no requirement or need to model at the individual patient level or to include random variability in the model. Finally, an analytical method would perhaps be more appropriate if there was no feedback between the different components of the system (or such feedback could be safely ignored for the purposes of the study) or potentially if the resultant model were to be embedded within a software tool.
While systems modelling and simulation methods are often used in isolation, there is an increasing awareness of the benefits of using a mixture of soft and hard methods in combination. For example, soft methods that allow for the use of a participative and facilitative approach have been used to generate the conceptual model of a care system and the study objectives of a discrete event simulation study. The approach has been illustrated using a case study on the surgical obesity care pathway.23
Case study example
Modelling the implementation of stroke thrombolysis
Stroke is the major cause of disability internationally, the leading cause of disability in England, and the third most common cause of death worldwide. In the UK there are over 150 000 strokes each year with 1.2 million poststroke survivors. A 2009 estimate placed annual costs of stroke in the UK at £9 billion (€12.3 billion; $13.6 billion).24
Healthcare systems need to be highly responsive to acute stroke emergencies in order to minimise the substantial costs and consequences associated with stroke survival and rehabilitation.25 In recent years, systems modelling and simulation has been used to aid the design of responsive stroke care systems to meet the time sensitive requirements of treating ischaemic strokes with recombinant tissue plasminogen activator (rtPA).25 ,26 ,27 ,28 ,29 ,30 ,31 ,32 ,33 There is growing evidence that this work leads to implementation changes and real system improvement.34 Here we provide an overview of the simulation and modelling work conducted by Monks et al.34 ,35
Strong clinical evidence, weakly implemented
The only licensed treatment for acute ischaemic stroke is stroke thrombolysis with rtPA. There have been nine rtPA trials and analysis of the positive and negative trials in two individual patient pooled meta-analyses36 ,37 demonstrate a time-dependent effect of treatment with the benefits of treatment diminishing with the passing of every 90 min up to 4 h and 30min (although the maximum time window of benefit is still uncertain38 ,39) where the harms of the treatment, specifically risk of symptomatic intercranial haemorrhage, outweigh the benefits. Although treatment for rtPA was originally licensed 19 years ago, treatment rates remain low internationally.32 Part of the explanation for this apparent low uptake is explained by the inhospital delays in delivering rtPA, for example, poor identification procedures in emergency departments and unnecessary steps before initiating an urgent CT scan or contacting a stroke specialist.
Methods
Our case study hospital is situated in a mixed rural location and cares for over 630 acute strokes per year. In early 2011, annual treatment rates stood at ∼4% of all acute strokes with an average arrival-to-treatment time of 100 min. The hospital treated patients up to 3 h after onset of stroke, which at the time followed European guidelines for rtPA. A detailed simulation model of prehospital and inhospital processes was developed that took account of the intraday and between-day variation seen in onset-to-arrival times, emergency department waiting times, scanning and staff availability. This meant for example, that the model would accurately capture when the emergency department was busiest, when suspected stroke patients were most likely to attend the emergency department and when stroke physicians were on call.
The first stage of the modelling was to reproduce the current inhospital processes and the typical performance seen in terms of treatment rates and onset-to-treatment times. In addition the model used clinical data about the time dependent effectiveness of rtPA to estimate poststroke disability and change in workload. The former was operationalised using modified Rankin Scores of 0 or 1 at 90 days40 while the latter was reported as the increase in prioritised scans and urgent callouts of stroke nurse practitioners. Once the project group was confident that the model mimicked the ‘status quo’ accurately the model was used to estimate the impact of alternative ways to implement the evidence in practice. The most prominent of these were: extending the rtPA treatment window from 3 h to 4 h and 30 min; senior triage nurses alerting the acute stroke unit of suspected stroke patients in ED; ambulance paramedics using a phone-ahead protocol to prealert of an imminent arrival of a suspected stroke patient; and extended stroke nurse practitioner hours.
Model results and recommendations
The model provided several important insights into the inhospital process for stroke. First, although extension of the treatment window from 3 h to 4 h and 30 min increased treatment rates this was substantially less than holding the treatment window at 3 h and implementing the inhospital measures that reduced delays. The most effective of these was a paramedic phone-ahead protocol, although this did not capture the population of strokes that are brought to the ED by a witness. On this basis, it was recommended that the ED nurse and paramedic protocols were implemented in addition to extending the treatment window. The most expensive option was extension of stroke nurse practitioner hours from 20:00 until 00:00. The model predicted minimal improvement in treatment rates during this time. This was due to the arrival profile of strokes to the hospital and the constraint that stroke physicians are not on site from 18:00. Overall the model predicted that if changes listed above were implemented treatment rates would increase from 5% to 15% of all acute strokes. Potential bias in modelled results was handled by an exhaustive sensitivity analysis of model input parameters. This demonstrated that variables that had the most significant impact were process variables: the adherence of paramedics and nurses to the early alert protocols. Detailed results are available in Monks et al.34 ,35
Service evaluation
The project was evaluated using a simple before and after design.34 This demonstrated that once implementation was complete door to needle times fell from an average of 100 min to 55 min (as of July 2013). Thrombolysis rates rose to 14.5% (figures 1 and 2). There was no concern that faster treatment had affected safety with a smaller proportion of complications in the after period (since this time there is new evidence that faster treatment reduces the risk of symptomatic intercranial haemorrhage41). The latest quarter figures from the Sentinel Stroke National Audit Programme illustrate that improvement has been sustained with a long-term thrombolysis rate of 16%.42
Thrombolysis rate before, during and after the modelling project.
Patients treated by 90 min epochs and predicted modified Rankin scores (mRS) 0–1 at 90 days. †The period evaluated following the modelling (21 weeks) has been annualised to 52 weeks.
The role of modelling and simulation in organisational learning
Implementation of stroke thrombolysis has proved challenging internationally. The modelling approach adopted in the stroke thrombolysis study directly involved clinicians who administered and facilitated the delivery of thrombolysis in the hospital. This was a key factor in building trust and ownership in the data and results, but also for facilitating organisational learning about where bottlenecks were located in the pathway and how to tackle them effectively. Pathway visualisation provided by the simulation model was also important in this context.
The insights into implementation derived from the modelling are quite general and applicable to other hospitals nationally and internationally. The work was followed up with three further rural hospitals in the UK. Experiences here indicate that basic process problems for the delivery of rtPA vary considerably and as such it was necessary to conduct bespoke projects (although the same methodological approach was adopted). The benefit of a systems modelling and simulation approach was again to take the project team on a journey from diagnosing problems to investigating solutions.
Discussion
The stroke thrombolysis case study illustrates the potential beneficial insights that systems modelling and simulation can contribute to the design of health services. The discrete event simulation deployed in the example can also be used to address other process-driven problems that are subject to substantial variability and to capacity constraints, and where there is a clear need to prospectively ask “what-if?” and explore the likely impact of any changes. For example, similar techniques have been used to address accident and emergency departments,43 ,44 ,45 hospital pharmacies,46 intensive care units47 and diabetic retinopathy screening services.48
Table 2 summarises a selection of projects where systems modelling and simulation techniques have been successfully used to address specific health service needs. These give a flavour of the potential range of application.
The majority of applications address operational issues although others tackle problems of a strategic nature, warranting a different type of approach. For example, the highly detailed micro-level simulation of an acute stroke pathway as illustrated above is rather different to macro-level techniques aimed at modelling broader sections of the health system over longer time horizons. Systems modelling and simulation projects of the latter nature often investigate the interconnectedness of healthcare systems, expose feedback loops that reinforce or self-correct behaviour and help to identify strategic intervention points in a system. This is nicely illustrated by the research of Brailsford et al51 in which they construct a macro-level model of the healthcare system in Nottingham (UK) covering primary, secondary and community based services. Their macro-level modelling was used strategically and identified that reducing acute admissions from a small group of long-stay patients was more effective in reducing hospital occupancy than reducing discharge delays for a large group of patients.
Applying modelling in healthcare settings has its challenges, however. Perhaps the most widely bemoaned by systems modellers is a lack of reliable and/or comprehensive data, although thoughtful sensitivity analyses can often identify the circumstances in which one can nonetheless be confident in drawing insights from a model. In any case, data-free or conceptual modelling can facilitate learning about how a system works and may behave under given changes, while the modelling process (eg, bringing diverse stakeholders together to think through a particular issue) may prove to be of intrinsic value to decision-makers in and of itself (eg, as in our case study).
Beyond data issues are the broader challenges of attempting to influence change within the complex, diverse and highly interactive dynamics of health service organisations. For example, systems modellers can encounter difficulties attempting to develop generic and standardised approaches that can be applied across institutions and for different periods of time. In our case study, the problems faced by different rural hospitals for the delivery of rtPA varied sufficiently to necessitate bespoke modelling (albeit using the same methodological approach), illustrating the importance within systems modelling of drawing appropriately from a range of possible approaches to address the specifics of the problem at hand. Indeed, within the systems modelling community, successful implementation is generally thought to result from appropriate problem diagnosis and choice of modelling strategy, coupled with effective engagement with service managers and practitioners.52
Wider learning in the field of quality improvement would suggest that modelling may also be influenced by many other contextual factors (eg, cultural, financial, political and regulatory) known to affect other healthcare interventions. However, the systems modelling literature focuses heavily on technical methodology and there is no natural home for dissemination about the process of implementation and contextual influences, nor accounts of projects that fail53; further research specifically relating to modelling is needed to understand better the factors that influence its effectiveness and generalisability across settings, and where possible take account of these. Additionally, few studies clearly and systematically evaluate the benefits or otherwise of modelling.54 Devising viable methods to isolate and assess the contribution of a modelling process to the overall outcome of an intervention is admittedly challenging, yet scepticism may legitimately remain as to the effectiveness of systems modelling unless methods for evaluating it are strengthened and applied.
A further key barrier to more widespread adoption of modelling is the current lack of capacity within health services. Few staff in most health organisations possess sufficient technical training and capability to conduct the modelling themselves. In addition, there needs to be a greater understanding about how and where modelling tools can support decision-making if policy makers are to become ‘intelligent clients’ and more aware of the benefits of adopting these techniques. Here there is a responsibility for researchers and developers to ensure that outputs are readily accessible and relevant if models are to have an impact in improving health. There are also clear implications in terms of raising awareness, developing skills and promoting training for systems modelling within the health service.
The rarity of effective patient and public involvement in modelling and simulation is also disappointing and an important area for further development.55 Strengthening partnerships with health organisations and service users is a priority for increasing the relevance and application of academic systems modelling.
Notwithstanding these challenges, systems modelling and simulation is arguably playing an increasing role in healthcare. Our case study and a growing number of publications56 ,57 demonstrate that these methods can be, and have been, applied within a number of healthcare areas. However, we are still a long way from systems modelling and simulation contributing widespread impactful change within healthcare. In looking to bridge the ‘implementation gap’, in which the majority of academic systems modelling fails to be applied effectively in practice,7 we may be wise to shift our focus from striving to develop ever better technical solutions to drawing on the knowledge and experience of other disciplines about organisational change and quality improvement in order to learn how to conduct and apply systems modelling more effectively.58
References
Footnotes
Funding Martin Pitt is funded by the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care (CLAHRC) South West Peninsula. Thomas Monks is funded by NIHR CLAHRC Wessex. Sonya Crowe is supported by an Improvement Science Fellowship with The Health Foundation.
Disclaimer The views expressed in this publication are those of the authors and not necessarily those of the National Health Service, the NIHR, The Heath Foundation, or the Department of Health.
Competing interests None declared.
Provenance and peer review Commissioned; internally peer reviewed.