Plans to accelerate innovation in health systems are less than IDEAL
====================================================================

* Paul M Wilson
* Ruth Boaden
* Gillian Harvey

*   Evaluation methodology
*   Health services research
*   Evidence-based medicine

## The drive for innovation

Across health systems there is a drive to roll out innovative models of care that will deliver better value for money and improve the quality of care. Innovation in service delivery has been defined as ‘a novel set of behaviours, routines, and ways of working that are discontinuous from previous practice, directed at improving health outcomes, administrative efficiency, cost effectiveness or users’ experience and that are implemented by planned and coordinated actions’.1 Undertaking this type of innovation at scale is increasingly viewed as crucial to the long-term sustainability of health systems.

In the USA, the Affordable Care Act has provided a legislative framework that promotes innovation in service delivery. Accountable care organisations (ACOs) have emerged from this as innovative payment and delivery models that aim to improve the coordination and quality of care, enhance population health while containing the growth in healthcare costs.2 In England,3 ,4 the Health and Social Care Act has made innovation in the provision of health services a statutory duty and a major initiative to drive innovation at scale is underway.5 Fifty ‘Vanguard’ sites are now starting to act as test beds for multicomponent innovations in service delivery (see table 1), supported by a £200 million transformation fund from National Health Service England.

View this table:
[Table 1](http://qualitysafety.bmj.com/content/25/8/572/T1)

Table 1 
New models of care: the NHS Vanguards5

## The danger of innovation without evaluation

The sheer scale of this initiative serves as a timely reminder that innovation without adequate evaluation can lead to misattribution of effects and worse, the wider adoption of technologies, practices and ways of working without proven benefits over existing alternatives. Health systems, and the NHS in particular, can ill afford compromised decision-making in relation to continuation or wider spread on such a scale. Previous large-scale policy initiatives such as the drive to rapidly implement telehealth technologies despite known uncertainties relating to complexity, costs and benefits6–8 have led to what might be considered inappropriate allocation of finite resources. While there is some recognition that previous roll out and testing of service innovation has been suboptimal4 and that there is a need to establish evidence on ‘what works’,9 this on its own will not be sufficient.

Service innovation at scale is inherently complex and context dependent; what may appear to be successful in one setting may not work the same way in a different context.1 ,10 The inter-relationship between the mix of intervention, process, workforce and system changes and the contextual setting within which they are introduced is likely to generate both intended and unintended consequences. Understanding this complexity and its impact on apparent outcomes necessitates more nuanced approaches to evaluation than are currently used. To date, this level of understanding has been lacking in early evaluations of the spending and performance of ACOs,11 ,12 which suggest modest reductions in total Medicare expenditures and differential improvements in the quality of care. Yet the underlying mechanisms within each ACO are complex and evolving and whether or how these will continue to impact on quality and costs in a sustained and positive way or whether they can be reproduced elsewhere is less clear.13

## What type of evaluation is needed?

Approaches that are context-sensitive and that address the what, why, how, where and for whom of innovation implementation are necessary if we are to be able to recognise and reproduce ‘success’ at the service level. There is always a tension between rigorous evaluation and ‘good enough’ evidence,14 ,15 and there will always be a trade-off between what evidence is desirable and what is possible. Nevertheless, those implementing complex service innovations should be considering what sort of outcome and process evaluation is necessary from the outset; to do otherwise, would be to reduce value and increase waste.16

Prospective pathways for undertaking rigorous outcome and process evaluation of complex and large-scale interventions have been well defined by the UK Medical Research Council (MRC).17–19 The aim of these frameworks is to help researchers and funders recognise and adopt appropriate methods to evaluate the systematic and planned introduction of complex interventions. But there are recognised weaknesses within the existing MRC framework, notably the time and resources needed to operationalise it fully.20 Others have criticised the focus on the intervention rather than on the dynamic influence of the context into which an intervention is introduced or from which it may emerge.21 This lack of acknowledgement that innovation can be unintended as well as intended raises the possibility that other alternative frameworks may be better suited to guiding more nuanced approaches to innovation development and evaluation.

## The IDEAL framework

The IDEAL framework was developed to provide a systematic and incremental pathway to aid the transfer of surgical innovations into practice.22–25 The framework presents a five-phase approach to innovation implementation encompassing: idea, development, exploration, assessment and long-term study (see table 2). Although broadly following the MRC framework in its latter phases, IDEAL better recognises the planned and unplanned nature of innovation and the need to adapt methods to the innovation process rather than doing the opposite.

View this table:
[Table 2](http://qualitysafety.bmj.com/content/25/8/572/T2)

Table 2 
Phases of the IDEAL framework22

IDEAL was developed to confront problems with the evaluation of surgical procedures but as the uncontrolled introduction of innovations is not specific to surgery, there is increasing interest in its potential value beyond this field. The Food and Drug Administration has been working with IDEAL to explore how best to advance the infrastructure and methodology for evaluating medical devices.26 In England, IDEAL is also being promoted as a tool to help commissioners (payers) to consider requests for funding for novel services and make other resource allocation decisions (about interventions not routinely funded) in a more objective and systematic way.27 ,28 IDEAL therefore has the potential to be adapted and applied to a wide range of situations in which complex innovations are in need of rigorous development and evaluation.

With new models of care some interventions, processes and ways of working are likely to be planned from the outset but others may emerge by accident or occur through the unintended consequences of planned innovation. A focus on explaining early intentions and describing what actually happens over time is therefore key. The idea and development phases of the IDEAL framework recognises this and advocates the iterative development and evaluation of innovations to define and test whether it is likely to succeed in a particular setting and allow for adaptation, refinement and system integration before more widespread and rigorous testing is pursued.23

With this initial focus on iterative development, IDEAL also recognises that rapid cycle evaluative approaches that traditionally feature in quality improvement initiatives may be most appropriate in these early phases.29 Common to these approaches is the generation of real-time data driven learning and decision-making. This makes them ideally suited to addressing questions such as ‘Should we do this again?’ and ‘Should we pursue this further?’ Approaches such as plan-do-study-act and statistical process control also have the benefit of being widely used as part of efforts to improve the quality and safety of care across health systems. But if iterative evaluation of this type is to be pursued, well documented shortcomings in planning, execution, analysis and reporting of these methods have to be avoided or their effects minimised.30–32

And this is where one of the key strengths of IDEAL can best be harnessed; that of transparency through registration and reporting. IDEAL recommends that all surgical innovations are prospectively registered, and that it is unethical not to do so.22 In England, infrastructure already exists to facilitate the prospective registration of innovations although the extent to which it is used is unclear, and it may not be intended for service innovations.33 For example, at present it is unclear whether a comprehensive register exists of areas of England where ‘integrated care’ in various forms and at different scales is being attempted. Prospective registration will aid transparency by providing a permanent record of how an innovation was intended to be introduced, whether it was adapted and how it was ultimately implemented in a given context.

IDEAL also recommends that the early development and evaluation of any innovation should be adequately reported (regardless of ‘success’ or ‘failure’). Insights and contextual information which could inform wider spread and learning should be systematically captured in a standardised format. There are now a vast number of reporting frameworks available but given the focus on reporting initial innovation development, refinement and implementation, the SQUIRE2 guidelines may be the most appropriate.34 Although designed for quality improvement, SQUIRE2 recognises that innovation can be complex, multidimensional and is inherently context dependent. With a focus on explaining issues such as ‘Why did you start?’, ‘What did you do?’, ‘What did you find?’ and ‘What does it mean?’, SQUIRE2 encourages more nuanced reporting of what happened over time and reflection on the meaning of the outcomes and events that were observed.

Beyond the idea and development phases, IDEAL recommends that plans for wider spread should be accompanied by robust and independent comparative evaluation. The small-scale iterative evaluations of the early phases can only ever give tentative indications of benefit and can be prone to misinterpretation and bias; as such they should only ever represent a starting point. Further corroboration, explanation and evaluation should be sought along the IDEAL pathway and prospective comparative quasi-experimental designs that will generate economic, outcome and process data should be the preferred method of evaluation. Here IDEAL recognises and adopts the methods advocated by the MRC for evaluating complex interventions.17–19

## Adapting IDEAL

As IDEAL was originally devised for surgical innovations it does have shortcomings when considered in relation to other complex and emergent innovations. One such shortcoming is the failure to advocate the use of clearly articulated theory to explain how and why an innovation would be expected to deliver benefits over existing practices. This may be due in part to the fact that in surgery the theory is usually implicit in the innovation itself. But with increasing complexity (surgical or otherwise) theory becomes an essential lens through which we can predict, identify and describe the key features that will influence the implementation of any innovation.35 ,36 Particularly important is ensuring that there is clarity about what the innovation is together with clarity over the wider system, process and contextual features that will be need to be addressed if its causal mechanism is to function as planned. Encouragingly, use of logic models is being promoted and constructed for each Vanguard site. But this clarity over purpose should also inform the selection of evaluation methods that reflect an innovation's phase of development.37 Applying theory in the idea and development phases should help to focus attention on those evaluation strategies that are best equipped to capture innovation adaptation and refinement over time. Doing so will also offer a more efficient and meaningful method to generalise and predict the potential for successful replication in other settings.

Another major shortcoming with IDEAL is consideration of cost. Although IDEAL incorporates cost effectiveness in its latter phases, it is absent from the idea and development phases. Yet innovation (large or small) is not without cost or consequences. There will be opportunity costs as innovation competes with other health activity and services for finite resources. Determining whether the expected gains in health benefits from innovation implementation will exceed those lost as other activities are displaced is therefore crucial.38 This tension has been perfectly illustrated by a recent analysis of the impact of making routine hospital services available 6 days a week in England; the costs of the proposed solution reportedly outweigh the health benefits that are likely to be realised.39 Early assessment therefore may help to identify and eliminate cost-ineffective innovations. At the very least, innovators should be encouraged to systematically capture and report resource use and costs associated with early stage implementation from a clearly stated perspective—especially important if judgements are to be made on wider spread.

## Conclusions

An adapted IDEAL framework offers a pathway for prospective longitudinal evaluation of the Vanguard test bed sites and similar complex and emergent innovations. Adopting a common analytical framework will encourage iterative development, reporting and evaluation of innovations from the outset and before more widespread and rigorous testing is pursued.

## Footnotes

*   Contributors This paper is based on a discussion paper produced by PMW, RB and GH for the MRC/NIHR Research Methodology Panel. PMW wrote the first draft of this paper and RB and GH commented on this and subsequent drafts. All authors approved the final version and PMW is the guarantor.

*   Funding PMW, RB and GH all receive funding from the he NIHR Collaboration for Leadership in Applied Health Research and Care Greater Manchester. The views expressed in this article are those of the authors and not necessarily those of the NHS, NIHR or the Department of Health.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)

## References

1.  Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82:581–629. [doi:10.1111/j.0887-378X.2004.00325.x](http://dx.doi.org/10.1111/j.0887-378X.2004.00325.x)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.0887-378X.2004.00325.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15595944&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000225794300001&link_type=ISI) 

2.  Fisher ES, Shortell SM. Accountable care organizations: accountable for what, to whom, and how. JAMA 2010;304:1715–6. [doi:10.1001/jama.2010.1513](http://dx.doi.org/10.1001/jama.2010.1513)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2010.1513&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20959584&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000283129700028&link_type=ISI) 

3.  Department of Health. Innovation health and wealth: accelerating adoption and diffusion in the NHS. York: University of York, 2011.
    
    

4.  NHS England, Public Health England, Health Education England, Monitor, Care Quality Commission, NHS Trust Development Authority *Five year forward view*. London: NHS England, 2014.
    
    

5.  England N. New care models—vanguard sites. Secondary new care models—vanguard sites, 2015. [http://www.england.nhs.uk/ourwork/futurenhs/5yfv-ch3/new-care-models/](http://www.england.nhs.uk/ourwork/futurenhs/5yfv-ch3/new-care-models/)
    
    

6.  Centre for Reviews and Dissemination. Telehealth for patients with long term conditions. CRD evidence briefing. York: University of York, 2013.
    
    

7.  Henderson C, Knapp M, Fernández JL, et al. Cost-effectiveness of telecare for people with social care needs: the Whole Systems Demonstrator cluster randomised trial. Age Ageing 2014;43:794–800. [doi:10.1093/ageing/afu067](http://dx.doi.org/10.1093/ageing/afu067)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=ageing&resid=43/6/794&atom=%2Fqhc%2F25%2F8%2F572.atom) 

8.  Steventon A, Bardsley M, Billings J, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ 2012;344:e3874. [doi:10.1136/bmj.e3874](http://dx.doi.org/10.1136/bmj.e3874)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=bmj&resid=344/jun21_3/e3874&atom=%2Fqhc%2F25%2F8%2F572.atom) 

9.  NHS England, The AHSN Network, Government Office for Science, Office of Life Sciences. *Real world testing of ‘combinatorial innovation’ A global invitation to innovators. NHS England*, Gateway reference: 03268, March, 2015.
    
    

10. Bate P, Robert G, Fulop N, et al. Perspectives on context. A selection of essays considering the role of context in successful quality improvement. London: Health Foundation, 2014.
    
    

11. Nyweide DJ, Lee W, Cuerdon TT, et al. Association of Pioneer Accountable Care Organizations vs traditional Medicare fee for service with spending, utilization, and patient experience. JAMA 2015;313:2152–61. [doi:10.1001/jama.2015.4930](http://dx.doi.org/10.1001/jama.2015.4930)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2015.4930&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25938875&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 

12. McWilliams JM, Chernew ME, Landon BE, et al. Performance differences in year 1 of pioneer accountable care organizations. N Engl J Med 2015;372:1927–36. [doi:10.1056/NEJMsa1414929](http://dx.doi.org/10.1056/NEJMsa1414929)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMsa1414929&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25875195&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 

13. Voigt J. Medicare's mission to change how health care is paid for and delivered: a cloud with a silver lining or just a dark cloud? Appl Health Econ Health Policy 2015;13:433–5. [doi:10.1007/s40258-015-0186-1](http://dx.doi.org/10.1007/s40258-015-0186-1)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s40258-015-0186-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26179939&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 

14. Auerbach AD, Landefeld CS, Shojania KG. The tension between needing to improve care and knowing how to do it. N Engl J Med 2007;357:608–13. [doi:10.1056/NEJMsb070738](http://dx.doi.org/10.1056/NEJMsb070738)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMsb070738&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=17687138&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000248584900016&link_type=ISI) 

15. Davidoff F. Systems of service: reflections on the moral foundations of improvement. BMJ Qual Saf 2011;20(Suppl 1):i5–10. [doi:10.1136/bmjqs.2010.046177](http://dx.doi.org/10.1136/bmjqs.2010.046177)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=qhc&resid=20/Suppl_1/i5&atom=%2Fqhc%2F25%2F8%2F572.atom) 

16. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014;383:166–75. [doi:10.1016/S0140-6736(13)62227-8](http://dx.doi.org/10.1016/S0140-6736(13)62227-8)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(13)62227-8&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24411645&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000329579600032&link_type=ISI) 

17. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655. [doi:10.1136/bmj.a1655](http://dx.doi.org/10.1136/bmj.a1655)
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=FULL&journalCode=bmj&resid=337/sep29_1/a1655&atom=%2Fqhc%2F25%2F8%2F572.atom) 

18. Craig P, Cooper C, Gunnell D, et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Commun Health 2012;66:1182–6. [doi:10.1136/jech-2011-200375](http://dx.doi.org/10.1136/jech-2011-200375)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=jech&resid=66/12/1182&atom=%2Fqhc%2F25%2F8%2F572.atom) 

19. Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015;350:h1258. [doi:10.1136/bmj.h1258](http://dx.doi.org/10.1136/bmj.h1258)
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=FULL&journalCode=bmj&resid=350/mar19_6/h1258&atom=%2Fqhc%2F25%2F8%2F572.atom) 

20. Lakshman R, Griffin S, Hardeman W, et al. Using the Medical Research Council framework for the development and evaluation of complex interventions in a theory-based infant feeding intervention to prevent childhood obesity: the baby milk intervention and trial. J Obes 2014;2014:646504. [doi:10.1155/2014/646504](http://dx.doi.org/10.1155/2014/646504)
    
    

21. Hawe P, Shiell A, Riley T, et al. Theorising interventions as events in systems. Am J Commun Psychol 2009;43:267–76. [doi:10.1007/s10464-009-9229-9](http://dx.doi.org/10.1007/s10464-009-9229-9)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s10464-009-9229-9&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19390961&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000265685500008&link_type=ISI) 

22. McCulloch P, Altman DG, Campbell WB, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet 2009;374:1105–12. [doi:10.1016/S0140-6736(09)61116-8](http://dx.doi.org/10.1016/S0140-6736(09)61116-8)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(09)61116-8&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19782876&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000270370900033&link_type=ISI) 

23. McCulloch P, Cook JA, Altman DG, et al. IDEAL framework for surgical innovation 1: the idea and development stages. BMJ 2013;346:f3012. [doi:10.1136/bmj.f3012](http://dx.doi.org/10.1136/bmj.f3012)
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=FULL&journalCode=bmj&resid=346/jun18_3/f3012&atom=%2Fqhc%2F25%2F8%2F572.atom) 

24. Ergina PL, Barkun JS, McCulloch P, et al. IDEAL framework for surgical innovation 2: observational studies in the exploration and assessment stages. BMJ 2013;346:f3011. [doi:10.1136/bmj.f3011](http://dx.doi.org/10.1136/bmj.f3011)
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=FULL&journalCode=bmj&resid=346/jun18_3/f3011&atom=%2Fqhc%2F25%2F8%2F572.atom) 

25. Cook JA, McCulloch P, Blazeby JM, et al. IDEAL framework for surgical innovation 3: randomised controlled trials in the assessment stage and evaluations in the long term study stage. BMJ 2013;346:f2820. [doi:10.1136/bmj.f2820](http://dx.doi.org/10.1136/bmj.f2820)
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=FULL&journalCode=bmj&resid=346/jun18_3/f2820&atom=%2Fqhc%2F25%2F8%2F572.atom) 

26. Administration UFaD. Bridging the IDEAL and TPLC Approaches for Evidence Development for Surgical Medical Devices and Procedures. Transcript for Public Workshop December 2, 2011. Secondary Bridging the IDEAL and TPLC Approaches for Evidence Development for Surgical Medical Devices and Procedures. Transcript for Public Workshop December 2, 2011. 03/02/2015 2011. [http://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm288553.htm](http://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm288553.htm)
    
    

27. Austin D. *Using IDEAL when dealing with proposed service developments*. RightCare, 2015.
    
    

28. Austin D. *Using IDEAL when dealing with individual funding requests (IFRs)*. RightCare, 2015.
    
    

29. Portela MC, Pronovost PJ, Woodcock T, et al. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf 2015;24:325–36. [doi:10.1136/bmjqs-2014-003620](http://dx.doi.org/10.1136/bmjqs-2014-003620)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=qhc&resid=24/5/325&atom=%2Fqhc%2F25%2F8%2F572.atom) 

30. Koetsier A, van der Veer SN, Jager KJ, et al. Control charts in healthcare quality improvement. A systematic review on adherence to methodological criteria. Methods Inf Med 2012;51:189–98. [doi:10.3414/ME11-01-0055](http://dx.doi.org/10.3414/ME11-01-0055)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.3414/ME11-01-0055&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22476327&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000304500400002&link_type=ISI) 

31. Taylor MJ, McNicholas C, Nicolay C, et al. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf 2014;23:290–8. [doi:10.1136/bmjqs-2013-001862](http://dx.doi.org/10.1136/bmjqs-2013-001862)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=qhc&resid=23/4/290&atom=%2Fqhc%2F25%2F8%2F572.atom) 

32. Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007;16:387–99. [doi:10.1136/qshc.2006.022194](http://dx.doi.org/10.1136/qshc.2006.022194)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=qhc&resid=16/5/387&atom=%2Fqhc%2F25%2F8%2F572.atom) 

33. NHS England. Innovation Exchange. Secondary Innovation Exchange, 2014. [https://nhs-ihw-colab.induct.no/maya/companymain.aspx](https://nhs-ihw-colab.induct.no/maya/companymain.aspx)
    
    

34. Ogrinc G, Davies L, Goodman D, et al. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2015. Published Online First 14 Sept 2015. [doi: 10.1136/bmjqs-2015-004411](http://dx.doi.org/10.1136/bmjqs-2015-004411)[doi:10.1136/bmjqs-2015-004411](http://dx.doi.org/10.1136/bmjqs-2015-004411)
    
    

35. Davidoff F, Dixon-Woods M, Leviton L, et al. Demystifying theory and its use in improvement. BMJ Qual Saf 2015;24:228–38. [doi:10.1136/bmjqs-2014-003627](http://dx.doi.org/10.1136/bmjqs-2014-003627)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=qhc&resid=24/3/228&atom=%2Fqhc%2F25%2F8%2F572.atom) 

36. Walshe K. Pseudoinnovation: the development and spread of healthcare quality improvement methodologies. Int J Qual Health Care 2009;21:153–9. [doi:10.1093/intqhc/mzp012](http://dx.doi.org/10.1093/intqhc/mzp012)
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink?linkType=ABST&journalCode=intqhc&resid=21/3/153&atom=%2Fqhc%2F25%2F8%2F572.atom) 

37. Parry GJ, Carson-Stevens A, Luff DF, et al. Recommendations for evaluation of health care improvement initiatives. Acad Pediatr 2013;13(Suppl 6):S23–30. [doi:10.1016/j.acap.2013.04.007](http://dx.doi.org/10.1016/j.acap.2013.04.007)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.acap.2013.04.007&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24268081&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom) 

38. Claxton K, Martin S, Soares M, et al. Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold. Health Technol Assess 2015;19:1–503, v–vi. [doi:10.3310/hta19140](http://dx.doi.org/10.3310/hta19140)
    
    

39. Meacock R, Doran T, Sutton M, et al. What are the costs and benefits of providing comprehensive seven-day services for emergency hospital admissions? Health Econ 2015;24:907–12. [doi:10.1002/hec.3207](http://dx.doi.org/10.1002/hec.3207)
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1002/hec.3207&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26010243&link_type=MED&atom=%2Fqhc%2F25%2F8%2F572.atom)