Context in uncontrolled experimental trials and PDCA tests

Collecting and reporting data about context is an added burden to research, and may not be possible for simple before/after Plan-Do-Study-Act (PDSA) practitioner reports, although two case report repositories do provide some of this ‘background’.11 12 Any such data collection in these designs needs to be focused on documenting the aspects which are most likely to affect the outcomes, and analysis focused on assessing their relative influence. Reviews of research to discover which aspects have been reported previously would help researchers and practitioners to select which aspects to collect data about and how to collect the data. For example, effectiveness studies of health information technology have found aspects of the host organisation's staffing, size, previous experience and financing to be important.9

Context in ‘naturalistic studies’

“Naturalistic studies” are a broad category of designs for studying QI change in natural settings and which are more suited to developing theory about context. Some designs describe implementation only (eg, case study),13 14 but some may also assess intermediate outcomes (eg, some types of programme evaluation).15 16 Such designs are more often used for studies of large-scale programmes, policies or regulatory changes than for studying smaller projects, although many such large-scale changes include local projects. Two recent examples are Benn et al's 2009 mixed method study of a UK patient safety initiative17 and Greenhalgh et al's 2009 study of large-scale complex improvement programme in London, UK.18

One approach to develop theory of context is to collect a cross section of informants' views about aspects of context which they suggest were important at different stages of implementation. Validity can be enhanced by selecting a sample of informants, from different organisational levels and perspectives, who are knowledgeable about the change and may be able to cite evidence to support their insights. Observers' views can then be crosschecked against documentary or other data to assess the influence of context factors repeatedly mentioned, thus building inductively an understanding of context influences.19 However, informants may not be aware of some influences, and their ‘observer theories’ need to be supplemented with scientific theory, which may direct attention to other data to test hypotheses about possible influences. This approach starts with a model or theory, preferably based on previous research into similar QI changes, which suggested aspects of context which were important. This can then be used to plan data collection about these aspects of context or specify these as hypotheses to be tested.

One of the challenges of this type of research is to capture changes over time and the dynamics of intervention–condition interactions. To do so requires documenting how any influencing conditions change over the period of implementation (eg, a senior manager who was supportive of the change leaves or there are sudden cuts to budget). There are also challenges in understanding exactly how these changes influence the QI implementation. Generalisation in these types of studies could be helped by better understanding exactly how a condition influence interacts with parts of the intervention. For example, why what a leader does affects how personnel respond to training about use of a checklist. This approach moves towards examining how an idea is adapted and evolves in a setting through ‘enabled adaptive change’. In some action- or collaborative-research studies, this may involve the researchers contributing to the change, for example, by giving feedback to implementers.

These types of studies can be enhanced by specifying the ‘change theory’ of the implementers—the assumptions about which actions lead to which results through which steps20—or by defining the researchers' ‘programme theory’ before and after data gathering.15 These ‘theories’ could also include ideas about which aspects of context help and hinder the implementation. It is also possible that pragmatic testing, using PDSA cycles21 would be enhanced by implementers stating their assumptions about the conditions they need and the steps through which changes might affect outcomes. By making explicit their assumptions (theories ‘T’) before testing, and revising these after testing (‘T-PDSA-T’), improvers could learn not just whether a change affected outcomes, but why.

Learn from other disciplines

A number of disciplines have developed methods to study ‘implementation’, including public health22, sociology,23 educational studies,24 business studies,25 programme evaluation26 and implementation and innovation science.27 28 Some approaches draw on critical realist philosophy which emphasises the study of mechanisms which trigger effects under certain circumstances rather than using other types of cause–effect understanding.29 These bodies of knowledge provide methods and frameworks for the description of intervention actions, and also seek to assess intermediate changes to behaviour or organisation, or even final outcomes using a theory about how the actions taken then influence outcomes. Such methods are being applied to understand improvements in healthcare. However, research including context factors in these ‘causal’ theories is at the early stages.

More specific assessments of context are required

Research would be more useful and cumulative if it moved beyond general statements about variations in implementation and outcomes being due to ‘leadership’ or ‘culture’. More specific data are needed about, for example, which aspects of leadership by which leaders are important for which QI changes and how what leaders do has this influence. In the case of culture as a context factor, there are different data gathering instruments which can be used.30 Helfrich et al describe a method to assess ‘organisational readiness to change’, which may be an important context factor.31 Some context factors are difficult to operationalise as measures, but the discipline of seeking measures or defining concepts more precisely is necessary for data gathering and for understanding the influence of these factors.

There are frameworks and research from different knowledge domains which can provide a starting point for researchers, ideally drawing on previous research into changes similar to the particular QI being studied. These frameworks often distinguish context influences external to the implementing organisation, and those internal to it,28 or separate context influences which originate at different organisational levels.32 French et al provide a useful general overview of frameworks from different knowledge domains,33 and Rycroft-Malone et al describe a study protocol for studying context in one study.34 Context factors found to affect implementation of evidence-based practices are described by Dopson and Fitzgerald35 and Rycroft-Malone et al.36 Other relevant frameworks include those described by Pettigrew et al,37 Kitson et al38 and Lukas et al.39

Categorising QI changes according to the impact of context

It is possible that the implementation of some types of QI depends more on context than others.7 Furthermore, similar context factors are important for certain groupings of QI (eg, the groupings proposed by Shojania et al40). For example, equipment or automation changes may form one grouping of improvements, with computer-based changes forming a sub-grouping. Taxonomies of improvement interventions and context influences are underdeveloped and the lack of a common language is hindering scientific progress. At present, researchers have little guidance from previous research about which aspects of context to document, and there is possibly an assumption that all types of QI are equally affected by the same types of context influences. Theoretical research is needed to produce groupings of QI interventions according to which aspects of context are important for their implementation and which are different from other groupings of QI. The synthesis of implementation frameworks by Damschroder et al might provide one starting point.41 Such a categorisation could then be the basis for specifying the context elements for reporting of uncontrolled studies (eg, the SQUIRE reporting guidance).42

Interventions, multiple component changes or system changes?

QIs have traditionally been conceptualised as interventions—discrete changes separated from their surroundings—in order to assess whether they cause other changes in outcome variables such as patient outcomes. If the change is an improvement, the assumption is they can be repeated elsewhere to cause the same outcome changes. Some limitations of this way of understanding QI have been noted when applied to complex interventions such as QI ‘bundles’.10 If an intervention is separated conceptually from its surroundings, then one research agenda is to explore how the intervention changes context, as well as vice versa.

Another research agenda is not to conceptualise such a sharp separation, and to view improvement less as a specific change but as an interdependent set of actions which result in many types of changes, which in turn may result in better patient outcomes. Including context in understanding implementation and in improvement theory can advance improvement science and practice. It allows exploration of whether and how aligned changes at different levels may result, through complex influences, in better outcomes, and how these can be sustained. It moves from cause–effect understanding to conditional attribution, which allows qualified generalisations by discovering which conditions were necessary for the improvement to be carried out and its effects. This in turn allows decision makers to assess better likely results locally and how to adapt the change.