Article Text
PDF
Statistics from Altmetric.com
Background
The WHO Surgical Safety Checklist (SSC) was developed in 2008 as part of the Safe Surgery Saves Lives campaign.1 Broadly mandated and put into practice in hospitals around the world, the SSC has been the focus of 8 years of extensive research. Initial studies reported positive outcomes on morbidity and mortality.2 3 Other studies have reported more limited impacts, for example,4 still others have reported no impact at all5 6 or questioned the effectiveness of SSC.7 Such results have prompted calls for the reconsideration of policies mandating the SSC as an organisational safety practice.8
Much is at stake here. The role of team communication in care quality is incontrovertible9 10; therefore, decisions to pursue or abandon the SSC are consequential and should be made by drawing from a robust knowledge base. With the significant difficulties associated with progressively improving patient safety, the decision to abandon established and promising initiatives should be taken only after careful consideration of what has been achieved and a systematic assessment of what remains to be done.11 A multitude of studies, commentaries and reviews have been published since the introduction of the SSC, making this a good moment to pause and ask: how has knowledge mobilised and accumulated across high impact papers in the SSC literature?
Methods
Literature search and selection
Studies where the SSC was the central concern, available in English and published between 2009 and 2016 were included in the review (figure 1). Our search strategy focused on the term ‘surgical safety checklist’ and used the h-index* both from Web of Science and Scopus to help us identify highly impactful articles between 2009 and 2016. The first Web of Science search took place on 19 February 2016, and the h-index value was 25, where 25 articles were cited 25 times or more. Of the 25 articles from the h-25, one was rejected as not directly relevant to the SSC. Further h-index searches were conducted in Web of Science and Scopus on 22 June, 27 June and 4 July 2016 to consolidate the top 25 list after discarding duplicates. The h-25 papers were then described and analysed (table 1).
Literature selection process. SSC, Surgical Safety Checklist.
h-25 Surgical Safety Checklist (SSC) articles
Analytical stages
The data analysis process was iterative and the categories described below emerged from a process of reading the papers and discussing patterns among the research team in a series of meetings. Once we had identified and defined the main thematic categories, one researcher (BM) categorised all 25 papers according to these definitions. These categorisations were reviewed by at least one other researcher (BBN, SC or LAL), and difficult or discrepant examples were discussed in group meetings until consensus was achieved and the three categories were sufficiently refined to account for all papers.
The first analytical categorisation asked the question: how does the literature conceptualise the SSC? We analysed the 25 papers for whether they conceptualised the checklist as a thing, as a process or as both. Specific verbal and visual markers supported the categorisation: SSC as a thing was signalled by images of the checklist, references to checklist items and use of terms such as ‘tool’ or ‘instrument’. SSC as a process was signalled by verbs emphasising what the checklist does, for example, coordinate perspectives and references to SSC as part of a broader set of practices such as team communication.
The second analytical categorisation asked the question: how are the surgical safety ‘problem’ and the SSC ‘solution’ characterised within individual papers? This analysis focused on each paper’s introduction and discussion sections. We categorised the ‘problem/SSC solution’ characterisation in one of two ways. Straightforward categorisation emphased the obviousness of the problem and simplicity of solution, while nuanced characterisation emphasising the complexity of the problem and multifactorial or ambiguous nature of solution.
The third analytical categorisation asked the question: how is SSC knowledge taken up from one paper to another as studies accumulate over this body of work? This analysis involved an intertextual analysis of the introduction and discussion sections of the papers. Specifically, we tracked both intertextual references (citations of earlier work) and recontextualisations (representations of earlier work) to identify how existing knowledge is characterised and how new knowledge is placed in relation to it.
Results
The articles analysed (n=25) comprised empirical studies (15 quantitative and 2 qualitative), commentaries (2), systematic reviews (5) and an editorial (1). The articles reported studies from 14 countries.
Conceptualisation of the SSC
Most articles used both conceptualisations—checklist as thing and checklist as process—but emphasised one: 12 papers predominantly conceptualised the checklist as a thing and 8 as a process. In five papers, it was not possible to discern greater emphasis on one conceptualisation or the other.
Papers that conceptualised the checklist as a thing commonly referred to it as a ‘tool’,1 12 and many included a visual representation of the checklist.2 13–15 When the checklist was conceptualised as a process, the emphasis was on practice rather than tool; for example, Carney et al 16 described ‘checklist driven briefings and debriefings’ (p727).
Articles conceptualising the SSC as both thing and process followed two patterns. In one, the checklist as thing orientation was presented for the purpose of arguing against it. For example, Weiser et al 17 problematised the ‘apparent simplicity’ of the checklist as ‘a formal list used to identify, schedule, compare or verify a group of elements’ (quoting Federal Aviation Administration, no year) and then described that their ‘goal was to create a tool that supported clinical practice without attempting to substitute a rigid algorithm for professional judgement’ (p365). In the other combined pattern, articles began with a thing orientation which evolved into a process orientation as the paper unfolded. Illustrating this pattern, Borchard et al 18 presented checklists as ‘effective and economic tools’ in their introduction(p925) but their discussion problematised this conceptualisation, acknowledging the need to integrate the checklist into the existing hospital system.
Characterisation of the problem and SSC solution within papers
Our analysis of how individual papers characterised the surgical safety ‘problem’ and the SSC ‘solution’ revealed three different patterns: 3 employed straightforward characterisation in both introduction and discussion, 9 used straightforward characterisation in the introduction with a more nuanced characterisation in the discussion and 13 used nuanced characterisation in both introduction and discussion. Kwok et al 19 provide an illustrative example of the first pattern. Their introductory hypothesis that ‘implementation of a hospital-wide checklist program will significantly reduce postoperative hazards and complications…’ (Kwork et al, p633)19 takes a straightforward stance on the SSC’s likelihood of success, with no foreshadowing of lurking complications. Their discussion maintains this stance, with assertions such as ‘It is unlikely that a case’s observed status motivated providers to change their behaviour’ (Kwok et al, p637)19 which simplify the potentially complex issue of observer effect.
For the pattern of straightforward and nuanced characterisation of problem/solution, there were some examples where straightforward introductory characterisation was ‘unpacked’ in the discussion. In the study by Haynes et al,2 the problem of surgical safety was introduced straightforwardly in a number of claims including this one: ‘at least half of all surgical complications are avoidable’ (p492). The discussion of this paper, however, considered a number of nuances not signalled in the introduction, such as ‘the exact mechanism of improvement is less clear and most likely multifactorial’ (Haynes et al, p497).2 Similarly illustrating the straightforward intro/nuanced discussion pattern of characterisation, Bliss et al introduced the SSC as ‘an inexpensive tool’ (p766),20 but discussed a number of nuances, including ‘the need for focus on deficiencies in communication…as well as disruptive behaviour’ (p722).20
An example of the third pattern of problem/solution characterisation can be found in the study by Urbach et al.5 Its introduction characterised both problem and solution in a nuanced manner with statements such as ‘the effect of the mandatory checklist implementation is unclear’ and ‘implementation of the surgical safety checklists is not uniform’ (Urbach et al, p1030).5 The discussion continued this nuanced characterisation; in fact, the authors argued that the existence of straightforward, simple characterisations of the SSC may derive from the fact that ‘studies showing improvements in outcomes after checklist implementation are more likely to be published than are negative studies (publication bias)’ (Urbach et al, p1037).5 Similarly, Fourcade et al’s paper14 offered a nuanced/nuanced characterisation: their introduction suggested that ‘…questions have arisen about their [checklists’] ease of introduction into workflow patterns and their true impact on safety’ (p1) and their discussion acknowledged that ‘A checklist is often put across as a tool to enhance communication and as a reminder in stressful circumstances but, like other operational tools, it impacts on the organisation of work’ (p6).
Knowledge mobilisation across the papers
Intertextual references to previous large SSC trials were common in the 25 analysed papers. However, the pattern of recontextualisation of knowledge from previous work was recurrently one of simplification, particularly in introduction sections. Haynes et al 2 is a case in point. The article by Haynes et al 2 was by far the most cited article in our sample, with citations numbering 1545 in Web of Science, 1979 in Scopus and ranking number 1 in Web of Science and Scopus. All empirical papers in our h-25 list referenced Haynes et al 2 most in the first few sentences of their introduction. In almost all cases, these introductory references recontextualised Hayne’s et al’s2 insights into a straightforward knowledge claim such as ‘Mortality significantly declined from 1.5% before the SSC to 0.8% afterwards and inpatient complications fell from 11% to 7% (Mahajan, p163)15 or ‘Large benefits have been reported following implementation of the checklist, including reductions in adverse events’ (Pickering et al, p1664).21 We characterised such recontextualisations as simplifications because they do not represent Haynes et al’s2 extended discussion of the complex, multifactorial nature of SSC in their interpretation of results.
Another pattern of recontextualisation was visible in discussion sections. All of the nuanced discussions acknowledged the complexities associated with interpreting SSC implementation and its impact. For instance, papers discussed that ‘the exact mechanism of improvement is unclear’2 or ‘organisational changes are needed while implementing a surgical checklist’.14 However, in half of cases, these discussions contained no intertextual references to previous work. For example, ‘Improved outcomes after implementation may be explained by a number of mechanisms’ (de Vries et al, p1933)22 and ‘Checklist adoption in isolation fails to maximise the potential impact’ (Bliss et al, p722)20 are both concepts that were raised in earlier papers such as Haynes et al 2 but neither is accompanied by a reference to earlier, similar insights.
For those papers that did reference previous insights regarding the complexities of SSC intervention, most did so perfunctorily. That is, a citation was provided, but the paper neglected to engage with or advance those insights. For instance, papers acknowledged what others have indicated previously, that ‘simply enforcing the use of checklists will likely not suffice when substantial improvements in safety culture are desired’.4 However, the discussion did not elaborate beyond the need for future research.
Discussion
Our results suggest that knowledge is not effectively mobilising across papers in the SSC literature. Given that later papers have cited earlier ones in the sample, this knowledge stasis is not due to lack of awareness of others’ work. Nor do we believe that the SSC literature suffers from biased study reporting. The problem is more subtle than that. The problem is one of inappropriate simplification. Many papers in the sample exhibit a pattern of simplifying the story of SSC in their introductions by recontextualising the findings from earlier work, even when that work may itself have discussed a more nuanced characterisation of SSC. In a significant subset of papers, this simplification persists into the discussion section such that insights about the social complexity of SSC are presented either as new or as repetition without development or elaboration. This produces the sense across this body of high impact literature that knowledge is stuttering, or even standing still, as studies repeatedly discover the social complexity of SSC.
We offer two possible explanations for this pattern of recontextualisation. The first relates to the genre of scientific publications in clinical journals. Medicine tends to publish short, multiauthored research papers (in average nine pages) as compared with other disciplines that favour long, single-authored articles (eg, computer science with 27 pages or law with 43 pages).23 Thus, the common generic structure of empirical papers in medical journals is for introductions to be short and to the point, and for discussions to draw out nuance and complexity, which may explain why existing knowledge of SSC complexity gets simplified in paper introductions. Genre does not, however, explain the pattern of discussion sections drawing out nuanced insights but either not noting that those insights resonated with previous work or merely noting the resonance without further discussion.
A further explanation may lie in how these high impact papers approach the SSC phenomenon. Most assume that the SSC is an object that can be inserted, controlled and measured, that the SSC is independent of its uptake by people in contexts and that SSC-based discussions can be explained in terms of universal laws, including cause and effect.24 Reflecting such positivist assumptions, most of the highly cited SSC papers are deductive, concerned with the control and prediction, grounded in hypothesis testing and experimental design, with a goal of producing generalisable knowledge.24 Controlled experiments of SSC outcomes approach the SSC as an objectively real and stable entity and are therefore not designed to explore the social complexities, ambiguities and inconsistencies associated with SSC implementation, and are not oriented to even consider these possibilities. This problem is not unique to SSC research; there is a growing recognition of the limits of positivist research for evaluating and understanding complex health services interventions.25 26
We would contend that this positivist orientation largely explains the failure, across this body of high impact SSC research, to accumulate increasingly refined insight into the finding that SSC implementation is complex, inconsistent and troublesome. When investigators are oriented towards the SSC as a stable object, they design studies reflective of that orientation, and therefore they have no basis for making sense of findings that suggest otherwise. Thus, the pattern of recontextulisation in which introductions assert that the SSC is a clear solution to a well-defined problem, and discussions acknowledge that it is more complicated than we thought, but we can’t say much more because our study was designed to measure the impact of a clear solution on a defined problem. While a small body of social science research explores the complexities and inconsistencies associated with altering the communication routines of surgical teams, much of this work is very recent27–29 and earlier publications30 31 do not appear to have influenced the highly cited SSC publications we reviewed. Nonetheless, research using a social science lens offers valuable lessons. For example, Vats et al identified detrimental effects on workplace culture when SSC implementation strategies are not carefully thought out.32 Such insight suggests that using checklists requires a scientifically grounded understanding of how organisations and people work,33 34 including whether there is explicit awareness of the multifactorial underlying mechanisms of safety, mainly around the ways in which organisational values drive communication and teamwork practices.28 35 And given that organisational values are made explicit via stories, Hilligoss and Moffatt-Bruce remind us of the affordances of narrative methods to explore the complexities of checklist-driven practices.27 One such complexity pertains to the relationship between team mobilisation practices and participation in safety checks. Using an innovative approach from sociolinguistics, conversation analysis and social semiotics,36 Korkiakangas recently found that mobilisation practices were most affected by the timing of the checklist (eg, after the patient was positioned or during anaesthetist activities), the presence and distribution of staff (eg, getting staff in the same place) and the kinds of instigation practices (eg, loud and clear ‘Inclusive calls’ that address everyone in the room vs ‘exclusive calls’ vs ‘no calls’).37 These findings continue to challenge the assumption that improvements in team interactions should automatically follow the introduction of a checklist within an organisation. Expanding the view to the global context has also revealed the complex interrelation between checklist procedures, context, culture and behavioural changes. An ethnographic study by Aveling et al in hospitals in low-income and high-income countries showed that local policies, institutional support and cultural views around transparency and accountability that are encoded within checklist practices require careful scrutiny.38 Until the influence of cultural and economic contexts is better understood, it will remain a challenge to straightforwardly measure the potential of the checklist for patient safety. While we are not advocating against controlled experiments of SSC, we do call for better uptake of existing social science approaches and knowledge to advance our understanding of the complex sociological dimensions of SSC implementation and impact.39
Limitations
The majority of literature included in this review was taken from a period of 8 years immediately following the publication of the first WHO SSC study. While this was regarded as sufficient to plot patterns of knowledge mobilisation concerning the SSC, it did not take into account some seminal articles from the exploratory research leading up to the WHO study.
We analysed only the top 25 cited papers, of a body of literature in the thousands of publications. This sample is appropriate to understand major advances in knowledge which would be expected to be found in the highest impact papers. However, the h-25 index excludes many recent papers and those published in less-prominent journals; therefore, we cannot know whether the recontextualisation pattern we describe holds across the entire body of SSC knowledge. We suspect that the pattern may not be stable; for example, the paper by Urbach et al,5 one of the most recent papers in our sample, does not exhibit the pattern of imprecise recontextualisation we observed in other papers. Future research should explore whether SSC knowledge is mobilising more effectively outside the h-25, and the influence of epistemology, article genre and journal impact factor on such mobilisation.
It was not the intention of this paper to judge the methodological rigour or quality of the research presented in the analysed papers, nor to answer the debate about the extent of SSC impact on surgical safety.
Conclusion
This analysis of the top 25 cited SSC papers as a body of knowledge reveals at the least a stuttering effect and at worst a standstill, where studies repeatedly (re)discover the complexity of the SSC phenomenon but are not designed to advance insights about that complexity. We conclude that knowledge mobilisation would be improved with a new emphasis on SSC research that explicitly aims to address the lack of sophisticated understanding of the sociocultural nuances of SSC practices. Such research would be commensurate with a shift underway in safety science that seeks to move beyond standardising and measuring organisational routines to understand the role of adaptive human and social practices in safety efforts.40 41 For SSC knowledge to build effectively, it will need to directly engage with complexity and variability, not as a surprise or a citation in the interpretation of results, but as a central feature of studying local, emergent, adaptive team behaviour.
References
Footnotes
↵* The h-index was developed by J.E. Hirsch and published in Proceedings of the National Academy of Sciences of the United States of America 15 Nov 2005;102:16569–72.
Funding This work was supported by the Economic and Social Research Council (grant number ES/J500136/1).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Correction notice This paper has been amended since it was published Online First. Owing to a scripting error, some of the publisher names in the references were replaced with ‘BMJ Publishing Group’. This only affected the full text version, not the PDF. We have since corrected these errors and the correct publishers have been inserted into the references.