• Qualitative research
• Quality measurement
• Evaluation methodology

Two related papers1 ,2 in this issue of BMJ Quality & Safety provide interesting insights into the difficulties of evaluating improvement activities, and also illustrate why improvement is so hard. In a carefully crafted set of controlled, interrupted time series experiments, the authors examined the effectiveness in the operating theatre of two popular improvement interventions: standardised procedures and teamwork training. The primary outcomes in both were process measures: the theatre teams’ non-technical skills performance, and the count of ‘glitches’—omissions, interruptions or other untoward events that disrupted flow and had potential to affect safety or quality. In both experiments, the investigators took care to ensure the interventions were ‘owned’ by the frontline workers, and not imposed from without by managers disconnected with the realities of the workplace (although this also means that higher level support important for sustainability may have been lacking).

The papers report insufficient evidence to support improved performance from introducing standard operating procedures, even when those procedures were developed and implemented by the frontline staff themselves.1 However, they also report a partial success, in that, when accompanied by teamwork training, the combination of standard operating procedures and teamwork significantly improved non-technical skills performance.2 Curiously, in the combined experiment, technical performance as measured by ‘glitches’ per hour improved in experimental and control groups. Taken as a whole, the two papers suggest an interaction, or synergism, between the two interventions. Standardisation alone was not effective, but standardisation in conjunction with teamwork training, was (although we cannot be certain whether teamwork alone might have been similarly effective).

These two papers make a valuable contribution to the safety and quality literature by showing that the same intervention (standardisation) can be ineffective in one …