Considering chance variation to inform hospital management teams and boards

Statistical process control (SPC) charts, a type of time series chart, are commonly used in quality improvement as they enable a clear distinction between statistical significance and random variation.13 14 An advantage of SPCs is that it allows detection of whether a process is ‘in-control’, that is, variations from a mean are part of a stable process when they occur between upper and lower control limits. SPC charts can therefore identify variation secondary to an external force or ‘special cause’ from change that occurs due to random chance. Such ‘special cause variation’ may be attributable to a quality improvement intervention and can provide convincing statistical evidence of change provided a stable process before introduction of the intervention.15 For example, an SPC chart of catheter-associated urinary tract infection rates on a medical ward showing special cause variation after the implementation of a quality improvement initiative aimed at early urinary catheter removal and catheter insertion avoidance would provide compelling evidence to the hospital management team and the board that the intervention was effective to improve care. Rather than tracking outcomes over time within the hospital or unit, funnel plots can be used for between-group comparisons, with units or hospitals outside the control limits identified as having outlier performance, either significantly worse or significantly better.16

In healthcare, the application of SPC charts has been encouraged among quality improvement teams, managers and executives for day-to-day decision-making.13 An example might be to review falls data to investigate datapoints outside the upper control limit to determine if the higher number is cause for concern. Another use may be to decide whether to commit to additional funding to spread a falls prevention intervention throughout the organisation that has clearly demonstrated impact beyond chance. Collectively, the hospital management and operations teams work to operationalise strategic quality improvement plans and should use SPCs to inform their work.

In contrast to management, hospital boards have a different role—one where they hold the hospital management team accountable for achieving results on outcomes linked to strategic priorities. To support this role, hospital boards should review performance data—but in what format and how? Should SPCs, run charts or other charts be used as methods for data display? There is evidence that regular board review of quality metrics is associated with strong hospital performance, although a causal link has yet to be established.17 However, specific recommendations on the use and impact of SPC charts at the level of hospital boards are unclear and uptake at the board level appears limited. In a previous study of hospital board data reports among randomly selected English National Health Service (NHS) trusts, only 6% of charts submitted to the board depicted the role of chance.12 The authors postulated that board members may be unaware of control charts and/or support staff may lack the skills to create such charts.

In this issue of BMJ Quality & Safety, Kudrna and colleagues evaluated the impact of SPC training on the use of SPC charts in hospital board documentation in England.18 This observational study sampled 20 intervention acute care hospitals that received the training and matched these on the number of patient attendances, degree of specialisation and deprivation level to 20 control hospitals. The intervention consisted of training on SPC chart creation and interpretation, when and how to use them, how SPC charts can inform decision-making about process variation and the limitations of other charts. The training was delivered to board members (lasting 90 min) and quality improvement staff (over one working day) to improve knowledge about SPC charts and improve their uptake. The primary outcome was the proportion of all charts in board documents generated using SPC charts. Other outcomes included the quality of SPC charts (eg, whether control limits were set), the proportion of time series charts that used SPC methods and the proportion of time series and between-group charts (funnel plots) that used SPC methods. Qualitative data included thematic analysis of comments from participants about the training sessions. Nearly one-half of the total 6287 charts (encompassing all intervention and control hospitals) were quality and safety charts. Of these, 75% were time series charts, 10% between group comparisons, 14% were both time series and between-group comparisons and only 7% were SPCs. The authors found that intervention hospitals had a greater increase in SPC use from pre-intervention to post-intervention periods (7% to 21%, relative risk ratio of 9; 95% CI 3 to 32) compared with controls (0.6% to 1.2%). A majority of the qualitative comments was positive: better awareness of SPC, understanding how poor data presentation can lead to poor decisions, changing how participants think about data and reporting and interpretation of data.

In advocating for SPC use at the board level, Kudrna and colleagues hope to prevent over-reaction to normal variation in data. In their study, the authors found a ‘high proportion (of) charts with time series information in board papers (90%)’, suggesting that many of the boards in the NHS were using time series data, although specifics of the charts were not provided. We also do not have information on how hospital management teams were displaying and interpreting data given that only board reports were reviewed for the study. Since operations teams are responsible for carrying out quality improvement (QI) work, it would be important to understand SPC use at cascading levels within the organisation, from executives to front-line staff and how it impacts on outcomes. It may be possible that SPC charts were used by the operations teams but not shared with boards or that the operations teams have little appreciation of SPC charts. Studies identifying board practices associated with high performing organisations support regular review of data by board members but stop short of recommending the format most useful in supporting the board in their data review.

Additionally, although statistically effective, uptake only reached 21% in the intervention arm of the study which suggests an enduring low appetite for SPC use, although the optimal number is unknown. Another factor to be considered, in relation to sustainability at the board level, is that members are appointed for fixed terms, necessitating ongoing orientation and education of new members in order to maintain proficiency in SPC use and interpretation. Longer board memberships like those in the UK could reduce the need for ongoing maintenance.

The use of SPC charts and other time series data is a cornerstone of institutional quality improvement and should be used widely by improvement teams and managers, including experts in analytical and statistical tools, to evaluate change. Hospital boards have a different role and the question is whether board members should become experts at interpreting statistical results, considering some boards may be predominantly composed of lay public rather than clinical experts. Advocating for the inclusion of SPCs in hospital board reports begs the question, what is the problem we are trying to solve? If the problem is that boards are not provided with tools needed to respond to questions stemming from their fiduciary role (ie, ‘is care changing in the right direction?’), then familiarity with time series such as run charts or SPC may be appropriate. If the goal is to achieve better clinical outcomes, SPC use is critical by the operations team on a day-to-day basis but may or may not be needed at the board level, given its role is to hold the management team accountable to achieve set targets.

Improving hospital board decision-making

Data that fail to distinguish between common and special cause variation can result in misinterpretation by boards and subsequently, erroneous advice and decisions. If the role of hospital boards is to oversee the implementation of strategic priorities and ensure risk is mitigated, how can boards best review data to achieve these objectives? Through our experiences in quality improvement and serving on hospital boards, we have identified three common challenges experienced by board members in making sense of information provided in board packages: (1) large numbers of included measures and data are overwhelming and often obscure summary messaging; (2) misinterpreting the data due to insufficient narrative to enable board members to quickly make sense of the information provided; and (3) the lack of understanding performance within organisational context (table 1). These three challenges are in addition to the challenge already described for boards that are only provided with data in colour-coded dashboards. Based on findings from the literature on governance and characteristics of successful boards19 and supplemented by our own experience, we advocate for a number of additional recommendations, that are of equal importance to the display of data, when deciding what data and format to use when presenting data to board members (table 1).

This will ensure the focus is on presenting data in a format that is easily interpretable and informative, to equip board members to effectively perform their role for organisational quality and safety.