Ensuring end users understand and use guidelines as intended

Jones and colleagues’ approach affords an opportunity to reflect on the benefits of user testing and simulation of guidelines. The design and evaluation of their revised guidelines provides an excellent example of a careful stepwise progression in the development and evaluation of a guideline as a type of decision support for clinicians. First, in a prior study,20 they user tested the original NHS guidelines to improve retrieval and comprehension of information. The authors produced a revised guideline, which included reformatted sections as well as increased support for key calculations, such as for infusion rates. The authors again user tested the revised guideline, successfully showing higher rates of comprehension. Note that user testing refers to a specific approach focused on comprehension rather than behaviour21 and is distinct from usability testing. Second, in the current study, Jones et al evaluated whether nurse and midwife end users exhibited the desired behavioural changes when given the revised guidelines (with addition of actionable decision supports), compared with a control group working with the current version of the guidelines used in practice. As a result, Jones and colleagues verify that end users (1) understand the content in the guideline and (2) actually change their behaviour in response to using it.

Simulation can play a particularly useful role in this context, as it can help identify problems with users’ comprehension of the guideline and also empirically assess what behavioural changes occur in response to design changes in the guidelines. The level of methodological control and qualitative detail that simulation provides is difficult to feasibly replicate with real-world pilot studies, and therefore simulation fills a critical gap.

Jones et al report successful changes in behaviour due to the revised guidelines in which they added actionable decision supports. For example, their earlier user testing found that participants using the initial guidelines did not account for displacement volume when reconstituting the powdered drug, leading to dosing errors. A second error with the initial guidelines involved participants using the shortest infusion rate provided (eg, guidelines state ‘1 to 3 hours’), without realising that the shortest rate is not appropriate for certain doses (eg, 1 hour is appropriate for smaller doses, but larger doses should not be infused over 1 hour because the drug would then be administered faster than the maximum allowable infusion rate of 3 mg/kg/hour). These two issues were addressed in the revised guidelines by providing key determinants for ‘action’ such as calculation formulas that account for displacement volume and infusion duration, thereby more carefully guiding end users to avoid these dose and rate errors. These changes to the guideline triggered specific behaviours (eg, calculations that account for all variables) that did not occur with the initial guidelines. Therefore, the simulation testing demonstrated the value of providing determinants for action, such as specific calculation formulas to support end users, by showing a clear reduction in dose and rate errors when using the revised guidelines compared with the initial guidelines.

The authors also report that other types of medication-specific errors remained unaffected by the revised guidelines (eg, incorrect technique and flush errors)—the changes made did not facilitate the desired actions. The initial guidelines indicate ‘DO NOT SHAKE’ in capital letters, and there is a section specific to ‘Flushing’. In contrast, the revised guidelines do not capitalise the warning about shaking the vial, but embed the warning with a numbered sequence in the medication preparation section, aiming to increase the likelihood of reading it at the appropriate time. The revised guidelines do not have a section specific to flushing, but embed the flushing instructions as an unnumbered step in the administration section. Thus, the value of embedding technique and flushing information within the context of use was not validated in the simulation testing (ie, no significant differences in the rates of these errors), highlighting precisely the pivotal role that simulation can play in assessing whether attempts to improve usability result in actual behavioural changes.

Finally, simulation can identify potential unintended consequences of a guideline. For instance, Jones and colleagues observed an increase in errors (although not statistically significant) that were not medication specific (eg, non-aseptic technique such as hand washing, swabbing vials with an alcohol wipe). Given that the revised guidelines were specific to the medication tested, it is unusual that we see a tendency toward a worsening effect on generic medication preparation skills. Again, this finding was not significant, but we highlight this to remind ourselves of the very real possibility that some interventions might introduce new and unexpected errors in response to changing workflow and practice6; simulations offer an opportunity to spot these risks in advance.

Now that Jones et al have seen how the revised guidelines change behaviour, they are optimally positioned to move forward. On one hand, they have the option of revising the guidelines further in attempts to address these resistant errors, and on the other, they can consider designing other interventions to be implemented in parallel with their user-tested guidance. At first glance, the errors that were resistant to change appear to be mechanical tasks that end users might think of as applying uniformly to multiple medications (eg, flush errors, non-aseptic technique). Therefore, a second intervention that has a more general scope (rather than drug specific) might be pursued. Regardless of what they decide to pursue, we applaud their measured approach and highlight that the key takeaway is that their next steps are supported with clearer evidence of what to expect when the guidelines are released—certainly a helpful piece of information to guide decisions as to whether broad implementation of guidelines is justified.

Caveats and conclusion

Simulation is not a panacea—it is not able to assess longitudinal adherence, and there are limitations to how realistically clinicians behave when observed for a few sample procedures when under the scrutiny of observers. Further, studies where interventions are implemented to assess whether they move the needle on the outcomes we care about (eg, adverse events, length of stay, patient mortality) are needed and should continue. However, having end users physically perform clinical tasks with the intervention in representative environments represents an important strategy to assess the degree to which guidelines and other decision support interventions in fact promote the desired behaviours and to spot problems in advance of implementation. Such simulation testing is not currently a routine step in intervention design. We hope it becomes a more common phenomenon, with more improvement work following the example of the approach so effectively demonstrated by Jones and colleagues.