Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication

https://doi.org/10.1016/j.ijmedinf.2008.09.003Get rights and content

Abstract

Background

Evidence regarding how health information technologies influence clinicians’ patterns of work and support efficient practices is limited. Traditional paper-based data collection methods are unable to capture clinical work complexity and communication patterns. The use of electronic data collection tools for such studies is emerging yet is rarely assessed for reliability or validity.

Aim

Our aim was to design, apply and test an observational method which incorporated the use of an electronic data collection tool for work measurement studies which would allow efficient, accurate and reliable data collection, and capture greater degrees of work complexity than current approaches.

Methods

We developed an observational method and software for personal digital assistants (PDAs) which captures multiple dimensions of clinicians’ work tasks, namely what task, with whom, and with what; tasks conducted in parallel (multi-tasking); interruptions and task duration. During field-testing over 7 months across four hospital wards, fifty-two nurses were observed for 250 h. Inter-rater reliability was tested and validity was measured by (i) assessing whether observational data reflected known differences in clinical role work tasks and (ii) by comparing observational data with participants’ estimates of their task time distribution.

Results

Observers took 15–20 h of training to master the method and data collection process. Only 1% of tasks observed did not match the classification developed and were classified as ‘other’. Inter-rater reliability scores of observers were maintained at over 85%. The results discriminated between the work patterns of enrolled and registered nurses consistent with differences in their roles. Survey data (n = 27) revealed consistent ratings of tasks by nurses, and their rankings of most to least time-consuming tasks were significantly correlated with those derived from the observational data. Over 40% of nurses’ time was spent in direct care or professional communication, with 11.8% of time spent multi-tasking. Nurses were interrupted approximately every 49 min. One quarter of interruptions occurred while nurses were preparing or administering medications.

Conclusions

This method efficiently produces reliable and valid data. The multi-dimensional nature of the data collected provides greater insights into patterns of clinicians’ work and communication than has previously been possible using other methods.

Introduction

Evidence regarding how health information technologies influence patterns of clinical work and support efficient work practices is limited. A systematic review [1] published in 2005 uncovered 23 studies since 1984 which examined the impact of system use on clinicians’ (doctors’ and nurses’) time. These studies in general adopted either work sampling or time and motion methods. Only six (26%) examined work on general wards, all in US hospitals, while the remainder focused on specialized settings (e.g. ICU and general practice). Overall, studies which compared electronic with paper systems and calculated task time per patient or consultation, reported that computer use increased time required to complete tasks. Studies which examined task time across multiple patients or working shifts found computer use was more time efficient than paper-based systems [1]. Lo et al. [2] more recently observed specialists using either paper or computer systems within outpatient clinics and reported no significant difference in time spent per patient visit. A similar but smaller study [3] measuring time taken for hand-written and computer prescriptions in an ambulatory setting in the US also found no significant difference in average time per task for physicians.

Studies of changes in work distribution and communication patterns following system use are less prevalent, but do include evidence of changes. For example in the US, where nurses usually transcribe hand-written medication orders, CPOE eliminates this task [4], [5]. Following the introduction of a CPOE system clinicians may sequence work differently. As Callen [6] found, clinicians reported “if you are waiting for something on the computer you go and do something else”. This may result in, for example, batching the ordering of patients’ tests to one time of the day. Shu et al. [7] found interns in a US hospital spent more time alone and less time with other doctors after system introduction. A French hospital study [8] of doctor–nurse communication around medications showed that a CPOE system, in comparison with paper-based medication records, resulted in a move from synchronous communication to asynchronous. This introduced opportunities for misunderstandings and increased the extent to which nurses had to make assumptions about orders. Carpenter and Gorman [9] also reported a tendency for doctors in a US hospital to talk with nurses less about medication orders following system implementation.

Understanding these shifts in patterns of communication between clinicians are important as poor communication wastes time, threatens patient care and may be one of the major causes of preventable adverse events in clinical practice [10]. Any potential negative consequences of changes in communication patterns may be more than offset by the improvements in information exchange provided by having legible, easily accessible information which computerized systems afford clinicians. However until we have better quality data about how systems enhance or disrupt existing patterns of clinical work and communication we cannot move to re-design work practices or systems in ways which avoid any possible negative outcomes.

This research agenda needs to continue to progress beyond answering the question, does use of a computer save a clinician time, to questions about how patterns of work are re-arranged in response to the introduction of health technologies. Where time is released, or additional time consumed, how do clinicians re-distribute their time among work tasks? What amount of variation exists among different clinical sub-groups and do work tasks get re-distributed across groups? For example, if senior clinicians are found to spend less time in patient ordering following computerization is this because the system is efficient or because they have re-allocated this task, either explicitly or implicitly, to their junior colleagues? We need to examine how system use interferes with communication processes and as Gorman et al. [11] suggest, ensure that such systems “… facilitate care without interfering with or eliminating aspects of the process that are essential to high reliability performance in the face of urgency, uncertainty and interruptions” (p. 383).

We require studies which investigate whether changes in patterns of clinical work result in improved care delivery, patient outcomes and the work experiences of health professionals. These questions require a multi-method approach [12], and work measurement studies form an important component of such investigation. Researchers should be able to build upon previous work undertaken in different settings and countries. This requires standardization of measurement approaches and the adoption of valid and reliable measurement tools. Major factors identified for the paucity of evidence in this area are the limitations and varieties of methods used [1], [13], [14], the difficulties of capturing the non-linear and interruptive nature of clinical work [15], [16], and the lack of consistency in the application of rigorous research methods.

Our objective was to design, apply and test an observational method for capturing clinician work and communication patterns which incorporated an electronic data collection tool. The purpose of the tool was to allow efficient, accurate and reliable data collection, while also capturing a greater level of work complexity than previous paper-based methods have allowed. Results from a small number of previous studies [2], [17], [18], [19] suggest that the use of handheld computers (including personal digital assistants—PDAs) may be useful for this task, but researchers have presented minimal information about application and reliability issues relating to these tools. Building upon our previous work designing a paper-based, multi-dimensional work classification tool for nurses [14], we sought to investigate how much additional detail and task complexity data we could add using a PDA without reducing data accuracy or reliability. In this paper we detail: the development of these methods and tool; the application of the method in a study of 52 nurses in an academic hospital and a summary of key findings; and tests of the reliability and validity of data produced. This research comprised the first stage in a study to measure the impact of a commercial electronic medication management system (e-MMS) on doctors’ and nurses’ work and communication patterns.

Section snippets

Design of a multi-dimensional work task classification

Our first objective was to develop a work task classification system which would be incorporated in the PDA data collection tool. As a basis for this we used a multi-dimensional work measurement classification which we had applied in a paper-based work-sampling study of nurses that showed high levels of inter-rater reliability and face validity [14]. We extended the classification to contain greater detail about work tasks which previous literature indicated may be most susceptible to change

Reliability and validity of data

Inter-rated reliability testing showed all data collectors maintained 85% or higher (range 85–98%) in the field. The work task classification allowed 99% of all tasks to be classified with only 1% of tasks allocated to the ‘other’ category. The patterns of work task time distribution for enrolled nurses and registered nurses were compared to determine whether expected differences were reflected in the observational data. As Fig. 3 shows enrolled nurses spent a greater proportion of their time

Discussion

Our aim was to design research methods and a tool to allow efficient, reliable and accurate data collection about clinicians’ patterns of work and communication, which could potentially be used across specialties, professional groups, settings and countries. The Work Observational Method by Activity Timing (WOMBAT) proved to be a reliable means for collecting valid, complex, multi-dimensional data about nurses’ work and communication patterns, based on the indicators we measured. High levels of

Conclusions

Capture and analysis of clinicians’ work and communication patterns are vital if we are to further understand the full and far-reaching effects of electronic systems. The use of the WOMBAT approach allows us to go further than gathering clinicians’ perceptions of their time in various tasks, providing valid and reliable measurements of task time distribution and with whom and how tasks are undertaken. Further it enables us to distinguish between multi-tasking and interruptions and the

Acknowledgments

This research was funded by an ARC linkage grant in partnership with NSW Health. JW is supported by an NHMRC Fellowship. We thank M. Williamson, L. Kearney, K. Nguyen, and J. Pich for their involvement in various stages of the research.

References (21)

There are more references available in the full text version of this article.

Cited by (112)

View all citing articles on Scopus

This paper is based upon a presentation and paper from 12th World Congress on Medical Informatics. Editors: Kuhn KA, Warren JR, Leong T. Amsterdam: IOS Press, 1083–1087; Westbrook JI, Ampt A, Williamson M, Nguyen K, Kearney L. (2007). Methods for measuring the impact of health information technologies on clinicians’ patterns of work and communication.

1

This research was undertaken while A. Ampt was a researcher at the Health Informatics Research & Evaluation Unit, Faculty of Health Sciences, University of Sydney, Australia.

View full text