Methods

Qualitative methods were used to identify behaviours linked to high and low obstetrical team functioning and to organise these into a coherent framework for assessment of team performance. In a previous study, 12 interdisciplinary obstetrical teams were repeatedly assigned to teams of five to six members, with each team managing one of four scenarios.19 At the end of the sessions, all participants were asked to list narratives of behaviours that contributed to the functioning of the teams. Also, a focus group of individuals not involved in the simulation was convened to explore teamwork safety issues in obstetrical settings. The audio recording of the focus group was transcribed verbatim. In addition, a literature review was performed to search for published behavioural marking systems for assessment of medical and obstetrical teams. The process for the development of the AOTP is summarised in fig 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The Assessment of Obstetrical Team Performance development process. After obtaining research ethics board approval, videotapes of the 12 obstetrical teams were used by reviewers to generate a list of behavioural aspects judged to negatively or positively affect the teams’ performances. Thirteen reviewers were selected representing (a) team expert/content naïve individuals (n = 2), (b) team expert/content experts (n = 6) and (c) team naïve/content naïve individuals (n = 5). The number of reviewers was determined based on the available funding to support the reviewers’ time.

Data were collated and analysed by the first author (DT) using NVivo A line-by-line method was used to search for common themes and subthemes. The coinvestigators also read the collated data to check for themes and subthemes and then met to compare findings. The process was dynamic, involving repeated analysis by the team. Once the themes and subthemes were determined, anchored descriptors for “excellent” and “poor” team performance were developed. The themes and subthemes were compiled to create a prototype of the AOTP tool, incorporating a 5-point Likert scale (where 1 is poor performance and 5 is excellent performance). A GAOTP was also created using only the themes.

Once the prototypes were judged to accurately reflect the effectiveness of teams in the high-risk obstetrical simulations, usability and comprehensiveness were assessed. Accordingly, 14 reviewers (three RNs, six MDs and five university students, not involved in the generation of items) reviewed the videotapes of 12 teams managing one of four scenarios in a high-fidelity simulation centre and immediately completed both the AOTP and GAOTP to assess the teams’ performances. After the reviewers assessed all of the videotaped performances, they completed a questionnaire regarding usability and time taken for completion.

In order to examine the effect of training in the use of the tools, three naïve raters were asked to assess team performances of three randomly selected scenarios. Following this process, raters watched an additional scenario with an expert facilitator present. During the viewing of the scenario, the videotape was stopped and discussion was encouraged to identify problem areas in the use of the tools. Once the raters felt that they had achieved a common understanding of performance expectations, each rater independently reviewed the three scenarios. Analyses of the internal consistency and inter-rater reliability results between pre-training and post-training was done using Cronbach’s α and intraclass coefficients.

Results

In total, simulation participants, focus group members and video reviewers identified 1294 behavioural items contributing to obstetric team performances. The 6 themes and 18 subthemes (table 1) were compiled into the AOTP and the GOATP. Examples of poor and excellent anchored descriptors are found in table 2.

A summary of the usability data is found in table 3. Some of the items in the usability questionnaire applied to the AOTP alone, whereas other items addressed both tools. The reviewers reported a high degree of satisfaction with usability and comprehensiveness (table 3). The median amount of time that participants spent completing the AOTP was 7.5 min (range 1.5 to 50 min) and 75% of reviewers thought the time requirement was moderate and manageable. Copies of the AOTP, GAOTP and the usability questionnaire are available from the corresponding author.

Internal consistency as measured by Cronbach’s α improved from pre-training to post-training in the three raters viewing the three randomly selected scenarios for the 18 subthemes of AOTP (pre-training, α = 0.829; post-training, α = 0.914). With respect to the six themes used in GAOTP, similar improvement occurred following training with α = 0.679 for pre-training and α = 0.868 for post-training. Inter-rater reliability as measured by intraclass coefficients was 0.54 for pre-training and 0.94 for post-training.

Discussion

The primary objective of this study was to develop a behavioural marking system for use in the performance assessment of interdisciplinary obstetrical teams and to assess its usability. In effect, two tools were developed: the AOTP providing a template for formative feedback of team performance and the GAOTP providing a summative team assessment. The purpose was not to assess the validity or reliability of the tools, nor to compare performance results between them but to develop a performance template that would undergo further research into their psychometric properties.

If simulation training for obstetrical teams is to be successful, we must have a behavioural marking system that is built on the theoretical underpinnings of the team concept and that can be used to guide improvement in specific dimensions of team effectiveness. Accordingly, a key assumption of the AOTP/GAOTP is that team effectiveness depends on balancing diverse professional roles and autonomy with mutually dependent communication and participatory decision-making.20 In addition, the AOTP/GAOTP assumes a high level of interdisciplinary integration, openness and scrutiny of disciplinary values and a common approach to the team’s goal.21

Our findings extend the work of Fletcher et al,2 who focused specifically on the role of anaesthetists, by moving assessment of non-technical skills from the level of the individual to that of the team. In addition, the AOTP addresses the importance of involving the patient and family in obstetrical care by integrating subthemes that speak directly to patient-centred team behaviours. However, the AOTP differs from the MHPTS18 in two key ways. The first difference is that the MHPTS consists of 16 elements of team behaviour, eight of which are required and eight of which may not apply in every situation. In contrast, the AOTP includes 18 subthemes of team behaviour grouped into six themes. The second difference is that the MHPTS was designed as a brief measure of a range of high performance skills for self-evaluation and self-reflection by learners. The AOTP was designed to provide a much finer grained assessment of team behaviours to be used to assess improvements in teamwork over time.

In terms of usability of the AOTP system, two issues were raised by the reviewers. The first issue speaks to the challenges experienced when trying to differentiate between the performance of an individual team member and the performance of the team as a whole. The following excerpts illustrate this concern: “Some behaviors are relevant to one member of the team only and not to the team in general. For example, when one member of the team focuses on one problem (i.e., fixation) to the exclusion of the big picture.” “I had some trouble when completing the AOTP when individual actions were detrimental but the rest of the team acts so that there is no disruption. I had difficulty in weighting my answer between the two actions (i.e., the individual’s performance and the team’s performance).” These comments highlight the complexity of teamwork and the challenge of making an overall assessment of the team’s behaviours especially when the contribution of a team member is suboptimal or counterproductive. In cases like this, examiners must remain vigilant to the understanding that the work of high-risk obstetrical teams requires constant and intense interdependence and continuous self-monitoring, feedback and correction. Use of the AOTP/GAOTP does not require examiners to weigh between individual and team performance; rather, they are asked to integrate their perceptions of the behaviours and contributions of each and every team member in order to assess the behaviours of teams as a whole. Although the comments demonstrated issues related to individual versus team behaviours as noted by untrained raters, the three raters who subsequently received training were able to come to a consensus as to overall team performance despite individual team member competencies.

The second concern about usability speaks to the training of the reviewers on the use of the tool. The reviewers suggested improved training through the provision of baseline scenarios demonstrating poor and excellent team behaviours. In addition, the content-naïve reviewers recommended information about performance expectations of the obstetrical teams so that they could direct their attention towards what was done to cause a delay or failure to comply with what was expected.

The purpose of the study was to incorporate as many opinions as possible into the development and usability of the tools recognising that training and refinement of the tools would be required and the psychometric properties determined for their ultimate use in clinical team performance evaluation. Further assessment of the tools with three trained raters indicated an acceptable inter-rater reliability, demonstrating that raters strongly agreed on the relative level of performances. In addition, internal consistency was good to excellent for all raters following training. A Cronbach’s α between 0.8 and 0.9 (as demonstrated for the 18 subthemes of the AOTP and the six themes of the GAOTP) give reasonable confidence that the items added together are tapping into a coherent construct and that training had a positive effect on these results. While it is recognised that this is a small sample on which to make assumptions, it supports the need and hopefully success of intensive training and feedback before the use of these tools in the evaluation of obstetrical team performances. In addition, results of this study underscore the difficulty in the development of team performance tools and the need for careful attention and analysis of their properties before use.

In summary, although the validity and reliability of the AOTP/GAOTP still need to be determined, initial feedback regarding their usability suggests that they allow for an accurate and comprehensive reflection of team behaviours. Results reported here provide support for future studies that will utilise these tools to investigate the effect of obstetrical team training in non-behavioural skills, specifically comparing the effects of high fidelity simulation team training to other education methodologies.