You are here

Article Text

Article menu
Original research
Mapping and assessing clinical handover training interventions
  1. Slavi Stoyanov1,
  2. Henny Boshuizen1,
  3. Oliver Groene2,
  4. Marcel van der Klink1,
  5. Wendy Kicken1,
  6. Hendrik Drachsler1,
  7. Paul Barach3,4,5
  1. 1Centre for Learning Sciences and Technologies, Open University of the Netherlands, Heerlen, The Netherlands
  2. 2Health Services Research Faculty of Public Health and Policy, London School of Hygiene & Tropical Medicine, London, UK
  3. 3University of Stavanger, Stavanger, Norway
  4. 4Utrecht Medical Centre, Utrecht, The Netherlands
  5. 5University College Cork, Ireland
  1. Correspondence to Dr Slavi Stoyanov, Centre for Learning Sciences and Technologies, Open University of the Netherlands, PO Box 2960, Heerlen 6401 AT, The Netherlands; slavi.stoyanov{at}ou.nl

Abstract

Background The literature reveals a patchwork of knowledge about the effectiveness of handover and transfer of care-training interventions, their influence on handover practices and on patient outcomes. We identified a range of training interventions, defined their content, and then proposed practical measures for improving the training effectiveness of handover practices.

Methods We applied the Group Concept Mapping approach to identify objectively the shared understanding of a group of experts about patient handover training interventions. We collected 105 declarative statements about handover training interventions from an exhaustive literature review, and from structured expert interviews. The statements were then given to 21 healthcare and training design specialists to sort the statements on similarity in meaning, and rate them on their importance and feasibility.

Results We used multidimensional scaling and hierarchical cluster analysis to depict the following seven clusters related to various handover training issues: standardisation, communication, coordination of activities, clinical microsystem care, transfer and impact, training methods and workplace learning.

Conclusions Ideas on handover training interventions, grouped in thematic clusters, and prioritised on importance and feasibility creates a repository of approaches. This allows healthcare institutions to design and test concrete solutions for improving formal training and workplace learning related to handovers, and addressing informal social learning at the organisational level, with the aim of increasing impact on handover practice and patient outcomes. Measures need to be taken to assure a continuum of handover training interventions from formal training through workplace learning through less formal social learning, and to embed this training in the design of the clinical microsystem.

  • Hand-off
  • Health professions education
  • Healthcare quality improvement
  • Evaluation methodology

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/3.0/ and http://creativecommons.org/licenses/by-nc/3.0/legalcode

https://doi.org/10.1136/bmjqs-2012-001169

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background

    When a patient's transition from the hospital to home is less than optimal, the repercussions can be far-reaching—hospital readmission, adverse medical events and even mortality. A number of factors have been found to contribute to ineffective handover processes, including (1) lack of formal policies and standard handover protocols regarding health-provider communications1 ,2; (2) a decrease in the time devoted to teaching and oversight in the workplace due to an increase in service workload3 and (3) attitudes and organisational culture, such as lack of responsibility to cross-cover patients, and a pervasive ‘culture of blame’.3 ,4 Education and training in handover are considered effective means to address these issues.5 However, research to assess the impact of educational interventions on patient outcomes is still limited and fragmented.4 ,6 ,7 There is no agreement on what constitute the core content areas to address, and what are the instructional methods to apply in formal handover training.

    Formal training is a systematic, planned, instructor-led learning approach to healthcare professionals, typically conducted in specific places and times, and leading to some form of recognition (diplomas or certificates) on successful completion of predefined learning objectives.8 ,9 Although the literature discusses mainly formal training for improving handovers, it is useful to check for informal training interventions, including those shaped by the hidden curriculum.10 In addition, little is known about the transfer of handover training knowledge and skills to the bedside, or the impact of handover training on actual clinical practice or patient outcomes.11 These limitations might explain, despite years of effort to improve hospital to community patient discharges, the limited impact on reducing hospital readmissions.12

    The study addresses the following research questions: (1) what are training interventions for improving handover; (2) what are the most important core topics and the training methods for handover formal training and (3) how can we increase the transfer, uptake and impact of training on handover practices and patient outcomes.

    Methods

    Setting and sample

    This study was undertaken as part of the European Handover Research Consortium as part of the European Commission 7th Framework sponsored project ‘Improving the Continuity of Patient Care through Identification and Implementation of Novel Handoff process in Europe (the HANDOVER Project)’. The study was conducted between January and July 2011. The sample consisted of 30 project members invited through electronic mail to participate in the study. The project members had prior knowledge about and experience in patient handover, as indicated by a survey conducted within the framework of the study.13 Subjects were informed about the purpose, the procedure and the time needed for completing the activities. The group was introduced to the Group Concept Mapping (GCM) approach applied to the study, during two of the project meetings. In a later stage, we invited 10 external experts (healthcare specialists with experience in handover) recommended by HANDOVER project members. These professionals received the same information about the approach and intent of the study.

    Instruments and procedures

    GCM is an integrated mixed method, including both qualitative and quantitative measures. It uses a structured approach to identify an expert group's understanding about the types, methods and characteristics of handover training interventions.14–16 Multivariate statistical techniques of multidimensional scaling and hierarchical cluster analysis (HCA) translate complex qualitative data into conceptual maps. A group concept map shows all the specific ideas about a particular topic (eg, handover educational interventions). The map also indicates how ideas are related to other ideas. In addition, the map indicates how much emphasis should be placed on a particular idea or cluster relative to other ideas (eg, how relatively important or feasible to implement a given intervention is, vis-a-vis other proposed approaches).

    Idea generation

    A literature search, in English, was performed on a number of databases in both the medical and educational domains, such as Academic Search Elite, Business Source Premier, PsyINFO, Web of Science, and Pubmed. The search resulted in 128 papers that were selected for further exploration. They were divided into four parts, and each of the four researchers (WK, MvdK, HB and SS) were tasked to independently look at one part, extract ideas about handover training interventions, and formulate them as statements.

    The statements typically were short phrases expressing an idea and, where appropriate, incorporate an active verb to give a sense of action and direction.17 Examples of statements include: ‘Look for a standard approach to handover communication’; ‘Adopt methods already used in other domains (ie, Crew Resource Management, I-SBAR, Five Ps, I-PASS-THE-BATON)’; and ‘Apply job aids’.

    The literature search generated 252 statements. After removal of duplicative and vague statements, the final list included 75 unique statements in the sample. We added 26 statements to this list from the structured interviews that were conducted with 35 healthcare training specialists from European Union nations. Examples of statements from the interview analysis were as follows: ‘Use active methods such as case studies and role playing’; ‘Train providers about attitudes for common responsibility of patients’; ‘Shift attention from one-doctor one-patient relationships to cross-cover patient commitments’; and, ‘Calculate the adverse events to measure the training effects’.

    Details about the search strategy for literature review, the interview procedure, and the questions in the interview script are presented in online supplementary appendix A.

    Sorting

    The 101 resulting statements were mailed to the 30 participants in the original European Handover Research Consortium (see full list under Acknowledgments). We asked participants to evaluate whether the statements covered the domain of handover educational interventions and to add new statements as needed. Four new statements were generated from this step. The final 105 statements were sent to 15 HANDOVER project members (out of 30) and six external participants (out of 10) who agreed to participate. We asked the 21 experts to first sort the statements into clusters that made conceptual sense, and then assign to each cluster a label that described its contents.

    Rating

    The expert group was instructed to rate each statement using a Likert scale of 1–5. The two rating questions were: (1) how important was the statement (1=not important, 5=extremely important) and (2) how feasible was it to implement these ideas in practice (1=not feasible, 5=most feasible).

    A web-based platform for sorting and rating based on Concept System Global platform was created to make the process more efficient.13 The survey also collected information about the educational background, professional experience, and prior knowledge of the participants about handover practices and research.

    Participant characteristics

    Forty participants were invited, of whom 21 (15 HANDOVER project's members and six external experts) accepted the invitation to participate and complete both the statement sorting and rating activities. Fourteen (66.7%) of the sample were healthcare professionals and seven (33.3%) were instructional designers, specialists in designing training in different professional domains, but without educational background in medicine. Ten of the participants (47.6%) had more than 10 years of professional experience. Five professionals (23.8%) reported between 6 and 10 years of experience. Six specialists (28.6%) declared 1–5 years of professional experience.

    The study received ethics approval by the ethical review board of the University Medical Centre, Utrecht, The Netherlands. The experts were consented before participating. To preserve participant anonymity, the files exported from Concept System Global to Concept System Core for further analyses18 contained only numeric data and no personal identifying information.

    Data analysis

    We applied multidimensional scaling (MDS) and generated a concept map depicting graphical representations of relationships among the 105 statements. Using the MDS solution, a HCA grouped the statements into conceptual clusters, based on similarity of ideas. Descriptive and non-parametric statistics were applied for the rating of data.

    Results

    We first describe the concept mapping study characteristics of our respondents, and then divide the results into the two major stages of data analysis: sorting and rating.

    Map construction

    Figure 1 represents the first output of the GCM analysis—a point map, which is combined with the cluster map and the labelled map.

    Figure 1
    Figure 1

    Combined point cluster map of patient handover educational interventions.

    Each point on the map represents one of the 105 original statements. The closer the statements are to each other, the closer in meaning they were perceived to be by the experts who performed the sorting. To make the map more informative, we used HCA, which increased the reliability of depicting thematic areas on the point map. We used the practical heuristics ‘20-to-5’, to find the optimal number of clusters, which is based on the rule that most of the other GCM projects identify clusters in the range between five and 20.14 ,15 We started from a 20-cluster solution with the goal of arriving at a five-cluster solution. At each iteration, we assessed whether the merging of clusters made sense. An additional criterion used was a routine multidimensional statistic called a bridging value. The analysis computes a bridging value (between 0 and 1) for each statement on the map. A bridging value closer to 0 means that a statement was grouped together with others close, while a value closer to 1 indicates that the statement was sorted ‘with some statements somewhat distant on one side of the same and some statements on the other side, and the algorithm located it in an intermediate position’(ref. 15 p. 101). A cluster is also assigned a mean bridging value, calculated on the basis of the bridging values of the statements in this cluster. The lower the bridging value the more coherent a cluster is, meaning that more people agree on the content. The process to define the optimal number of clusters using this approach produced a seven-cluster solution as the best representation of the data (figure 1).

    The size of each cluster does not reflect the importance or strength of a cluster. Clusters represent distinct conceptual areas that participants identified as key issues of handover training interventions. The statements within each cluster, therefore, ‘co-sort’ statistically and conceptually. The closer the clusters are to each other the closer they are conceptually.

    There are three methods to define clusters thematically, and the best solution is to combine all three methods. The first method looks at the statements that constitute a particular cluster; the second checks the bridging values for the statements in a cluster; and the third considers the suggestions that emerge from the Concept System Core software18 for the best fitting labels of the clusters (as defined by participants). The following seven clusters were identified: Standardisation, Communication, Coordination of activities, Clinical Microsystem, Transfer and Impact, Training Methods and Workplace Learning. Online supplementary appendix B presents all clusters with statements included, and the bridging values for both statements and clusters. The values of statements on importance and feasibility are also included.

    The resulting map objectively represents the group's common understanding of issues related to handover training interventions. The focus of the sorting analysis is on this common understanding and shared vision rather than on differences between subsets of the samples shown by different maps. Clusters help identify distinctive themes, but they do not ‘rate’ the ideas, for instance, compare clusters with high-rated statements to clusters containing low-rated ideas.15 ,19

    Importance and feasibility of handover educational interventions

    Exploring the rating data provided useful information for interpreting the results as well. Clusters that scored high on importance received lower scores on feasibility, and vice versa. The exception was Standardisation, which scored high on both dimensions. The highest score on importance was attributed to the clusters Clinical Microsystem (M=3.89; SD=0.3), Standardisation (M=3.88; SD=0.2), and Transfer/Impact (M=3.81; SD=0.3). The other clusters, Communication (M=3.61; SD=0.3), Coordination (M=3.60; SD=0.4), Training Methods (M=3.57; SD=0.5) and Workplace Learning (M=3.58; SD=0.5) received a somewhat lower score. A Kruskal–Wallis test revealed no significant differences between the clusters on importance (χ2=9.332; df=6; p>0.05).

    The feasibility rating figures show a different configuration. Standardisation received the highest score (M=3.89; SD=0.2), and Communication (M=3.63; SD=0.3), Coordination (M=3.60; SD=0.4), and Training Methods (M=3.69; SD=0.4) also received a high score, while Workplace Learning (M=3.32; SD=0.4), and particularly Transfer/Impact (M=3.16; SD=0.4) and Clinical Microsystem (M=3.12; SD=0.6) received much lower scores. A Kruskal–Wallis test indicates a significant difference between the clusters on the feasibility dimension (χ2=32.279; df=6; p<0.001).

    We applied a detailed post-hoc Mann–Whitney U test to pinpoint where the differences reside, and also used the Bonferroni correction for multiple comparisons, to adjust the critical α value and prevent a type I error. The test reveals that a significant difference existed between Standardisation and Clinical Microsystem (p<0.001; r=−0.72); Standardisation and Transfer/Impact (p<0.001; r=−0.67); Standardisation and Workplace Learning (p<0.001; r=−0.69); Communication and Transfer/Impact (p<0.005; r=−58); Clinical Microsystem and Training Methods (p<0.05; r=−0.48); and Training Methods and Transfer/Impact (p<0.001; r=−0.54). The analyses indicate a large effect size for all tests as well as significant findings.

    The analysis also depicted a significant difference between the values of importance and feasibility in two clusters: Clinical Microsystem (mean rankimportance=20.43; mean rankfeasibility=8.57; χ2=14.560; df=1; p<0.001) and Transfer/Impact (mean rankimportance=26.66; mean rankfeasibility=12.34; χ2=15.817; df=1; p<0.001). Analysing interventions for their relative importance and feasibility seems to be useful, but it might not be sufficiently sensitive in terms of specifying which interventions for adoption a statement suggests. To explore the relationships between statements on importance and feasibility within a particular cluster further, we used the average of each statement of both values to plot a bivariate graph. The graphic is divided into four quadrants above and below a mean value of each rating variable within a cluster. GCM methodology calls this graphic a ‘go-zone’, because it suggests actions and identifies possible implementation challenges. An example of a go-zone is presented in figure 2.

    Figure 2
    Figure 2

    Position of statements in a cluster determined by their rating values.

    Typically, statements in the upper-right quadrant are the most ‘actionable’ and with high-priority ideas in the short term, as they score above the mean on both variables (eg, statement ‘29. Relate handover training to real-life situations’). The lower-right quadrant, statements with higher importance and lower feasibility indicates interventions are score-high on priority, but may be challenging from an implementation perspective. The upper-left and the lower-left quadrants contain ideas with a lower priority. Online supplementary appendix C presents all the cluster go-zones. Online supplementary appendix D lists all statements that score above the means of both importance and feasibility (all statements from all clusters in the upper-right quadrant).

    Most statements that score high on both values come from the clusters that represent formal handover training interventions, including Training Method, 10; Standardisation, 8; Communication, 4 and Coordination, 4. The cluster Clinical Microsystem is represented by five statements; Workplace Learning and Transfer/Impact by two. Ideas that are important but difficult to implement represent mostly the clusters Clinical Microsystem (issues with regard to changing attitudes and culture) and Impact (challenges with measuring impact of handover training interventions).

    A cluster could contain statements with relatively higher or lower rating on importance and feasibility.

    Differences with regard to professional groups and experience

    The analysis of the professional occupation and past experience of the raters revealed no significant differences between the ratings of healthcare professionals versus those of training experts (χ2importance=0.669; df=1; p>0.05; χ2feasibility = 1.397; df=1; p>0.05), nor was there a difference in ratings based on experience (‘more than 10 years’, ‘between 6 and 10 years’ and ‘between 1 and 5 years’; χ2importance=0.013; df=2; p>0.05; χ2feasibility=0.881; p>0.05). This suggests that all participants in this study were a homogeneous group with a high degree of agreement on the valuation of different statements and clusters.

    Discussion

    The discussion is organised around the three research questions: (1) what are training interventions for improving handover; (2) what are core topics and training methods for handover formal training and (3) how can we increase the transfer, uptake and impact, of training on handover practices and patient outcomes.

    What are training interventions for improving handover?

    The concept mapping study identified three types of handover training interventions: formal training in handover, workplace learning and clinical microsystem-based interventions. The clusters, Standardisation, Communication, Coordination and Training Methods are identified as separate clusters, but since they cluster close together, we feel they fall into a more global category consistent with a concept of ‘formal training’ (‘zone’ of formal training in handover). Standardisation, Communication and Coordination are about what to teach, Teaching Methods are about how to teach. While the literature discusses formal training in handover as the only training intervention, this study identified two other training interventions: workplace learning and interventions related to redesigning the clinical microsystem.

    The cluster workplace learning suggests that learning needs to be integrated in professional practice. Workplace learning does not need to be formal or entail organised training events, but must be guided by explicit learning goals to be achieved and that can be measured. Job aids, handover electronic performance systems, supervision and guided practice on a one-to-one basis, discussion of cases, and workplace observation were recommended by participants. Future research should determine what would be the most effective approaches to ensure sustainable workplace learning.

    At first glance, it may seem that redesigning the clinical microsystem has little to do with implementing handover training interventions. A more careful examination suggests that the statements in this cluster outline the contours of a ‘handover community of practice’. ‘Communities of practice’ is a term associated with training, but it is also considered an integral part of the work of professionals and their professional formation.10 A community of practice is a different learning model than formal training and workplace learning. It is social, informal and integrated into the professional practice and organisational culture. Community of practice requires a group of professionals not necessarily bound to a particular department. (‘Involve different professions, such as doctors, nurses and allied professions, in order to reflect the complexity of real life handovers’). Learning is embedded in shared professional practice and occurs in real-life contexts. A community of practice develops a repertoire of sharing resources to support learning at the workplaces (‘Provide support of handover practices on work places’; ‘Apply job aids (to-do lists, help about content and format of handover procedure, check lists) to support handover in work environments’; ‘Use existing information systems for an effective handover practice’; ‘Adopt methods of high-performance teams’). The best way to acquire particular handover attitudes is by socialisation through immersion into the culture of a community of practice (‘Create appropriate attitudes, climate and role models’; ‘Shift attention from one-doctor one-patient relationships to cross-cover patient commitments and transfer of professional responsibility’; ‘Effective handovers require changing mentality of [the] professionals involved’).

    Regarding the lower ratings of feasibility for the cluster, Microsystems, it is not realistic to expect that training alone can change a clinical microsystem, although it can contribute to change and help establish more effective handover practice and culture that supports social and informal learning.

    The high-feasibility scores of all clusters defining formal training suggests that the participants in this study consider formal training as the easiest handover educational intervention to organise.

    What are the most important core topics and the training methods for handover formal training?

    The concept mapping study identified at least three important handover training themes to address: standardisation of practice, communication and coordination of activities. The results are in line with the findings of Cheah et al,1 Laugaland et al5 and Shojania et al.7 Although the literature discussed standardisation of practices as part of communication, the current study underscores the importance to consider standardisation as a separate topic.

    The statements in the cluster, Training Methods, refer to different instructional design approaches needed (Problem-Based Learning,20 ,21 Four Components Instructional Design Model (4C/ID Model),22 ,23 Cognitive Apprenticeship Approach,24 ,25 Theory of Deliberate Practice,26 and Cognitive Flexibility Theory.27 They also suggest considering combinations of these methods according to the first principles of instructional design.28 The statements in the clusters Standardisation, Communication, Coordination and Training Methods present a rich repository of ideas for selecting content and instructional methods when designing formal training in handover. The statements can be considered building blocks that can be combined in different ways to design customised training that reflects specific goals and contexts. Defining which of these combinations are most effective is a subject for future research.

    How to increase the transfer, uptake and impact of training on handover practices and patient outcomes?

    This study identified a separate cluster that indicates issues with the transfer of formal training knowledge and skills to the workplace and the impact of formal training on real handover practices. The fact that these two issues are included in one cluster suggests that they are interdependent. Stated another way, if there is no transfer of training, there cannot be an impact.

    The distances between each of the clusters that compose formal training zones (Standardisation, Communication, Coordination and Training Methods) and the cluster, Transfer/Impact, is relatively large, which suggests that the participants in this study do not associate formal training with transfer of knowledge, skills and attitudes, and they do not believe that formal training would automatically impact handover practice. This result is in accordance with the findings of other studies and needs to be addressed in any future intervention.11 ,29–31 Making the transfer of handover knowledge and skills more effective, which eventually, might generate an impact on the flow and process of the clinical microsystem, requires measures to be taken for providing effective support to clinicians in their workplaces. More research is needed on what these measures could be, and how best to enable them to support training interventions.

    Although formal training, Clinical Microsystem and Transfer/Impact are not directly related, an indirect link between them exists through the bridging role of Workplace Learning. The Workplace Learning scores on feasibility were lower than the formal training clusters, but higher than the Clinical Microsystem and Transfer/Impact. The importance of workplace learning for transfer of knowledge, skills and attitudes, and the impact it has on handover practices is supported by other research.32

    Our study has several limitations. The sample included a limited number of participants, was non-randomly selected, and had an unequal representation of health profession occupations. The sample of 21 participants should be judged in comparison with the typical GCM practice which allows a smaller number of people to sort and rate statements.14 ,15 In addition, our statements came from 128 scientific papers on handovers, supplemented with the data from 35 interviews. We checked the sorting analysis using the responses of 10, 12, 15, 17 and 21 participants at different stages of the study, and we found no substantial differences between the group sizes suggesting a valid and reliable process. There seems to be a point of saturation in the number of participants, after which no substantial benefit is accrued, and no substantive changes occur in the results. These findings were confirmed by a recent meta-analytical study on 69 GCM projects conducted over the last 10 years, which found that 20–30 sorters produce the optimal goodness-of-fit between the aggregated similarity matrix and its representation as a conceptual map.33 This observation is also in line with results from research in other domains.34

    Conclusions

    One of the main conclusions and contributions of this study is agreement on the benefits of a continuum of handover training, ranging from formal training to workplace learning to participation in a community of practice related to handovers, and to considering the training context offered by the clinical microsystem.

    We believe that the results of our study will contribute to the development of more effective design of handover training interventions. A combination of various research approaches, and a larger sample of training experts and clinicians, could provide valuable perspectives and further insights into the theory and practice of educational interventions to improve patient handovers.

    Our study was exploratory, and sought to provide empirical ground for formulating hypotheses, not for testing hypotheses. Individual statements, grouped in clusters, are a rich source of information for researchers and practitioners to look at, select and combine ideas to design and test handover training interventions in different contexts, and at different levels. One particular idea that comes out of this study which we want to further elaborate and empirically test, is a training approach that combines principles of different instructional methods (Problem-Based Learning, Cognitive Apprenticeship, Four Components Instructional design, and Cognitive Flexibility Approach). Another idea worth investigating is the effect of electronic handover performance support systems on increasing performance in the workplace.

    This is not the first time GCM is used in the healthcare domain. We emphasise in our analysis the powerful feature of the GCM method to produce a common understanding (conceptual map) of a group of experts to help drive reflection in action in improving clinical care in general, and in this study on handover training interventions.

    Healthcare projects are increasingly applying this approach for research and also for informing decision making and planning of clinical interventions.35

    Acknowledgments

    We thank all the members of the European HANDOVER Research Collaborative for their participation in this study. Loes Pijnenborg, Julie Johnson, Beryl Göbel, Cor Kalkman, Richard Lilford, Nicola Novielli, Yen-Fu Chen, Semira Manaseki-Holland, Basia Kutryba, Halina Kutaj-Wasikowska, Ewa Dudzik-Urbaniak, Marcin Kalinowski, Francesco Venneri, Giulio Toccafondi, Antonio Molisso, Sara Albolino, Hub Wollersheim, Gijs Hesselink, Lisette Schoonhoven, Myrra Vernooij, Marieke Zegers, Helen Hansagi, Mariann Olsson, Susanne Bergenbrant, Maria Flink, Gunnar Ohlen, Carola Orrego, Rosa Sunol, Jerry Andriessen.

    References

    1. ↵
      1. Cheah LP,
      2. Pollard J,
      3. Watters DAK
      . Electronic medical handover: towards safer medical care. Med J Aust 2005;7:369–73.
      OpenUrl
    2. ↵
      1. Iedema R,
      2. Merrick ET,
      3. Kerridge R,
      4. et al
      . Handover-enabling learning in communication for safety (HELiCS): a report on achievements at two hospital sites. Med J Aust 2009;11:133–6.
      OpenUrlWeb of Science
    3. ↵
      1. Barach P,
      2. Philibert I
      . The July effect: fertile ground for systems improvement. Ann Intern Med 2011;155:331–2;PMID 21747094.
      OpenUrlPubMedWeb of Science
    4. ↵
      1. Arora VM,
      2. Johnson JK,
      3. Meltzer DO,
      4. et al
      . A theoretical framework and competency-based approach to improving handoffs. Qual Saf Health Care 2008;17:11–14.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Laugaland KA,
      2. Aase K,
      3. Barach P
      . Interventions to improve patient safety in transitional care—A review of the evidence. Work 2012;4:2915–24.
      OpenUrl
    6. ↵
      1. Johnson J,
      2. Barach P
      . Patient care handovers: what it will take to ensure quality and safety during times of transition. Med J Aust 2009;11:110–12.
      OpenUrl
    7. ↵
      1. Shojania KG,
      2. Fletcher KE,
      3. Sanjay S
      . Graduate medical education and patient safety: A busy and occasionally hazardous intersection. Ann Intern Med 2006;8:592–8.
      OpenUrl
    8. ↵
      1. Milano M,
      2. Ullius D
      . Designing powerful training. San Francisco, CA: Jossey-Bass Pfeiffer, 1998.
    9. ↵
      1. Rothwell WJ,
      2. Kazanas HC
      . Human resource development. A strategic approach. Amherst, MA: HRD Press, 1994.
    10. ↵
      1. Etienne W
      . Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press, 1998.
    11. ↵
      1. Phillips JJ
      . How much is the training worth? Training Dev 1996;4:20–4.
      OpenUrl
    12. ↵
      1. Levinson D
      . Hospital incident reporting system do not capture most patient harm. Department of Health and Human Services, Office of Inspector General. January 2012. OEI-06–09-00091. http://oig.hhs.gov/oei/reports/oei-06-09-00091.asp (accessed 6 Aug 2012).
    13. ↵
      Concept Systems Global [Computer Software]. Ithaca, NY: Concept Systems, Inc, 2011.
    14. ↵
      1. Trochim W
      . An introduction to concept mapping for planning and evaluation. Eval Program Plann 1989;12:1–16.
      OpenUrlCrossRefWeb of Science
    15. ↵
      1. Kane M,
      2. Trochim W
      . Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage Publishing, 2007.
    16. ↵
      1. Stoyanov S,
      2. Hoogveld B,
      3. Kirschner PA
      . Mapping major changes to education and training in 2025. JRC Technical Note European Communities Institute for Prospective Technological Studies 2010:JRC59079:2–52.
    17. ↵
      1. Eden C,
      2. Ackermann F
      . Making strategy. The journey of strategic management. London, UK: Sage Publications, 2002.
    18. ↵
      Concept System Core [Computer Software]. Ithaca, NY: Concept Systems, Inc, 2011.
    19. ↵
      1. Bartholomew DJ,
      2. Steele F,
      3. Moustaki I,
      4. et al
      . The analysis and interpretation of multivariate data for social scientists. London, UK: Chapman & Hall/CRC, 2002.
    20. ↵
      1. Dolmans DHJM,
      2. Wolfhagen IHAP,
      3. Van der Vleuten CPM,
      4. et al
      . Solving Problems with group work in problem- based learning: hold on to the philosophy. Med Educ 2001;5:84–9.
      OpenUrl
    21. ↵
      1. Hmelo-Silver CE
      . Problem-based learning: What and how do students learn? Educ. Psychol Rev 2004;3:235–66.
      OpenUrl
    22. ↵
      1. Van Merriënboer JJG
      . Training complex cognitive skills: a four-component instructional design model for technical training. Englewood Cliffs, NJ: Educational Technology Publications, 1997.
    23. ↵
      1. Van Merriënboer JJG,
      2. Kirschner PA
      . Ten steps to complex learning. Mahwah, NJ: Erlbaum/Taylor and Francis, 2007.
    24. ↵
      1. Brown J,
      2. Collins A,
      3. Duguid P
      . Situated cognition and the culture of learning. In: McLellen H, ed. Situated learning perspectives. Englewood Cliffs, NJ: Educational Technology Publications, 1996:19–44.
    25. ↵
      1. Brown JS,
      2. Duguid P
      . The social life of information. Boston, MA: Harvard Business School, 2000.
    26. ↵
      1. Ericsson KA
      . The influence of experience and deliberate practice on the development of superior expert performance. In: Ericsson, KA, Charnes N, Feltovich PJ, Hoffman RR, eds. The Cambridge handbook of expertise and expert performance. New York, NY: Cambridge University Press, 2006:683–703.
    27. ↵
      1. Spiro RJ,
      2. Jehng J
      . Cognitive flexibility and hypertext: theory and technology for the non-linear and multidimensional traversal of complex subject matter. In: Nix D, Spiro R.eds. Cognition, education and multimedia. Hillsdale, NJ: Erlbaum, 1990:163–205.
    28. ↵
      1. Merrill D
      . First principles of instruction. Educ Technol Res Dev 2002;50:43–59.
      OpenUrlCrossRef
    29. ↵
      1. Kirkpatrick DL,
      2. Kirkpatrick JD
      . Evaluating training programs. 3rd edn. San Francisco, CA: Berrett-Koehler Publishers, 2006.
    30. ↵
      1. Chiabura DS,
      2. Marinova SV
      . What predicts skill transfer? An exploratory study of goal orientation, training self-efficacy and organizational supports. Int J Training Dev (IJTD) 2005;9:110–23.
      OpenUrlCrossRef
    31. ↵
      1. Gaudine A,
      2. Saks AM
      . A longitudinal quasi-experiment on the effects of post training transfer intentions. Human Resource Dev Quart 2004;15:57–76.
      OpenUrlCrossRef
    32. ↵
      1. Raybould B
      . Building performance centered web-based systems, information systems, and knowledge management systems in the 21st century. In: Rossett A, ed. The ASTD E-learning handbook. Best practices, strategies, and case studies for an emerging field. New York, NY: McGraw-Hill, 2002:338–53.
    33. ↵
      1. Rosas SR,
      2. Kane M
      . Quality and rigor of the concept mapping methodology: a pooled study analysis. Eval. Program Plann 2012;35:36–245.
      OpenUrl
    34. ↵
      1. Turner CW,
      2. Lewis JR,
      3. Nielsen J
      . Determining usability test sample size. In: Karwowski W, ed. International Encyclopedia of Ergonomics and Human Factors. Boca Raton, FL: CRC Press, 2006:3084–8.
    35. ↵
      Concept mapping methodology biography and recent publications. Examples of concept mapping projects. Concept System incorporated 2012. http://www.conceptsystems.com/files/all/bibliography_examples_of_cm_projects_5.pdf (accessed 9 Sep 2012).

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors SS: substantial contributions to conception and design, acquisition of data, analysis and interpretation of data; drafting the article (all sections), revising it critically; and final approval of the version to be published. HB: substantial contributions to design, acquisition of data and interpretation of data; revising the article critically; and final approval of the version to be published. OG: substantial contributions to acquisition of data and interpretation of data; revising the article critically; and final approval of the version to be published. MvdK: substantial contributions to acquisition of data and interpretation of data; drafting the article (sections: Background, Methods and Discussion); and final approval of the version to be published. WK: substantial contributions to acquisition of data and interpretation of data; drafting the article (sections: Background, Methods and Discussion); and final approval of the version to be published. HD: substantial contributions to acquisition of data and interpretation of data; drafting the article (sections: Background, Methods and Results); and final approval of the version to be published. PB: substantial contributions to design, acquisition of data and interpretation of data; revising the article critically; and final approval of the version to be published.

    • Funding The study was supported by a grant from the European Union, the 7th Framework Programme of the European Commission (FP7-HEALTH-F2-2008-223409). The study sponsor had no role in the study design, collection, analysis, interpretation of the data, or any other contribution.

    • Competing interests All authors have completed the Unified Competing Interest form in http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Open Access This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/