Article Text
Abstract
Objectives: To determine the extent to which general practitioners (GPs) were aware of a recent significant event and whether a structured analysis of this event was undertaken to minimise the perceived risk of recurrence.
Design: Cross sectional survey using a postal questionnaire.
Setting: Greater Glasgow primary care trust.
Participants: 466 principals in general practice from 188 surgeries.
Main outcome measures: GPs’ self-reported personal and practice characteristics, awareness of a recent significant event, participation in the structured analysis of the identified significant event, perceived chance of recurrence, forums for discussing and analysing significant events, and levels of primary care team involvement.
Results: Four hundred and sixty six GPs (76%) responded to the survey. GPs from single handed practices were less likely to respond than those in multi-partner training and non-training practices. 401 (86%) reported being aware of a recent significant event; lack of awareness was clearly associated with GPs from non-training practices. 219 (55%) had performed all the necessary stages of a structured analysis (as determined by the authors) of the significant event. GPs from training practices were more likely to report participation in the structured analysis of the recent event, to perceive the chance of this event recurring as “nil” or “very low”, and to report significant event discussions taking place.
Conclusions: Most GPs were aware of a recent significant event and participated in the structured analysis of this event. The wider primary care team participated in the analysis process where GPs considered this involvement relevant. There is variation in the depth of and approach to significant event analysis within general practice, which may have implications for the application of the technique as part of the NHS quality agenda.
- adverse events
- significant event analysis
- clinical audit
- general practice
Statistics from Altmetric.com
Significant event analysis (SEA) is a relatively new method of clinical audit that is now widely promoted in primary health care for various professional1 and organisational reasons.2,3 It differs from conventional criterion based audit primarily because it requires practitioners to use a qualitative rather than a quantitative approach to monitoring and improving patient care. The contemporary method of SEA is based on a combination of single case review—traditionally practised by many healthcare professionals and teams—and the philosophy and principles that underpin the critical incident technique.4 This provides the SEA method of audit with a simple structure to follow and a degree of scientific rigour that is not necessarily present during largely informal case review discussions.5 Although single case review as an educational activity is encouraged, it has been subject to criticism from within the medical profession when used as a proxy for regular “audit” of patient care.6
The potential of SEA as a method of audit in general practice was first tested in the mid 1990s when the relative strengths and weaknesses of this approach were compared with conventional criterion based audit.7 The conclusions reached suggested that SEA was a feasible and acceptable technique which, with a few caveats (such as good team dynamics and facilitation skills), could be successfully applied in different types of general medical practices as a complementary technique to the more familiar quantitative method of audit.
SEA has also been recommended as a method for enhancing patient safety and risk management and in facilitating adverse incident reporting.8,9 The multifaceted potential of SEA makes it attractive to individual practitioners, healthcare teams, and NHS organisations as an educational learning mechanism and as an instrument for implementing and monitoring change. Applied in a consistent, rigorous, and structured manner, it may make a substantial contribution to the ongoing clinical governance agenda10 and the work of the new National Patient Safety Agency (NPSA).11 However, compared with the use of similar “analysis” techniques to inform on safety and error—for example, in the aviation industry12—the application of SEA in general practice is in its infancy. Questions about the complexity and understanding of the technique, the level of expertise required, and the adequacy of the analysis all merit further research.
In addition, the impact of SEA on patient care, practitioner behaviour, and healthcare practices has yet to be adequately evaluated. An initial qualitative study concentrated on the perceptions of SEA among members of the healthcare team and in developing recommendations to improve the process.13 Other research has shown SEA to be well supported by the staff involved, who thought it worthwhile in making improvements to patient care. Potential barriers to participation such as topic selection, confidentiality, and demonstrating outcomes have been identified.14 Two studies involving the evaluation of peer assessed SEA reports voluntarily submitted by GPs for educational purposes have also recently been published.15,16 Both were concerned with the quality of SEA reports and the factors involved in gaining a successful peer assessment. Further work has shown SEA to be a useful tool for reflective practice,17 improving teamwork in primary health care,18 involving patients in assessing the quality of their care,19 and in developing clinical governance in the prison healthcare service.20
The relevance of addressing significant events and the ability to apply the SEA technique has been brought into sharp focus as a professional issue for all GPs with the impending system of appraisal and revalidation for all UK doctors. This will require GPs periodically to provide documentary evidence of their involvement in identifying and analysing significant events as part of demonstrating their fitness to practice. At present SEA is a conditional requirement in retaining GP training practice status in Greater Glasgow. However, little is known about those involved in the identification and analysis of significant events in general practice and how this is approached and applied. Indeed, the important issue of why an event is deemed “significant” enough to be identified for analysis (or not) requires in depth study.21 More evidence is also required to gain a better understanding of these and other factors if SEA is to be established as a worthwhile and effective technique. A major difficulty in doing this is the complexity and uncertainty that characterises much of day to day general practice, making it a difficult area to research adequately and leaving many relevant questions unanswered.22 Another difficulty is the confusing terminology applied in this area. For example, the terms “adverse event”, “critical incident”, “error”, “near miss”, and “significant event” appear to be used arbitrarily and interchangeably in relation to risk and safety issues in primary care.21
We conducted an exploratory survey to determine the professional and practice characteristics of a group of GPs who reported being aware of a recent significant event associated with their practice. We aimed to assess whether a structured analysis of this specific event was undertaken and whether this impacted on the perceived risk of the event recurring. In addition, we ascertained how significant events are discussed in practice, the forums used for analyses of these events, and the level of involvement by various members of the primary care team. A further objective was to determine if there were any reported differences in the responses from GPs in training and non-training practices.
METHODS
Definition of a significant event
A broad definition of a significant event was provided on the front page of the questionnaire to assist those GPs who may have been unfamiliar with both the term and its meaning (box 1).
Box 1 Definition of a significant event
-
Any event thought by anyone in the team to be significant in the care of patients or the conduct of the practice....7
Data collection
A postal questionnaire was sent to all 617 principals in 222 general practices in Greater Glasgow primary care trust. Non-respondents were sent two additional reminders and the survey was completed in January 2002. The questionnaire was devised by the authors and pre-tested on five departmental colleagues who are part time GPs. A pilot survey involving 12 GPs based in the eastern Glasgow Local Health Care Cooperative (LHCC) was undertaken in October 2001. Minor amendments were made to the design of the questionnaire based on the feedback received from pilot participants.
Data were collected on GPs’ professional and practice characteristics, reported awareness of a recent significant event, whether a structured analysis of the event was undertaken, and the perceived risk of this event recurring. Additional data on the discussion forums for significant events and the composition of the meetings were also collected.
Structured analysis of a significant event
Respondents were asked to indicate if certain activities were carried out during their analysis of the recently identified significant event. We adapted the format used in previous work to determine if a structured analysis of this significant event had taken place.15,16 If all of the relevant stages of analysis outlined in box 2 were reportedly carried out by respondents, then we agreed a structured analysis of the significant event in question had been achieved. Because of the work involved this could not be independently verified, nor could the overall effectiveness of the analysis being measured.
Box 2 Stages involved in the structured analysis of a significant event
-
A good description of the actual event is obtained.
-
A clear reason for the event occurring is sought.
-
Insight into the event is demonstrated by those involved.
-
A change in practice is considered but not implemented, or
-
A change in practice is implemented.
-
An analysis of this significant event is formally documented.
Statistical methods
The denominator in this study was the principal in general practice. Although more than one principal from the same general practice could have returned the questionnaire, with the exception of training practice status, number of partners and list size, no other data linking individual GPs with their surgery were recorded.
Characteristics of the GPs including sex, time as a principal, and professional status were summarised. The characteristics of the practices to which the GPs were attached were also summarised but were not linked; thus, for a practice returning the experience of, for example, three GPs, the same data on practice characteristics would appear identically three times.
Statistical associations between the proportion of GP respondents from training practices, multi-partner and single handed non-training practices with 95% confidence intervals (CI) were calculated. The frequency of the occurrence of the different stages of the previously defined “structured analysis of a significant event” was tabulated and the outcome “Did the GP undertake a structured analysis?” derived. Univariate logistic regression was used to explore the relationship between a number of covariates which it was felt might be important predictors for this outcome—for example, time as GP, training practice status, number of practice partners, and perceived risk of significant event recurring. The odds ratios (ORs) for the stated unit of comparison or change—for example, time as a GP ⩽5 years or training practice versus non-training practice—were calculated (with 95% CI) and the associated p value for the significance test of unity of the OR. A multivariate logistic regression model was then fitted to adjust for all of these factors simultaneously.
Fisher’s exact tests were calculated to determine if there was an association between the types of significant event meetings held, levels of primary care staff involvement, and training practice status. Caution should be exercised when interpreting the p values for these comparisons because the analyses make no adjustment for the possible clustering of responses from within a practice, nor do they adjust for the other practice characteristics.
All analyses were performed using SAS 8.2 for Windows NT. No adjustment has been made for multiple comparisons.
RESULTS
Response rate and characteristics of respondents
Of the 617 GPs in 222 surgeries, 466 (76%) from 188 surgeries (85%) completed and returned the questionnaire. A higher response rate was achieved from training practices (162/186, 87%) than from multi-partner non-training practices (273/376 (73%); difference 14.5%, 95% CI 7.9 to 21.1) or single handed non-training practices (31/55 (56%); difference 30.7%, 95% CI 16.8 to 44.7). Table 1 summarises the characteristics of respondents and their practices.
Characteristics of GP respondents (n = 466) and their practices
Awareness of a recent significant event
401 of the 466 principals who responded (86%) reported being aware of a recent significant event associated with their practice, while 65 (14%) indicated that they were unaware. GPs who reported being unaware of a recent significant event were more likely to be based in non-training practices (adjusted OR 4.00 (95% CI 1.71 to 9.35), p<0.001).
Structured analysis of the recent significant event and perceived risk of recurrence
Table 2 shows that most GPs who were aware of a recent significant event indicated to varying degrees that the individual stages involved in the analysis of a significant event were undertaken; 219 (55%) reported participating in all of the stages of a structured analysis as previously described (box 2). Both univariate and multivariate logistic regressions (table 3) show that GPs from training practices were much more likely to undertake a structured analysis than those based in non-training practices (adjusted OR 0.25 (95% CI 0.15 to 0.41), p<0.0001). In addition, GPs who indicated that the risk of recurrence of a significant event was either nil or very low were more likely to have undertaken a structured analysis than those who perceived a higher risk of recurrence (adjusted OR 2.19 (95% CI 1.18 to 4.07), overall p = 0.014).
Stages of structured analysis of significant event undertaken and perceived chance of recurrence among GPs who were aware of a recent significant event (n = 401)
Univariate and multivariate logistic regressions for subjects who had been aware of a recent significant event and who performed a structured analysis (n = 219/401)
Team involvement in significant event analyses and the forums used
A few respondents (5%), almost exclusively from non-training practices, reported that significant events were never discussed in their practices (p<0.001). GPs from training practices were much more likely to report that the discussion of significant events took place at partners’ meetings (p<0.001), practice team meetings (p<0.001), or dedicated audit/significant event meetings (p = 0.013) than colleagues from non-training practices (table 4).
Discussions, forums, and staff composition of significant event meetings by training practice status
The majority of GPs (94%) reported always being involved in the discussion of significant events associated with the practice compared with 58% of practice managers and 31% of practice nurses. Fewer than 10% of respondents reported that other health professionals such as health visitors and district nurses were always involved in the discussion of significant events within the practice. However, much greater levels of involvement in significant event discussions were reported for these staff groups only if required by the practice partners.
DISCUSSION
The main findings from this study provide a measure of the variation in the awareness and analyses of significant events among GPs. Most were aware of a recent significant event associated with their practice, although it may be of concern that a few were not. Of those who recalled a recent significant event, a full structured analysis was reportedly undertaken by just over half. Analysis of this significant event in a structured manner was associated with a decrease in the perceived risk of the event recurring. Most GPs formally discussed significant events at various types of meetings in the practice, although only a small number have dedicated SEA meetings. Members of the primary care team only tend to be involved in these discussions when considered appropriate by GPs, although the practice manager has much greater involvement than other staff.
Our survey generated a good response rate from the GPs involved. However, a number of limitations are associated with this type of descriptive cross sectional study. There may have been response bias because GPs from training practices were more likely to respond and single handed GPs less so. This may reflect the knowledge and experience of the subject matter of those from the training environment. We also rely on respondents to self-report, which potentially limits the reliability of the data because there is no independent means of verifying responses. Caution should therefore be exercised when extrapolating these data for general purposes. Traditional SEA allows for significant events that highlight good practices to be identified for discussion;7 however, respondents taking this approach may have had difficulty undertaking our version of a “structured analysis”. We also provided one definition of a significant event but acknowledge that respondents might interpret this and their own concept of “event analysis” differently, which may again influence the reported data. If we compare the time and resources put into the root cause analysis of individual events in other industries, it may be that SEA in general practice is applied superficially, particularly as there is as yet no standardised method of event investigation or data collection in general practice.12 This may also reflect the complexity and uncertainty that characterises much of primary health care.22
A small number of GPs reported being unaware of a recent significant event, although if extrapolated nationally these figures would be quite substantial. It is possible that some of these doctors had genuinely not experienced a recent significant event. However, it seems more likely that inexperience and lack of knowledge are more plausible explanations for them being unaware. Another possibility is that some of these doctors lack insight in this area and may find it difficult to imagine they are ever involved in significant events. The introduction of appraisal for all doctors may provide an opportunity to address this particular concern.
Most GPs reported undertaking a recent SEA. Previous informal estimates of SEA participation rates among GPs and practices have ranged from 6% to 20%.23,24 However, it is also clear that a substantial minority indicated that a full structured analysis of the recent event did not take place. Worryingly, one fifth of respondents who attempted an analysis perceived the risk of the event recurring as moderate to very high. Merely identifying and discussing the event may be insufficient if lessons are to be learned and changes considered or introduced. The main purpose of SEA is that it goes further than simple discussion and should show that insight into the event has been established and action taken to minimise recurrence. The formal documentation of the analysis highlights that the significant event is probably more likely to be taken seriously by the practice and that a permanent record of it exists—especially if it is linked to a complaint or potential litigation.
The necessary stages of a structured analysis we outline are open to further debate and scrutiny. We also recognise that not every SEA will involve the implementation of change. However, the limited research available indicates that all of these factors are normally present when an effective analysis of a significant event is performed.15,16 Further education for GPs has already been highlighted as an issue if SEA is to be applied effectively and is to have an impact on practitioner behaviour and improving patient care.16 Our findings also appear to confirm that a number of GPs may not be applying SEA as robustly as other colleagues.
Most practices appear to discuss significant events at a specified internal forum, with a minority favouring informal discussions only. Informal or unstructured discussion of significant events may be ineffective for that very reason.6 GPs are present at most significant event discussions, but there appears to be wider team involvement where it is felt appropriate, with the practice manager being most involved. This is not surprising given the administrative nature of many significant events.15 It is interesting that SEA is promoted as a key team activity with the underlying assumption that non-medical staff should be involved in most significant event discussions.7,10,13 However, our evidence appears to suggest that GPs are selective about which staff members attend these meetings.
Key messages
-
A cross sectional survey of 466 GP principals from 188 surgeries showed that a substantial majority were aware of a recent significant event associated with their practice.
-
Over half reported undertaking a structured analysis of this significant event.
-
Analysing the significant event in a structured manner decreased the perceived risk of the event recurring.
-
Training practice GPs were reportedly more likely to undertake a structured analysis of the identified significant event than non-training practices.
-
In most practices significant events are discussed in a formal group meeting and involve relevant members of the primary care team, when required.
-
Significant event analysis (SEA) may be more established in general practice than previously reported.
-
SEA is widely encouraged in primary health care and is now a requirement of revalidation. However, there is variation in how SEA is undertaken and applied which may lead to inconsistency and ineffectiveness in addressing patient care and safety issues.
The local training practice environment appears to be an important factor in the participation and analyses of significant events. This finding is not unexpected given that participation in SEA is a training practice requirement in Greater Glasgow. It is also known that GP trainers participating in an educational peer assessment model of SEA are more likely to undertake satisfactory analyses than non-trainers.16 The personal ability of the trainer in this area together with the compulsory focus on quality improvement required of a training practice may directly influence the development of SEA within the practice and the involvement of the primary care team.
A potential difficulty is the small proportion of GPs (and, conceivably, practices) not engaging in the discussion or analyses of significant events, even on an informal basis. The reasons for this merit further exploration but, given that the finding is almost exclusively related to non-training practices, it may point to an educational issue for the practitioners involved, especially with the imminent introduction of appraisal and revalidation. A clinical governance issue may also have been identified for those local primary care organisations wishing seriously to address potential risk management and patient safety concerns.
We have shown that the SEA technique is reportedly practised by a small majority, but with mixed results in terms of application and potential impact. There is a potential concern that GPs who fail to address significant events or who are ineffective with their analyses might be missing opportunities to improve patient care and safety. Thus, an important aspect of the clinical governance agenda may not be adequately addressed. If confirmed by further research, an educational strategy may be required to maximise the application and effectiveness of the technique.
Setting aside the need for research into the underlying causes of significant events, more evidence is required into GPs’ understanding and interpretation of these events and the effectiveness of their analyses. In addition, more research into the impact of SEA on practitioner behaviour, the healthcare quality agenda, and how to optimise the sharing of best practice in this area is needed. Ultimately, this will inform the robustness or otherwise of SEA as a tool to enhance patient care and safety.