Article Text
PDF
Abstract
Background Patients are a valuable source of information about ways to prevent harm in primary care and are in a unique position to provide feedback about the factors that contribute to safety incidents. Unlike in the hospital setting, there are currently no tools that allow the systematic capture of this information from patients. The aim of this study was to develop a quantitative primary care patient measure of safety (PC PMOS).
Methods A two-stage approach was undertaken to develop questionnaire domains and items. Stage 1 involved a modified Delphi process. An expert panel reached consensus on domains and items based on three sources of information (validated hospital PMOS, previous research conducted by our study team and literature on threats to patient safety). Stage 2 involved testing the face validity of the questionnaire developed during stage 1 with patients and primary care staff using the ‘think aloud’ method. Following this process, the questionnaire was revised accordingly.
Results The PC PMOS was received positively by both patients and staff during face validity testing. Barriers to completion included the length, relevance and clarity of questions. The final PC PMOS consisted of 50 items across 15 domains. The contributory factors to safety incidents centred on communication, access to care, patient-related factors, organisation and care planning, task performance and information flow.
Discussion This is the first tool specifically designed for primary care settings, which allows patients to provide feedback about factors contributing to potential safety incidents. The PC PMOS provides a way for primary care organisations to learn about safety from the patient perspective and make service improvements with the aim of reducing harm in this setting. Future research will explore the reliability and construct validity of the PC PMOS.
- Primary care
- Patient safety
- Surveys
Statistics from Altmetric.com
Introduction
Internationally, primary care is the first point of contact with the healthcare system for the majority of patients. There is a misconception that primary care is a low technology environment where safety is not considered to be problematic. A growing body of evidence suggests that this is not the case. For example, patient safety incidents have been predicted to occur in approximately 2% of consultations1 with other reports suggesting that it is more likely to be around 10%.2 There has been a range of error types identified with the majority of errors in primary care falling within the categories of medication errors,3 diagnostic errors4 and communication errors.5 ,6
Few studies are available to help us understand how to improve safety in primary care.7 One of the ways to address this gap is to involve patients. Indeed, there is an emergent consensus that patients are able to identify potential errors in secondary care,8 and we are beginning to learn more about this in primary care.9–11 Patients have a different perspective on safety and can provide insights to prevent errors that health professionals may not have known about.12 ,13
Not only can patients directly comment on error and patient harm, they can also identify factors that contribute to patient safety incidents.11 ,14 A tool to collect patient feedback on the factors contributing to safety incidents has been developed in secondary care.14 ,15 This tool is one element of an intervention currently being assessed in a large trial investigating patient involvement in patient safety in hospitals.16 The patient measure of safety (PMOS) is a theory and evidence-based tool14 ,15 derived from James Reason's Swiss Cheese model of accident causation,17 and also from a systematic review of the literature, which resulted in the Yorkshire Contributing Factors Framework (YCFF).18 The YCFF outlines 20 factors contributing to safety incidents, which range from active failures to latent external factors in hospital settings. Some of the domains of the framework include communication systems, patient factors, physical environment, external policy context, and equipment and supplies.18
Patient feedback in primary care tends to focus on experience and satisfaction,19 or on after event reporting and incident disclosure.20 Systematic reviews have shown that there are some existing patient-reported instruments to measure patient safety in primary care,21 ,22 but no tool has been explicitly developed that comprehensively measures factors contributing to patient safety incidents. Furthermore, only a minority of these tools have an underlying theoretical foundation to support their development and use. Primary care health professionals could use patient feedback to improve safety in the same way health professionals are doing within hospital settings.16 In addition, evidence has shown that general practitioners are willing to embrace patient feedback but are unsure about how to use this information effectively.23 A theory-driven, evidence-based tool is needed to capture information about the potential ‘latent’ weaknesses in primary care, which could contribute to future patient safety incidents. Therefore, the aim of this study was to develop a primary care patient measure of safety that can be used as a basis for proactively managing safety and service improvement.
Methods
The questionnaire was developed in two stages. Stage 1 comprised domain and item generation of the questionnaire, and stage 2 involved testing the face validity with patients and primary care staff (figure 1 illustrates the methods).
Flow diagram outlining questionnaire development.
Stage 1: consensus process for domain and item generation
Procedure
A modified Delphi technique24 was used to reach consensus on the domains and items to include in the questionnaire. This technique was chosen as it has been shown to be an effective method for gaining consensus and has been used successfully to develop quality indicators in primary care.25 Modification included the addition of two round-table discussions with panel members after rating and review rounds were completed. The round-table discussions applied a nominal group technique (NGT)26 approach to agree on the final domains and items to be included in the draft version of the questionnaire. NGT is a group process that usually involves sharing and discussion of reasons for the choices made by each group member.
The modified Delphi process consisted of three rounds of rating and review over a 3-week period and the addition of two discussion round tables with panel members. The discussions were held 2 weeks apart. They were used to elicit any further contributions from the panel members and to facilitate final consensus on the questionnaire domains and items prior to face validity testing. Modification of the Delphi technique was incorporated into the study design as discussion among panel members was required to generate different ideas about the subject matter and reduce group conformity that can influence the quality of findings. The social interaction during the round-table discussions was thought to facilitate group acceptance and ownership of the results.27 The combination of two consensus methods has been advocated by contrast with a Delphi process that has preceded NGT for initial item generation.28 ,29
The multidisciplinary expert panel contained members with extensive experience and knowledge in the field of patient safety in primary care from Australia, the USA and the UK. Panel members’ backgrounds included general practice academics (n=4); a consumer representative who is the CEO of an organisation that embodies >50 Australian consumer and advocacy organisations for people with chronic illness (n=1); nursing, sociology, health systems and safety researchers (n=4); and a representative from the Australian Commission on Safety and Quality in Health Care (n=1).
Three sources of information were provided to the expert panel members over a 3-week period. These included the domains and items from the PMOS questionnaire for hospital use,14 ,15 a brief description of the themes and subthemes from previous enquiry of the research team investigating patients’ views of safety in general practice9 ,11 and a brief description of the themes from the literature about threats to patient safety in primary care.3 ,30–38
The three sources of information were presented electronically to panel members in Excel spreadsheets. Modification to the Delphi rating scale has been used successfully in previous research.29 For each source of information, panel members were asked to simply answer ‘yes’, ‘no’ or ‘unsure’ as to whether they considered the domain or item to be a contributory factor to patient safety in primary care. For each response, panel members were also asked to provide comments or justification at a level they felt necessary for their response. The categorical response options were a further modification from the traditional Delphi rating scale of 0–9. This modification was made to give panel members more discrete and clearer categories to choose from. It was thought to assist with or prompt discussion about the ‘unsure’ domains or items during the round tables with panel members. A measure of dispersion or median score was not required for feedback to panel members as each domain and item was discussed at length during the round tables where consensus was reached. Modification to the Delphi rating scale has been used successfully in previous research.29 All panel members were instructed to provide responses anonymously via email to the facilitator (ALH). Anonymous reporting aimed to allow panel members to freely express their opinions and avoided the potential dominance by eminent or highly opinionated panel members.
Analysis
Responses were analysed by the facilitator and presented back to the panel during the first round-table discussion. During the round table, panel members examined the results and reached consensus about which domains to include in the questionnaire. The facilitator used these results to generate example items for each domain where there was not already an existing PMOS questionnaire item.14 ,15 During the second round-table discussion, panel members considered the example questionnaire items and refined these until consensus was reached.
Stage 2: face validity testing
Sample
There were two participant groups for this stage of the study: (1) patients and (2) healthcare professionals, managers and administrators from primary care settings within Australia and the UK. Primary care patients and staff were purposively sampled from networks known to the research team. The network from which the sample was selected included participants with past experience and knowledge of the research area and who had formal established partnerships with the relevant universities undertaking the study. These patients and staff were selected from various patient demographics and different professional groups with the aim of increasing diversity of the sample and to ensure that the questionnaire would be valid and usable across countries.
Procedure
A ‘think aloud’ methodology was employed to test the face validity of the draft version of the questionnaire.39 Twenty ‘think aloud’ interviews were conducted with patients (n=11) and staff (n=9). ‘Think alouds’ were arranged at a time and place convenient for the participants. Staff included general practitioners, practice nurses, community pharmacists, practice manager and administration staff. As staff and patients are potential users of the survey, it was important to gain both their views.
The ‘think aloud’ method involved asking participants to talk aloud about their thoughts and feelings as they read and decided how to respond to each question in the draft version of the questionnaire.39 ,40 Participants were also asked questions following the ‘think aloud’ process, which included the perceived barriers to questionnaire completion, time taken to answer the questionnaire and questionnaire format.
All interviews were digitally recorded and transcribed verbatim. Interviews were between 20 min and 1 h in length.
Participants were also asked to complete a demographic questionnaire as part of the consent process.
Analysis
Revisions were made to the questionnaire following the ‘think aloud’ procedure. A smaller working group (ALH, SJG, JKOH, JF, JJ and JAD) considered the findings from the interviews to reach consensus on the items to be included in the final version.
Two researchers (AH and SG) independently reviewed all the transcripts. Patient transcripts were analysed to identify when participants were able to understand and respond to the questions on the draft questionnaire. This was a dichotomous assessment consisting of ‘yes’ or ‘no’ for ability to understand and/or respond to the question. Staff transcripts were analysed to identify the relevance and importance of each item for practice. Any particular issues with questions from both patients and staff were also considered during the analysis.
Responses to questions following the interviews were collated and reviewed. These, together with the interview transcripts, were used to make revisions to the draft questionnaire.
The final version of the questionnaire was assessed for readability using the Flesch Reading Ease and Flesch-Kincaid Grade level.41
Results
Stage 1
Three rounds of rating and review and two discussion meetings with expert panel members resulted in a draft questionnaire with 24 domains and 77 questionnaire items. The domains and items are listed in online supplementary appendix 1.
Stage 2
Understanding and responding
Results from the interviews with patients are presented in online supplementary appendix 2. This table shows the number of patients who could understand (comprehension of the statement) or respond (experience of the statement), and both understand and respond (both comprehend and experience) to the questionnaire items. Generally, patients could understand and respond to most items. Demographics of patients and general practice staff are provided in table 1.
Demographic information of participants (patients n=11 and staff n=9)
Time to complete the questionnaire
On average, participants took 15 min to complete the questionnaire, apart from one participant with low literacy level who took longer (40 min).
Strengths of questionnaire and barriers to completion
Participants felt that patients would be willing and able to complete the questionnaire. Staff also agreed that this type of tool would be useful for improving safety in primary care. Both patients and staff identified some potential barriers to completing the questionnaire, mostly concerning the length of the questionnaire and the level of attention required to remain engaged for a long period of time. Patients were specifically apprehensive about their responses being attributed to them individually. Other patients thought that elderly people and those with low literacy levels would need external facilitation to complete the questionnaire. Some patient participants used the ‘neither agree or disagree’ option when the statement was not applicable to them or they did not know how to respond. Negatively worded items were also problematic for some patients but not staff. Some participants were unfamiliar with the terminology used in items, for example, ‘adherence’ and ‘after hours’. Responses were similar between participants from different cultural groups.
Questionnaire format and implementation
Participants thought that providing a range of formats of the questionnaire would be beneficial. Participants believed that younger people would prefer the questionnaire to be online or electronic and older people would prefer a paper-based questionnaire or require external facilitation. There were various responses about where and when to complete the questionnaire. Many believed the time in the waiting room prior to their consultation would be an ideal time. Others felt it would be better to complete the questionnaire at home because they would be more comfortable and have more time in a neutral setting. They acknowledged that getting these questionnaires back would be problematic. The intended time for patients to complete the questionnaire would be just after a recent primary care visit.
Developing the final version of the questionnaire
Based on the findings of the interviews described above, the draft questionnaire was revised. The expert panel (AH, SG, JKOH, JF and JD) discussed each of the 77 items with the aim of strengthening the questionnaire and reducing the length.
In total, 50 items were retained from the draft questionnaire. The wording was changed for 13 items to improve clarity, based on patient and staff feedback from the interviews (see online supplementary appendix 1). Questions related to a specific primary care professional were also grouped together in the final version for ease of completion. The remaining questions were randomly ordered to remain true to the original format of the PMOS questionnaire for hospital settings where there was little effect of question order shown in participant responses.14 The random order of items was also thought to reduce participant fatigue or confusion when similarly grouped items appear together on the questionnaire. A mix of both positive and negatively worded items was also included in the questionnaire to minimise the possibility of acquiescent response bias and assist with future reliability and construct validity testing.42
In the draft questionnaire, there were 24 domains. Some of these were collapsed into existing domains or renamed to be consistent with PMOS domains identified from hospital settings,15 and from the YCFF on which the PMOS is based.18 This resulted in 15 domains. The ‘Access to resources’ domain was collapsed into ‘Access’; ‘Coordination of care’ and ‘Provider performance’ domains were collapsed into a renamed domain ‘Task performance’; ‘Medicare system and structure’ was renamed ‘External policy context’; ‘Staff training’ was renamed ‘Training and education’; ‘Team-work’ was renamed ‘Team factors’; and ‘Type and layout of practice’ was renamed ‘Physical environment’ (see online supplementary appendix 3).
Readability of the questionnaire
The Flesch Reading Ease of the final version of the questionnaire was 59.0. This means that the questionnaire is easily understandable by 13-year-old to 15-year-old students. Readability of the questionnaire using the Flesch-Kincaid Grade level was 7.8. This means that the text is expected to be understandable by an average student in the seventh grade (ages 12–13).
The final version of the questionnaire is the primary care PMOS (PC PMOS) (see online supplementary appendix 4).
Discussion
Patient safety in primary care is a growing field, and patients are emerging as a potential source of information about safety in this setting, but no evidence-based or theory-driven tool currently exists that captures this information from patients.
This paper describes the development of the first patients’ questionnaire measuring factors contributing to safety that has been developed for the primary care setting. The PC PMOS could be a tool to provide feedback on patients’ understanding of the factors contributory to safety incidents in primary care. This tool is a resource to further the involvement of patients in safety in primary care.43 The development of this tool meets a gap in the body of evidence, which suggests that only 3% of all patient safety in primary care tools use the patients’ perspective.22
The PC PMOS is innovative as patients have traditionally had limited opportunity to participate in prevention of harm other than in more formal ways of identifying risk, such as incident reporting.8–10 This tool goes one step beyond current practice, which at best limits or ignores the patients’ potential for involvement in safety in primary care.
There were four unique primary care domains that were additional to the original PMOS questionnaire for secondary care settings. The unique domains were continuity of care, external policy context, primary–secondary interface and referrals. These domains reflect the structural diversity and broader scope of primary care,5 and hence the contributing factors to safety incidents and the errors that occur are likely to be different from hospitals.44 Given these differences, there is clearly a need to develop tools that are specific for the primary care context.
The PC PMOS contains a balance of domains that are considered to be both error producing and latent contributory factors to safety incidents, unlike its secondary care counterpart.14 ,15 ,17 The PMOS for hospital settings contains domains that are mainly centred on local working conditions (error producing) factors, with the exception of the cross-cutting ‘communication’ domain.14 ,15 These latent domains are important because it demonstrates that patients are aware and have understanding of a wide range of safety factors that contribute to their care.
As measurement and monitoring of patient safety is a current challenge in healthcare,45 the PC PMOS tool may be diagnostic, useful and practical for primary care staff to undertake safety improvement work, as well as a means of monitoring changes over time. We anticipate that primary care practices will use the survey results from the PC PMOS to identify areas of weakness and to plan continuous quality improvements, much like they do in quality improvement collaboratives46 and hospital applications of the tool that use action planning cycles of activity.16 Patient feedback collected on the PC PMOS may be regarded as the extra piece of the patient safety intelligence ‘jigsaw’.8 ,47 This tool could enhance or complement current data collection methods used in primary care to identify and prevent safety incidents such as significant event or root-cause analysis already required for accreditation.48 Patient and staff feedback, and other data sources such as complaints, will together provide a more comprehensive picture of patient safety.
Primary care staff report that they struggle to make changes based on patient survey feedback alone,49 and qualitative data can provide important contextual information that is in some cases a more preferable option for staff.50 ,51 The PC PMOS was intentionally designed with free text space for patients to provide qualitative responses to further elaborate each item when necessary. The PC PMOS alleviates some of these challenges identified by primary care staff who use patient experience surveys,49 as this tool specifically focuses on areas of safety that are not only important to patients, but the contributory factors are derived from theory and the literature, and can therefore impact on practice.
There are a number of potential barriers to implementation of the PC PMOS. Recruitment of patients is always a major barrier to data collection. Our findings suggested that patients preferred to receive the tool in a variety of different formats that may increase response rates. Particularly vulnerable patient groups may experience difficulties completing the questionnaire. For example, those with low literacy, a visual impairment, physical or other disabilities, and the elderly or very young people may experience problems when voicing concerns about the safety of their care. This may impact the use and usefulness of the tool as feedback from these vulnerable groups is important because they are considered the most at risk groups for safety incidents.52 In those cases, we recommend external facilitation, although we recognise the challenges of how to operationalise this in practice. Patient peers and volunteers have been used successfully for patient data collection in hospitals and may be an avenue for exploration in primary care.53 We addressed participants’ concerns about the length of the questionnaire that was reduced during expert panel meetings, but we were still able to retain sufficient domains and items required for further validation and testing. Specific items on the PC PMOS relating to cost of services or access may not be relevant to all primary care contexts, and we recommend the removal or adaption of these items where necessary.
Strengths and limitations
The major strength of this study is that both inductive and deductive approaches were undertaken to be as inclusive as possible when developing the questionnaire domain and items. Three sources of information were used to create the content of the tool. We recognise that not all factors contributing to patient safety incidents are included in the tool. Practitioners should use the PC PMOS in conjunction with other safety measurement tools such as significant event analysis or patient complaints. Furthermore, while the face validity testing was conducted on a small number of participants, these were from two countries. The sample was predominantly female, which is a direct result of the purposive sampling methodology employed. There are greater proportions of females working in the healthcare industry,54 and women are more likely to attend primary care55 and respond to surveys than men.56 Further work will test the reliability and validity of the questionnaire on a larger scale with a more representative sample of the general population. The implications for using a modified Delphi process for the development of the PC PMOS were considered to be beneficial to the study overall and produced a strong tool that is based in both the evidence and expert opinion. The inclusion of panel members in round-table discussions was particularly useful for achieving consensus on the domains and items. We recognise that the sample size and composition of the expert panel members are different from the traditional Delphi processes that usually comprise large randomly sampled populations. This may have influenced the potential range and scope of ideas generated and the level of agreement reached. We attempted to mitigate this by including a diverse range of experts who provided anonymous feedback during the rating and review rounds, and who had a high response rate between rounds and attendance at discussion meetings. The sample size also followed the ideal group size as specified by the NGT approach and was pragmatic to implement.27
Conclusion
This work presents the first tool to allow the systematic collection of patient feedback on the safety of care, within a primary care setting. The PC PMOS is theory-based, and evidence-based, with data gathered both inductively (interviews with patients) and deductively (literature review and use of a theoretical framework). The PC PMOS provides a way for patients to identify various factors contributing to safety incidents. Patients are potentially a valuable source of information to help prevent harm. Patient feedback would not be used exclusively to improve patient safety, but this tool should form part of a comprehensive approach to safety management in primary care. Future research will explore the reliability and validity of the PC PMOS with a larger sample of patients in order to develop an intervention that improves patient safety in primary care.
Acknowledgments
We greatly appreciate the time and effort of the expert panel members (in addition to the authors) who participated in the Delphi process to develop the domains and items of the PC PMOS; Dr Christine Walker, Dr Mark Morgan, Dr Paresh Dawda, Dr Dale Ford and Ms Naomi Poole. We are grateful to the patients and primary care staff members who participated in the face validity testing.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online appendix 1
- Data supplement 2 - Online appendix 2
- Data supplement 3 - Online appendix 3
- Data supplement 4 - Online appendix 4
Footnotes
Twitter Follow Jane O'Hara at @janekohara and Julie Johnson at @JulieJKJohnson
Contributors ALH conceived and designed the study. ALH and SJG were responsible for the data collection and analysis, and also for creating the first draft of the manuscript. ALH, SJG and JKOH developed the first draft of the PC PMOS, and all the authors contributed to the final version. JKOH, JF, JJ and JAD contributed to specific sections of the manuscript. All authors read and approved the final version of the manuscript.
Funding The research reported in this article forms part of the research programme of the Australian Primary Health Care Research Institute (APHCRI) Centre of Research Excellence in Primary Health Care Microsystems, and is supported by a grant from the Australian Government Department of Health and Ageing. This paper summarises independent research partly funded by the National Institute for Health Research (NIHR) Greater Manchester PSTRC.
Disclaimer The information and opinions contained in the paper do not necessarily reflect the views or policy of the APHCRI, the Australian Government or the Department. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests None declared.
Patient consent Participants from stage 2 provided informed written consent and received a shopping voucher for their time and travel expenses.
Ethics approval Flinders University Social and Behavioural Research Ethics Committee (project no. 5667), and the University of Manchester (project no. 14339).
Provenance and peer review Not commissioned; externally peer reviewed.