Article Text
PDF
Abstract
Background Patient-reported outcomes (PROs) can promote patient engagement, shared-decision making and improve the overall experience of care. However, PRO integration in the primary care clinical setting is limited. Exploring the perspectives of primary care physicians (PCPs) on PROs is key to understanding how they are being used in the clinical setting. We sought to elucidate this clinical perspective at one of the largest US health systems that has integrated a wide range of PROs into routine primary care.
Methods Mixed methods study with both anonymous online surveys and in-person qualitative semistructured interviews conducted with PCPs to understand their clinical perspectives on the applications of the existing PROs. PCPs from the 19 affiliated clinics were prompted to complete the survey. Interviewed PCPs were selected via a combination of random and purposive selection from the PCP directory.
Results Of 172 PCPs, 117 (68%) completed the online survey and 28 completed semistructured interviews. Most PCPs (77%) reviewed PRO responses with their patients. PCPs endorsed that PROs improve clinic efficiency and clinical management. However, PCPs have heterogeneous perspectives on the relevance of PROs in clinical practice, likely due to variations in clinic practice. For specific PRO instruments, PCPs reported anxiety and depression screening PROs to be most helpful. PCPs felt that PROs assisted with completing screening questions that are required by regulatory bodies. Barriers to using PROs include poor user-interface for both clinicians and patients and inadequate training.
Conclusions Most PCPs regularly use PRO data though there are mixed opinions about their clinical relevance. An adaptable, user-friendly PRO system has the potential to have meaningful clinical applications in primary care.
- primary care
- implementation science
- patient-centred care
- quality improvement
- qualitative research
Data availability statement
Data are available on reasonable request. Deidentified data will be made available to those who request it for one year after the publication from the corresponding author.
Statistics from Altmetric.com
Introduction
Patient-reported outcomes (PROs) are measurements based on a report that comes directly from the patient about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else.1 PROs can help with patient engagement, promote shared decision-making and improve the overall experience of care.2–4 However, widespread clinical application of PROs is limited.5 6 Applying PROs in primary care setting has been particularly challenging, in part due to primary care’s longitudinal management of a large and diverse array of physical and mental health problems.7 8
Despite these obstacles, there are encouraging examples of standardised PRO utilisation in primary care. In the UK, the National Health System has several examples of incentivised nation-wide PROs implemented in primary care. For example, the Patient Health Questionnaire-9 (PHQ-9) depression screen instrument and the Hospital Anxiety and Depression Scale were components of the pay-for-performance Quality and Outcomes Framework between 2006 and 2013.9–11 Additionally, primary care physicians (PCPs) are required to complete the Oxford Knee Score before referring patients to orthopaedic surgeons for knee replacement surgery.12 In contrast, the adoption of PROs has been less standardised in the USA.13
However, PRO collection alone does not reflexively lead to meaningful clinical use of PROs. UK general practitioners (GPs) often report knowledge gaps in the selection, administration, interpretation and implementation of PROs.14 A recent systematic review revealed that effective PRO implementation requires adequate infrastructure, minimal workflow disruption, seamless technology and clinician guidance.15 However, the primary care literature included in the reviews focused primarily on one PRO: depression screening. The reality is that PCPs are tasked with managing a broad spectrum of preventive care, chronic disease and mental health. Additionally, primary care serves as the locus of measurement and reporting for a diverse array of clinical quality measures and regulatory requirements. Therefore, in order to meaningfully implement PROs in the primary care setting, it is critical to understand the PCP’s clinical perspective about PROs that cover a wide range of clinical and regulatory domains. We explored the PCP’s clinical perspective on PROs by surveying and interviewing PCPs from a large US health system that has established a centralised PRO platform that assesses patients across a wide range of clinical domains.
Methods
Study design
We conducted a mixed methods study with a sequential explanatory design where the quantitative data are further explored with the qualitative data.16 The study was approved by our organisation’s Institutional Review Board (Protocol number: 2019P000881). For the quantitative study, we distributed optional, anonymous online surveys to PCPs to elicit their perspectives of the existing PRO platform. For the qualitative study, we used the survey data to develop an interview guide that was used to conduct semistructured interviews with PCPs to gain an in-depth understanding of their perspectives (online supplemental efigure 1).
Supplemental material
Overview of the existing PRO platform at the Massachusetts General Hospital
Massachusetts General Hospital (MGH) is a large not-for-profit academic medical centre that is part of a larger health system comprised of a second academic medical centre and 15 other community-based hospitals that serve 2–3 million patients annually. In 2014, our organisation implemented a system-wide PRO initiative that has since collected over five million questionnaires across 231 clinics, 56 specialties and 98 geographical locations. PROs are administered through our electronic health record (EHR) on both a digital patient portal and tablets in clinic (Epic, Verona, Wisconsin, USA).17 Results are immediately available for review within the patient chart. Our organisation uses this platform for its PRO collection.
MGH has 19 affiliated PCP clinics that serve over 200 000 patients. In 2018, all our organisation’s PCP practices launched the Primary Care Screening Bundle (PCSB), a standardised 70-item questionnaire that is administered at annual visits and new patient visits. The PCSB contains PROs that include but are not limited to the PHQ-2 for depression screening, General Anxiety Disorder (GAD) for anxiety screening and drug use screening test (online supplemental efigure 2).18 19 It also includes other patient-reported data (PRD) such as Social Determinants of Health (SDOH) screening, which assesses food, housing and transportation insecurity and other questions related to socioeconomic hardship (online supplemental efigure 3) and Medicare Health Risk Assessment (HRA) questionnaire, which assesses activities of daily living, home safety and cognition (online supplemental efigure 2). Each of these PRO and PRD questionnaires were designed to examine areas of concern for individual patients and then facilitate clinician assessment and intervention. Of note, the SDOH and Medicare HRA questions are not technically considered PROs; we include them in this analysis to demonstrate the comprehensiveness of our PCSB.
The original goal of the PCSB was to collect electronic, population-wide screening questionnaires that were either recommended by the United States Preventive Services Task Force based on grade A and grade B evidence (eg, PHQ-2 for depression, AUDIT-C for alcohol consumption) or clinical assessments that were required by regulatory and compliance to meet our contractual obligations (eg, Medicare HRA). Prior to implementation of the PCSB, each of our primary care practices had non-standardised, paper-based questionnaires that each patient was asked to complete in the waiting room prior to a visit. Patient responses on these paper forms were often scanned into the EHR or manually transcribed into the EHR, precluding any ability to perform data analysis across the population or trend responses at the patient level over time. The impetus for implementation of the PCSB was to inform clinicians in real-time about concerning individual patient responses and then facilitate clinical intervention while also allowing our health system to systematically collect data to improve the quality of care at a population level.
In 2019, 114 521 out of 154 758 qualifying patients (74%) successfully completed the PCSB. The high patient completion rate of PROs is due to several factors. First and foremost, all PROs are available electronically at the time of the visit and also within the online portal in the week prior to the visit. Immediately on completion, they are available for the clinician to use in the EHR. This standardises the workflow throughout our practices and ensures that physicians know where to find the results. In addition, our set of PROs includes screens that are either recommended as standard of care or are required for billing and compliance reasons. Patients completing these screens prior to the visit saves time during the clinical encounter, as the results will have already been documented in the EHR. As such, physicians are motivated for them to be completed and also prompt patients to complete them. Despite the systematic collection of PROs since 2018, feedback from clinicians regarding their usefulness is lacking.
Survey data
Survey design and data collection
We conducted semistructured qualitative interviews with three key clinical and administrative stakeholders of the organisation’s PRO programme to understand the operational elements of the current PRO platform at our PCP clinics. These three stakeholders include the Senior Medical Director and Assistant Medical Director for PROMs, who together manage the teams that enable PRO collection and data dissemination and analysis and the Director of Population Health and Quality for the Division of General Internal Medicine, who developed and implemented the PCSB. This information guided the development of a 15-question survey that asked PCPs about their demographics and experiences with the PRO platform. Open-ended free-text questions were included when appropriate (table 1). These questions were reviewed with the PRO programme, primary care and hospital leadership to ensure relevance and comprehensibility. A unique REDCap (Nashville, Tennessee, USA) survey link was then emailed to all clinically active affiliated PCPs between May 2019 and July 2019. The survey invitation conveys that it is optional and that results will be anonymous. Non-responders received up to three biweekly e-mail reminders.
A 15-question online survey was developed with input from PCPs, PRO programme leadership and hospital leadership
Data analysis
The perceived helpfulness of PROs were compared. The survey responses were dichotomised into helpful (answer options include ‘helpful’ or ‘very helpful’) and not helpful (answer options include ‘neutral’, ‘unhelpful’ or ‘very unhelpful’). These data were analysed with χ² (GraphPad Prism, San Diego, California, USA). The free text responses were categorised into themes using content analysis.20 Two doctoral-level researchers with experience in qualitative methods (DM, MC) developed a preliminary codebook and independently coded the open-ended responses and categorised them based on content. Consensus was reached through deliberation, with disagreements arbitrated by a third qualitative expert (KL, Nvivo 12, QSR International).
Qualitative Interviews
Data Collection
PCPs were recruited per random selection from our PCP directory.21 Purposive sampling criteria included PCPs in leadership positions to elicit a management perspective. Semistructured interviews were conducted between June 2019 and October 2019 either in-person or over the phone, per participant preference.22 Verbal consent was obtained prior to all interviews. All interviews were conducted by a physician researcher with qualitative interview training (DM). The interviews were audio-recorded and transcribed verbatim. Interviews continued until data saturation was achieved and confirmed through deliberation by the qualitative team (DM, MC and KL).23 Data saturation as the point where additional interviews added minimal additional insights as determined through deliberation by the qualitative team. This was achieved after 20 interviews.
An interview guide was developed based on a literature review, input from PRO leadership (RCS, MH) and findings from the survey. The qualitative data were used to deepen and contextualise the interpretation of the quantitative findings, including usefulness of PROs, barriers to using PROs and unmet needs. During the interview, participants were asked to provide in-depth descriptions of their clinical experiences with the PRO platform, perceived barriers and facilitators to using PROs and supporting examples.
Data analysis
DM, MC and KL used deductive coding to create a preliminary codebook with the first 10 interviews.24 25 Deidentified transcripts were independently coded by two authors (DM, MCA) line by line, allowing new codes to emerge inductively.26 Iterative deliberation yielded team consensus about coding discrepancies, emergent codes and amended coding descriptions. The qualitative team applied the final codebook to the initial 10 interviews and remaining 18 transcripts. Subsequently, codes were organised into themes through a consensus process using pattern and focused coding (Nvivo 12, QSR International). These themes were subsequently organised broadly under benefits and barriers of using PROs in the clinical setting.
Results
Survey data
Of 172 PCPs who were contacted, 117 (68%) completed the survey. Respondents were primarily women (61%) and had an average of 16–30 years in practice. Nearly all PCPs (90%) endorsed knowing where to find the PRO data in the EHR. A majority (77%) reported reviewing PROs with patients during clinic visits. There were notable differences in the perceived usefulness of different PROs (n=117, χ² p<0.0001; figure 1A). A majority of PCPs deemed depression (84%) and anxiety (78%) PROs to be helpful. Around half of the PCPs found the screening questions for domestic violence (62%), falls (54%), alcohol use (61%), substance use (56%), SDOH questions (55%) and Medicare Questionnaire (58%) to be helpful. Only 14% of PCPs found the pain screening question to be helpful. Seventy-six per cent of PCPs deemed the review of systems question to be helpful. Despite variations in clinical relevance across different PROs, there is also variation in the degree of perceived clinical relevance within each individual PRO (figure 1B).
(A) PCPs considered certain PROs to be more helpful than others. PCPs responses of ‘helpful’ or ‘very helpful’ were considered ‘helpful’ (blue bars), whereas responses of ‘neutral’, ‘unhelpful’ and ‘very unhelpful’ were considered ‘unhelpful’ (red bars). *The review of systems responses were binary responses of ‘helpful’ and ‘not helpful’. (B) Within individual PROs, there is significant variation of perceived helpfulness. PCP, primary care physician; PRO, patient-reported outcome; SDOH, Social Determinants of Health.
PCPs ranked the top three barriers to using PROs descending order of importance ‘difficult data format’, ‘survey too long’ and ‘iPad technical problems’ (figure 2). When asked about ease of interpretability of the PRO data as rendered in the provider-facing EHR (figure 3A), 47% of PCPs responded ‘difficult or very difficult’, 16% responded ‘neutral’ and 37% responded ‘easy or very easy’ (table 1). In the free text comments, PCPs reported that their patients find the tablets difficult to use, the PCSB is too long, and the questions are often poorly phrased (table 2). Furthermore, PCPs discussed the need for more detailed patient history questions (eg, family history, smoking history) and better data formatting that readily highlights concerning or abnormal PRO findings to clinicians. Finally, PCPs wanted to minimise disruption to clinic flow related to PRO completion, ensure that PROs only elicit issues from patients that are addressable by the PCP and provide PROs in other languages beyond English and Spanish (table 2).
Primary barriers to PROs use included data format and survey length. PRO, patient-reported outcome.
Sample of PRO data output before (A) and after (B) PCP feedback on data display was implemented. Text font size is increased, pertinent findings are highlighted in red and answers are located closer to the questions. PCP, primary care physician; PRO, patient-reported outcome.
Survey free text responses were categorised by themes
Qualitative interviews
We conducted 28 semistructured interviews with PCPs via a combination of random selection from the PCP directory and purposive selection for PCPs in hospital leadership positions. The interview time lasted between 15 and 31 min. Interviewed PCPs had practiced medicine for an average of 17 years and saw an average of 34 patients per week. The transcribed interviews were coded and organised into themes that were then categorised as benefits and barriers of PROs in clinical applications. There was heterogeneity in terms whether PCPs found clinical value in PROs. This is likely attributable to workflow and clinical management variations across different PCP practices.
Heterogeneous views on clinical benefits of PROs
PCPs voiced variable usefulness of PROs in terms of improving clinic efficiency and enhancing clinical decision-making. PCPs endorsed that PROs can facilitate completion of screening questions that are required by regulatory bodies, which may improve clinic efficiency:
I love the fact that I'm not actually having to ask all these [PRO] questions… [I] would have to spend an hour and a half with someone if [I] were doing all these little questions one by one. (PCP 2)
A lot of these screening questions we have to ask anyway because it’s a regulatory requirement… Can you imagine if you only have eight min with the doctor and the doctor has to do [all] this? I know you're here for a sinus infection but let me ask you this: Have you ever had thoughts of suicide? (PCP 5)
Additionally, the existing PRO platform may further promote clinic efficiency by facilitating EHR documentation:
[From a] data entry perspective [PROs are] really helpful… think about what it’s like to be a patient in a doctor’s office when your doctor is sitting there staring at the computer just asking you hundreds of questions. (PCP 1)
My favorite is the annual wellness visit because all the pertinent questions can get pre-populated into the notes. That’s incredibly helpful. (PCP 2)
However, not all PCPs agree that the current PRO platform promotes efficiency. This variation appears to stem from different office clinic workflows and clinician habits.
I could say 25% of the time [the PROs] don't work. I know my front desk staff hates them [because it delays clinic flow]. My front desk staff would argue that it less efficient. (PCP 4)
The average wait time for a patient between walking in the door and being [placed in the exam room] is about 90 seconds… So, what ends up happening is a lot of the time [is that] I'll look in the room and the questions are only half done… I end up going through some of the questions myself. (PCP 12)
Maybe if… we got into the habit of [reviewing] our patients’ questionnaires… we would use them more. (PCP 16)
Some PCPs endorsed that PROs may enhance clinical decision-making by eliciting patients’ issues that may otherwise go ignored:
Sometimes the patient will point out something [on PROs] that they normally wouldn't have just brought it up… so there has been occasion where a headache will come up or abdominal pain… That does add to the visit experience. (PCP 5)
I briefly look at [the PROs to make sure] any outstanding questions like any metrics are not grossly abnormal like the GAD 7 or PHQ9… if those scores are very high, I will perhaps dig in a little harder. (PCP 3)
I'll say [to my patient], ‘hey you mentioned you had joint pain and some weight changes, can you tell me about that?’ So, I will absolutely bring that up in the in the discussion, based on what I see [in the PRO data]. (PCP 14)
Additionally, the SDOH questions may be helpful to identify patients with limited resources. PCPs use this information in different ways that range from facilitating further conversation on relevant social topics to referring patients to social workers.
I find I always review [PROs] and use it as a jumping off point [for questions]… especially for the Social Determinants of Health to make sure that we’re identifying patients that could benefit from extra resources. (PCP 1)
I think the social determinants of health is very useful. We now have social workers and different programs to support patients. (PCP 15)
However, some PCPs have reported that PROs do not meaningfully impact clinical management:
[PROs] quite rarely truly impact [my patients’] clinical care… it’s not a huge factor in how I'm going to manage them. (PCP 7)
My suspicion that most providers don't even look at [PROs]… they [take] time away that they could have been doing something else. (PCP 3)
Barriers to PRO use
Despite clinical applications, there are obstacles to using PROs in clinical practice. PCPs believed that the existing PRO platform may be confusing and frustrating for patients:
I've had many, many patients tell me ‘Doc why am I filling this out?’. I’ve had a couple tell me they feel like I'm giving them an exam like a like a school exam and it makes them nervous and anxious and they don't know why I'm asking these questions. [They say] ‘Doc all I'm here for is sinus infection but you're asking me all these questions about how my depression’… It gets in the way of the patient really feeling like they're engaging with me. (PCP 5)
[Regarding SDOH questions], some of the people are upset about it because they think they're being profiled as someone who might have food insecurity. (PCP 9)
PCPs claimed that the existing PRO platform has a poor patient-facing user interface:
[Patients] get frustrated with [the PRO tablets] if they log themselves out. They have to enter the encounter ID again… [which] they don’t remember. (PCP 3)
For elderly patients or people that are visually impaired and [PROs] can be stressful for them to click through all of the screens because it’s a lot of questions. (PCP 1)
PCPs also remarked that the PRO platform is not user-friendly for clinicians, both in terms of accessing the data and interpreting the data:
The frustration of even finding the PROs is pretty high. [Once I find the data] I can't even tell which [PRO question] is which because they're just all like shoved down there into one big blob of text. I wish there were a way to just visually tell like a like a good dashboard in your car if it’s a red light that means there’s something wrong. (PCP 5)
[The PRO] questions should be formatted and asked in a way so that at a glance you can see [the] problems. (PCP 4)
PCPs further commented that they would have liked to have more education and training related to PROs prior to its launch:
When [PROs] started there was not a great… education of the primary care community in terms of what they were going to be and the best workflow… to use them. The worst thing is occasionally I get a message in my [EHR] in-basket that one of my patients answered, [indicating] that they wanted to kill themselves… I don't believe primary care has been trained enough in terms of what to do with those answers. (PCP 2)
I'd love to see… somebody [sitting] elbow to elbow with [PCPs] and watched [to see] what their workflow is to figure out how to how to make [PROs work]. (PCP 10)
Discussion
Most PCPs from a large healthcare system with an integrated PRO platform found PROs to be helpful, with 77% of PCPs reporting that they reviewed PRO responses with their patients. However, there was significant heterogeneity in PCP responses regarding PRO clinical applications. There was variation in perceived clinical helpfulness for different PROs (figure 1A), and within a single PRO, there is variation of perceived clinical helpfulness across different PCPs (figure 1B). Interviews and surveys revealed that certain PCPs found that PROs improve clinic efficiency by facilitating completion of screening questions required by regulatory bodies and expediting EHR documentation, whereas others claimed that PROs worsen clinic efficiency by obstructing clinic flow (table 2). Certain interviewed PCPs used PROs to enhance clinical management by identifying medical or social issues that require intervention, while others endorsed minimal clinical impact of PROs. Thirty-four per cent of surveyed PCPs endorsed not using the review of systems data while several interviewed PCPs consistently used these data. Interview data suggest that these variations are likely due to differences in clinic workflow and clinical management. Some PCP offices have short wait times, which may not allow for patients to complete the PCSB in a timely matter. Other PCPs endorse minimal clinical relevance and also do not habitually review PRO responses. In terms of barriers to using PROMs clinically, PCPs complained about poor user-interface, an overly lengthy survey and insufficient clinician training. Despite these limitations, only a minority of PCPs (20%) felt that PROs were unhelpful. These findings suggest that PCPs found PROs to be of clinical benefit, but there is clearly room for improvement in the implementation of the PCSB.
This is one of the few studies to investigate the PCP’s perspective on PROs in a large US healthcare system with a mature, EHR-integrated PRO programme that contains a wide range of PROs. Most prior studies have been limited to assessing the clinical utility of a single PRO, whereas our study assessed a comprehensive 70-question questionnaire containing PROs across multiple domains.15 Our findings support prior literature showing that PCPs often find value in PROs as screening tools.14 Specifically, most surveyed PCPs endorsed that screening for depression and anxiety was valuable, which is in line with GPs’ opinions in England.24 Interestingly, we found conflicting PCP opinions on the utility of specific PROs and the impact of PROs on clinic efficiency and clinical management. This may be in part due to the fact that the PCSB we studied assesses a wide array of clinical and regulatory domains.
PCPs highlighted barriers to using PROs that have been reported in literature, including clinic flow interference, poor user-interface and insufficient clinician training.14 15 25 26 Of note, the top three identified barriers were all readily modifiable: poor PRO data display format, long survey length and IT issues. In fact, feedback from our PCP survey has already resulted in substantially improved PRO display format that has been implemented in the EHR since completion of our study (figure 3B). Overall, the PRO user experience has been limited by a suboptimal EHR IT infrastructure that limits our ability to make incremental improvements to the PRO display and usability.
Our findings highlight several important recommendations for others interested in establishing PRO programmes. Given the heterogeneous responses regarding the clinical utility of PROs, PRO programmes should be developed with an understanding of existing clinic workflows and be adaptable to variations in clinical operations. Furthermore, after implementing PROs, it is critical to elicit feedback from clinical end-users to ensure their meaningful use and have a robust IT infrastructure in place to quickly make incremental improvements. Finally, investment in clinician training will facilitate PRO adoption.
Strengths of our study include a mixed method design with a rigorously designed survey and in-depth interviews. Limitations include the fact that this was a single system study, which limits generalisability. However, our institution is one of the largest US healthcare organisations that systematically collect standardised, comprehensive PROs.17 Additionally, our survey had a 68% response rate, which exposes the study to response bias. Finally, we do not offer objective data on the patient’s experience interacting with PROs. This area should be investigated further.
In conclusion, PROs have the potential to improve clinic efficiency in completing preventive screenings and meeting regulatory requirements, and they could enhance clinical management in primary care. Despite implementation obstacles, the majority of our institution’s PCPs regularly review PROs with their patients and find them helpful, thus demonstrating proof of concept of PROs in the US primary care setting.
Data availability statement
Data are available on reasonable request. Deidentified data will be made available to those who request it for one year after the publication from the corresponding author.
Ethics statements
Patient consent for publication
Ethics approval
This study was approved by our Healthcare Institutional Review Board (Protocol number: 2019P000881). Informed consent was obtained from all participating PCPs.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
DM and DMH are joint first authors.
Twitter @DannyMouMD, @danielmhorn, @marilyn_heng
DM and DMH contributed equally.
Contributors All authors contributed to the planning, conduct, analysis and writing of this study. DM is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. If allowed, DM and DMH should be listed as cofirst authors as they have contributed equally to the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.