Article Text

Download PDFPDF

Implementing electronic patient-reported outcomes measurements: challenges and success factors
  1. Lisa Nordan1,
  2. Lorrie Blanchfield1,
  3. Shehzad Niazi1,2,
  4. Juliet Sattar1,3,
  5. Courtney Elizabeth Coakes1,
  6. Ryan Uitti1,4,
  7. Michael Vizzini1,5,
  8. James M Naessens1,6,
  9. Aaron Spaulding1,7
  1. 1 Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Jacksonville, Florida, USA
  2. 2 Psychiatry and Psychology, Mayo Clinic, Jacksonville, Florida, USA
  3. 3 Operational Support, Mayo Clinic, Jacksonville, Florida, USA
  4. 4 Neurology, Mayo Clinic, Jacksonville, Florida, USA
  5. 5 Administration, Mayo Clinic, Jacksonville, Florida, USA
  6. 6 Division of Health Care Policy and Research, Mayo Clinic, Rochester, Minnesota, USA
  7. 7 Division of Health Care Policy and Research, Mayo Clinic, Jacksonville, Florida, USA
  1. Correspondence to Dr Aaron Spaulding, Health Sciences Research, Mayo Clinic, Jacksonville, FL 32224, USA; spaulding.aaron{at}

Statistics from

Determining how to collect and use patient-reported outcome measures (PROMs) continues to be an area of discussion, and in some cases frustration.1–5 Gaining a greater depth of knowledge concerning a patient’s initial health status as well as improvement after a medical or surgical intervention, would provide a clearer understanding of needed care paths and outcomes of treatments, oftentimes missing from our current healthcare processes.6 7 While PROMs are not a new idea, the ability to electronically collect, report and use the data has become more relevant in recent years. As such, this work focuses on the challenges and lessons learnt from implementing electronic PROMs (ePROMs) within a destination medical centre which provides team-based comprehensive care for patients.

Implementations in multiple departments and disease specific areas of care throughout the organisation took place between January 2016 and March 2018. The International Consortium for Health Outcomes Measurement (ICHOM) standard sets were used in each department as the primary survey tool, as scores could easily be calculated and the disease specific questionnaires administered at the point of care demonstrated a good fit for the organisation’s practices.6 8 ICHOM Standard Sets are global, standardised and validated questionnaires that have been combined into a set that captures information for the provider at a level that patients understand. Pre-existing intake assessments were evaluated within each department and service line and subsequently were merged or replaced with the ICHOM sets to create customised comprehensive condition-specific survey sets.

In all cases, collection of the survey sets at patient intake provides a baseline for changes to be monitored as well as provides context for patient concerns, needs and areas in which clinicians may need to further inquire. Follow-up data are collected after operations or procedures and changes in patient responses are monitored and assessed. In the first year of implementation, 2771 ePROMs were completed with a 99.8% capture rate over four different disease conditions. Ten additional condition-specific areas were added in subsequent years, resulting in over 50 000 ePROMs by June 2018 (see figure 1).

Figure 1

ePROM collection over time.


Naturally, implementing ePROMs proved more difficult in some specialties than others. This, in some cases, was due to physicians being reluctant to use the ICHOM standard sets.5 Many departments used tools developed and endorsed by specialty societies which promoted hesitation when considering conversion to another data collection tool. In addition, bringing front desk staff on board prior to implementation also proved challenging due to reluctance to take on additional responsibility as well as perceived disruptions of the workflow. In both cases, mapping workflows, eliminating redundant steps as well as defining how the new standard sets could be either combined with or collected in place of previous sets provided evidence on how changes could prove beneficial. Once the net zero impact, and in some cases, time savings on work throughputs were demonstrated, the staff was typically receptive to the new process. This evidence demonstration and work process flow communication has been previously indicated as important to managing change, particularly with respect to ePROM implementation.5 9

Furthermore, some staff’s history with previous attempts to implement and collect ePROMs created implementation barriers. Initial trials with homegrown iPad solutions were unstable due to problems with WI-FI internet connection, complex logins designed to create secure connections, and the fact that computer adaptive testing and branching logic capabilities were not possible. The new platform and technology solved these issues; however, dedicated communication, demonstration and training were still required to help overcome these barriers. Again, this is consistent with previous research focused at change management and technology acceptance.10–12

Patient acceptance of remote capture of questionnaires also presented a challenge. It was found that when questionnaires were sent to the patient’s email or cell phone for completion rather than being presented on site, capture rates were lower. This may have been partially due to the email appearing as a generic email coming from the organisation and as such was disregarded as ‘junk’ mail. As a result, it was necessary to request clinicians interacting with the patient to provide context and set expectations with patients for the remote completion of questionnaires. This process of communication and expectation setting dramatically increased remote capture rates, which is consistent with findings of Atherton et al.13 In addition, customising the email to come from the provider versus the organisation was a key driver in acceptance and capture. The patient received an email from the provider to complete a questionnaire that was then used as part of their appointment. That personalisation created a connection between the patient and provider, as it was no longer presented like other organisational questionnaires that previously may have been deleted.

Finally, there were concerns revolving around data and security. Initial concerns about data storage and security revolved around using a third-party vendor rather than an internal technology solution. This was resolved through use of a HIPAA compliant data cloud owned by the organisation to store and maintain the data. Subsequently, there were concerns around ensuring the information collected in the ePROMs tool were included in the electronic health record (EHR). These concerns were resolved in two different manners. In the absence of an interface with the current EHR, physicians were required to login to an additional browser window, they were then able to import narrative reports into their notes in the EHR. Additionally, patient-completed questionnaires were routed through the organisation’s Health Information Management Systems so that appropriate documents could be scanned into the EHR. Each of these barriers is a reality of a complex organisational IT environment and is an important consideration for cost controls and securities. As such, it is necessary to allocate time and energies to ensure the framework of committees and organisational policies and procedures are followed.

Lessons Learnt

Minimising workflow disruption is essential in creating positive reception of the process changes by staff.5 14 In order to minimise workflow disruption, we observed and interviewed staff in order to better understand their current processes prior to implementation. In addition, we completed workflow mapping activities both preintervention and postintervention to show time and workflow savings.9 Ultimately, the key to gaining staff members’ participation was enthusiasm and support from leadership, face to face support, collaborative efforts in customising the tool as well as data that demonstrated tangible benefits to the practice and process of care.15

The turning point of the implementation was finding a physician champion to engage his colleagues by demonstrating the flexibility of the tool, highlighting efficiencies in the overall work process and convincing his colleagues of the value of the ePROM sets. Consistent with previous inquiry, engagement with physicians, nurses and administrators, that have an interest in using the tools and serve as champions within their own practices, has proven successful in each of the areas we have implemented the ePROMs.16 In particular, front-line staff perceptions of ePROMs usability are critical to implementation, as they are providing the iPad to the patient, explaining the process and are responsible for ensuring that patients complete questionnaires. Including front-line staff in the process flow discussions increased acceptance and willingness to support the ongoing project. In addition, having a coordinator or manager on the implementation project is extremely beneficial. Peer-to-peer communication has proven influential in increasing user acceptance as one’s colleagues are best able to express practice and time benefits.

Flexibility with the ePROM platform was also crucial in the successful adaptation of the tool by staff members. The platform had to be customisable in a number of ways to meet each practice’s specific needs, provider preferences and workflows; a team’s ability to interact in the process and make changes to better suit their needs also creates a collaborative environment that allows physicians and staff to become invested in the project.17 The patient questionnaires were built with a toolkit that allowed multiple-choice questions, multiselect questions, sliding scales, drawing and diagramming capabilities, free text as well as the ability to provide visuals and graphics to better engage patients. Also appealing to providers was the ability to send questionnaires remotely via SMS text messaging or email. This allows the patient to complete the questionnaires before their visit, and as a backup, patients were able to complete questionnaires on an iPad in the lobby on arriving for their appointment.18

In addition, providers could customise the ePROM system to meet their research or practice needs. For instance, they are able to define the ‘weight’ of individual questions, enabling them to create custom scoring systems tailored to particular conditions or the needs of specific patient populations. Prompts for additional follow-up questionnaires or provider interventions can be set based on specified scoring flags and thresholds. By applying scoring algorithms, patient responses can then be displayed to the provider with colour coding, allowing the provider to quickly identify the patient’s top concerns. For example, a red, orange, yellow and green system could be used to illustrate symptom severity to flag patients’ priority concerns in individual symptoms such as sleep, appetite, energy and concentration. Furthermore, the ePROM platform has the ability to prompt follow-up questionnaires should the patient’s score reach a predefined risk threshold for related disease or psychosocial condition. Results were immediately available and populated the provider’s inbox, accessible on the platform through any web browser. As a result, the provider could easily review patient responses in advance of the care visit to use that information with the patient to better focus on their specific needs. Moreover, triggers set to specific question responses or scores can result in an email to clinical staff prompting immediate contact with the patient when needed. The ability to easily add new and modify existing questionnaires allows for a smoother implementation process as specific practice needs can be addressed in real time. This also facilitates the provision of a scalable solution which can expand to additional condition specific areas of care.

Finally, encouraging individual physicians to customise the output summary using natural language processing was also important in the adoption of the new ePROM system. The natural language output examines the way a provider normally dictates their clinical note and uses that information to build a summary from the patient’s questionnaire that could be used as the basis for the clinical note for the patient’s visit. In turn, this decreases the physician work effort, and it is worth noting that there is substantial evidence showing that patient-reported symptoms more accurately reflect their health status than physician elicited responses.19 Incorporating ePROM results into the clinical encounter helps to engage the patient and provider in more relevant discussion that is focused on the patient’s needs.4 5 This allows for a more fruitful and focused conversation concerning patient needs, desires and thoughts about their visit or about the status of their recovery. It also increases the depth of the conversations allowing for more shared decisions regarding treatment paths. This opportunity, plus the ability to track scores for a specific patient, for physicians or for a department in general, allowed for greater buy-in from patients, physicians and administrators. Ultimately, these data may also prove beneficial when negotiating insurance contracts, physician recertification and managing referring providers relationships. A summary of the challenges and lessons learnt is available in table 1.

Table 1

Challenges and lessons learnt summary


This ePROM system has helped capture how patients perceive their health and has proven to be essential in assessing healthcare quality and outcomes in our organisation. As we continue to expand to additional conditions within our organisation, lessons learnt and staff input will continue to be our main drivers. Having key stakeholders at the design session will continue to improve our implementation strategy and success. By expanding our collection of ePROMs to other conditions, we hope to transform the practice with the data collected, to improve patient care, patient outcomes and ultimately provide patients with valuable information to allow them to make decisions on their healthcare supported by shared decision-making models.



  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent Not required.

  • Provenance and peer review Not commissioned; internally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.