Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information

https://doi.org/10.1016/j.pec.2014.05.027Get rights and content

Highlights

  • The PEMAT a new instrument to assess patient education materials.

  • It assesses understandability and actionability of print and audiovisual materials.

  • The PEMAT demonstrated strong internal consistency and reliability.

  • Consumer testing and readability assessments produced evidence of construct validity.

Abstract

Objective

To develop a reliable and valid instrument to assess the understandability and actionability of print and audiovisual materials.

Methods

We compiled items from existing instruments/guides that the expert panel assessed for face/content validity. We completed four rounds of reliability testing, and produced evidence of construct validity with consumers and readability assessments.

Results

The experts deemed the PEMAT items face/content valid. Four rounds of reliability testing and refinement were conducted using raters untrained on the PEMAT. Agreement improved across rounds. The final PEMAT showed moderate agreement per Kappa (Average K = 0.57) and strong agreement per Gwet's AC1 (Average = 0.74). Internal consistency was strong (α = 0.71; Average Item-Total Correlation = 0.62). For construct validation with consumers (n = 47), we found significant differences between actionable and poorly-actionable materials in comprehension scores (76% vs. 63%, p < 0.05) and ratings (8.9 vs. 7.7, p < 0.05). For understandability, there was a significant difference for only one of two topics on consumer numeric scores. For actionability, there were significant positive correlations between PEMAT scores and consumer-testing results, but no relationship for understandability. There were, however, strong, negative correlations between grade-level and both consumer-testing results and PEMAT scores.

Conclusions

The PEMAT demonstrated strong internal consistency, reliability, and evidence of construct validity.

Practice implications

The PEMAT can help professionals judge the quality of materials (available at: http://www.ahrq.gov/pemat).

Introduction

Health literacy is the capacity to “obtain, process and understand basic health information and services needed to make appropriate health decisions” [1]. Health literacy is well recognized as a challenge for public health, with many adults lacking the requisite skills to engage successfully in their health care. Recent systematic reviews have confirmed that low health literacy is strongly associated with poorer use of health care and subsequent health outcomes, leading to higher use of emergency departments and inpatient beds [2], [3], [4].

While the skills of individuals are an important part of health literacy, the field has come to recognize that the demands placed on individuals by the health system and professionals are an important determinant [5], [6], [7], [8], [9]. To address health literacy, the U.S. Department of Health and Human Services’ National Action Plan to Improve Health Literacy (National Action Plan) promotes a multisector effort to improve health literacy, including reducing the demands placed on individuals [10]. A key goal of the National Action Plan is “to develop and disseminate health and safety information that is accurate, accessible, and actionable” [10]. Studies assessing the readability, suitability or comprehensibility of patient education materials on a myriad of topics are abound, and the evidence is clear that most education materials are too complex for patients with low health literacy.

Organizations and professionals aspiring to produce low-demand patient education materials have a selection of guides that provide instruction [11], [12], [13], [14], [15]. Readability formulas [13], [16] are commonly relied upon to assess whether written materials are in fact low-demand. Readability formulas provide quantitative estimates, in the form of a grade level, of the reading difficulty of written information based on word and sentence difficulty. Yet readability formulas ignore several factors that contribute to comprehension [13].

In recognition of the shortcomings of readability formulas, several checklists and instruments that assess the health literacy demand of materials have been developed [17], [18], [19], [20], [21], [22], [23], including two newly-developed instruments [24], [25]. These assessment tools, however, have not shown inter-rater reliability [17], were developed or tested with a specific topic or aim [18], [19], [20], or were tested using only raters trained in the use of the instrument [24], [25]. Furthermore, most are applicable only to print material, and none measure whether materials are actionable – an important characteristic of materials called for in the first goal of the National Action Plan [10].

The aim of this study was to develop a reliable and valid Patient Education Materials Assessment Tool (PEMAT) to be used by untrained lay and health professionals alike to assess the understandability and actionability of both printable (e.g., printed materials like brochures or pamphlets or materials that can be printed from websites like PDFs) and audiovisual (e.g., video or multi-media presentation with or without narration) patient education materials. We defined understandability and actionability as follows:

  • Understandability: Patient education materials are understandable when consumers of diverse backgrounds and varying levels of health literacy can process and explain key messages.

  • Actionability: Patient education materials are actionable when consumers of diverse backgrounds and varying levels of health literacy can identify what they can do based on the information presented.

Section snippets

Methods

The PEMAT was iteratively and systematically developed with repeated input from a panel of experts in health literacy; health communications; content creation, including different modalities; patient education; communication; patient engagement, and health information technology. The experts included clinicians, researchers, policymakers, academicians, and staff from non-profit, for-profit and governmental organizations. (See Acknowledgements for the list of experts.)

Once we defined

Review of existing instruments and guides

From an initially identified set of 41 instruments and guides, 22 had potentially relevant concepts and items for understandability or actionability. From these instruments, 7 topics (i.e., content, word choice and style, numeracy, quality visual aids, organization, layout and design, ease of use for audiovisual materials only) and 64 associated items were initially compiled. The project team reviewed and narrowed the pool of items based on relevance to understandability or actionability and

Discussion

This article reports on the development and reliability and validity testing of the Patient Education Materials Assessment Tool (PEMAT), a new instrument to assess the understandability and actionability of both printable and audiovisual patient education materials on diverse topics. The development of the PEMAT was guided by earlier work defining and measuring desirable characteristics of patient education materials and a multidisciplinary expert panel. The PEMAT had strong internal

Funding

The information upon which this publication is based was performed under Contract #HHSA290200900012I, TO 4 “Improving EHRs Patient Education Materials” funded by the Agency for Healthcare Research and Quality (AHRQ), Department of Health and Human Services. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does the mention of trade names, commercial products, or organizations imply endorsement by the U.S.

Conflicts of interest

All authors state that there are no other actual or potential conflicts of interest including any financial, personal or other relationships with other people or organizations within 3 years of beginning the submitted work that could inappropriately influence, or be perceived to influence, their work.

Acknowledgments

We would like to acknowledge the raters from Abt Associates, AHRQ, Massachusetts General Hospital, and Northwestern University who helped to establish the reliability of the PEMAT, Allyson Ross Davies for her guidance on instrument development, and Ken Carlson and Mark Spranca from Abt Associates for their valuable engagement with the reliability and validity testing of the PEMAT.

We would like to thank the technical expert panel who helped to shape this instrument by providing guidance and

References (37)

  • A.R. Feinstein et al.

    High agreement but low kappa: I. The problems of two paradoxes

    J Clin Epidemiol

    (1990)
  • U.S. Department of Health and Human Services

    Healthy people 2010

    (2000)
  • N.D. Berkman et al.

    Low health literacy and health outcomes: an updated systematic review

    Ann Intern Med

    (2011)
  • D.A. Dewalt et al.

    Literacy and health outcomes: a systematic review of the literature

    J Gen Intern Med

    (2004)
  • D.L. Keller et al.

    Impact of health literacy on health outcomes in ambulatory care patients: a systematic review

    Ann Pharmacother

    (2008)
  • U.S. Department of Health and Human Services

    Research-based web design and usability guidelines

    (2006)
  • C. Brach et al.

    Ten attributes of a health literate health care organization

    (2012)
  • D.W. Baker

    The meaning and the measure of health literacy

    J Gen Intern Med

    (2006)
  • R. Rudd

    Objective 11-2. Improvement of health literacy

    Communicating health: priorities and strategies for progress

    (2003)
  • U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion

    National action plan to improve health literacy

    (2010)
  • Pfizer Clear Health Communication Initiative

    Pfizer principles for clear health communication

    (2004)
  • U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion

    Quick guide to health literacy

    (2010)
  • Centers for Medicare & Medicaid Services

    The toolkit for making written material clear and effective

    (2012)
  • National Institutes of Health, National Cancer Institute

    Clear & simple: developing effective print materials for low-literate readers

    (2013)
  • National Institutes of Health, U.S. National Library of Medicine

    How to write easy-to-read health materials

    (2013)
  • D.B. Friedman et al.

    A systematic review of readability and comprehension instruments used for print and web-based cancer information

    Health Educ Behav

    (2006)
  • C.C. Doak et al.

    Teaching patients with low literacy skills

    (1996)
  • Cited by (0)

    View full text