Article Text
PDF
Abstract
Background Organisational culture affects physician behaviours. Patient safety culture surveys have previously been used to drive care improvements, but no comparable survey of high-value care culture currently exists. We aimed to develop a High-Value Care Culture Survey (HVCCS) for use by healthcare leaders and training programmes to target future improvements in value-based care.
Methods We conducted a two-phase national modified Delphi process among 28 physicians and nurse experts with diverse backgrounds. We then administered a cross-sectional survey at two large academic medical centres in 2015 among 162 internal medicine residents and 91 hospitalists for psychometric evaluation.
Results Twenty-six (93%) experts completed the first phase and 22 (85%) experts completed the second phase of the modified Delphi process. Thirty-eight items achieved ≥70% consensus and were included in the survey. One hundred and forty-one residents (83%) and 73 (73%) hospitalists completed the survey. From exploratory factor analyses, four factors emerged with strong reliability: (1) leadership and health system messaging (α=0.94); (2) data transparency and access (α=0.80); (3) comfort with cost conversations (α=0.70); and (4) blame-free environment (α=0.70). In confirmatory factor analysis, this four-factor model fit the data well (Bentler-Bonett Normed Fit Index 0.976 and root mean square residual 0.056). The leadership and health system messaging (r=0.56, p<0.001), data transparency and access (r=0.15, p<0.001) and blame-free environment (r=0.37, p<0.001) domains differed significantly between institutions and positively correlated with Value-Based Purchasing Scores.
Conclusions Our results provide support for the reliability and validity of the HVCCS to assess high-value care culture among front-line clinicians. HVCCS may be used by healthcare groups to identify target areas for improvements and to monitor the effects of high-value care initiatives.
- Healthcare quality improvement
- Quality measurement
- Safety culture
Statistics from Altmetric.com
Background
The pressure on delivery systems to provide high-value care, defined as maximising health outcomes achieved per dollar spent,1 has never been higher due to sweeping payment reforms such as the Value-based Purchasing (VBP) Program of 2012 and the Medicare Access and Children's Health Insurance Program (CHIP) Reauthorization Act of 2015.2–4 Despite mounting pressures like these, most organisations lack strategies for identifying targeted areas for improvement and engaging physicians in the necessary cultural changes needed for reform. Traditionally, innovations to meet these challenges have focused on tactics to streamline protocols and reduce practice variation.
Organisational culture—defined as a system of shared assumptions, values, beliefs and norms existing within an environment—should be a target for intervention since it affects physician behaviours.5–8 Currently, the prevalent culture in medicine contributes to the overuse of resources.9 For example, a recent study showed that 97% of emergency medicine physicians surveyed order unnecessary imaging tests, largely reflecting a cultural response to uncertainty rather than a lack of medical judgement.10 In addition, the environment in which physicians train correlates with their practice patterns later in life, including the extent to which they practice high-quality, cost-effective care.11 ,12 Thus, to create changes in clinician behaviours that promote high-value care delivery, organisational culture needs to be meaningfully addressed.
In the field of patient safety, validated surveys of organisational safety culture have been useful in identifying opportunities for improvement and motivating change.13–15 In fact, higher Patient Safety Scores are associated with changes in clinician behaviours and with improved safety outcomes in hospital wards and across entire health systems.16–19
With the international imperative to provide high-value care, a survey to evaluate aspects of an organisational culture supporting high-value care delivery could have a similarly important role. Our aim was to develop and complete initial psychometric evaluation of a High-Value Care Culture Survey (HVCCS) for use by leaders of healthcare delivery systems and training programmes to capture front-line provider experiences and target future interventions for improvement in delivering value-based care.
Methods
We created a conceptual model for high-value care culture and used a modified Delphi process to identify items for inclusion in the survey, followed by a cross-sectional survey of internal medicine residents and hospitalists at two large academic medical centres in California. We linked these results to hospital-level data from the Center for Medicare and Medicaid Services (CMS) Impact, VBP and Hospital Compare files from fiscal year 2014. All analyses were conducted using SAS software (V.9.4).
Conceptual model of high-value care culture
We conducted a literature review on organisational culture, decision-making and value-based care to create the initial domains and subdomains relevant to the culture of value-based decision-making.12–14 ,20–25 Making a high-value decision is complex and influenced by a wide range of cultural and structural factors from the provider, patient and organisation. Our conceptual model, therefore, was adapted from two previously used models in organisational culture (Ferlie and Shortell model) and provider-level culture and decision-making (Schein iceberg model) that address these factors and guided development of survey domains.20 ,21 We also adapted the Institute of Medicine definitions for quality and cost (ie, negative financial, physical and emotional effects to patients and the health system) to guide survey development.26 (Refer to online appendix II for full definitions and online appendix III for a list of all domains and subdomains in the original conceptual model.)
Supplementary appendices
Modified Delphi process to identify items for inclusion in the study
In March 2015, seven practising clinicians in the research team generated survey items that reflected the conceptual model subdomains, and evaluated content validity.27–29 This group included national leaders in quality and value, as well as local leaders in health system operations and health service research.
We used a modified Delphi process to evaluate this comprehensive list of survey items,30 conducted in April through May 2015. We identified a national panel of 28 high-value care experts through purposive sampling. Experts represented all five regions of the USA, four types of medical centres (academic, county, community, Veterans Affairs), various training levels (residents through senior clinicians), 11 medical and surgical subspecialties and key stakeholders related to healthcare value (ie, leaders from CMS, Institute for Healthcare Improvement, the Choosing Wisely campaign, Lown Foundation, Costs of Care, American College of Physicians and Accreditation Council for Graduate Medical Education). We approached a broad range of participants to reduce bias and provide greater generalisability. During round 1 of the Delphi process, panel members were asked to rate the importance of each item for survey inclusion on a 7-point Likert Scale (‘1’ indicating definitely not important to ‘7’ definitely important). Agreement was determined when ≥70% of the participants rated an item within 2 points on the Importance Scale for inclusion or exclusion in the survey.31 Experts had space to submit additional items that they felt should have been included in the survey, or to give other unstructured feedback. Only experts who responded to round 1 were eligible to participate in the second round Delphi process. The second survey included only indeterminate potential survey questions (ie, <70% of experts did not rate importance within 2 points on the scale) and histograms of group responses for each survey question in round 1.30 We calculated means, medians and Agreement Scores for each potential survey item.
Cognitive interviewing
Next, we administered 79 items identified from the Delphi process to eight internal medicine residents and hospitalists at each site. We assessed participant comprehension, interpretation, readability and perception of ambiguity of each item in the draft survey using a 5-point Categorical Response Scale (‘strongly disagree’ to ‘strongly agree’). We iteratively revised survey items that were problematic. We also tested our high-value care definition among participants. They favoured a simple definition: care that tries to maximise quality while minimising costs. Participants also preferred the Institute Of Medicine (IOM) adapted definitions for quality and cost (refer to online appendix II).
Psychometric evaluation of the survey
We administered the resulting 38-item survey via Qualtrics 2015 (Provo, Utah) between July and September 2015. We approached second-year and third-year residents and hospitalists practising internal medicine at two study sites (Centres A and B) by email to complete the HVCCS. Exclusion criteria for residents included preliminary or transitional residents who temporarily participate in internal medicine resident teams and residents who spent less than 8 weeks in the prior year providing direct patient care on inpatient internal medicine services at the studied institution. Exclusion criteria for hospitalists included those who primarily identified as working in subspecialty divisions and those who spent less than 8 weeks in the prior year providing direct patient care on inpatient internal medicine services at the studied institutions. Centre A had 81 residents and 47 hospitalists, and Centre B had 81 residents and 44 hospitalists that were eligible.
We first calculated descriptive statistics for demographic characteristics of study participants including age, gender, race/ethnicity, residency track, resident career plans, payment structure and years practising internal medicine. We then conducted a series of psychometric analyses, described below.
Exploratory factor analysis
We used exploratory factor analysis to help identify the underlying structure of the items. To determine the number of underlying dimensions, we examined Guttman's weakest lower-bound principal components with eigenvalues of at least 1 and a scree plot of eigenvalues to locate the point at which the slope of decreasing eigenvalues approaches 0 and eliminating additional factors would not eliminate significant variance.32
After identifying the number of underlying dimensions, we used principal axis factoring (PFA) and Promax oblique rotation to obtain a simple solution.33 ,34 Based on the results of the factor analysis, we formed multi-item scales. We then estimated correlations of each item with the sum of other items in the same scale and with other scales. Items should correlate significantly and highly with the scale they represent (0.40 or higher) and correlate more highly with the hypothesised scale than with other scales (item discrimination across scales).35 We estimated internal consistency reliability (coefficient α) of the multi-item scales and used a benchmark of α ≥0.70 as a minimum threshold for acceptable reliability for group-level comparisons.33 ,36
Confirmatory factor analysis
We conducted categorical confirmatory factor analysis to test emergent factor solutions from exploratory factor analysis and compared them with the originally hypothesised scales. We evaluated model fit using the Normed Fit Index (NFI), Goodness of Fit Index and root mean square residual (RMSR). Members of the research team independently suggested and came to consensus on four final factor names.
Construct validity
We estimated product-moment correlation of overall HVCCS Scores and Domain Scores with the CMS institutional VBP Scores. The 2015 CMS VBP Program Total Performance Score is a publicly reported institution-level aggregate measure of clinical value representing aspects of clinical care, patient satisfaction, patient outcomes and cost. Currently, there is no gold standard to measure value. While other clinical measures of care intensity have been used as proxies of value-based care,37 ,38 they are limited since they exclude many county and community hospitals, and the data are not recent. We used the VBP publicly reported measure of value since it is widely used, can be followed over time and affects reimbursements for 80% of hospitals in the CMS VBP Program in 2014, affecting more than 3500 hospitals (ie, more than half of hospitals) in the USA.39 We averaged Individual Question Scores to obtain each of the new Factor Scores. Weighing each factor equally, we then averaged each Factor Score to obtain the overall HVCCS Score.
We hypothesised that Centre B would perform better in the overall HVCCS Score and all component factors since this group had instituted a hospital-wide value committee and curriculum over the previous 5 years, while Centre A had not yet implemented these at the time of the study. Centre B's VBP Score was also noted in the top third of academic hospitals in the state of California while Centre A's Score was in the bottom third. We also hypothesised that institutional VBP Scores would positively correlate with HVCCS Scores, estimated at the respondent level. We obtained institution-level data from the CMS VBP Program and Hospital Compare files.
Results
Modified Delphi process
In the modified Delphi process, 26 (93%) experts responded to the round 1 survey and 22 (85%) to round 2. Participants ranged in age from 31 years to 68 years with 19 (0.68) male, 6 (0.21) in procedural fields and 7 (0.25) representing key healthcare value stakeholder groups. Eleven (0.39) Delphi process participants were from the west, 10 (0.36) from the north-east, 3 (0.11) from the Midwest, 2 (0.07) from the south-east and 2 (0.07) from the south-west. Of 79 potential survey items representing eight hypothesised domains, 38 items reached ≥70% consensus from experts for inclusion and were retained.
Exploratory and confirmatory factor analysis
Seventy-nine (98%) Centre A residents, 62 (77%) Centre B residents, 39 (83%) Centre A hospitalists and 34 (77%) Centre B hospitalists completed the survey, resulting in an overall response rate of 85%.
Hospital characteristics
Centres A and B are both urban academic hospitals. Centre B has a higher average daily census, disproportionate share payment and overall VBP Performance Score driven by Hospital Consumer Assessment of Healthcare Providers and Systems and Patient Outcome Scores. Centre A has a higher Process of Care Score and slightly higher Case Mix Index (table 1).
Medical centre characteristics
Fifty-nine per cent of hospitalists at Centre A believed that they were paid by a combination of salary with adjustments for productivity compared with none of the hospitalists at Centre B (p value <0.001). More residents at Centre A planned to go into hospital medicine compared with Centre B (p value =0.01). There were no statistically significant differences between institutions in age, gender, residency track, years practising internal medicine or resident versus attending status (table 1).
Exploratory factor analysis
Guttman's weakest lower bound suggested up to eight factors (principal component eigenvalues >1), and the scree plot showed breaks at three, four, five and eight factors. We extracted eight factors initially, but the factor loading matrix supported four dimensions represented by 24 items that met our minimum factor loading criterion of 0.40 and produced scales with acceptable internal consistency reliability. We repeated the categorical PFA with unweighted least squares (ULS) extraction and Promax rotation with four factors. The rotated factor loading matrix provided a simple four-factor solution: (1) leadership and health system messaging, (2) data transparency and access, (3) comfort with cost conversations and (4) blame-free environment (see online appendix I and table 2).
High-Value Care Culture Survey factor characteristics
Confirmatory factor analysis also supported this four-factor solution according to the NFI (=0.975), Goodness of Fit Index (GFI=0.980) and RMSR (=0.058) (table 2). Coefficient α exceeded the minimum of 0.70 for the resulting four multi-item scales (table 2). (Please refer to online appendix II for the final survey instrument.)
Assessment of construct validity
Three of the four institutional HVCCS Factor Scores were significantly higher at Centre B, which is consistent with our original hypothesis since Centre B had instituted numerous high-value care initiatives and curricula. This finding suggests the HVCCS is sensitive to differences in high-value care culture. These three HVCCS factors positively correlated with VBP Scores, including leadership and health system messaging (r=0.57, p<0.001), data transparency and access (r=0.15, p<0.001) and blame-free environment (r=0.37, p<0.001). The factor ‘comfort with cost conversations’ showed no differences between sites and did not correlate significantly with our two study sites' VBP Scores. The overall HVCCS Score also positively correlated with VBP Scores (r=0.39, p<0.001) (table 3).
Construct validity of High-Value Care Culture Survey and institutional differences
Discussion
We developed a survey to measure areas that contribute to a culture of high-value care within a clinical group or training environment. The HVCCS development has methodological strengths, including broad clinician and expert input for item development and rigorous psychometric evaluation. While we began with a conceptual model to determine potential survey items, we allowed data to guide survey development to reduce bias. Front-line physicians and nurses representing a broad array of specialties, types of medical centres, training levels, regions and stakeholders contributed to the development of this survey. The four factors were reliable and provided distinct information. The overall HVCCS Score and three of its four component factors showed correlation with the CMS VBP Overall Performance Score. These results provide support for the construct validity of the HVCCS and suggest promise of the instrument in deciphering differences between and within institutions for use across health system and training programme leaders, researchers and policy makers. The HVCCS is ready for further construct validity evaluation across various specialties and hospital types, and based on our current results, may be applied as an evaluation instrument for high-value care initiatives, including curricula and improvement efforts, among hospitalists and internal medicine residents in the USA.
As this tool was created to measure culture among front-line clinicians (ie, physicians, nurses) at the division or practice level, we believe it will provide nuanced information at a local level and be useful for identifying targeted interventions that address culture change. Hospital divisions, practices and training programmes with low VBP Scores can use the HVCCS to evaluate areas needed to support behaviour change.
The four HVCCS domains—(1) leadership and health system messaging, (2) data transparency and access, (3) comfort with cost conversations and (4) blame-free environment—capture specific areas for targeted value-improvement interventions. In addition to health system managers, the HVCCS could be helpful to other key stakeholders as well. For example, health system and training programme leaders might use it to identify the processes and structures that best support a high-value care culture. Researchers and policy makers might evaluate and compare the cultural component of value within healthcare centres over time.
This information can help to further identify drivers of high-value outcomes. These efforts would be similar to how patient safety culture surveys have been widely used to monitor and improve patient safety.16–19 Experience with patient safety surveys show that culture varies considerably between organisations, and within individual units or clinical groups.13 For an organisation struggling to improve the value of its care, results on the HVCCS might become a key intervention target and a centre component of a learning healthcare system. The HVCCS consists of four factors as described below.
Leadership and health system messaging
Our first factor consists of 18 survey questions and covers leadership visibility, engagement with front-line clinicians, support for improvement efforts, role modelling, open communication about quality and costs of care, formal value training, and pride in delivering high-value care. While some of these items have been included in published patient safety culture surveys,7 ,13 ,14 others were drawn from our conceptual model and literature review.10–12 ,21 ,22 ,24 The HVCCS differs from existing patient safety culture surveys in that some survey items tested from the safety culture survey did not perform well in the modified Delphi process or psychometric analysis and were ultimately excluded from the HVCCS.
Low scores among individual questions in this factor can identify areas for improvement, such as increasing formal training related to value or providing transparent access to quality outcomes. Leaders can explicitly state that providing high-value care is a priority and may provide specific resources or incentives related to value improvement activities. In academic medical centres, creating and supporting leaders that bridge healthcare delivery and education silos may prove to be a critical mechanism for spurring culture change and improving HVCCS Scores related to this domain.40
Data transparency and access
Our second factor focuses on transparency and access to cost data for front-line clinicians. Prior safety culture surveys focused on data transparency and access to quality outcome performance.13 ,14 However, we found that access and transparency to cost data specifically performed better in psychometric evaluation compared with data about quality performance. Our findings echo recent studies suggesting that front-line providers find access and transparency to cost and utilisation data important.10 ,22 Some institutions have already determined the local cost for virtually every service and resource in their health system.41 Centres with lower scores in this domain may need to prioritise creating reliable cost databases and providing this data to clinicians.
Comfort with cost conversations
Three questions were highly correlated with clinicians' comfort with cost conversations. While prior culture surveys have included domains for staff communication, they have not addressed patient-clinician communication.13 ,14 In psychometric evaluation, items about patient-clinician communication performed better than items about staff communication that were therefore excluded from the survey. Recent studies have identified the importance of better assessment of patients’ wishes,10 ,22 ,24 and patients' underlying attitudes and beliefs to guide care.21
We believe this area provides clear opportunities for intervention. Based on regulatory pressures and perceived needs, training programmes now teach residents how to discuss challenging topics such as domestic violence, sexual histories and advanced care planning.42 ,43 Similar programmes could focus on cost conversations introduced into patient-provider discussions by using standardised financial harm screening tools or via formal training using standardised patients or interactive modules. Recent studies suggest patients are increasingly worried about the costs of medical bills for routine care,44 and two-thirds of patients say they want to talk about costs with their physicians.45
Blame-free environment
One key lesson from the patient safety field is that fear of blame often serves as a powerful obstacle to clinicians' willingness to report errors and engage in safety improvement activities.13 ,14 ,46–49 While the concept of blame has a slightly different meaning in the context of value-improvement, we hypothesised that a healthy value culture would minimise blame.10 ,22 ,24 Here, blame may involve a resident being criticised by an attending physician for failing to order a low-value imaging study, or the threat of a malpractice suit after a reasonable and evidence-based decision to forego a test. We therefore included two questions focusing on blame within divisions, practices and training programmes. Centres that score low on this domain will likely focus on promoting evidence-based practice and giving clinicians information so they can fairly weigh the risks of defensive medicine with the benefits. Medical centres may also create a non-punitive environment in which discussions of errors or poor outcomes are not unduly biased towards defensive medicine.50
Limitations
The major limitations of this study involve generalisability and validity testing. This study evaluated HVCCS among internal medicine hospitalists and residents within two academic medical centres. While they both differ in CMS VBP Scores, a multisite study will be required to evaluate the range of variation among different types of health systems and to further assess construct validity. Members of this evaluation team are currently conducting a follow-up study across academic, community and county medical centres with differing size, location, ownership and value-promoting resources. We will further evaluate HVCCS validity and assess the role of culture in predicting institution-level value outcomes. While our modified Delphi process did include multiple specialties and nurses working in various clinical settings, further validation studies are needed to assess HVCCS across other specialties and in ambulatory settings. At this current stage, this survey is specifically intended for use at a division level or practice level to promote targeted interventions.
Another limitation is the lack of a clear gold standard metric to assess high-value care culture in the field. The VBP measure is currently the best measure available with recent value data for over half of hospitals in the USA representing a broad array of hospital types, that can be followed over time and that presents true policy implications. While VBP includes both medical and surgical outcomes, clinical cultures between specialties likely do overlap. However, we have not yet evaluated this survey for use in other specialties outside of internal medicine. As the field of value metrics develops, this survey tool can undergo further evaluation and iterative refinement. Other value metrics reported in literature were felt to be less appropriate since they are not available for many county and community hospitals, do not provide recent data, focus only on the last 2 years of patients’ lives, address overutilisation but not other domains of value such as quality and patient experience, or are not linked to policy implications. Future research should consider administering HVCCS along with other culture surveys (eg, Hospital Survey on Patient Safety Culture) to examine the associations between them. This study is also cross-sectional and may benefit from further evaluation of organisational culture over time.
Conclusion
There are few issues more important to healthcare systems than the promotion of high-value care. HVCCS highlights opportunities for health system managers to identify areas for improvements within the local organisational culture. The survey also provides a unique instrument to help ensure that clinicians are supported in their efforts to deliver high-value care.
Acknowledgments
The authors thank Bradley Sharpe, MD and Michelle Mourad, MD from the Department of Medicine at UCSF, and Sandy H Berry, MA, a behavioural scientist at the RAND Corporation, for their guidance and participation with the study.
References
Footnotes
Twitter Follow Reshma Gupta at @ReshmaGuptaMD
Contributors All authors took part in conceptualising and designing the study, interpreting data and revising the manuscript. RG, JDH, VV, NS and RDH analysed data.
Competing interests RG is the Director of Outreach and Evaluation and the Director of the Teaching Value in Healthcare Learning Network at Costs of Care. She is supported by the VA Office of Academic Affiliations through the VA/Robert Wood Johnson Clinical Scholars Program. CM receives royalties from McGraw Hill for the textbook ‘Understanding Value-based Healthcare’, outside of the submitted work and is the Director of Implementation at Costs of Care. RDH was supported in part by grants from the NIA (No. P30-AG021684) and the NIMHD (No. P20-MD000182). CHB is the Vice Dean for Education at the David Geffen School of Medicine at UCLA and the chair of the American Board of Internal Medicine. RW reports that he is a member of the Lucian Leape Institute of the National Patient Safety Foundation (for which he receives no compensation); is currently chairing an advisory board to England's National Health Service reviewing the NHS’ digital health strategy (no compensation); has a contract to UCSF from the Agency for Healthcare Research and Quality to edit a patient-safety website; receives compensation from John Wiley and Sons for writing a blog; receives royalties from Lippincott Williams & Wilkins and McGraw-Hill for writing/editing several books; received a stipend and stock options for having previously served on the Board of Directors of IPC Healthcare; receives stock options for serving on the board of Accuity Medical Management Systems; serves on the scientific advisory boards for Amino.com, PatientSafe Solutions, QPID Health, Twine and EarlySense (for which he receives stock options); and holds the Benioff endowed chair in hospital medicine from Marc and Lynne Benioff. RG and CM had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Ethics approval University of California Los Angeles Institutional Review Board and University of California San Francisco Committee on Human Research.
Provenance and peer review Not commissioned; externally peer reviewed.