Article Text
PDF
Abstract
Background While substantial public and scholarly attention has focused on the ethical review and oversight of quality improvement (QI) initiatives, there are no systematic data regarding the institutional mechanisms currently in place to review the conduct of QI and the ethical considerations guiding this work.
Methods/analysis The authors recruited quality improvement practitioners (QIP) affiliated with the Institute for Healthcare Improvement (IHI) ‘100 000 Lives’ Campaign to participate in a web-based survey. Standard statistical methods were used to analyse the data.
Results Surveys were completed by 132 QIPs (response rate=26%). Most respondents indicated that QI initiatives conducted with their organisation are subject to review prior to implementation. Respondents strongly agreed that ensuring minimal risk to patients, and privacy and confidentiality are relevant ethical considerations for QI initiatives conducted at their institution. A majority of respondents also agreed that assessing established practices, scientifically sound design, transparency, and the identification and minimisation of potential conflicts are relevant ethical considerations for QI initiatives.
Conclusion Many QI efforts seem to be getting some oversight, and those engaged in the QI initiatives endorse a range of ethical considerations that are suited to this work. Yet most of these reviews are not independent of those conducting the intervention and are not conducted by people trained in ethics. These findings should facilitate the development of a conceptual and policy framework that is informed by the realities of QI.
- Quality improvement
- oversight
- ethics
- survey
- practitioner
- decision-making
- healthcare quality improvement
Statistics from Altmetric.com
Introduction
Substantial attention in the professional and popular literature has focused on the ethical oversight of quality improvement (QI) initiatives. Perhaps the most vocal of these debates followed actions taken by the federal Office of Human Research Protection in response to a QI initiative directed at extending the successful use of a checklist to improve safety in intensive care units.1–5 While this particular case has been resolved, the larger questions of whether and when QI initiatives ought to be considered ‘research’ as well as how QI initiatives that are not research should be reviewed and monitored to ensure they are conducted in an ethically sound manner remain.6
While there are national efforts aimed at addressing the ethical review and oversight of QI initiatives generally7 8 and various proposals for how human subject research and QI initiatives ought to be distinguished in ethics review,9–11 there are no systematic data regarding the institutional mechanisms currently in place to review the conduct of QI initiatives or the ethical considerations that guide quality improvement practitioners (QIP) in their efforts. This is unfortunate, as attempts to develop policy regarding the ethical review and oversight of QI initiatives ought to be evidence-based.12 To this end, we conducted a survey of active QIPs to determine what types of institutional mechanisms are used to review QIP initiatives and identify what QIPs believe to be the ethical norms that guide their QI initiatives. Our ultimate goal is to inform deliberations regarding the appropriate ethical oversight for QI initiatives.
Research design and methods
Sample
Our sampling frame included the 3000 hospital/health systems that participated in the Institute for Healthcare Improvement (IHI) ‘100 000 Lives’ Campaign and, as such, are actively involved in QI work (http://www.ihi.org) The Campaign enlisted these US hospitals/health systems in adopting evidence-based interventions to reduce medical harm. IHI staff assisted in identifying a random sample of 500 QIPs active in the ‘100 000 Lives’ Campaign.
Survey development
In May 2008, we conducted a focus group of QI stakeholders (eg, patient safety officers, QI programme managers and Institutional Review Board (IRB) chairs or administrators from urban, suburban, and rural community hospitals). Among other factors that lead QI practitioners to consult/seek approval from an IRB, focus-group members identified an absence of an alternative institutional oversight body that is available to review and ensure that QI initiatives are ethically acceptable. In addition, focus-group participants indicated a number of ethical considerations they believe guide their work in QI including: transparency, minimal risk to patients and patient privacy. Based on these findings and the study team's knowledge of the field, our research team developed a survey instrument that included questions about the types of oversight mechanisms in place at their institution, their opinion of the effectiveness of these mechanisms, whether the publication of the article describing the successful intensive care unit checklist13 and the policy discussions that followed had an effect on their QI initiatives, and their assessment of a set of factors considered relevant to the ethical conduct of QI.
The draft survey was reviewed by the study team and by three experts in QI.
The survey was uploaded into Survey Monkey for administration (Survey Monkey, Portland, OR, USA). An email letter of invitation signed by PJP (co-author) and D Berwick (President, IHI) with an embedded web-link to the survey was sent to the sample of 500 eligible QIPs. Three weeks later, a reminder email including the survey web-link was sent from D Berwick.
Data analyses
All responses entered on-line were automatically transferred to an Excel (Microsoft, Seattle, Washington) database that was then uploaded into STATA for analysis (STATA, College Station, Texas). For binary (eg, yes/no) or categorical (eg, agree, strongly agree, or disagree) response variables, bivariate statistical associations (cross-tabulations using the Fisher exact test) between the responses and the independent variables were estimated. A variety of methods as delineated below were used to further explore possible associations between predictor variables (eg, respondents' attributes and opinions) and the outcomes of interest (eg, opinions regarding guiding ethical considerations). We attempted to reduce the number of predictor variables using cluster analysis of respondent attributes and opinions. We also attempted to reduce the number of ethical consideration outcomes using factor analysis and a global score (unweighted and weighted). However, these efforts were unsuccessful in identifying useful variables to pursue in multiple regression models.
Results
The initial email was sent on 21 April 2009. A total of 132 respondents completed the survey (response rate=26%); 81 responded to the initial email invitation and 51 to the follow-up message 3 weeks later.
Demographic characteristics
Of the 102 subjects who completed the demographic questions on the survey, 92 (90%) were women, 91 (90%) were white, half hold a Masters Degree, and one-third hold a nursing degree. The majority of respondents self-identify as managers either in a Quality Improvement/Safety department (n=44) or in another hospital department (n=19). Forty-five per cent have been in their current position for more than 5 years, and half indicate that 30 or more hours of their work week is dedicated to QIP initiatives. Two-thirds of respondents have participated in multisite QI initiatives, and yet only one-third completed a course in the ethics of human subject research. Forty per cent of respondents indicated that their QI initiatives have been supported by internal funding sources. Twenty per cent indicated they did not know what funds were used to support QI initiatives. Fewer than fifteen per cent of respondents indicated initiatives have been supported by foundation, State or Federal resources.
Review mechanisms
One hundred and ten respondents (83%) indicated that QI initiatives conducted by faculty/staff affiliated with their organisation are subject to some type of review prior to implementation. Seventeen respondents (13%) indicated that QI initiatives are not reviewed prior to implementation, and five did not know (4%).
Of those who indicated QI initiatives are subject to review, one-third (n=43) reported that a review is always conducted, and one-half (n=64) that a review is conducted most of the time.
Respondents were asked to indicate whether one of a number of predefined review mechanisms was in place at the institution for the review of QI initiatives and whether they thought the review mechanism was effective. The three most common review mechanisms reported were: (1) QI management team/office (n=94), (2) clinical leadership conducting QI (n=92) and (3) an Advisory Board (or equivalent) created for the purpose of reviewing QI (n=73; see table 1)
Institutional oversight mechanism for review of quality improvement (QI) initiatives
Almost all indicated that these mechanisms were very or somewhat effective. Sixteen per cent of respondents indicated that their IRB was involved in the review of QI and on average rated the IRB as more effective when compared with how effective respondents rated other review mechanisms (see table 1). Most respondents (69%) indicated that the QI review mechanism in place at their institution identifies and considers ethical issues related to QI well or very well (See table 2).
How well does the QI oversight mechanism identify and consider related ethical issues?
That is, while the data indicate that QI initiatives are routinely reviewed prior to implementation for any number of reasons (eg, administrative coordination, quality of science, budget allocation) not all institutions routinely consider ethical issues prior to implementation. Respondents were also asked what they believed was the best way to provide effective review for QI initiatives at their institution. The most common responses were the same as those which they indicated are currently in place at their institutions, though in a slightly revised order: (1) QI management team/office (n=63), (2) an Advisory Board (or equivalent) created for the purpose of reviewing QI (n=54) and (3) clinical leadership conducting QI (n=50; see table 3).
Best mechanism for oversight of QI initiatives (n=110)
Effect of QI policy discussions
Respondents were asked a series of questions to determine their familiarity with publication of the Intensive Care Unit safety checklist study13 and the ensuing policy discussion about the review of QI initiatives. On a scale from 1 to 10 (1 being not familiar, 10 being very familiar), slightly more than half indicated they were not familiar (response=1) with the ICU project, and 25% assessed their level of familiarity as a score of 6 or above. Forty per cent indicated they were not familiar (response=1) with the ensuing policy discussions, and 20% assessed their level of familiarity as a score of 6 or above. Because respondents reported a higher level of familiarity with the ensuing policy discussions than the ICU article, we explored trends related to the former. That is, we summarise below the ways in which the policy discussions had an influence on institutional policy and respondents' experiences.
Only seven respondents indicated that in response to recent policy discussions about the review of QI initiatives in general, their institution developed guidance for QIPs to help them decide whether their work needed to be submitted to an IRB. Six of the seven were among those who indicated they were familiar with the policy discussions (score of 6 or above). Respondents were also asked a series of questions about whether recent QI policy discussions in general had any effect on their ongoing or planned QI initiatives. Approximately 10% of all respondents agreed (‘sort of’ or ‘strongly’) that they were reluctant to systematically collect data on QI initiatives in which they were involved, were more likely to submit QI initiatives to the IRB, changed the way in which they conduct QI initiatives, or that a QI initiative in which they were involved was delayed, experienced a substantive change, was stopped (and later restarted or not) or was cancelled. We found no differences in responses comparing those who indicated they were familiar with the policy discussions and those who were not.
Ethical considerations
Respondents were given a set of considerations thought to be relevant to the ethical conduct of QI initiatives. They were asked to indicate whether each consideration was relevant to QI initiatives implemented at their institution and to indicate how important they believed each consideration was for the ethical conduct of QI generally. Table 4 displays the respondents' level of agreement with and ranking of each consideration.
Agreement with and ranking of ethical considerations
More than 80% of respondents strongly agreed that minimal risk to patient and privacy and confidentiality are relevant ethical considerations for QI initiatives conducted at their institution. More than 90% of respondents ‘strongly’ or ‘sort of’ agreed that assessing established practices, scientifically sound design, transparency, and the identification and minimisation of potential conflicts are relevant ethical considerations for QI initiatives conducted at their institution. Minimal risk to patients, privacy and confidentiality, and assessing established practices had an average ranking of 9.0 or greater (1=not important, 10=very important). None of the considerations were ranked below 8.5. Table 5 displays data indicating that respondents are relatively consistent in their strong agreement and ranking of the ethical considerations.
Comparing ranking of agreement with and ranking of ethical considerations
Of note, respondents rank rigorous measurement higher as a general ethical consideration than a relevant ethical consideration at their institution.
Those respondents who indicated they were more familiar with the policy discussions regarding the oversight of QI (score of 6 or greater) were more likely to ‘strongly agree’ than ‘sort of’ agree that balancing risk and benefit (p=0.019) and rigorous measurement (0.006) are relevant ethical considerations for their institution and that both considerations are important (score of 9 or 10) to the ethical conduct of QI more generally (p=0.045 and p=0.028 respectively). Rigorous measurement is a relevant ethical consideration. As an essential component of study design, poorly defined and applied measurements can compromise the integrity of any systematic collection of data, for research or QI purposes. Respondents with multisite QI initiative experience were more likely to ‘strongly agree’ than ‘sort of’ agree that shared learning (p=0.020) and stewardship (p=0.042) are important guiding consideration for their institution and that shared learning is very important to the ethical conduct of QI more generally (p=0.003).
Discussion
Findings from this national survey indicate that most (83%) QI initiatives are routinely reviewed by a variety of mechanisms at hospitals and health systems actively engaged in QI initiatives. What we do not know is whether and to what extent an ethical review of QI initiatives is routinely conducted. Few institutions report submitting some or all QI initiatives to their local IRB for an ethical review. We do know that a broader array of review mechanisms is commonly employed in practice. These include review by QI management team or QI office, clinical leadership conducting QI and an advisory board established for this purpose. The respondents, who are generally very experienced in QI, endorse the use of such models as being the best ways to conduct ethical review of QI. While more than two-thirds of respondents indicated that the review mechanisms at their institutions do a good job of considering ethical issues, only one-third of respondents indicated they had training in research ethics. In addition, how this review is conducted and whether reviewers systematically pay attention to the types of ethical considerations that are relevant to QI is unclear and should be pursued in future research efforts. Further, we do not know the effectiveness of these reviews, and given that many are conducted by staff connected to the effort and without training in research ethics, the effectiveness of the reviews should be studied. Policy makers and others engaged in considering QI need to be aware of these alternative models in beginning to assess properly whether in fact there is something that needs to be fixed before proposing a solution—for example, whether the management or clinical team actively involved in the implementation of QI initiatives is best suited to provide the ethical review and oversight of the initiatives conducted at their home institution. For the same reason that IRBs were developed to address the potential conflict of interest that investigators conducting research with human subjects may have if made responsible for the ethical review of their own work, there may be reason to consider whether similar or related conflicts exist in this arena that need to be managed by an appropriately constituted group of advisors. At a minimum, it would seem to be appropriate to suggest that some mechanism be in place at all institutions; moreover, these mechanisms should be independent of the group implementing the effort and conducted by qualified staff.
Of course, while a variety of review mechanisms are in place, and most respondents indicated that the review mechanisms in place are working well, it would be inappropriate to simply assume that this is the case. Accordingly, additional work should be conducted regarding how these mechanisms actually operate and the kinds of QI initiatives that are reviewed by them to help determine whether they are suited to the tasks at hand. Of note, few respondents indicated that QI initiatives were reviewed by IRBs at their institution. This may reflect a belief among our respondents that routine QI does not meet the regulatory definition of human subject research.14 While we did not ask respondents whether they believe any of the QI initiatives they conduct meet the definition of human subject research, Lindenauer et al found that quality officers were less likely than IRB chairs surveyed to conclude that IRB review was required for the adoption of professional practice guidelines by a single institution, evaluated through a chart review and telephone calls to patients.15
Nevertheless, the results of this survey may be helpful in considering the types of ethical issues that any robust oversight system will need to accommodate. For instance, among respondents, there is a high level of endorsement of ethical considerations thought to be relevant to the conduct of QI initiatives. That the ethical considerations minimising risk and protection of patient privacy/confidentiality were ranked highest may reflect appropriately a clear commitment to the welfare of patients served by the organisations represented among our respondents. As such, it is likely that asking explicitly about such issues in any oversight mechanism will be unlikely to encounter much resistance. In contrast, that attention to study design and rigorous measurement were not ranked as highly may indicate a need for intervention to ensure that all of those engaged in QI understand the need for such attention so that QI initiatives help meet their promises and resources appropriately. The importance of study design and rigorous measurement is increasingly important as healthcare organisations commonly make QI data public. If results are biased, consumers can be misled. Furthermore, the data we report regarding ethical considerations in oversight may be useful to those engaged in developing conceptual approaches to QI that may answer the frequent calls in the literature and in policy discussions for mid-level ethical considerations concerning QI.
Regardless, the results of this survey need to be interpreted with some limitations in mind. First, our low response rate may indicate a bias related to knowledge of this area. Those who did not respond may not consider this topic worthy of attention, even less worth their time to respond. On the other hand, those who responded may be more aware of and committed to excellence in this area, and lead to a conclusion that active oversight is more common than it actually is in practice. It is also the case that our respondents came from a sample meant to be representative of organisations active in the conduct of QI, further limiting our confidence that attention to these issues is widespread. Accordingly, it will be important to conduct similar research at a broader range of institutions. Finally, the data collected on respondent agreement with and importance of the ethical considerations we identified may indicate some response bias in favour of ethical conduct.
Conclusion
Many QI efforts seem to be getting some oversight, and those engaged in the QI initiatives endorse a range of ethical considerations that are suited to this work. Yet most of these reviews are not independent of those conducting the intervention and not conducted by people trained in research ethics. The results of this survey provide the first systematic look at these issues and direct our attention to both additional research questions and potential interventions to enhance current oversight mechanisms. Future efforts should be directed at understanding these issues at a broader range of institutions that have not necessarily committed to QI in the same way as IHI organisations. Similarly, data are needed about how effective these oversight mechanisms are in practice. These sorts of data will help move towards a conceptual and policy framework that is informed by the realities of QI.
Acknowledgments
The authors would like to thank N Kass, for her role in the focus group conducted in advance of the survey, and to N Kass and R Faden, for their review of an earlier version of the manuscript. We would also like to thank J Reed and C Gunther-Murphy at IHI, for creating the sample and coordinating correspondence with the sample; K Van Veen, for research assistance; A-M Audet, at The Commonwealth Fund for her advice regarding the survey instrument; and R Thompson and N James, from the Johns Hopkins Biostatistics Center, Johns Hopkins Bloomberg School of Public Health, for assistance with the statistical analyses.
References
Footnotes
Funding Funding for this project was provided by The Commonwealth Fund and was conducted in collaboration with the Institute for Health Improvement.
Competing interests None.
Ethics approval Ethics approval was provided by the Johns Hopkins Medicine Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Quality lines