Background The SQUIRE (Standards of QUality Improvement Reporting Excellence) guidelines were developed to improve the reporting of quality improvement (QI) projects. The effect of the guidelines on the completeness of reporting in the QI literature is unknown.
Objectives Our primary objective was to determine if the completeness of reporting in the QI literature has been improved[OUP_CE13] since the introduction of the SQUIRE guidelines.
Methods We performed a before-and-after evaluation of QI articles selected from four prominent journals of healthcare quality. Twenty-five articles published in each of two time periods (2006–2008 and 2010–2011) were confirmed to be QI projects using a standardised definition and were independently evaluated by two investigators as an interim evaluation of a planned larger sample. Articles were assessed using 50 statements of the SQUIRE guidelines, and the overall change in the completeness of reporting between the two groups was determined. The value of p<0.05 was considered significant.
Results Both groups were similar in characteristics. There was no significant difference in the mean (SD) number of SQUIRE statements completed by authors before and after publication of the SQUIRE guidelines, 20.2 (5.0) versus 20.4 (7.0), p=0.9. The study was stopped early due to the absence of any significant trend in the completeness of reporting.
Discussion There was no overall improvement observed in the completeness of reporting of QI projects after the publication of the SQUIRE guidelines, and the study was stopped early. There is potential for improvement in reporting standards, particularly for those guideline items or statements specific to QI projects.
- Healthcare quality improvement
- Quality improvement methodologies
- Quality improvement
Statistics from Altmetric.com
Reporting guidelines are endorsed by journals, editors and experts in reporting standards as effective methods for improving the quality of manuscripts published in the healthcare literature.1–3 To improve the quality of published research reports within different medical and research specialties, there are an increasing number of specialty-specific reporting guidelines being developed and implemented by journals to meet the varying demands of specific study types and populations.4–6 While quality improvement (QI) projects typically focus on improving care at a local level, many aspects of QI projects have external validity and it is vital that project methodologies and interventions are transparent and reproducible to facilitate the effective interpretation and implementation of published QI findings and recommendations.
The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines were published in 2008 to improve the quality and completeness of reporting in the QI literature.7 ,8 They were developed and tested extensively prior to introduction using a modified Delphi process and consensus opinion from experts and stakeholders such as journal editors and authors involved in QI and reporting standards in the medical literature.9–12
The SQUIRE guidelines comprise a checklist of 19 items described in six different text sections (see online supplementary appendix). These 19 items comprise a total of 50 specific statements that, when applicable, contribute to reproducible reporting standards across published QI projects. The guidelines aim to provide a comprehensive framework for reporting QI projects in the medical literature but may also assist in the planning phase of QI interventions as they contain elements that can facilitate the development and implementation of QI projects.13–15 Despite consensus recommendations by experts for the use of the SQUIRE guidelines when preparing QI manuscripts, and the adoption or endorsement by some editorial boards within their ‘Instructions for Authors’, it is not known if the SQUIRE guidelines have improved the completeness of reporting of QI projects.16
We hypothesised that QI projects published in the healthcare literature after introduction of the SQUIRE guidelines would demonstrate increased completeness of reporting compared with projects published prior to SQUIRE publication. The primary aim of this study was to evaluate the effect of the SQUIRE guidelines on the completeness of reporting of QI projects published in prominent medical journals evaluating QI in healthcare before and after the introduction of SQUIRE guidelines in 2008.
Based on expert opinion and the Eigenfactor ranking of journals in the Journal Citation Reports of Web of Science (Thompson Reuters), we identified four prominent healthcare quality journals that regularly publish QI articles. The Eigenfactor was chosen as it estimates the relative influence of journal references based on cross-citation data for 5 years.17 Regularly publishing QI reports was defined as at least one QI report in each journal issue. The journals identified were the American Journal of Medical Quality, BMJ Quality and Safety (formerly Quality and Safety in Healthcare), the International Journal for Quality in Healthcare and the Joint Commission Journal on Quality and Patient Safety.
None of the journals included make adherence to the SQUIRE guidelines mandatory for the reporting of QI projects. BMJ Quality and Safety and the Joint Commission Journal on Quality and Patient Safety encourage authors to use the SQUIRE publication guidelines. Neither the International Journal for Quality in Healthcare nor the American Journal of Medical Quality makes reference to SQUIRE within their instructions to authors, although the American Journal of Medical Quality did publish an editorial in 2010 advocating the use of the guidelines for QI reports.14
We searched the electronic archives of these journals for all publications published between July 2006 and June 2008 (pre-SQUIRE) and between July 2010 and June 2012 (post-SQUIRE). The 24-month period between the two timeframes was excluded to allow time for guideline dissemination and implementation. A single investigator (CM) reviewed all titles, and abstracts when indicated, for all articles published during these study timeframes to determine article eligibility. Articles were reviewed within each of the two timeframes until the minimum numbers of eligible QI projects were identified or the study window ended, whichever event occurred first. QI articles were defined as original reports that were primarily concerned with systematic approaches to making changes leading to better patient outcomes, stronger system performances or enhanced professional development. To be eligible for inclusion in this study, we classified a QI project and report as an activity or intervention initiated with the specific goal of improving the performance of medical practice or delivery of healthcare in relation to an established standard. The project design should use established QI methodology (such as Plan-Do-Study-Act cycles) whereby a systematic action or pattern of actions are applied, observed and analysed with the purpose of optimising productivity, communication and/or healthcare delivery. The QI report should describe implementation, measurement of effect, analysis of outcome and sustainability of efforts.18–20 Articles that described quality assurance projects, editorials, reviews, case reports, letters and short communications were excluded.
Two investigators (AES and VH) independently extracted data from all included articles using a standardised template. Studies were assessed in chronological order from the start of the timeframe for each group. Descriptive data collected included the month and year of publication, journal title, location and setting of the study. The completeness of reporting was assessed by reviewing articles for each of the 50 statements of the SQUIRE guidelines. The reviewers assessed whether each aspect of the guideline item was reported (complete, incomplete, not applicable) in the manuscript. When guideline statements contained multiple descriptors, they were recorded as complete when the majority (>50%) of the descriptors were reported. Text relevant to a specific SQUIRE statement that was completed in a different section than that recommended was recorded as ‘complete’ during analysis. In the event of disagreement between both data extractors, a consensus was reached or a third investigator (CM or JDO) was consulted to resolve any dispute. Both investigators evaluating completeness of reporting of study articles received training in the SQUIRE guidelines prior to the study to ensure consistency and accuracy of data extraction. The education sessions were led by a faculty member of the University of Toronto's Centre of Quality Improvement and Patient Safety (CM) and consisted of directed reading and discussion of the SQUIRE literature. Data collectors then independently scored five QI articles for completeness of reporting of SQUIRE statements using the same criteria as the study, for which substantial interobserver agreement was observed (overall agreement for 250 SQUIRE statements in five papers was 88%, κ=0.77).
As this study is the first to investigate the effect of the SQUIRE guidelines on the completeness of QI reporting in the healthcare literature, the minimum sample size needed was difficult to determine a priori. High-impact medical journals previously demonstrated completeness of reporting rates varying from 35% to 73% prior to publication of the CONSORT (Consolidated Standards for Reporting of Trials) statement, and the completeness of reporting tended to be lower again for some subspecialty publications.6 Based on the findings of other studies evaluating reporting guidelines, a relative increase in overall statement reporting of at least 50% was considered to be significant and anticipated by other studies investigating the effect of reporting guidelines. We estimated that a minimum sample size of 97 publications per group (pre-SQUIRE and post-SQUIRE) would be required to detect a difference in the proportion of overall SQUIRE statements reported from 0.4 to 0.6, at two-tailed values of α=0.05 and β=0.2. Using these data, we anticipated that a total of 194 articles would be required for inclusion in this study. We planned an interim analysis using the first 25 articles in each of the pre-SQUIRE and post-SQUIRE groups to determine the accuracy of our initial sample size estimation as it was anticipated that significant variation in the completeness of reporting among publication types and specialties might occur and our sample size estimation was based on studies investigating the effectiveness of the CONSORT statement.
The primary outcome was the overall number of SQUIRE statements (n=50) addressed by authors. Data were recorded in Microsoft Excel 2010 and analysed using Prism 6 (GraphPad Software, San Diego, California, USA). Data are presented as medians or means with measures of variance appropriate to the data distribution. The differences in the number of statements completed between groups were compared using the χ2 test or Fisher's exact test. OR and 95% CIs were determined where appropriate. The value of p<0.05 was considered statistically significant. We report our study according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.21
From the four journals selected for this study, 1182 articles were identified and reviewed to determine study eligibility (figure 1). The most common reasons for exclusion of reviewed abstracts were not having a QI study design (eg, case reports) or articles incorrectly classified as QI reports (eg, retrospective studies not conducted using accepted QI methodology).
In the post-SQUIRE group, we identified 97 potential articles eligible for inclusion between July 2010 and October 2011; therefore no further articles were screened for eligibility. Conversely, in the pre-SQUIRE group, within the 2-year window from July 2006 to June 2008, only 56 articles meeting study inclusion criteria were identified. We present the results of our interim analysis of those 50 articles selected from the pre-SQUIRE and post-SQUIRE groups.
The characteristics of the articles included are summarised in table 1. Pre-SQUIRE and post-SQUIRE groups had similar demographics, and the proportion of articles from each of the four journals was similar for each group. The majority of articles included in this study were published in BMJ Quality and Safety (n=26, 52%), originated from the United States of America (n=35, 70%) and the project setting was an academic, university-affiliated hospital (n=25, 50%).
Data for the primary outcome, the number of SQUIRE statements completed, were normally distributed. The mean (SD) number of SQUIRE statements completed by authors in both groups did not differ significantly before and after publication of the SQUIRE guidelines, 20.2 (5.0) vs 20.4 (7.0), p=0.9. Only one article in the post-SQUIRE group reported using the SQUIRE guidelines.
Comparing each of the 50 statements between groups, there was no significant difference in individual statement reporting pre-SQUIRE and post-SQUIRE, with the exception of Item 17a (‘Interpretation: Explores possible reasons for differences between observed and expected outcomes’). This item was reported less frequently in the post-SQUIRE group than the pre-SQUIRE group, 6 versus 16 (OR 0.2; 95% CI 0.0 to 0.7). Completeness of reporting of SQUIRE statements for the six text sections of the guidelines is reported in table 2.
The number of guideline statements reported in a section of the article different from that recommended by the SQUIRE guidelines was significantly reduced in the post-SQUIRE group (32 vs 17; p=0.04).
Due to the lack of statistical difference between the groups in the overall number of SQUIRE statements reported and the anticipated sample size needed to detect any real difference between the pre-SQUIRE and post-SQUIRE groups, it was decided not to include further QI reports for analysis and to terminate the study early.
This study attempted to evaluate the effects of the SQUIRE guidelines on the completeness of reporting of QI projects in the healthcare literature. In a small sample (n=50) of QI reports published in four prominent quality and safety journals, we demonstrated no overall improvement in the completeness of reporting of articles published after dissemination of the SQUIRE guidelines. The study was stopped early due to the absence of any statistical or meaningful difference between groups and the unfeasible sample size that would be required to detect the smallest of differences.
In similar studies investigating the impact of STROBE and CONSORT statements on observational and randomised trials, respectively, a small positive effect was found but this was potentially confounded by secular variation in the quality of reporting in medical journals.4 ,6 The SQUIRE guidelines and checklist were developed over a period of time, first appearing as draft guidelines in 2005, then undergoing several iterations of discussion and revision before final publication in 2008. During those 3 years, many articles describing the development of the guidelines were published. Therefore, it is reasonable to expect that by the time of final publication, the SQUIRE statement and its aim to improve the completeness and quality of reporting were already well disseminated among the community most likely to adopt the guidelines.7 ,22 However, in our study, completeness of reporting was found to be low and was comparable prepublication and postpublication of the SQUIRE guidelines, suggesting that SQUIRE did not have any demonstrable effect on the completeness of QI reporting in the specialty journals chosen for this study.
Of note, 2 years after publication of the STARD (STAndards for the Reporting of Diagnostic accuracy studies) [OUP_CE14] statement (guidelines to improve completeness of reporting of diagnostic accuracy studies), an observational cohort study demonstrated only a small improvement in quality of reporting.5 It was proposed that it was likely too early to detect a more significant improvement in this study. In contrast, the findings of a before-and-after analysis of the effect of the CONSORT statement reported an increase in the quality of reporting of randomised controlled trials published 2 years after publication of the guideline.6 There is a tendency for the completeness of reporting in the biomedical literature to increase over time, and as a result of this phenomenon we anticipated that our study design had potential to erroneously attribute a positive effect of the SQUIRE guidelines on completeness of reporting of QI projects.4 ,23 However, despite a 2-year interval between the study groups, we observed no trend towards increased completeness of reporting after the introduction of SQUIRE.
While there were no significant improvements in the reporting of any of the 50 individual statements of the SQUIRE checklist, there were some statements that were not regularly included that could make it impossible to replicate and successfully implement an apparently effective QI intervention (see online supplementary appendix). Such items included the following: the ability to assess intervention implementation (Items 10a and 11a); internal and external validity (Item 10e); data quality and accuracy (Items 11b and c); internal feedback (Item 13a iv); and harm, unexpected failure and limitations (Items 13b iii, 16c, 16d, 16e). Comparative before-and-after evaluations of the impact of CONSORT statement on randomised controlled trials, and STROBE statement on observational studies, confirmed concerns regarding how discussions and comment sections are reported and demonstrated that reporting rates were lowest for items describing efforts to limit potential sources of bias, discussion of external validity, discussion of limitations, management of missing data, confounding and sample size estimation.4 ,6 Our study does not measure the quality of reporting in QI publications but does provide some focus for similar areas of concern with regard to QI reporting. It is unfortunate that many such items are vital components required for adopting and successfully implementing the findings of published quality interventions, suggesting that there is a greater role for SQUIRE in improving the completeness of QI reporting and an argument to be made[OUP_CE15] for placing the SQUIRE statement more centrally in the planning and development phases of QI projects. Future studies could expand on the findings suggested by this small study and investigate the completeness of reporting of QI projects across a wider selection of journals.
While other studies evaluating reporting guidelines have used Medical Subject Headings (MeSH) to identify specific publication subject categories or types, we were unable to use this approach due to the diverse subject matter of QI reports and the lack of applicable MeSH vocabulary. To overcome this, a single investigator (CM) performed a hand search of the included journals and reviewed the titles, and abstracts when needed, of all articles published to determine eligibility.
The central limitation of this study is the small sample size on which we base our findings. The lower-than-expected number of QI projects published in the period before SQUIRE precluded us from meeting the initial sample size estimation or analysing the time trend for completeness of reporting without extending the timeframe. The search methodology used for this study to identify eligible journals may have contributed to this by not considering subspecialty journals which regularly publish QI reports. Another limitation of this study is that blinding of the reviewers to the date of publication was not possible due to time being an important component of QI interventions which can be mentioned repeatedly throughout reports. This study evaluated completeness of reporting and did not aim to measure quality of reporting according to specific SQUIRE statements, but a more complete manuscript is not necessarily a surrogate for superior quality of reporting.24 Similarly, for this study all 50 SQUIRE statements were given an equal weighting for analysis; however, each statement may not be of equal importance or relevance as there are components of the guidelines that are particularly associated with QI and are unique to SQUIRE. One criticism of reporting guidelines for the QI literature is that they may limit the autonomy of authors and journals on how to best present QI reports. Editorial styles differ between journals and QI reports do not have the same homogeneity of settings or methodology compared with some other study designs; however, the authors of SQUIRE recognise this limitation and caution against adhering to the guidelines too strictly.13 Nevertheless, it is of particular importance when reporting QI projects that interventions are presented in such a way that every reader can judge the applicability, validity and ability to implement reported interventions in their respective centres with the expectation of similar outcomes.3
In conclusion, this small before-and-after study demonstrated no statistical or meaningful difference in the completeness of reporting of QI projects after publication of the SQUIRE guidelines. The lack of positive findings in this study suggests that the SQUIRE guidelines are not being optimally implemented by authors and journals publishing QI reports. In 2015, the next iteration of SQUIRE guidelines will be published, providing new opportunities for better promotion and increased advocacy of SQUIRE by authors and journals that regularly publish QI projects.25
Review history and Supplementary material
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contributors All team members contributed a significant part to the design and development of this study, the methodology, data analysis, and the writing and editing of the manuscript.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.