Original ArticleSingle data extraction generated more errors than double data extraction in systematic reviews
Introduction
The purpose of a systematic review is to provide a comprehensive, valid, and reliable synthesis of the evidence regarding a particular question. There are numerous measures used during the conduct of systematic reviews to ensure that they meet these criteria. The stage of data extraction involves transcribing information reported in the primary studies to a standard form that has been developed to capture all relevant information in a format specific to the review question. This stage is critical to the results and conclusions of the review, as the subsequent analysis and interpretation stem from the information extracted.
Recent investigations have raised questions regarding the accuracy and reliability of the data extraction process in systematic reviews. Strang et al. [1] found substantial discordance between two experts extracting data from 12 clinical trials. Discrepancies occurred with respect to extraction of samples sizes, design, start and end dates, selection criteria, and secondary outcomes. The authors recommended enhanced rigor in data extraction including systematic, double extraction. In another investigation, data extraction was repeated by a medical statistician for 28 systematic reviews published by the Cochrane Cystic Fibrosis and Genetic Disorders Group [2]. The rate of inaccuracies was found to be “unacceptably high,” which has led to new policies to reduce errors associated with data extraction. Contrary to these findings, Haywood et al. [3] reported good agreement in data extraction between three reviewers with different expertise using an electronic database with careful instruction and training.
The currently recommended method of data extraction in systematic reviews is for two researchers to independently extract data and come to consensus over discrepancies through discussion or in consultation with a third party (double data extraction) [4], [5]. The consensus process involves resolving differences between two sets of data extraction. This method is labor-intensive, time-consuming, and costly. An alternative method is for one researcher to independently extract data, a second researcher to verify the data extraction, and consensus over discrepancies is reached through discussion or in consultation with a third party (single data extraction).
To our best knowledge, there is no empirical evidence regarding the types and magnitude of errors arising from these two methods and their relative impact on the results of meta-analysis. Our objective was to conduct a pilot study to describe and compare the types and frequency of errors that accompany single vs. double data extraction; compare the estimate of treatment effect derived from single vs. double data extraction; and compare the time required for single vs. double data extraction.
Section snippets
Methods
Our Center recently conducted a systematic review of the efficacy and safety of melatonin for the management of sleep disorders [6], under contract with the Agency for Healthcare Research and Quality. The efficacy review included 30 randomized controlled trials of the effect of melatonin on people with sleep disorders. The outcomes of interest were sleep onset latency, sleep efficiency, wakefulness after sleep onset, total sleep time, sleep quality, REM latency and percent time in REM sleep.
Rate of disagreement
The overall rate of disagreement between the two methods was 28.0 % [95% confidence interval (CI): 25.4, 30.7] (Table 1). The rate ranged from 11.1 to 47.2% among studies (median: 27.8%) and from 0.0 to 76.7% among variables (median: 23.3%). The variables with the lowest rates of disagreement were language of publication, gender distribution of population, type of sleep restriction disorder, and study interventions (0.0%). The variable with the highest rate of disagreement was outcome reporting
Discussion
The rate of disagreement between methods of data extraction was high. A number of factors may have contributed to this finding. The majority of reviewers had minimal knowledge of the content area, and some reviewers had minimal experience in data extraction and data verification, which may have led to an increase in the number of errors, and therefore, in the number of disagreements. Although detailed instruction was provided to reviewers with respect to the items to be extracted for each
Conclusion
In summary, single data extraction generates more errors but requires less time than double data extraction. The decision to employ single vs. double data extraction for a review depends on a number of factors, including the time and human resources available for the review. If there are adequate resources to employ double data extraction, then this method should be used. In the case that single data extraction is used, reviewers and readers need to be mindful of the possibility for more errors
Acknowledgments
N.B. participated in designing the study, data collection, data analysis, and writing and editing the manuscript. L.H. participated in designing the study, data analysis, and writing and editing the manuscript. B.V. participated in designing the study, statistical analysis of data, and writing and editing the manuscript. L.T. participated in writing and editing the manuscript. T.P.K. reviewed and provided feedback on the manuscript. The authors acknowledge Nicola Hooton, Brad Johnston, Daniel
References (6)
- Strang WN, Boissel P, Uberla K. Inter-reader variation. 2nd International Conference. Scientific Basis of Health...
- Jones A. Improving the reliability of data extraction [abstract]. XI Cochrane Colloquium: evidence, health care and...
- et al.
Reviewing measures of outcome: reliability of data extraction
J Eval Clin Pract
(2004)