Background Systematic reviews are the foundation for evidence-based guidelines. Rigorous standards exist, but there is wide variation in implementation, highlighting the need for a more comprehensive quality assessment tool for systematic reviews.

Objective To develop a tool that sufficiently evaluates major biases relevant to experimental and observational study designs.

Methods The Documentation and Appraisal Review Tool (DART) was developed using epidemiologic principles of study design and the following resources: Overview Quality Assessment Questionnaire (OQAQ), Assessment of Multiple Systematic Reviews (AMSTAR), the Cochrane Handbook, and the standards promoted by the Agency for Healthcare Research and Quality, and the Institutes of Medicine (IOM). DART underwent multiple rounds of testing and revisions.

Results Compared to OQAQ and AMSTAR, DART includes two unique questions and several questions covered by OQAQ or AMSTAR but not both. OQAQ and DART had the highest reporting consistency. Four AMSTAR questions elicited inconsistent responses. Identifying reviewer rationale was most difficult using the OQAQ tool, and easiest using DART.

Discussion DART allows for documentation of reviewer rationale, facilitating reconciliation between reviewers and documentation for future updates. DART also allows for evaluation of major biases relevant to observational study designs and the assessment of standards recommended by the March 2011 IOM Standards for Systematic Review.

Implications for Guideline Developers/Users The proliferation of systematic reviews provides guideline developers the opportunity to utilise pre-existing research to produce evidence-based guidelines. However, the wide variation in quality means developers will need to carefully assess the quality of systematic reviews using a tool such as DART.