Background Currently there are two widely used databases, PubMed and Google Scholar, are used for guidelines development. Research suggests PubMed is superior, however recent evidence suggests Google Scholar may have closed that gap. One family of journals reports 60% of their traffic is coming from Google Scholar.

Objectives Assess efficiency and completeness of searching for known moderate and high quality RCTs in PubMed and Google Scholar.

Methods Searches were performed by two experienced researchers using the same search terms to identify RCTs for a specific treatment. In a crossover design, one researcher performed the search in PubMed (PM1), the other in Google Scholar (GS1). Subsequently each performed the same searches in the other database (PM2 and GS2). Total numbers of articles identified, relevant articles found, and the time to complete were collected. Articles were compared to a known comprehensive list of 5 RCTs used for guideline preparation that was drawn from 6 exhaustive database searches.

Results GS1 identified 2 and GS2 identified 3 of the RCTs. PM1 identified 2 and PM2 identified 2 RCTs. PubMed and Google Scholar searches averaged 63 and 194 minutes to complete respectively.

Discussion Each database consistently identified one of the two highest quality studies, but neither database identified both. Difference search time is nearly 3-fold. No single search identified all quality studies. Additional trials are planned.

Implications for Guideline Developers/Users For comprehensive literature searches both databases should be searched.