Abstract
The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type “randomized controlled trial.” Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3% (±26.1) for interventions and 57.1% (±19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58% while the same rate for textword searches was significantly lower (11%, p < .001). The difference in overall precision rates was nonsignificant (26% versus 33%, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12% versus 52%, p < .001 and 41% versus 69%, p < .001). Two textwords appeared in more than 25% of the benchmark collection: reminder (25.7%) and cost (25.0%). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
Berwick, D.M., Continuous improvement as an ideal in health care. N. Eng. J. Med. 320(1):53, 1989.
Horowitz, R.I., The experimental paradigm and observational studies of cause-effect relationships in clinical medicine. J. Chron. Dis. 40(1):91, 1987.
Friedman, L.M., Furberg, C.D., & DeMets, D.L., Fundamentals of Clinical Trials (second edition), PSG, Littleton, MA, 1985.
North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N. Engl. J. Med. 325:445, 1991.
McPhee, S.J., Bird, J.A., Jenkins, C.N., & Fordham, D., Promoting cancer screening. A randomized, controlled trial of three interventions. Arch. Intern. Med. 149:1866, 1989.
Tape, T.G., and Campbell, J.R., Computerized medical records and preventive health care: success depends on many factors. Am. J. Med. 94:619, 1993.
Melin, A.L., & Bygren, L.O., Efficacy of the rehabilitation of elderly primary health are patients after short-stay hospital treatment. Med. Care 30:1004, 1992.
Dickersin, K., Min, Y., & Meinert, C. L., Factors influencing publication of research results. JAMA 267(3):374, 1992.
Funk, M.E., Reid, C.A., & McGoogan, L.S., Indexing consistency in MEDLINE. Bull. Med. Libr. Assoc. 71(2):176, 1983.
Silagy, C., Developing a register of randomized controlled trials in primary care. BMJ 306:897, 1993.
Dickersin, K., Hewitt, P., Mutch, L., Chalmers, I., & Chalmers, T.C., Perusing the literature: comparison of MEDLINE searching with a perinatal trials database. Contr. Clin. Trials 6:306, 1985.
Gøtzsche, P.C., & Lange, B., Comparison of search strategies for recalling double-blind trials from MEDLINE. Dan. Med. Bull. 38(6):476, 1991.
Kirpalani, H., Schmidt, B., McKibbon, K.A., Haynes, R.B., & Sinclair, J.C., Searching MEDLINE for randomized clinical trials involving care of the newborn. Pediatrics 83(4):543, 1989.
Dickersin, K., Scherer, R., & Lefebvre, C., Identifying relevant studies for systematic reviews. BMJ 309(6964):1286, 1994.
Hersh, W., and Hickam, D., Use of a multi-application computer worksatation in a clinical setting. Bull. Med. Libr. Assoc. 82(4):382, 1994.
McKinin, E.J., Sievert, M.E., Johnson, E.D., & Mitchell, J.A., The Medline/full-text research project. J. Am. Soc. Inf. Sci. 42(4):297, 1991.
Balas, E.A., Stockham, M.G., Mitchell, J.A., Austin, S.M., West, D.A., Ewigman, B.G., The Columbia registry of information and utilization management trials. JAMIA 2:307, 1995.
Balas, E.A., In Health Services in the United States. (D. Wedding, ed.), Behavior and Medicine, Mosby, St. Louis, 1995, p. 461.
Cohen, J., A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1):37, 1960.
National Library of Medicine. Medical subject headings, annotated alphabetic list, 1994. The Library, Bethesda, 1993.
Bernstein, F., The retrieval of randomized clinical trials in liver diseases from the medial literature: manual versus MEDLARS searches. Contr. Clin. Trials 9:23, 1988.
Jadad, A.R., McQuay, H.J., A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J. Curr. Clin. Trials 1993 (Doc No. 33), 1993.
Poynard, T., Conn, H.O., The retrieval of randomized clinical trials in liver disease from the medical literature: a comparison of MEDLARS and manual methods. Contr. Clin. Trials 6:271, 1985.
Haynes, R.B., Wilczynski, N., McKibbon, K.A., Walker, C.J., & Sinclair, J.C., Developing optimal search strategies for detecting clinically sound studies in MEDLINE. JAMIA 1:447, 1994.
L'Abbé, K.A., Detsky, A.S., & O'Rourke, K., Meta-analysis in clinical research. Ann. Int. Med. 107:224, 1987.
Sacks, H.S., Berrer, J., Reitman, D., Ancona-Berk, V.A., & Chalmers, T.C., Meta-analyses of randomized controlled trials. N. Engl. J. Med. 316:450, 1987.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Balas, E.A., Stockham, M.G., Mitchell, J.A. et al. In Search of Controlled Evidence for Health Care Quality Improvement. Journal of Medical Systems 21, 21–32 (1997). https://doi.org/10.1023/A:1022887224126
Issue Date:
DOI: https://doi.org/10.1023/A:1022887224126