Article Text
Abstract
Objective To develop search strategies for identifying papers on patient safety in MEDLINE, EMBASE and CINAHL.
Methods Six journals were electronically searched for papers on patient safety published between 2000 and 2006. Identified papers were divided into two gold standards: one to build and the other to validate the search strategies. Candidate terms for strategy construction were identified using a word frequency analysis of titles, abstracts and keywords used to index the papers in the databases. Searches were run for each one of the selected terms independently in every database. Sensitivity, precision and specificity were calculated for each candidate term. Terms with sensitivity greater than 10% were combined to form the final strategies. The search strategies developed were run against the validation gold standard to assess their performance. A final step in the validation process was to compare the performance of each strategy to those of other strategies found in the literature.
Results We developed strategies for all three databases that were highly sensitive (range 95%–100%), precise (range 40%–60%) and balanced (the product of sensitivity and precision being in the range of 30%–40%). The strategies were very specific and outperformed those found in the literature.
Conclusion The strategies we developed can meet the needs of users aiming to maximise either sensitivity or precision, or seeking a reasonable compromise between sensitivity and precision, when searching for papers on patient safety in MEDLINE, EMBASE or CINAHL.
- Patient safety
- medical errors
- safety management
- review literature as topic
- information storage and retrieval
- risk management
- evidence-based medicine
Statistics from Altmetric.com
- Patient safety
- medical errors
- safety management
- review literature as topic
- information storage and retrieval
- risk management
- evidence-based medicine
After the Institute of Medicine in the USA released its 1999 report To Err Is Human, publications on patient safety proliferated.1 Given the growing emphasis on evidence-based practice and decision making, accessing these resources is an important factor in improving patient safety.2 However, end users—researchers, health professionals and decision makers—encounter several barriers to retrieving and using this knowledge: difficulties in accessing journals, lack of time for an effi-cient search, inadequate technical skills, database complexity, cost of information searches and others.3–5
This is why many authors strongly recommend engaging a librarian for information searches.2 6–8 However, with the increased availability of electronic databases, most users conduct their own bibliographic searches,9 10 generally producing results with poor sensitivity (missing relevant items) or precision (with a great number of irrelevant items).3 4 9
To optimise electronic database searches, many authors advise using search filters.11–13 These filters combine textwords (or terms in the title and abstract) and specific index terms to retrieve relevant articles or documents of a given type or on a given subject in a particular database. Thus, the literature increasingly offers search filters developed to facilitate the process of retrieving information that could otherwise take months to develop and would mobilise significant resources.14 Most are methodological filters aimed to retrieve studies carried out using a specific methodology, such as randomised controlled trials.9 The methods used to develop them vary greatly. First-generation filters are derived subjectively with no objective measure of their efficacy. Filters derived from second-generation methods are constructed the same way but evaluated against a “gold standard”—a set of relevant records used to test efficacy. In third-generation methods, the quest for objectivity begins in the first stages of developing the gold standard, continues with measurements of filter efficacy and culminates in a validation using an independent set of relevant records.15
This article offers, to those interested in patient safety in the context of healthcare processes, effective search filters developed using third-generation methodology that can save time, effort and resources when searching for papers on patient safety—that is, academic publications explicitly mentioning the term “patient safety” or dealing with subjects such as medical, diagnostic or medication errors; adverse or undesirable events; effects or outcomes; or medical injuries, in three databases: MEDLINE (OVID interface), EMBASE (OVID interface) and CINAHL (EBSCOhost interface).
Method
The search filters were constructed and validated in several stages.
First, to develop the gold standard, journals were selected from a corpus of articles on patient safety known to the authors from their previous work in the field. This initial list was expanded to include journals from the extensive bibliography on the National Patient Safety Foundation's website,i as well as from Ulrich's Periodicals Directory,ii which lists international journals on many subjects, including patient safety.
Of the 55 journals thus listed, we retained six (BMJ (2000), JAMA (2001), New England Journal of Medicine (2003), Quality & Safety in Health Care (2002–2006), American Journal of Nursing (2002) and Nursing Management (2005–2006)) using these criteria: the journal is indexed in either MEDLINE, EMBASE or CINAHL; complete texts were easily available online; the journal published at least one article on patient safety between 2000 and 2006, as verified by a quick search in MEDLINEiii; and nursing journals were represented.
In a second phase, the journals selected were simultaneously searched in MEDLINE, EMBASE and CINAHL. The search produced 11 380 articles, which were imported into TrialStat's SRS platform for systematic reviews,16 iv an Internet Technology solution that automates the workflow and data management aspects of systematic reviews through the screening and data abstraction phases. Two research assistants separately reviewed the 7003 records left after eliminating duplicates. An initial screening of these articles based on titles and abstracts eliminated 6535 articles that did not explicitly deal with patient safety. After a full reading of the remaining 468 articles, 147 that dealt specifically with patient safety in the healthcare system were retained and constituted our gold standard. Because they were not uniformly listed in all three databases, the size of the gold standard varied: MEDLINE (147), EMBASE (140) and CINAHL (108).
To allow us to validate the search filters with journals not used in their construction, the gold standard was divided into two convenience samples: a “composition gold standard” using JAMA (2001), Quality & Safety in Health Care (2004–2005 and 2006) and Nursing Management (2005), and a “validation gold standard” using BMJ (2000), New England Journal of Medicine (2003), Quality & Safety in Health Care (2002–2003) and Nursing Management (2006). After an in-depth screening, no relevant reference was found in the American Journal of Nursing (2002). The composition gold standard was made up of 83, 79 and 56 references from MEDLINE, EMBASE and CINAHL, respectively, while the validation gold standard had 64, 61 and 52.
In a third phase, EndNotev files containing the articles of the composition gold standard for each database were created to identify terms to retain in constructing the search filters: free text from article titles or abstracts and specific index terms used by the databases to classify the documents. Lists of terms from each source (titles, abstracts, keywords) were created using EndNote's “define term lists” tool. Two reviewers examined these lists independently to select all potentially relevant terms and then classified them according to a conceptual design that would guide their AND/OR linkages. Any disagreements between the two assessments were resolved by a third reviewer. Three concepts constituted the skeleton of this design: PATIENT for terms referring to patients, SAFETY for terms referring to safety and CARE for terms referring to care processes.
New lists were developed in Microsoft Excel to ascertain the number of gold standard articles in which these terms appeared at least once, to calculate a certain number of indicators and to do the definitive selection of the terms that would be used to develop the search filters.
Five indicators were used to evaluate the individual terms and final searches:
Sensitivity: (number of relevant articles retrieved by the search)/(total number of relevant articles in the universe of articles (gold standard))*100.
Precision: (number of relevant articles retrieved by the search)/(total number of articles extracted by the search)*100.
Specificity: (number of non-relevant articles not retrieved by the search)/(number of non-relevant articles in the universe of articles)*100.
Sensitivity*precision: the product of sensitivity and precision, which allows us to reach an equilibrium between sensitivity and precision.
Number of articles to read: 1/precision, which allows us to identify the number of non-relevant articles that must be read to find one relevant article.
To validate the search filters, we used three distinct procedures. First, we measured the performance of filters developed in the validation gold standard to evaluate their internal validity—that is, the integrity of the approach used to develop them. Second, an important step in our validation process consisted of verifying whether there were already high-performance search strategies in the literature for retrieving articles on patient safety. Westwood et al's17 systematic review for the National Health Service published in 2002 presented various strategies used in several databases to retrieve studies on patient safety. Their definition of patient safety was close to ours and, like us, in the search strategies they proposed for MEDLINE, EMBASE and CINAHL, they put no restrictions on the type of publication retrieved. We reproduced these strategies as faithfully as possible in the OVID and EBSCOhost interfaces (see Appendix A) to retrieve papers in our two gold standards. Finally, to establish external validity, a new gold standard was generated from readings suggested by the National Patient Safety Foundation between 1999 and 2008. This is a corpus of articles entirely different from those of the study and covering several more years. This gold standard will allow us to determine the extent to which our filters can be generalised. The results of this last phase will be presented in a second article.
Results
All individual terms with >10% sensitivity were retained for inclusion in combinations to be used in constructing final search strategies. Figures 1–3 present the individual terms that were chosen based on the conceptual design for MEDLINE, EMBASE and CINAHL. Figures for sensitivity, precision and specificity are indicated, respectively, in parentheses after each term.
By combining terms using “OR”, our goal was to achieve search strategies with 100% sensitivity and 100% specificity, and by using “AND”, to achieve at least 50% precision.
All combinations illustrated in figures 1–3 were tested and classified according to sensitivity, precision, the product of sensitivity and precision, and specificity. Together, sensitivity and specificity are used to assess the intrinsic value of the strategies: high sensitivity means nothing without high specificity. A strategy is considered useful if the sum of sensitivity and specificity exceeds 100%.18
The objective was to present three types of search strategies for three different types of users, based on Boynton's19 terminology. The most sensitive strategies are for “sensitivity maximizers”—users prepared to read a large number of documents for the certainty of identifying the majority of documents on the topic. The more precise strategies are aimed at “precision maximizers”—users who want to identify a maximum number of documents with minimum resource outlay. Between these extremes are strategies for users seeking a reasonable compromise between sensitivity and precision.
Tables 1–3 present these different groups of strategies for the three databases. The performances of the search strategies in the validation gold standard are presented in parentheses. The last row presents the capacity of the filters developed using Westwood et al's study to identify the composition gold standard and validation gold standard articles.
We should mention that none of these filters impose any restrictions on types of articles, language, age groups, subject groups and others.
Using a conceptual design allowed us to create some interesting combinations. The strategies devised were very sensitive and highly precise. The fact that they were as effective in the validation group attested to the construction procedure's strong internal validity. The sum of sensitivity and specificity largely exceeds 100% for most of the strategies.
Westwood et al's strategy compares very well to the most precise strategy developed for MEDLINE. On the other hand, only two of the strategies developed had a lower sensitivity than Westwood et al's strategy.
The search strategies developed for EMBASE are systematically more sensitive than that proposed by Westwood et al, and their precision is equal to that of the most precise strategy proposed for EMBASE.
As for CINAHL, the strategy obtained is more precise and just as sensitive as the most sensitive strategy we developed. On the other hand, we propose other, more precise strategies that offer a better balance between sensitivity and precision.
Discussion
Search filters developed using third-generation methods offer numerous advantages over filters developed more intuitively. Nevertheless, they are not without shortcomings, some of which apply to the specific case presented in this article.
First, the method is still somewhat subjective. Some decisions were taken arbitrarily: the selection of keywords to be retained from the texts for the combination of terms; the classification of the index terms in the conceptual design when the databases did not clearly define these terms; the decision to use the “explode” function, which sacrifices precision, for certain index terms; and the decision to set 10% as the minimal sensitivity level required for including individual terms in the search filters (combining the most sensitive terms does not necessarily produce more sensitive strategies).
We also did not do any preliminary study to determine the size of the gold standard. Because individual terms making up the search filters were selected based on frequency, the size of the gold standard clearly has an impact on what terms are ultimately selected.
Another potential for bias is in the choice of journals from which the gold standard is extracted. We chose mostly academic journals in medicine and nursing that publish articles in English. Clearly, these are not representative of all journals that could provide material on patient safety, neither in terms of content, or of disciplines represented, nor in quality of indexing.
Despite these biases, the search filters proposed in this article can lay claim to several strengths. First, an effort was made to ensure temporal representativity. The gold standard articles span from 2000 to 2006. Given the dynamic nature of index terms, this is an advantage over filters developed using articles from a single year. An effort was also made to assess the internal validity, which was shown to be very good. The use of the conceptual design allowed us to combine very effective individual terms logically, which ultimately resulted in filters that performed very well with regard to sensitivity, specificity and precision.
Conclusion
We offer a range of high-performance search filters to users wishing to retrieve articles on patient safety in three well-known and often-used databases: MEDLINE, EMBASE and CINAHL. The choice of which strategy to use will depend on the needs of the users and their temporal, financial and human resource constraints.
Search filters are not a panacea; however, if used appropriately, they are a valuable asset in carrying out a systematic literature review or for clinicians or researchers requiring quick searches with high precision.
Acknowledgments
This research was funded by the Canadian Institutes of Health Research (CIHR). Scholarships were offered by
▶ the programme AnÉIS (Analyse et Évaluation des Interventions en Santé/Analysisand Evaluation of Health Interventions) funded by CACIS (Chair in Community Approaches and Health Inequities)/GETOS (Chair in Governance and Transformation of Health Organisations)/CHSRF (Canadian Health Services Research Foundation)/CIHR
▶ the SSHRC (Social Sciences and Humanities Research Council of Canada)
▶ the GIRU (Groupe Interuniversitaire de Recherche sur les Urgences / Interuniversity Research Group on Emergencies)
▶ the Government of Côte d'Ivoire
▶ the University of Montreal and
▶ the GRIS (Groupe de Recherche Interdisciplinaire en Santé /Interdisciplinary Team for Research in Health). The team includes Affaud Anaïs Tanon, François Champagne, André-Pierre Contandriopoulos, Marie-Pascale Pomey, Alain Vadeboncoeur, Hung Nguyen, Derson Bruno and Vary Jacquet.
Appendix A The original strategies used by Westwood et al and their translation into the OVID and EBSCOhost interfaces
Presentationvi
Westwood et al's study, published in 2002, was designed to draw together a diverse collection of research and provide a map of the patient safety research literature. In particular, the following questions were addressed:
What have been the goals of patient safety research?
What methods have been used in patient safety research?
What types of studies have shown what kinds of results?
Studies included were those in which patient safety falls somewhere on the continuum from what have been called errors and deviations, through dangerous situations and near misses to accidents. Research studies that have investigated the roots of individual or system failure, detection of error and direct interventions designed to reduce accidents have been included. Hospital care has been the main focus (as this is where the majority of research has been conducted) although other healthcare settings have been considered.
Efforts to improve safety require an understanding of the organisational culture and working conditions that influence safe conduct; therefore, research studies addressing this issue have also been included. All types of research have been considered, quantitatively and qualitatively, to reflect the approaches used in patient safety research. Publications that are potentially of interest but which fall outside of being empirical research have not been included because of time constraints. In particular, there are a number of key texts—for example, Clinical Risk Management: Enhancing Patient Safety, edited by Vincentvi, which presents useful material, ranging from the principles of risk management through to the implementation of risk management strategies—which have not been included as they present little, if any, direct research. The searches have dated back to 1990, as this is when a body of research documenting the problem of medical errors began to emerge.
Initial strategy | Our translation |
MEDLINE (Silverplatter/ARC searched—03/07/01)1966–2000/2002 | MEDLINE OVID 1950–2008 |
1. “Accident-Prevention” | 1 Accident prevention/ |
2. “Safety”/ all subheadings | 2 Safety/ |
3. “Accident-proneness” | 3 Accident proneness/ |
4. “Medical-errors”/ all subheadings | 4 Medical errors/ |
5. “Medication-errors”/ all subheadings | 5 Medication errors/ |
6. Patient safety in ti ab | 6 Patient safety.ti,ab. |
7. Medical error* in ti ab | 7 Medical error$.ti,ab. |
8. Medication error* in ti ab | 8 Medication error$.ti,ab. |
9. Human error* in ti ab | 9 Human error$.ti,ab. |
10. Detect* near ((error* or incident* or accident*) in ti ab) | 10 (detect$ adj (error$ or incident$ or accident$)).ti,ab. |
11. Reporting near ((error* or incident* or accident* or system*) in ti ab) | 11 (Reporting adj (error$ or incident$ or accident$)).ti,ab. |
12. Information system* in ti ab | 12 Information system$.ti,ab. |
13. Prescribing system* in ti ab | 13 prescribing system$.ti,ab. |
14. “Organizational-culture” | 14 Organisational culture/ |
15. “Fatigue”/ all subheadings | 15 Fatigue/ |
16. Explode “Stress-Psychological”/ all subheadings | 16 Exp stress, psychological/ |
17. Explode “Information-Systems”/ all subheadings | 17 Exp information systems/ |
18. “Risk-management”/ all subheadings | 18 Risk management/ |
19. “Safety-management”/ all subheadings | 19 Safety management/ |
20. Risk management in ti ab | 20 Risk management.ti,ab. |
21. Safety management in ti ab | 21 Safety management.ti,ab. |
22. (Fatigue or workload* or tired*) near ((error* or harm or incident* or accident* or patient safety or job performance or work performance) in ti ab) | 22 ((Fatigue or workload$ or tired$) adj (error$ or harm or incident$ or accident$ or patient safety or job performance or work performance)).ti,ab. |
23. Incident* report* in ti ab | 23 Incident$ report$.ti,ab. |
24. #10 or #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 | 24 10 or 11 or 12 or 13 or 14 or 15 or 16 or 17 or 18 or 19 or 20 or 21 or 22 or 23 |
25. #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 | 25 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 |
26. #25 and #24 | 26 24 and 25 |
Initial strategy | Our translation |
EMBASE (Silverplatter/ARC—searched—06/07/01) 1980–2001/2006 | EMBASE OVID 1996–2008 |
1. “Medical-Errors”/ all subheadings | 1 Medical Error/ |
2. “Medication-Errors”/ all subheadings | 2 Medication Error/ |
3. Patient safety in ti ab | 3 patient safety.ti,ab. |
4. Medical error* in ti ab | 4 medical error$.ti,ab. |
5. Medication error* in ti ab | 5 medication error$.ti,ab. |
6. Human error* in ti ab | 6 human error$.ti,ab. |
7. Detect* near ((error* or incident* or accident*) in ti ab) | 7 (detect$ adj (error$ or incident$ or accident$)).ti,ab. |
8. Reporting near ((error* or incident* or accident* or system*) in ti ab) | 8 (reporting adj (error$ or incident$ or accident$ or system$)).ti,ab. |
9. Information system* in ti ab | 9 information system$.ti,ab. |
10. Prescribing system* in ti ab | 10 prescribing system$.ti,ab. |
11. “Risk-Management”/ all subheadings | 11 risk management/ |
12. Risk Management in ti ab | 12 Risk Management.ti,ab. |
13. Safety Management in ti ab | 13 Safety Management.ti,ab. |
14. (Fatigue or workload* or tired*) near ((error* or harm or incident* or accident* or patient safety or job performance or work performance) in ti ab) | 14 ((fatigue or workload$ or tired$) adj (error$ or harm or incident$ or accident$ or patient safety or job performance or work performance)).ti,ab. |
15. Incident* report* in ti ab | 15 incident$ report$.ti,ab. |
16. (“Accident-Prevention”) or (“Safety”/ all subheadings) or (“Accident-Proneness”) or #1 or #2 or #3 or #4 or #5 or #6 | 16 Accident Prevention/ or Safety/ or Accident Proneness/ or 1 or 2 or 3 or 4 or 5 or 6 |
17. #7 or #8 or #9 or #10 or (“Fatigue”/ all subheadings) or #11 or #12 or #13 or #14 or #15 or (explode “stress”/ all subheadings) | 17 7 or 8 or 9 or 10 or Fatigue/ or 11 or 12 or 13 or 14 |
18. #1 or #2 or #3 or #4 or #5 or #6 | 18 1 or 2 or 3 or 4 or 5 or 6 |
19. #7 or #8 or #9 or #10 or #11 or #12 or #13 or #14 or #15 | 19 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 |
20. #18 and #19 | 20 18 and 19 |
Initial strategy | Our translation | |
CINAHL (Silverplatter/ARC—searched—06/07/01) 1982–2001/2004 | CINAHL EBSCOhost 1982-2008) | |
1. “Patient-Safety”/ all topical subheadings/ all age subheadings | S1 | (MH “Patient Safety”) |
2. Explode “Treatment-Errors”/ all topical subheadings/ all age subheadings | S2 | (MH “Treatment Errors+”) |
3. Explode “Diagnostic-Errors”/ all topical subheadings/ all age subheadings | S3 | (MH “Diagnostic Errors+”) |
4. Patient safety in ti ab | S4 | TI (patient safety) OR AB (patient safety) |
5. Medical error* in ti ab | S5 | TI (medical error*) OR AB (medical error*) |
6. Detect* near ((error* or incident* or accident*) in ti ab) | S6 | TI (detect* AND (error* OR incident* OR accident*)) OR AB (detect* AND (error* OR incident* OR accident*)) |
7. Reporting near ((error* or incident* or accident* or system*) in ti ab) | S7 | TI (reporting AND (error* OR incident* OR accident* OR system*)) OR AB (reporting AND (error* OR incident* OR accident* OR system*)) |
8. Information system* in ti ab | S8 | TI (information system*) OR AB (information system*) |
9. Prescribing system* in ti ab | S9 | TI (prescribing system*) OR AB (prescribing system*) |
10. “Organisational-Culture” | S10 | (MH “Organisational Culture”) |
11. “Fatigue”/ all subheadings/ all age subheadings | S11 | (MH “Fatigue”) |
12. Explode “Stress-Psychological”/ all subheadings/ all age subheadings | S12 | (MH “Stress, Psychological+”) |
13. Explode “Information-Systems”/ all subheadings/ all age subheadings | S13 | (MH “Information Systems+”) |
14. “Risk-Management”/ all subheadings/ all age subheadings | S14 | (MH “Risk Management”) |
15. Safety Management in ti ab | S15 | TI (Safety management) OR AB (Safety management) |
16. (Fatigue or workload* or tired*) near ((error* or harm or incident* or accident* or patient safety or job performance or work performance) in ti ab) | S16 | TI ((fatigue or workload* or tired*) AND (error* or harm or incident* or accident* or (patient AND safety) or (job AND performance) or (work AND performance))) or AB ((fatigue or workload* or tired*) AND (error* or harm or incident* or accident* or (patient AND safety) or (job AND performance) or (work AND performance))) |
17. Incident* report* in ti ab | S17 | TI (incident* report*) OR AB (incident* report*) |
18. #1 or #2 or #3 or #4 or #5 | S18 | S1 OR S2 OR S3 OR S4 OR S5 |
19. #6 or #7 or #8 or #9 or #10 or #11 or #12 or #13 or #14 or #15 or #16 or #17 | S19 | S6 OR S7 OR S8 OR S9 OR S10 OR S11 OR S12 OR S13 OR S14 OR S15 OR S16 OR S17 |
20. #18 and #19 | S20 | S18 OR S19 |
References
Footnotes
Funding Canadian Institutes of Health Research, 160, rue Elgin, 9e étage Indice de l'adresse 4809A Ottawa, Ontario, Canada K1A 0W9.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
↵iii For each of the 55 journals, and for the years 2000, 2002, 2004 and 2006, a quick search was performed in MEDLINE using the strategy (Medical Errors/ or (Safety/ and exp Patient/)). The records thus found were called “patient safety records”. The journals were then sorted according to the number of patient safety records.
↵v A software tool for publishing and managing bibliographies. http://www.endnote.com
↵vi from Westwood et al's study report: Westwood M, Rodgers M, Sowden A. Patient safety: a mapping of the research literature. York, UK: National Health Service Centre for Reviews and Dissemination.
Linked Articles
- Quality lines