FindZebra: A search engine for rare diseases

https://doi.org/10.1016/j.ijmedinf.2013.01.005Get rights and content

Abstract

Background

The web has become a primary information resource about illnesses and treatments for both medical and non-medical users. Standard web search is by far the most common interface to this information. It is therefore of interest to find out how well web search engines work for diagnostic queries and what factors contribute to successes and failures. Among diseases, rare (or orphan) diseases represent an especially challenging and thus interesting class to diagnose as each is rare, diverse in symptoms and usually has scattered resources associated with it.

Methods

We design an evaluation approach for web search engines for rare disease diagnosis which includes 56 real life diagnostic cases, performance measures, information resources and guidelines for customising Google Search to this task. In addition, we introduce FindZebra, a specialized (vertical) rare disease search engine. FindZebra is powered by open source search technology and uses curated freely available online medical information.

Results

FindZebra outperforms Google Search in both default set-up and customised to the resources used by FindZebra. We extend FindZebra with specialized functionalities exploiting medical ontological information and UMLS medical concepts to demonstrate different ways of displaying the retrieved results to medical experts.

Conclusions

Our results indicate that a specialized search engine can improve the diagnostic quality without compromising the ease of use of the currently widely popular standard web search. The proposed evaluation approach can be valuable for future development and benchmarking. The FindZebra search engine is available at http://www.findzebra.com/.

Highlights

► Many clinicians use information retrieved from the web to aid the diagnostic process. Tools such as Google and PubMed are the preferred interfaces for seeking such information. ► We wanted to answer the question of how suitable these tools are in the diagnostic setting. In order to do that we sat up an evaluation approach. ► Made a dedicated easy-to-use rare disease search engine FindZebra to benchmark the web tools against. ► FindZebra is built upon open source information retrieval software and curated publicly available data. ► The evaluation performed on 56 difficult real life cases highlights the shortcomings of the web search engines and makes the case for the use of specialized search engines for specialized tasks.

Introduction

The web has become a primary source of information about illnesses or treatments [1], with an exponential growth in both volume and amount of entries available [2]. An important resource in locating medical information online is information retrieval systems, more commonly known as search engines. A December 2009 poll found that 66% of web users have searched for medical information online [3]. This class of search activities, which goes beyond simple fact retrieval, is referred to as exploratory health search [4], [5]. It can be carried out by both expert and non-expert medical users.

A typical example of an expert medical user is a clinician. Diagnostic health search can also be seen as a coarse form of hypothetico-deductive reasoning [1], where web search engines guide the iterative cycle of hypotheses about a disease being formulated from evidence and those hypotheses then guide the collection of additional discriminating evidence. According to recent studies, an increasing number of clinicians use web search engines to assist them in solving difficult medical cases, for instance when confronted with rare (or orphan) diseases [6]. The exact definition of rare diseases in terms of prevalence threshold and requirement for severity vary across the globe, but a disease is, in general, said to be rare if it affects fewer than approximately one in two thousand individuals. A study1 conducted by the European Organisation for Rare Diseases (EURORDIS), showed that 40% of rare disease patients were wrongly diagnosed before the correct diagnosis was given, and that 25% of patients had diagnostic delays between 5 and 30 years.

The current popularity of web search engines (primarily Google) and medical databases (primarily PubMed) for aiding diagnosis may appear a bit surprising as these tools are not optimised for this task. For example, (a) a diagnostic query may be quite long, whereas web search engines are typically optimised for very short queries (2–3 terms long). (b) Queries consist of lists of patient symptoms, often expressed as multi-word units. However, search engines often make term independence assumptions in order to increase efficiency. For instance, web search engines may not distinguish between “sleep deficiency, increased sexual appetite” and “sexual deficiency, increased sleep”, hence returning non-relevant results. (c) Some symptoms listed in the clinician's query may not apply to the correct disease, and conversely, some pertinent symptoms for the correct disease may be missing from the query because they are masked under different conditions. However, search engines are designed to maximise the match between all the query terms and the returned documents.

In short, the clinicians’ queries on rare diseases are likely to be more feature-rich but also harder for a search engine than ordinary web search queries, and should ideally be processed as such. Furthermore, the popularity-based metrics derived from hyperlinking (PageRank), user visit rates, or other forms of user recommendation that are commonly used by search engines are not likely to benefit the retrieval of rare diseases. These practices tend to favour webpages with many in-links (backlinks) or results often viewed by users. Information on rare diseases, on the other hand, is generally likely to be very sparse and less hyperlinked than other medical content. Finally, often efficiency concerns may lead to brute-force index pruning for web search, e.g. by removing from the index terms of low frequency or terms that are unusually long, such as “hydrochlorofluorocarbons” ([7], Chapter 5). Such practices may be particularly damaging for rare disease search, as the medical terminology involved may be exceptionally rare or formed by heavy term compounding. It is probably fair to conclude that the general popularity and ease of use compared to traditional information search and diagnostic support systems (reviewed below) are the main contributing factors to the current popularity.

Motivated by these observations we asked to what degree can web search engines actually be used for diagnosis and what are the main contributing factors that determine success and failure. To try to answer these questions it is necessary to go through a number of steps. First of all an evaluation approach has to be set up. It should consist of cases of varying degrees of difficulty and retrieval performance measures to allow for quantitative comparisons between methods. Furthermore, the web search engine algorithms are not public so one can only to a limited degree change settings and thus decipher why a query returns a given set of results. Google offers a search engine customisation product called Google Custom Search Engine,2 which has a few options for customisation that can be used to emphasise particular resources and thus determine how the choice of the information source (the index) influences the performance. If emphasising resources known to be authoritative in the rare disease domain improves the performance then one can conclude the huge index used by Google Web Search introduces noise. However, this will not give information along the “algorithm dimension”. We therefore made FindZebra, a search engine specifically designed to retrieve rare disease information for clinicians. It uses a specially curated dataset of rare disease information, which is crawled from freely available online authoritative resources. This means that FindZebra searches for rare disease information from a repository of “clean”, specialized resources, unlike web search engines that search the whole web and are hence likely to return spurious, commercial and less relevant results. The same index will be used for the customised versions of Google thus allowing us to gain insight about the adequacy of the Google Search algorithm in rare disease diagnosis.

The rest of this article is organised as follows: Section 2 discusses background work on collecting and retrieving medical information automatically with a focus on rare disease data. Section 3 presents the evaluation approach. Section 4 presents our search engine, FindZebra and the information resources used for its index. Section 5 describes the evaluation, benchmarking FindZebra, different versions of Google Search and PubMed against each other. Section 6 discusses the results and finally Section 7 summarises the findings of the paper.

Section snippets

Background

Historically, the task of retrieving medical information has relied on authoritative resources, such as the 1879 Index Medicus (which ceased publication in 2004) [2]. Since that time, the amount, availability and authority of medical resources has changed radically. More medical information is becoming freely available on the web; however, the authority of this information is not always easy to trace. A study by Eysenbach and Kohler [8] found that many users searching for medical information

Evaluation approach

The evaluation follows the standard paradigm of measuring functions of precision and recall at certain cut-off levels on a set of user queries [23]. In the evaluation we want to address two properties of the different systems simultaneously, namely the quality of the dataset (the index) and quality of information retrieval algorithms for our particular task. We can to a large degree separate the two by using the Google Custom Search functionality. In this Section we describe the diagnostic

FindZebra: a search engine for rare diseases

Our search engine is called FindZebra, as zebra is a name often given to rare diseases by medical professionals [26]. The interface of the search engine located at findzebra.com is very similar to that of standard web search engines so it should be straightforward to use by anyone familiar with web search. FindZebra is based on Indri [27], a state-of-the-art open source experimental information retrieval system. Specifically, we use Indri's indexing and retrieval functions, on top of which we

Evaluation

We evaluate and compare FindZebra and the four other systems from two perspectives. On the one hand following the standard paradigm of computing statistical measures of precision and recall, a commonly used approach in evaluating information retrieval systems. Table 2 shows the retrieval precision at rank k (P@k) and the mean reciprocal rank (MRR) of our experiments averaged for all 56 queries. The result for each query is given in Supplementary Table 2 [31], [32], [33], [34], [35], [36], [37],

Discussion

One of Google's advantages in web search is its specialized ranking algorithm optimised to work with a large sized index. Our finding, that FindZebra outperforms Google overall for this task and especially when restricted to the sites of our collection (Google Restricted), suggests that Google ranking algorithm is suboptimal for the task at hand. The poor Google Restricted results highlight this because in this case FindZebra and Google are using the same limited, focused data. When broadening

Summary and conclusion

Effective text processing tools are very important to aid biomedical researchers. There has been a remarkable surge of new advances in biomedical language processing, and web search engines in particular are becoming increasingly popular for the task of diagnosing difficult cases. In this article we have asked ourselves how efficient is web search actually for diagnosis? We therefore designed an evaluation approach and focussed on the most popular resources used namely Google Search and PubMed.

Authors contributions

R.D., P.P., C.L. and O.W. overall contributed equally to the project. R.D. and P.P. collected the data, set up the search engine, performed the evaluation under the guidance of C.L. and O.W. C.L. and O.W. wrote the paper assisted by R.D. and P.P. The remainder of the authors commented on the paper. O.W. conceived the project and the other authors primarily R.D., P.P. and C.L. contributed to further development of the project.

Conflicts of interest

None declared.

Summary points

What was already known

  • Many clinicians use information retrieved from the web to aid diagnosis.

  • It is not clear that standard web search is well suited for the task. Diagnostic queries tend to be quite unspecific whereas web search algorithms are optimized for typical web searches. Furthermore, the searched index is not restricted to relevant resources so a lot of irrelevant and non-authoritative answers may be retrieved.

What this study added to our knowledge

  • An

Acknowledgements

The authors wish to thank MD Lennart Friis-Hansen for the initial inspiration for the project. Thank you to the two anonymous referees and the editor for valuable comments.

References (50)

  • R.W. White et al.

    Cyberchondria: studies of the escalation of medical concerns in Web search

    ACM Trans. Inf. Syst.

    (2009)
  • M.G. Bouwman et al.

    Doctor Google ending the diagnostic odyssey in lysosomal storage disorders: parents using internet search engines as an efficient diagnostic strategy in rare diseases

    Arch. Dis. Child.

    (2010)
  • C.D. Manning et al.

    Introduction to Information Retrieval

    (2008)
  • G. Eysenbach et al.

    How do consumers search for and appraise health information on the world wide web? Qualitative studies using focus groups, usability test, and in-depth interviews

    Br. Med. J.

    (2002)
  • T. Lewis

    Seeking health information on the Internet: lifestyle choice of bad attack of cyberchondria?

    Med. Cult. Soc.

    (2006)
  • J. Schwarz et al.

    Augmenting web pages and search results to support credibility assessment

  • W.R. Hersh et al.

    How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review

    J. Am. Med. Assoc.

    (1998)
  • S.K. Bhavnani

    Domain-specific search strategies for the effective retrieval of healthcare and shopping information

    CHI ‘02 Extended Abstracts on Human Factors in Computing Systems, CHI EA ‘02

    (2002)
  • G.O. Barnett et al.

    DXplain. An evolving diagnostic decision-support system

    J. Am. Med. Assoc.

    (1987)
  • A. Fryer

    London dysmorphology database

    J. Med. Genet.

    (1991)
  • W.R. Hersh et al.

    Factors associated with success in searching MEDLINE and applying evidence to answer clinical questions

    J. Am. Med. Inform. Assoc.

    (2002)
  • C. Lombardi et al.

    Search engine as a diagnostic tool in difficult immunological and allergologic cases: is Google useful?

    Intern. Med. J.

    (2009)
  • H. Tang et al.

    Googling for a diagnosis – use of Google as a diagnostic aid: internet based study

    Br. Med. J.

    (2006)
  • W.B. Croft et al.

    Search Engines: Information Retrieval in Practice

    (2010)
  • C. Buckley, E.M. Voorhees, Retrieval evaluation with incomplete information, in: SIGIR, 2004, pp....
  • Cited by (0)

    View full text