You are here

Article Text

Article menu
Narrative review
Use of health information technology to reduce diagnostic errors
  1. Robert El-Kareh1,2,
  2. Omar Hasan3,
  3. Gordon D Schiff4,5
  1. 1Division of Biomedical Informatics, UCSD, San Diego, California, USA
  2. 2Division of Hospital Medicine, UCSD, San Diego, California, USA
  3. 3American Medical Association, Chicago, Illinois, USA
  4. 4Division of General Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
  5. 5Harvard Medical School, Boston, Massachusetts, USA
  1. Correspondence to Dr Robert El-Kareh, Division of Biomedical Informatics, UC San Diego, 9500 Gilman Dr, #0505, La Jolla, CA 92093-0505, USA; relkareh{at}ucsd.edu

Abstract

Background Health information technology (HIT) systems have the potential to reduce delayed, missed or incorrect diagnoses. We describe and classify the current state of diagnostic HIT and identify future research directions.

Methods A multi-pronged literature search was conducted using PubMed, Web of Science, backwards and forwards reference searches and contributions from domain experts. We included HIT systems evaluated in clinical and experimental settings as well as previous reviews, and excluded radiology computer-aided diagnosis, monitor alerts and alarms, and studies focused on disease staging and prognosis. Articles were organised within a conceptual framework of the diagnostic process and areas requiring further investigation were identified.

Results HIT approaches, tools and algorithms were identified and organised into 10 categories related to those assisting: (1) information gathering; (2) information organisation and display; (3) differential diagnosis generation; (4) weighing of diagnoses; (5) generation of diagnostic plan; (6) access to diagnostic reference information; (7) facilitating follow-up; (8) screening for early detection in asymptomatic patients; (9) collaborative diagnosis; and (10) facilitating diagnostic feedback to clinicians. We found many studies characterising potential interventions, but relatively few evaluating the interventions in actual clinical settings and even fewer demonstrating clinical impact.

Conclusions Diagnostic HIT research is still in its early stages with few demonstrations of measurable clinical impact. Future efforts need to focus on: (1) improving methods and criteria for measurement of the diagnostic process using electronic data; (2) better usability and interfaces in electronic health records; (3) more meaningful incorporation of evidence-based diagnostic protocols within clinical workflows; and (4) systematic feedback of diagnostic performance.

Keywords
  • Diagnostic errors
  • clinical decision support systems
  • health information technology
  • clinical informatics
  • patient safety

this is an open access article distributed in accordance with the creative commons attribution non commercial (cc by-nc 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. see: http://creativecommons.org/licenses/by-nc/3.0/

https://doi.org/10.1136/bmjqs-2013-001884

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Keywords

    Introduction

    Unaided clinicians often make diagnostic errors. Vulnerable to fallible human memory, variable disease presentation, clinical processes plagued by communication lapses, and a series of well-documented ‘heuristics’, biases and disease-specific pitfalls, ensuring reliable and timely diagnosis represents a major challenge.1–3 Health information technology (HIT) tools and systems have the potential to enable physicians to overcome—or at least minimise—these human limitations.

    Despite substantial progress during the 1970s and 1980s in modelling and simulating the diagnostic process, the impact of these systems remains limited. A historic 1970 article4 predicted that, by 2000, computer-aided diagnosis would have ‘an entirely new role in medicine, acting as a powerful extension of the physician's intellect’.5 Revisiting this prediction in 1987, the authors conceded that it was highly unlikely this goal would be achieved and that ‘except in extremely narrow clinical domains (using computers for diagnosis) was of little or no practical value’.5 In 1990 Miller and Masarie noted that a fundamental issue with many of these systems was that they were based on a ‘Greek Oracle’ paradigm whereby clinical information was provided to the computer with the expectation that it will somehow magically provide the diagnosis.6 They suggested that a more useful approach would be to use computer systems as ‘catalysts’ to enable physicians to overcome hurdles in the diagnostic process rather than have the system become the diagnostician itself.

    To understand and summarise how diagnostic accuracy can be enhanced, one needs a conceptual framework to organise HIT tools and their potential applications as ‘catalysts’ to known hurdles in the diagnostic process. Our objectives were to develop one such conceptual framework based on a review of published evidence and recent examples of HIT tools that have been used to improve diagnosis and to highlight particular areas in need of future research.

    Background

    Early leaders in computer-aided diagnosis developed statistical methods7 ,8 and models9 ,10 to serve as underpinnings for diagnostic systems. Shortliffe and colleagues skillfully organised these approaches into categories including: clinical algorithms, databank analysis, mathematical modelling of physical processes, statistical pattern recognition, Bayesian techniques, decision theory approaches and symbolic reasoning.11 Additional summaries and categorisations of the various possible approaches are also well-described in other reviews.12–14 Several applications emerged to tackle medical diagnosis in a variety of contexts, including Present Illness Program (PIP),15 MYCIN,16 INTERNIST-1/Quick Medical Reference (QMR),17 ,18 Iliad,19 DXplain20 and several others. These pioneering efforts provided a foundation for much of the current work on diagnostic systems.

    We describe recent contributions to the field, building upon the work and context provided by prior reviews of computerised diagnostic systems. In 1994 Miller summarised the work of diagnostic decision support21 and suggested that focused diagnostic systems such as those for ECG or arterial blood gas analysis were likely to proliferate. In order for more general diagnostic systems to succeed, he identified key steps which included: (1) development and maintenance of comprehensive medical databases; (2) better integration with HIT to avoid extensive data entry; and (3) improved user interfaces. Three subsequent reviews of computerised decision support22–24 identified a relatively small number of studies of diagnostic systems with only a handful showing improvement in clinician performance and only one demonstrating improved patient outcomes.25

    Methods

    Article selection

    We initially searched for studies related to diagnostic decision support systems and diagnosis-related HIT published since 2000 (see search strategy in online supplementary appendix). Because we found only modest advances during this time, we broadened the search to include some important work from earlier decades, largely obtained from previous reviews of computer-aided diagnosis.

    Taxonomy development, data extraction and categorisation

    We adapted models of the diagnostic process from Schiff et al,1 ,26 Croskerry27 and Klein28 to create a model for categorising steps in the diagnostic process addressed by HIT and similar tools (figure 1) and linked each step with categories from the Diagnosis Error Evaluation and Research (DEER) taxonomy (figure 2).1 ,26 Based on this model, we created a condensed set of categories describing different steps or aspects of diagnosis targeted by HIT tools (box 1). During data abstraction, each study was linked to one or more of these categories.

    Figure 1
    Figure 1

    Model of diagnostic process with Diagnosis Error Evaluation and Research (DEER) categories of potential errors.

    Figure 2
    Figure 2

    Diagnosis Error Evaluation and Research (DEER) taxonomy.

    Box 1

    Condensed set of categories describing different steps in diagnosis targeted by diagnostic health information technology (HIT) tools

    • Tools that assist in information gathering

    • Cognition facilitation by enhanced organisation and display of information

    • Aids to generation of a differential diagnosis

    • Tools and calculators to assist in weighing diagnoses

    • Support for intelligent selection of diagnostic tests/plan

    • Enhanced access to diagnostic reference information and guidelines

    • Tools to facilitate reliable follow-up, assessment of patient course and response

    • Tools/alerts that support screening for early detection of disease in asymptomatic patients

    • Tools that facilitate diagnostic collaboration, particularly with specialists

    • Systems that facilitate feedback and insight into diagnostic performance

    We developed a customised data extraction form using Microsoft Access 2010. Following in-depth review, we determined the following information for each study: (1) whether the study met our inclusion criteria; (2) clinical problem/question addressed; (3) type of HIT system described; (4) whether it was evaluated in a clinical setting; (5) target of the HIT intervention/tool; (6) duration/sample size of the study; (7) study outcomes; and (8) results.

    Results

    We summarised the main types of diagnostic HIT tools and mapped each type to steps in the diagnostic process that it currently or potentially targets (figure 3). Below we provide details of our findings in the 10 categories of interventions.

    Figure 3
    Figure 3

    Main types of diagnostic health information technology (HIT) tools and steps in diagnosis targeted by each type.

    Tools that assist in information gathering

    The value of a high-quality history and physical examination is well-recognised,29–31 but time pressures and reliance on clinician memory pose a major barrier to their performance. Beginning in the 1960s, various systems have been devised to assist history-taking through computer-based patient interviewing.32–34 Interestingly, these were mainly reported before the timeframe of our review, suggesting a loss of research interest for unclear reasons.35 Several recent studies have examined automated patient interviewing in specialised settings including home,36 emergency department waiting rooms37 ,38 and online visits in primary care.39 One study found that physician-acquired history and computer-based systems each elicited important information that the other missed,40 reaffirming the role of technology in complementing rather than replacing the physician-acquired history. To augment the clinician's physical examination there have been systems designed to support interpretation of auscultation, both cardiac41–43 and pulmonary.44 The state of this research also remains underdeveloped with a paucity of recent or rigorous studies.

    Cognition facilitation by enhanced organisation and display of information

    The increasing volume of electronically available patient information creates significant challenges and necessitates tools to enable efficient review of patient information and pattern recognition. One logical direction to pursue is the graphical representation of numerical data.45 One usability study found that graphical laboratory value displays led to reduced review times and that graphical and tabular representations were each more effective for answering different clinical questions.46 However, different clinical settings may benefit from differing data summary formats. For example, in a neonatal ICU, automatically generated textual summaries supported decision-making as well as graphical representations but not as well as their human-generated counterparts.47 Overall, improved organisation and display of data might facilitate identification of temporal patterns as well as helping to ensure that items do not get overlooked, especially to offset electronic health record (EHR) data hypertrophy,48 but the evidence base to date is quite limited.

    Aids to generation of a differential diagnosis

    One repeatedly demonstrated contributor to diagnostic errors is the lack of a sufficiently broad differential diagnosis.49 One suggested approach to support this process is to provide diagnostic checklists with common, ‘don't miss’ or commonly missed diagnoses for various presenting symptoms and signs.50 This approach can be facilitated with computer-based differential diagnosis list generators. While work in this area has spanned decades, we focus on recent additions to the field. Four systems in current use (Isabel, DXplain, Diagnosis Pro and PEPID) were recently reviewed and evaluated on test cases.51 There have also been various evaluations of these systems and earlier counterparts (eg, QMR and Iliad) including retrospective52–58 and simulated cases53 ,59 ,60 as well as pre–post61 and prospective62 studies. In general, these studies—although not always rigorously performed—demonstrate that the systems include the gold standard diagnosis within the output list of up to 30 diagnoses in 70–95% of cases. Whether undifferentiated lists of this length are clinically helpful requires further evidence. One study found that using such a system led to a similar number of diagnoses changed from correct to incorrect as from incorrect to correct.58

    Tools and calculators to assist in weighing diagnoses

    Once a differential diagnosis is generated, weighing the likelihood of candidate diagnoses is subject to various challenges and cognitive pitfalls.49 ,63 Several of the differential diagnosis generators described above provide rankings of their diagnostic suggestions.51 ,64 ,65 Another more quantitative approach is the use of ‘clinical prediction rules’, which are scoring systems to calculate the likelihood of diagnoses based on sets of clinical symptoms, signs or test results.66 ,67 Examples that have been recently evaluated in clinical settings include prediction rules for pulmonary embolism,68 ,69 deep vein thrombosis,70 paediatric appendicitis,71 meningitis,72–74 cervical spinal injury,75 ,76 intra-abdominal injury after blunt trauma77 and osteoporosis.78

    EHRs can embed algorithms into the workflow to determine whether one condition is present or select one diagnosis from a small predetermined set of potential candidates. Examples of such embedded algorithms have evaluated patients for pneumonia,79 ,80 acute myocardial infarction,81 postoperative infections82 and, in one broad effort, to diagnose general paediatric patients with one of 18 potential conditions.83 Although several systems showed promising results, the acute myocardial infarction system did not impact decision-making in the emergency department in a pre–post evaluation,84 and we were unable to find evaluations of the use of the other systems in clinical care.

    Support for intelligent selection of diagnostic tests/plan

    Diagnostic protocols can facilitate evidenced-based diagnostic strategies. Often they can be embedded or integrated into various electronic tools. One well-designed study found that a handheld diagnostic algorithm of evaluation of suspected pulmonary embolism with integrated clinical decision support improved the appropriateness of investigations,85 and a validation study of a chest pain protocol confirmed the safety of referring patients with low-risk chest pain to outpatient stress testing.86 We also found examples of protocols without impact, including a cluster randomised controlled trial of a protocol for evaluating skin lesions using instant cameras in primary care which failed to improve the proportion of benign lesions excised.87

    One targeted use of such algorithms and embedded electronic clinical decision support is to use the order entry function of EHRs to improve the appropriateness of diagnostic tests88 ,89 although, when tested, it failed to demonstrate an impact on the proportion of radiology tests with positive findings or improve patient outcomes.88 One group designed a ‘Smart Form’ for acute respiratory illnesses to standardise and harness clinical documentation and integrate it with diagnostic decision support.72 Usage of the system was low, with resulting minimal impact on diagnostic decision-making or antibiotic prescription appropriateness.73

    Enhanced access to diagnostic reference information and guidelines

    Simply providing access and time to review a medical textbook can support a diagnostician by avoiding exclusive reliance on memory. Various electronic approaches and products aim to support timely access to context-specific information, and these can be active (requiring the user to look up information) or more passive (information is automatically pushed to the user). One popular approach to make relevant reference information readily available is the ‘infobutton.’90 This functionality provides context-specific links from clinical systems to reference systems and is often designed to anticipate clinicians’ information needs. Infobuttons have the potential to provide diagnosis-specific information without requiring clinicians to exit the EHR to perform a separate search.91 However, studies of infobuttons to date have focused mainly on medications, with little published evidence on how they might support the diagnostic process.92 ,93

    Tools to facilitate reliable follow-up, assessment of patient course and response

    Patient follow-up and assessment of response over time is often a crucial part of ensuring an accurate diagnosis.94 An important related issue is follow-up of test results, especially those with long or variable turnaround times (eg, microbiology tests, pathology results, ‘send-out’ tests). Other studies have used tools to facilitate longitudinal automated assessments of asthma symptoms,95 visualise imaging for neuro-oncology patients over time,96 and an interactive voice response (IVR) system integrated into EHRs to provide systematic follow-up of walk-in clinic patients to screen for misdiagnoses.97

    To help improve the reliability of follow-up of the high volume of test results, electronic result managers have been created—both comprehensive systems98 ,99 as well as test-specific systems such as tools related to cancer screening or follow-up.100 ,101 Other approaches target test result follow-up for specific high-risk scenarios such as microbiology cultures pending at the time of discharge from the hospital102 or automatic gastroenterology consultations for positive faecal occult blood tests.103 When evaluated, these systems often showed improvements in process measures, although they have been insufficiently powered to show impact on clinical outcomes.

    Tools/alerts that support screening for early detection of disease in asymptomatic patients

    An important aspect of timely diagnosis is early disease detection via screening of appropriate populations,104 ,105 for which there is an extensive literature.106 Here we highlight illustrative examples. One approach involves generation of unsolicited alerts informing providers of recommended or overdue screening tests. Studies have evaluated alerts designed to screen for a wide range of conditions including cancer,107–113 osteoporosis,114–116 diabetes,117 overdue vaccinations118 and others.119–131 When studied in clinical settings, these alerts often show statistical improvements in provider performance. However, improvements are often surprisingly modest (typically 3–15% absolute improvement in screening rates). In addition to alerts targeting individual providers, population management informatics tools (eg, panel managers that list and facilitate contacting overdue patients) have been shown to be moderately effective in improving diagnostic screening rates.118 ,132

    Tools to facilitate diagnostic collaboration, particularly with specialists

    Just as instantaneous access to information and reference resources is likely to improve diagnosis, timely expert consultations can support diagnosis quality. Driven mainly by desires to support more remote/rural clinicians in obtaining consultations, ‘tele-medicine’ specialty consultation systems have been widely deployed and tested. Given the expanding numbers of articles, we cite excellent reviews rather than detail individual published studies.133 ,134 An entire journal is now devoted to this approach, featuring uses such as tele-dermatology, tele-radiology and tele-pathology.135 The objectives are not necessarily to improve a specialist's diagnosis but to achieve comparable accuracy for remote patients.136 ‘Store-and-forward’ (asynchronous) and real-time consultation technology have been reported to result in a more timely diagnosis for patients than a conventional referral process.133 An exciting, largely untapped potential for diagnostic support is facilitated collaboration and coordination among different members of the care team, including patients and their families, for facilitated access for concerning symptoms and collaborative diagnostic decision-making.137–140

    Systems that facilitate feedback and insight into diagnostic performance

    Systematic provision of feedback (immediate or longer term) to individual providers (or organisations) represents a powerful potential for improving diagnosis.94 For generations, autopsies and/or ‘second opinions’ have been used for this purpose in selected patients. Automating systematic feedback, despite its great potential, is mostly non-existent, making current medical practice largely an ‘open loop’.94 While several examples of decision support to facilitate feedback of management and screening exist,141–143 we found only one qualitative evaluation of the impact of systematic feedback of clinician diagnostic performance.144

    Discussion

    The goals of this review were to provide an overview of the current state of diagnostic HIT tools and systems and to outline a conceptual framework that can serve to suggest areas for further exploration. We adapted prior models of the diagnostic process and reviewed the published literature to create a map showing steps of the diagnostic process targeted by each group of tools. Through this iterative process, we identified areas with gaps in evidence as well as common themes to guide future work.

    Overall, we found that progress in diagnostic HIT has been slow and incremental with few significant ‘game-changing’ approaches emerging in the last decade. While there were representative studies in each of our 10 categories of tools, rigorous studies in clinical settings were very infrequent. When clinical studies were performed, benefits shown in retrospective, simulated or controlled environments have rarely been demonstrated in actual clinical practice due in part to well-described barriers common to decision support systems in general.145–147 We found limited evidence to support diagnostic protocols to guide investigations and alerts and panel management tools to improve performance of screening tests. However, for the majority of the categories of HIT tools, the evidence base was too scant to determine their utility in clinical settings.

    We believe that the field of diagnostic HIT research can move forward by focusing on a few areas. First, we need to develop the electronic ‘yardstick’ to measure the accuracy of the diagnostic process. Improved measurement will enable both targeted decision support as well as more robust and useful feedback to clinicians. Ideally, this needs to be done in a way that is well-integrated into the clinical workflow rather than requiring extensive manual data collection. Second, we should expand collaboration with cognitive science and human/computer interaction experts to improve the structure and interfaces of EHRs.148 Design and implementation of enhancements will need to be done thoughtfully to become useful in everyday practice.149–151 Third, there is an urgent need to integrate evidence-based diagnostic investigations more effectively into computerised order entry systems. The challenge is to create diagnostic protocols with enough flexibility to allow clinicians to exercise their clinical judgement but to avoid unnecessary or suboptimal diagnostic strategies as well as over-alerting. Fourth, support for systematic feedback of diagnostic performance is underdeveloped and warrants more attention. As this field evolves, evaluations of diagnostic HIT tools should assess the strength of the evidence behind them. We propose a five-level hierarchy based on the model of Fryback and Lusted152 as a way to approach such critical and evidence-based assessments (box 2).

    Box 2

    Proposed levels of evidence for evaluating diagnostic HIT tools*

    Level I. Appear useful for suggesting, weighing, or in other ways helping physicians in diagnosis-related tasks (face validity)

    Level II. Clinicians (or students) report they like and find helpful in directing them to correct diagnosis in a more timely, reliable, useful way (and ideally, regularly use them).

    Level III. Compared to not using these tools (ideally concurrent, or at least historical controls) physicians arrive at the correct diagnosis more often, sooner or more safely.

    Level IV. Improved outcomes in patients (ideally randomly assigned) for whom tools are used—fewer errors, more timely diagnosis, or more efficient or cost-efficient diagnostic evaluation process

    Level V. Tools show both improved patient outcomes and produce sufficiently greater marginal benefit to justify investment resources expended (money, clinician time) on the tools vs. other places that those resources could be invested. (ie, ROI).

    *Based on model of Fryback and Thornbury.152

    Our review has several limitations. We focused on recent work, largely excluding studies prior to 2000. While we reviewed a broad representation of tools and systems, we recognise the list was not exhaustive. Although we covered many types of interventions and approaches, we could not cover all because of time and space considerations and excluded various important domains such as computer-aided diagnostic tools for radiology studies, alarms and alerts built into monitoring equipment and support tools targeting patients and non-clinicians.

    In conclusion, we found that the field of diagnostic health information technology is still in its early stages and there has been minimal development over the past decade in various promising realms. Many aspects of the diagnostic process have been targeted, but few tools and systems have been shown to improve diagnosis in actual clinical settings. We can move the field forward by developing and testing interventions in real-world settings using cross-disciplinary research and systematic feedback of diagnostic performance.

    References

    1. ↵
      1. Schiff GD,
      2. Kim S,
      3. Abrams R,
      4. et al
      . Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project and Methodology. 2005.
    2. ↵
      1. Berner ES,
      2. Graber ML
      . Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008;121(5 Suppl Diagnostic Error in Medicine):S2–23.
      OpenUrlCrossRefPubMed
    3. ↵
      1. Newman-Toker DE,
      2. Pronovost PJ
      . Diagnostic errors: the next frontier for patient safety. JAMA 2009;301:1060–2.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Schwartz WB
      . Medicine and the computer. N Engl J Med 1970;283:1257–64.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Schwartz WB,
      2. Patil RS,
      3. Szolovits P
      . Artificial intelligence in medicine. Where do we stand? N Engl J Med 1987;316: 685–8.
      OpenUrlPubMedWeb of Science
    6. ↵
      1. Miller RA,
      2. Masarie FE
      . The demise of the Greek Oracle model for medical diagnostic systems. Methods Inf Med 1990;29:1–2.
      OpenUrlPubMedWeb of Science
    7. ↵
      1. Ledley RS,
      2. Lusted LB
      . Reasoning foundations of medical diagnosis. Science 1959;130:9–21.
      OpenUrlFREE Full Text
    8. ↵
      1. Gorry G
      . Strategies for computer-aided diagnosis. Math Biosci 1968;2:293–318.
      OpenUrlCrossRef
    9. ↵
      1. Gorry GA,
      2. Barnett GO
      . Experience with a model of sequential diagnosis. Comput Biomed Res 1968;1:490.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Spitzer RL,
      2. Endicott J
      . Diagno—a computer program for psychiatric diagnosis utilizing differential diagnostic procedure. Arch Gen Psychiatry 1968;18:746.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Shortliffe EH,
      2. Buchanan BG,
      3. Feigenbaum EA
      . Knowledge engineering for medical decision-making—review of computer-based clinical decision aids. Proc IEEE 1979;67:1207–24.
      OpenUrlCrossRefWeb of Science
    12. ↵
      1. Rogers W,
      2. Ryack B,
      3. Moeller G
      . Computer-aided medical diagnosis—literature review. Int J Biomed Comput 1979;10:267–89.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Szolovits P,
      2. Pauker SG
      . Categorical and probabilistic reasoning in medical diagnosis. Artif Intell 1978;11:115–44.
      OpenUrlCrossRefWeb of Science
    14. ↵
      1. Szolovits P,
      2. Patil RS,
      3. Schwartz WB
      . Artificial-intelligence in medical diagnosis. Ann Intern Med 1988;108:80–7.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Pauker SG,
      2. Gorry GA,
      3. Kassirer JP,
      4. et al
      . Towards simulation of clinical cognition—taking a present illness by computer. Am J Med 1976;60:981–96.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Shortliffe EH,
      2. Davis R,
      3. Axline SG,
      4. et al
      . Computer-based consultations in clinical therapeutics: explanation and rule acquisition capabilities of the MYCIN system. Comput Biomed Res 1975;8:303–20.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Miller RA,
      2. McNeil MA,
      3. Challinor SM,
      4. et al
      . The INTERNIST-1/QUICK MEDICAL REFERENCE project—status report. West J Med 1986;145:816–22.
      OpenUrlPubMedWeb of Science
    18. ↵
      1. Miller RA,
      2. Pople HE Jr..,
      3. Myers JD
      . Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 1982;307:468–76.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      Warner H Jr, Warner HR, Haug P, et al. ILIAD as an expert consultant to teach differential diagnosis. Proceedings of the Annual Symposium on Computer Application in Medical Care; American Medical Informatics Association, 1988.
    20. ↵
      1. Barnett GO,
      2. Cimino JJ,
      3. Hupp JA,
      4. et al
      . DXplain. An evolving diagnostic decision-support system. JAMA 1987;258:67–74.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Miller RA
      . Medical diagnostic decision support systems—past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994;1:8–27.
      OpenUrlAbstract/FREE Full Text
    22. ↵
      1. Johnston ME,
      2. Langton KB,
      3. Haynes RB,
      4. et al
      . Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med 1994;120:135–42.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Hunt DL,
      2. Haynes RB,
      3. Hanna SE,
      4. et al
      . Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280:1339–46.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Garg AX,
      2. Adhikari NK,
      3. McDonald H,
      4. et al
      . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223–38.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Chase CR,
      2. Vacek PM,
      3. Shinozaki T,
      4. et al
      . Medical information management: improving the transfer of research results to presurgical evaluation. Med Care 1983;21:410–24.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Schiff GD,
      2. Hasan O,
      3. Kim S,
      4. et al
      . Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881–7.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Croskerry P
      . A universal model of diagnostic reasoning. Acad Med 2009;84:1022–8.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      Klein GA. A recognition-primed decision (RPD) model of rapid decision making. In: Klein GA, Orasanu J, Calderwood R, et al. eds. Decision Making in Action: Models and Methods Norwood, NJ: Albex, 1993:138–47.
    29. ↵
      1. Hampton JR,
      2. Harrison MJ,
      3. Mitchell JR,
      4. et al
      . Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. Br Med J 1975;2:486–9.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Paley L,
      2. Zornitzki T,
      3. Cohen J,
      4. et al
      . Utility of clinical examination in the diagnosis of emergency department patients admitted to the department of medicine of an academic hospital. Arch Intern Med 2011;171:1394–6.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Sandler G
      . The importance of the history in the medical clinic and the cost of unnecessary tests. Am Heart J 1980;100(6 Pt 1 Diagnostic Error in Medicine): 928–31.
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Slack WV,
      2. Hicks GP,
      3. Reed CE,
      4. et al
      . A computer-based medical-history system. N Engl J Med 1966;274:194–8.
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. Lucas RW,
      2. Card WI,
      3. Knill-Jones RP,
      4. et al
      . Computer interrogation of patients. Br Med J 1976;2:623–5.
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Locke SE,
      2. Kowaloff HB,
      3. Hoff RG,
      4. et al
      . Computer-based interview for screening blood donors for risk of HIV transmission. JAMA 1992;268:1301–5.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Bachman JW
      . The patient-computer interview: a neglected tool that can aid the clinician. Mayo Clin Proc 2003;78:67–78.
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Slack WV,
      2. Kowaloff HB,
      3. Davis RB,
      4. et al
      . Evaluation of computer-based medical histories taken by patients at home. J Am Med Inform Assoc 2012;19:545–8.
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Benaroia M,
      2. Elinson R,
      3. Zarnke K
      . Patient-directed intelligent and interactive computer medical history-gathering systems: a utility and feasibility study in the emergency department. Int J Med Inform 2007;76:283–8.
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Porter SC,
      2. Cai Z,
      3. Gribbons W,
      4. et al
      . The asthma kiosk: a patient-centered technology for collaborative decision support in the emergency department. J Am Med Inform Assoc 2004;11:458–67.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. Adamson SC,
      2. Bachman JW
      . Pilot study of providing online care in a primary care setting. Mayo Clin Proc 2010;85:704–10.
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Zakim D,
      2. Braun N,
      3. Fritz P,
      4. et al
      . Underutilization of information and knowledge in everyday medical practice: Evaluation of a computer-based solution. BMC Med Inform Decis Mak 2008;8.
    41. ↵
      1. Mahnke C
      . Automated heart sound analysis/computer-aided auscultation: a cardiologist's perspective and suggestions for future development. Conf Proc IEEE Eng Med Biol Soc 2009;2009:3115–18.
      OpenUrlPubMed
    42. ↵
      1. Thompson WR,
      2. Hayek CS,
      3. Tuchinda C,
      4. et al
      . Automated cardiac auscultation for detection of pathologic heart murmurs. Pediatr Cardiol 2001;22:373–9.
      OpenUrlPubMedWeb of Science
    43. ↵
      1. Pretorius E,
      2. Cronje ML,
      3. Strydom O
      . Development of a pediatric cardiac computer aided auscultation decision support system. Conf Proc IEEE Eng Med Biol Soc 2010; 2010:6078–82.
      OpenUrlPubMed
    44. ↵
      1. Abbas A,
      2. Fahim A
      . An automated computerized auscultation and diagnostic system for pulmonary diseases. J Med Syst 2010;34:1149–55.
      OpenUrlCrossRefPubMed
    45. ↵
      1. Powsner SM,
      2. Tufte ER
      . Graphical summary of patient status. Lancet 1994;344:386–9.
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Bauer DT,
      2. Guerlain S,
      3. Brown PJ
      . The design and evaluation of a graphical display for laboratory data. Neuron Glia Biol 2010;6:416–24.
      OpenUrl
    47. ↵
      1. Portet F,
      2. Reiter E,
      3. Gatt A,
      4. et al
      . Automatic generation of textual summaries from neonatal intensive care data. Artif Intell 2009;173:789–816.
      OpenUrlCrossRefWeb of Science
    48. ↵
      1. Stead W
      . Healthcare culture in the United States. In: Grossmann C, Goolsby WA, Olsen LA, McGinnis JM, eds. Engineering a learning healthcare system: a look at the future: workshop summary. National Academies Press, 2011:120–5.
    49. ↵
      1. Croskerry P
      . The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003;78:775–80.
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Ely JW,
      2. Graber ML,
      3. Croskerry P
      . Checklists to reduce diagnostic errors. Acad Med 2011;86:307–13.
      OpenUrlCrossRefPubMedWeb of Science
    51. ↵
      1. Bond WF,
      2. Schwartz LM,
      3. Weaver KR,
      4. et al
      . Differential diagnosis generators: an evaluation of currently available computer programs. J Gen Intern Med 2012;27:213–19.
      OpenUrlCrossRefPubMed
    52. ↵
      1. Ramnarayan P,
      2. Cronje N,
      3. Brown R,
      4. et al
      . Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emerg Med J 2007;24:619–24.
      OpenUrlAbstract/FREE Full Text
    53. ↵
      1. Ramnarayan P,
      2. Tomlinson A,
      3. Rao A,
      4. et al
      . ISABEL: a web-based differential diagnostic aid for paediatrics: results from an initial performance evaluation. Arch Dis Child 2003;88:408–13.
      OpenUrlAbstract/FREE Full Text
    54. ↵
      1. Bavdekar SB,
      2. Pawar M
      . Evaluation of an Internet-delivered pediatric diagnosis support system (ISABEL(R)) in a tertiary care center in India. Indian Pediatr 2005;42:1086–91.
      OpenUrlPubMedWeb of Science
    55. ↵
      1. Thomas NJ,
      2. Ramnarayan P,
      3. Bell MJ,
      4. et al
      . An international assessment of a web-based diagnostic tool in critically ill children. Technol Health Care 2008;16:103–10.
      OpenUrlPubMed
    56. ↵
      1. Grober ML,
      2. Mathew A
      . Performance of a web-based clinical diagnosis support system for internists. J Gen Intern Med 2008;23:37–40.
      OpenUrlPubMedWeb of Science
    57. ↵
      1. Graber ML,
      2. Tompkins D,
      3. Holland JJ
      . Resources medical students use to derive a differential diagnosis. Med Teacher 2009;31:522–7.
      OpenUrlCrossRefWeb of Science
    58. ↵
      1. Berner ES,
      2. Maisiak RS,
      3. Heuderbert GR,
      4. et al
      . Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system. Annual Symposium Proceedings/AMIA Symposium; 2003:76–80.
    59. ↵
      1. Ramnarayan P,
      2. Roberts GC,
      3. Coren M,
      4. et al
      . Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study. BMC Med Inform Decis Mak 2006;6:22.
      OpenUrlCrossRefPubMed
    60. ↵
      1. Ramnarayan P,
      2. Kapoor RR,
      3. Coren M,
      4. et al
      . Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score. J Am Med Inform Assoc 2003;10:563–72.
      OpenUrlAbstract/FREE Full Text
    61. ↵
      1. Ramnarayan P,
      2. Winrow A,
      3. Coren M,
      4. et al
      . Diagnostic omission errors in acute paediatric practice: impact of a reminder system on decision-making. BMC Med Inform Decis Mak 2006;6:37.
      OpenUrlCrossRefPubMed
    62. ↵
      1. Graber M,
      2. VanScoy D
      . How well does decision support software perform in the emergency department? Emerg Med J 2003;20:426–8.
      OpenUrlAbstract/FREE Full Text
    63. ↵
      1. Kassirer JP,
      2. Kopelman RI
      . Cognitive errors in diagnosis: instantiation, classification, and consequences. Am J Med 1989;86:433–41.
      OpenUrlCrossRefPubMedWeb of Science
    64. ↵
      1. Lemaire JB,
      2. Schaefer JP,
      3. Martin LA,
      4. et al
      . Effectiveness of the Quick Medical Reference as a diagnostic tool. CMAJ 1999;161:725–8.
      OpenUrlAbstract/FREE Full Text
    65. ↵
      1. Demeester RP,
      2. Bottieau E,
      3. Pini A,
      4. et al
      . Prospective multicenter evaluation of the expert system “KABISA TRAVEL” in diagnosing febrile illnesses occurring after a stay in the tropics. J Travel Med 2011;18:386–94.
    66. ↵
      1. Ebell M
      . AHRQ White Paper: use of clinical decision rules for point-of-care decision support. Med Decis Mak 2010;30:712–21.
      OpenUrlCrossRefWeb of Science
    67. ↵
      1. Grady D,
      2. Berkowitz SA
      . Why is a good clinical prediction rule so hard to find? Arch Intern Med 2011;171:1701.
      OpenUrlCrossRefPubMedWeb of Science
    68. ↵
      1. Runyon MS,
      2. Webb WB,
      3. Jones AE,
      4. et al
      . Comparison of the unstructured clinician estimate of pretest probability for pulmonary embolism to the Canadian score and the Charlotte rule: a prospective observational study. Acad Emerg Med 2005;12:587–93.
      OpenUrlCrossRefPubMedWeb of Science
    69. ↵
      1. Douma RA,
      2. Mos IC,
      3. Erkens PM,
      4. et al
      . Performance of 4 clinical decision rules in the diagnostic management of acute pulmonary embolism: a prospective cohort study. Ann Intern Med 2011;154:709–18.
      OpenUrlCrossRefPubMedWeb of Science
    70. ↵
      1. Oudega R,
      2. Moons K,
      3. Hoes AW
      . Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost 2005;94:200–5.
      OpenUrlPubMedWeb of Science
    71. ↵
      1. Escriba A,
      2. Gamell AM,
      3. Fernandez Y,
      4. et al
      . Prospective validation of two systems of classification for the diagnosis of acute appendicitis. Pediatr Emerg Care 2011;27:165–9.
      OpenUrlCrossRefPubMedWeb of Science
    72. ↵
      1. Baty V,
      2. Viel JF,
      3. Schuhmacher H,
      4. et al
      . Prospective validation of a diagnosis model as an aid to therapeutic decision-making in acute meningitis. Eur J Clin Microbiol Infect Dis 2000;19:422–6.
      OpenUrlCrossRefPubMedWeb of Science
    73. ↵
      1. Nigrovic LE,
      2. Malley R,
      3. Kuppermann N
      . Meta-analysis of bacterial meningitis score validation studies. Arch Dis Child 2012;97:799–805.
      OpenUrlAbstract/FREE Full Text
    74. ↵
      1. Cohn KA,
      2. Thompson AD,
      3. Shah SS,
      4. et al
      . Validation of a clinical prediction rule to distinguish Lyme meningitis from aseptic meningitis. Pediatrics 2012;129:e46–53.
      OpenUrlAbstract/FREE Full Text
    75. ↵
      1. Stiell IG,
      2. Clement CM,
      3. Grimshaw J,
      4. et al
      . Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ 2009;339:b4146.
      OpenUrlCrossRefPubMed
    76. ↵
      1. Bandiera G,
      2. Stiell IG,
      3. Wells GA,
      4. et al
      . The Canadian C-spine rule performs better than unstructured physician judgment. Ann Emerg Med 2003;42:395–402.
      OpenUrlCrossRefPubMedWeb of Science
    77. ↵
      1. Holmes JF,
      2. Wisner DH,
      3. McGahan JP,
      4. et al
      . Clinical prediction rules for identifying adults at very low risk for intra-abdominal injuries after blunt trauma. Ann Emerg Med 2009;54:575–84.
      OpenUrlCrossRefPubMed
    78. ↵
      1. Geusens P,
      2. Dumitrescu B,
      3. van Geel T,
      4. et al
      . Impact of systematic implementation of a clinical case finding strategy on diagnosis and therapy of postmenopausal osteoporosis. J Bone Miner Res 2008;23:812–18.
      OpenUrlCrossRefPubMed
    79. ↵
      1. Aronsky D,
      2. Fiszman M,
      3. Chapman WW,
      4. et al
      . Combining decision support methodologies to diagnose pneumonia. J Am Med Inform Assoc 2001:12–16.
    80. ↵
      1. Schurink CAM,
      2. Visscher S,
      3. Lucas PJF,
      4. et al
      . A Bayesian decision-support system for diagnosing ventilator-associated pneumonia. Intensive Care Med 2007;33:1379–86.
      OpenUrlCrossRefPubMedWeb of Science
    81. ↵
      1. Baxt WG,
      2. Shofer FS,
      3. Sites FD,
      4. et al
      . A neural computational aid to the diagnosis of acute myocardial infarction. Ann Emerg Med 2002;39:366–73.
      OpenUrlCrossRefPubMedWeb of Science
    82. ↵
      1. Sakaguchi S,
      2. Takifuji K,
      3. Arita S,
      4. et al
      . Development of an early diagnostic system using fuzzy theory for postoperative infections in patients with gastric cancer. Dig Surg 2004;21:210–14.
      OpenUrlCrossRefPubMed
    83. ↵
      1. Grigull L,
      2. Lechner WM
      . Supporting diagnostic decisions using hybrid and complementary data mining applications: a pilot study in the pediatric emergency department. Pediatr Res 2012;71:725–31.
      OpenUrlCrossRefPubMedWeb of Science
    84. ↵
      1. Hollander JE,
      2. Sease KL,
      3. Sparano DM,
      4. et al
      . Effects of neural network feedback to physicians on admit/discharge decision for emergency department patients with chest pain. Ann Emerg Med 2004;44:199–205.
      OpenUrlCrossRefPubMedWeb of Science
    85. ↵
      1. Roy PM,
      2. Durieux P,
      3. Gillaizeau F,
      4. et al
      . A computerized handheld decision-support system to improve pulmonary embolism diagnosis a randomized trial. Ann Intern Med 2009;151:677–86.
      OpenUrlCrossRefPubMedWeb of Science
    86. ↵
      1. Scheuermeyer FX,
      2. Innes G,
      3. Grafstein E,
      4. et al
      . Safety and efficiency of a chest pain diagnostic algorithm with selective outpatient stress testing for emergency department patients with potential ischemic chest pain. Ann Emerg Med 2012;59:256–64.e3.
      OpenUrlCrossRefPubMed
    87. ↵
      1. English DR,
      2. Burton RC,
      3. del Mar CB,
      4. et al
      . Evaluation of aid to diagnosis of pigmented skin lesions in general practice: controlled trial randomised by practice. BMJ 2003;327:375.
      OpenUrlAbstract/FREE Full Text
    88. ↵
      1. Solberg LI,
      2. Wei F,
      3. Butler JC,
      4. et al
      . Effects of electronic decision support on high-tech diagnostic imaging orders and patients. Am J Manag Care 2010;16:102–6.
      OpenUrlPubMedWeb of Science
    89. ↵
      1. Curry L,
      2. Reed MH
      . Electronic decision support for diagnostic imaging in a primary care setting. J Am Med Inform Assoc 2011;18:267–70.
      OpenUrlAbstract/FREE Full Text
    90. ↵
      1. Cimino JJ
      . Use, usability, usefulness, and impact of an infobutton manager. AMIA Annu Symp Proc 2006:151–5.
    91. ↵
      1. Cimino JJ,
      2. Jing X,
      3. Del Fiol G
      . Meeting the electronic health record “Meaningful Use” criterion for the HL7 Infobutton standard using OpenInfobutton and the Librarian Infobutton Tailoring Environment (LITE). AMIA Annu Symp Proc 2012;2012:112–20.
      OpenUrlPubMed
    92. ↵
      1. Collins SA,
      2. Currie LM,
      3. Bakken S,
      4. et al
      . Information needs, Infobutton Manager use and satisfaction by clinician type: a case study. J Am Med Inform Assoc 2009;16:140–2.
      OpenUrlAbstract/FREE Full Text
    93. ↵
      1. Del Fiol G,
      2. Haug PJ,
      3. Cimino JJ,
      4. et al
      . Effectiveness of topic-specific Infobuttons: a randomized controlled trial. J Am Med Inform Assoc 2008;15:752–59.
      OpenUrlAbstract/FREE Full Text
    94. ↵
      1. Schiff GD
      . Minimizing diagnostic error: the importance of follow-up and feedback. Am J Med 2008;121(5 Suppl Diagnostic Error in Medicine):S38–42.
      OpenUrlPubMed
    95. ↵
      1. Hoeksema LJ,
      2. Bazzy-Asaad A,
      3. Lomotan EA,
      4. et al
      . Accuracy of a computerized clinical decision-support system for asthma assessment and management. J Am Med Inform Assoc 2011;18:243–50.
      OpenUrlAbstract/FREE Full Text
    96. ↵
      1. Hsu W,
      2. Taira RK
      . Tools for improving the characterization and visualization of changes in neuro-oncology patients. AMIA Annu Symp Proc 2010;2010:316–20.
      OpenUrlPubMed
    97. ↵
      1. Willig JH,
      2. Krawitz M,
      3. Panjamapirom A,
      4. et al
      . Closing the feedback loop: an interactive voice response system to provide follow-up and feedback in primary care settings. J Med Syst 2013;37:9905.
      OpenUrlCrossRefPubMed
    98. ↵
      1. Poon EG,
      2. Wang SJ,
      3. Gandhi TK,
      4. et al
      . Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36:80–91.
      OpenUrlCrossRefPubMedWeb of Science
    99. ↵
      1. Dalal AK,
      2. Poon EG,
      3. Karson AS,
      4. et al
      . Lessons learned from implementation of a computerized application for pending tests at hospital discharge. J Hosp Med 2011;6:16–21.
      OpenUrlCrossRefPubMed
    100. ↵
      1. Dupuis EA,
      2. White HF,
      3. Newman D,
      4. et al
      . Tracking abnormal cervical cancer screening: evaluation of an EMR-based intervention. J Gen Intern Med 2010;25:575–80.
      OpenUrlCrossRefPubMed
    101. ↵
      1. Verberne CJ,
      2. Nijboer CH,
      3. de Bock GH,
      4. et al
      . Evaluation of the use of decision-support software in carcino-embryonic antigen (CEA)-based follow-up of patients with colorectal cancer. BMC Med Inform Decis Mak 2012;12:14.
      OpenUrlCrossRefPubMed
    102. ↵
      1. El-Kareh R,
      2. Roy C,
      3. Williams DH,
      4. et al
      . Impact of automated alerts on follow-up of post-discharge microbiology results: a cluster randomized controlled trial. J Gen Intern Med 2012;27:1243–50.
      OpenUrlCrossRefPubMed
    103. ↵
      1. Humphrey LL,
      2. Shannon J,
      3. Partin MR,
      4. et al
      . Improving the follow-up of positive hemoccult screening tests: an electronic intervention. J Gen Intern Med 2011;26:691–7.
      OpenUrlCrossRefPubMed
    104. ↵
      UK National Screening Committee. Programme appraisal criteria. http://www.screening.nhs.uk/criteria (accessed 3 Feb 2013).
    105. ↵
      1. Samson D,
      2. Schoelles KM
      . Chapter 2: Medical tests guidance (2) developing the topic and structuring systematic reviews of medical tests: utility of PICOTS, analytic frameworks, decision trees, and other frameworks. J Gen Intern Med 2012;27(Suppl 1 Diagnostic Error in Medicine):S11–19.
      OpenUrlCrossRefPubMed
    106. ↵
      1. Smith RA,
      2. Cokkinides V,
      3. Brawley OW
      . Cancer screening in the United States, 2012. CA Cancer J Clin 2012;62:129–42.
      OpenUrlCrossRefWeb of Science
    107. ↵
      1. Atlas SJ,
      2. Grant RW,
      3. Lester WT,
      4. et al
      . A cluster-randomized trial of a primary care informatics-based system for breast cancer screening. J Gen Intern Med 2011;26:154–61.
      OpenUrlCrossRefPubMed
    108. ↵
      1. El-Kareh RE,
      2. Gandhi TK,
      3. Poon EG,
      4. et al
      . Actionable reminders did not improve performance over passive reminders for overdue tests in the primary care setting. J Am Med Inform Assoc 2011;18:160–3.
      OpenUrlCrossRefPubMedWeb of Science
    109. ↵
      1. Leffler DA,
      2. Neeman N,
      3. Rabb JM,
      4. et al
      . An alerting system improves adherence to follow-up recommendations from colonoscopy examinations. Gastroenterology 2011;140: 1166.
      OpenUrlCrossRefPubMedWeb of Science
    110. ↵
      1. Sequist TD,
      2. Zaslavsky AM,
      3. Marshall R,
      4. et al
      . Patient and physician reminders to promote colorectal cancer screening a randomized controlled trial. Arch Intern Med 2009;169: 364–71.
      OpenUrlCrossRefPubMedWeb of Science
    111. ↵
      1. Nease DE,
      2. Ruffin MT,
      3. Klinkman MS,
      4. et al
      . Impact of a generalizable reminder system on colorectal cancer screening in diverse primary care practices: a report from the prompting and reminding at encounters for prevention project. Med Care 2008;46:S68–73.
      OpenUrlCrossRefPubMedWeb of Science
    112. ↵
      1. Saleem JJ,
      2. Haggstrom DA,
      3. Militello LG,
      4. et al
      . Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation. BMC Med Inform Decis Mak 2011;11:74.
      OpenUrlCrossRefPubMed
    113. ↵
      1. Wagholikar KB,
      2. MacLaughlin KL,
      3. Henry MR,
      4. et al
      . Clinical decision support with automated text processing for cervical cancer screening. J Am Med Inform Assoc 2012;19: 833–9.
      OpenUrlCrossRefPubMedWeb of Science
    114. ↵
      1. Rolnick SJ,
      2. Jackson JM,
      3. Amundson JH
      . Development, implementation and evaluation of an electronic medical record prompt for bone density testing. Health Inform J 2009;15:296–304.
      OpenUrlCrossRefWeb of Science
    115. ↵
      1. Feldstein A,
      2. Elmer PJ,
      3. Smith DH,
      4. et al
      . Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial. J Am Geriatr Soc 2006;54:450–7.
      OpenUrlCrossRefPubMedWeb of Science
    116. ↵
      1. DeJesus RS,
      2. Angstman KB,
      3. Kesman R,
      4. et al
      . Use of a clinical decision support system to increase osteoporosis screening. J Eval Clin Pract 2012;18:89–92.
      OpenUrlCrossRefPubMed
    117. ↵
      1. Kenealy T,
      2. Arroll B,
      3. Petrie KJ
      . Patients and computers as reminders to screen for diabetes in family practice: randomized controlled trial. J Gen Intern Med 2005;20: 916–21.
      OpenUrlCrossRefPubMedWeb of Science
    118. ↵
      1. Loo TS,
      2. Davis RB,
      3. Lipsitz LA,
      4. et al
      . Electronic medical record reminders and panel management to improve primary care of elderly patients. Arch Intern Med 2011;171:1552–8.
      OpenUrlCrossRefPubMedWeb of Science
    119. ↵
      1. Steele AW,
      2. Eisert S,
      3. Davidson A,
      4. et al
      . Using computerized clinical decision support for latent tuberculosis infection screening. Am J Prev Med 2005;28:281–4.
      OpenUrlCrossRefPubMed
    120. ↵
      1. Apkon M,
      2. Mattera JA,
      3. Lin ZQ,
      4. et al
      . A randomized outpatient trial of a decision-support information technology tool. Arch Intern Med 2005;165:2388–94.
      OpenUrlCrossRefPubMedWeb of Science
    121. ↵
      1. Gill JM,
      2. Chen YX,
      3. Grimes A,
      4. et al
      . Using electronic health record-based tools to screen for bipolar disorder in primary care patients with depression. J Am Board Fam Med 2012;25:283–90.
      OpenUrlAbstract/FREE Full Text
    122. ↵
      1. Harinstein LM,
      2. Kane-Gill SL,
      3. Smithburger PL,
      4. et al
      . Use of an abnormal laboratory value-drug combination alert to detect drug-induced thrombocytopenia in critically ill patients. J Crit Care 2012;27:242–9.
      OpenUrlCrossRefPubMedWeb of Science
    123. ↵
      1. Lee NJ,
      2. Chen ES,
      3. Currie LM,
      4. et al
      . The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters. Adv Nurs Sci 2009;32:211–21.
      OpenUrlCrossRefPubMed
    124. ↵
      1. Chaudhry R,
      2. Tulledge-Scheitel SM,
      3. Parks DA,
      4. et al
      . Use of a web-based clinical decision support system to improve abdominal aortic aneurysm screening in a primary care practice. J Eval Clin Pract 2012;18:666–70.
      OpenUrlCrossRefPubMed
    125. ↵
      1. Walker J,
      2. Fairley CK,
      3. Walker SM,
      4. et al
      . Computer reminders for chlamydia screening in general practice: a randomized controlled trial. Sex Transm Dis 2010;37:445–50.
      OpenUrlPubMedWeb of Science
    126. ↵
      1. Lo HG,
      2. Matheny ME,
      3. Seger DL,
      4. et al
      . Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc 2009;16:66–71.
      OpenUrlAbstract/FREE Full Text
    127. ↵
      1. Schriefer SP,
      2. Landis SE,
      3. Turbow DJ,
      4. et al
      . Effect of a computerized body mass index prompt on diagnosis and treatment of adult obesity. Fam Med 2009;41:502–7.
      OpenUrlPubMed
    128. ↵
      1. Meurer WJ,
      2. Smith BL,
      3. Losman ED,
      4. et al
      . Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc 2009;57:40–5.
      OpenUrlCrossRefPubMedWeb of Science
    129. ↵
      1. Gance-Cleveland B,
      2. Gilbert LH,
      3. Kopanos T,
      4. et al
      . Evaluation of technology to identify and assess overweight children and adolescents. J Spec Pediatr Nurs 2010;15:72–83.
      OpenUrlCrossRefPubMed
    130. ↵
      1. McIntosh N,
      2. Becher JC,
      3. Cunningham S,
      4. et al
      . Clinical diagnosis of pneumothorax is late: use of trend data and decision support might allow preclinical detection. Pediatr Res 2000;48:408–15.
      OpenUrlCrossRefPubMedWeb of Science
    131. ↵
      1. Pasquale LR,
      2. Asefzadeh B,
      3. Dunphy RW,
      4. et al
      ., Ocular TeleHealth Team. Detection of glaucoma-like optic discs in a diabetes teleretinal program. Optometry 2007;78:657–63.
      OpenUrlCrossRefPubMed
    132. ↵
      1. Kesman RL,
      2. Rahman AS,
      3. Lin EY,
      4. et al
      . Population informatics-based system to improve osteoporosis screening in women in a primary care practice. J Am Med Inform Assoc 2010;17:212–16.
      OpenUrlAbstract/FREE Full Text
    133. ↵
      1. Warshaw EM,
      2. Hillman YJ,
      3. Greer NL,
      4. et al
      . Teledermatology for diagnosis and management of skin conditions: a systematic review. J Am Acad Dermatol 2011;64:759–72.
      OpenUrlCrossRefPubMed
    134. ↵
      1. Eminovic N,
      2. de Keizer NF,
      3. Bindels PJ,
      4. et al
      . Maturity of teledermatology evaluation research: a systematic literature review. Br J Dermatol 2007;156:412–19.
      OpenUrlCrossRefPubMedWeb of Science
    135. ↵
      Telemedicine and e-Health. http://www.liebertpub.com/TMJ (accessed 3 Feb 2013).
    136. ↵
      1. van der Heijden JP,
      2. Spuls PI,
      3. Voorbraak FP,
      4. et al
      . Tertiary teledermatology: a systematic review. Telemed J E Health 2010;16:56–62.
      OpenUrlCrossRefPubMedWeb of Science
    137. ↵
      1. Armstrong AW,
      2. Kwong MW,
      3. Chase EP,
      4. et al
      . Teledermatology operational considerations, challenges, and benefits: the referring providers’ perspective. Telemed J E Health 2012;18:580–4.
      OpenUrlCrossRefPubMedWeb of Science
    138. ↵
      1. Baumlin KM,
      2. Genes N,
      3. Landman A,
      4. et al
      . Electronic collaboration: using technology to solve old problems of quality care. Acad Emerg Med 2010;17:1312–21.
      OpenUrlCrossRefPubMedWeb of Science
    139. ↵
      1. Delbanco T,
      2. Walker J,
      3. Bell SK,
      4. et al
      . Inviting patients to read their doctors’ notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012;157:461–70.
      OpenUrlCrossRefPubMedWeb of Science
    140. ↵
      1. Delbanco T,
      2. Walker J,
      3. Darer JD,
      4. et al
      . Open notes: doctors and patients signing on. Ann Intern Med 2010;153:121–5.
      OpenUrlCrossRefPubMedWeb of Science
    141. ↵
      1. Thomas KG,
      2. Thomas MR,
      3. Stroebel RJ,
      4. et al
      . Use of a registry-generated audit, feedback, and patient reminder intervention in an internal medicine resident clinic—a randomized trial. J Gen Intern Med 2007;22:1740–4.
      OpenUrlCrossRefPubMedWeb of Science
    142. ↵
      1. Linder JA,
      2. Schnipper JL,
      3. Tsurikova R,
      4. et al
      . Electronic health record feedback to improve antibiotic prescribing for acute respiratory infections. Am J Manag Care 2010;16(12 Suppl HIT Diagnostic Error in Medicine):e311–19.
      OpenUrlPubMedWeb of Science
    143. ↵
      1. Roberts DH,
      2. Gilmartin GS,
      3. Neeman N,
      4. et al
      . Design and measurement of quality improvement indicators in ambulatory pulmonary care: creating a “culture of quality” in an academic pulmonary division. Chest 2009;136:1134–40.
    144. ↵
      1. Berner ES
      . Closing the Feedback Loop to Improve Diagnostic Quality: AHRQ, 2010.
    145. ↵
      1. Bates DW,
      2. Kuperman GJ,
      3. Wang S,
      4. et al
      . Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10:523–30.
      OpenUrlAbstract/FREE Full Text
    146. ↵
      1. Sittig DF,
      2. Teich JM,
      3. Osheroff JA,
      4. et al
      . Improving clinical quality indicators through electronic health records: it takes more than just a reminder. Pediatrics 2009;124:375–7.
      OpenUrlFREE Full Text
    147. ↵
      1. Lester WT,
      2. Zai AH,
      3. Grant RW,
      4. et al
      . Designing healthcare information technology to catalyse change in clinical care. Inform Prim Care 2008;16:9–19.
      OpenUrlPubMed
    148. ↵
      1. Schiff GD,
      2. Bates DW
      . Can electronic clinical documentation help prevent diagnostic errors? N Engl J Med 2010;362:1066–9.
      OpenUrlCrossRefPubMedWeb of Science
    149. ↵
      1. Flanagan ME,
      2. Saleem JJ,
      3. Millitello LG,
      4. et al
      . Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inform Assoc 2013;20:e59–66.
      OpenUrlCrossRefPubMedWeb of Science
    150. ↵
      1. Saleem JJ,
      2. Patterson ES,
      3. Militello L,
      4. et al
      . Using human factors methods to design a new interface for an electronic medical record. AMIA Annu Symp Proc 2007:640–4.
    151. ↵
      1. Yen PY,
      2. Bakken S
      . Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012;19:413–22.
      OpenUrlCrossRefPubMedWeb of Science
    152. ↵
      1. Fryback DG,
      2. Thornbury JR
      . The efficacy of diagnostic imaging. Med Decis Making 1991;11:88–94.
      OpenUrlAbstract/FREE Full Text

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors All three authors made significant contributions to the conception, design, drafting and revision of the manuscript, as well as providing final approval of the version to be published.

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.