Effects of neural network feedback to physicians on admit/discharge decision for emergency department patients with chest pain

doi:10.1016/j.annemergmed.2004.02.037

Annals of Emergency Medicine

Volume 44, Issue 3, September 2004, Pages 199-205

https://doi.org/10.1016/j.annemergmed.2004.02.037 Get rights and content

Abstract

Study objective

Neural networks can risk-stratify emergency department (ED) patients with potential acute coronary syndromes with a high specificity, potentially facilitating ED discharge of patients to home. We hypothesized that the use of “real-time” neural networks would decrease the admission rate for ED chest pain patients.

Methods

We conducted a before-and-after trial. Consecutive ED patients with chest pain were evaluated before and after implementation of a neural network in an urban university ED. Data included 40 variables used in neural networks for acute myocardial infarction and acute coronary syndrome. Data were obtained in real time, and neural network outputs were provided to the treating physician while patients were in the ED. On hospital discharge, attending physicians received feedback, including neural network output, their initial clinical impression, cardiac test results, and final diagnosis. The main outcome was the actual admit/discharge decision made before versus after the implementation of the neural network.

Results

Before implementation, 4,492 patients were enrolled; after implementation, 432 patients were enrolled. Implementation of the neural network did not decrease the hospital admission rate (before: 62.7% [95% confidence interval (CI) 61.3% to 64.1%] versus after: 66.6% [95% CI 62.2% to 71.0%]). Additionally, the ICU admission rates were not different (11.4% [95% CI 10.5% to 12.3%] versus 9.3% [95% CI 6.6% to 12.0%]). Physician query found that the neural network changed management in only 2 cases (<1%).

Conclusion

The use of real-time neural network feedback did not influence the admission decision for ED patients with chest pain, most likely because the neural network output was delayed until the return of cardiac markers, and the disposition decision had already been made by that time.

Introduction

Rapid and accurate identification of low-risk emergency department (ED) patients with chest pain syndromes remains problematic.1., 2. Clinical and computer algorithms can successfully risk-stratify patients; however, they are not able to identify a group of patients at such low risk that they could be safely and immediately released from the ED.3., 4., 5., 6., 7., 8., 9., 10., 11., 12.

A number of approaches have been developed to assist physicians in improving their diagnostic accuracy for myocardial infarction.8., 9., 13., 14., 15. It has been suggested that physicians will not use any method that purports to improve diagnostic accuracy unless it is easy to use and dramatically and consistently improves their performance.16., 17. Most approaches developed to aid in the diagnosis of myocardial infarction have not met these requirements. Many of them have failed when applied in the field,¹⁷ in part because they depend on complete and accurate data to maintain optimum performance, which in real-time application is rarely possible.

The artificial neural network is a powerful modality for the recognition of complex patterns that can maintain accuracy when some input data are missing.¹⁰ The network maps a complex multidimensional space defined by the constellation of input information, characterizing patterns that are provided during training. During testing, the network determines the goodness of fit of unknown patterns to that mapped space. Network function allows for an infinite variability of input information weighting. It is thought that the network's ability to assign the weight given to a specific input variable being determined by the values of all other input variables is what enables it to perform more accurately than other statistical approaches.

A number of studies have revealed that the artificial neural network can accurately identify the presence of myocardial infarction in patients with chest pain.10., 11., 18., 19., 20., 21. The artificial neural network achieves better diagnostic accuracy for acute myocardial infarction and acute coronary syndrome than other widely accepted risk-stratification tools.²² Despite data showing the superiority of neural networks over other risk-stratification algorithms, they have yet to be used in “real time.” We sought to determine the impact of real-time artificial neural network use on ED chest pain patient disposition and physician decisionmaking.

Section snippets

Study design

We performed a before-and-after trial to assess the impact of neural network feedback on physician decisionmaking for ED patients with chest pain, as assessed by their admit/discharge decision. At the conclusion of the study, we queried physicians to assess their satisfaction and trust with the neural network.

Setting

Patients were enrolled at an urban, tertiary care hospital ED with an annual census of approximately 53,000 adult visits. The “before” portion of the study was conducted between July 1999

Results

There were 4,492 patient visits for whom data collection instruments were completed before implementation of the neural network. After neural network implementation, an additional 432 patient visits were evaluated. Patients in both portions of the study were generally similar with respect to demographic, historical, and other clinical characteristics used for computation by the neural networks (Table). Neural network outputs were produced for 100% of patients in the neural network phase of the

Limitations

There are several limitations to our study. Although patients in both portions of our trial appeared similar with respect to demographic and clinical characteristics at presentation, the group that presented after implementation of the neural network was less likely to sustain an acute myocardial infarction or have a discharge diagnosis of acute coronary syndrome, which, however, should have biased the “after” group toward a lower admission rate.

We did not conduct a comprehensive instructional

Discussion

One of the strengths of the neural network is that its accuracy is not compromised when some input data are missing. Despite requiring more input data (and therefore being more likely to have occasional missing data), the neural network outperforms other risk-stratification algorithms, such as the Goldman and acute cardiac ischemia time-insensitive predictive instrument algorithms, 11., 23. which makes it a potentially valuable tool in the ED, where the emergency physician rarely has access to

References (28)

J.L. Tatum et al.
Comprehensive strategy for the evaluation and triage of the chest pain patient
Ann Emerg Med
(1997)
W.G. Baxt et al.
A neural computational aid to the diagnosis of acute myocardial infarction
Ann Emerg Med
(2002)
W.G. Baxt et al.
A neural network aid for the early diagnosis of cardiac ischemia in patients presenting to the emergency department with chest pain
Ann Emerg Med
(2002)
W.G. Baxt et al.
Prospective validation of artificial neural network trained to identify acute myocardial infarction
Lancet
(1996)
R.L. Kennedy et al.
An artificial neural network system for diagnosis of acute myocardial infarction (AMI) in the accident and emergency department: evaluation and comparison with serum myoglobin measurements
Comp Methods Prog Biomed
(1997)
M. Sunemark et al.
Serial VCG/ECG analysis using neural networks
Comput Biomed Res
(1998)
J.E. Hollander
The continuing search to identify the very-low-risk chest pain patient
Acad Emerg Med
(1999)
J.H. Pope et al.
Missed diagnosis of acute cardiac ischemia in the emergency department
N Engl J Med
(2000)
L. Goldman et al.
A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain
N Engl J Med
(1982)
L. Goldman et al.
A computer protocol to predict myocardial infarction in emergency department patients with chest pain
N Engl J Med
(1988)

L. Goldman et al.

Prediction of the need for intensive care in patients who come to emergency departments with acute chest pain

N Engl J Med

(1996)

M.W. Pozen et al.

A predictive instrument to improve coronary care unit admission practices in acute ischemic heart disease

N Engl J Med

(1984)

H.P. Selker et al.

Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia

Ann Intern Med

(1998)

H.P. Selker et al.

An evaluation of technologies for identifying acute cardiac ischemia in the emergency department: a report from the National Heart Attack Alert Program Working Group

Ann Emerg Med

(1997)

Cited by (32)

Implementation of Prediction Models in the Emergency Department from an Implementation Science Perspective—Determinants, Outcomes, and Real-World Impact: A Scoping Review
2023, Annals of Emergency Medicine
Prediction models offer a promising form of clinical decision support in the complex and fast-paced environment of the emergency department (ED). Despite significant advancements in model development and validation, implementation of such models in routine clinical practice remains elusive. This scoping review aims to survey the current state of prediction model implementation in the ED and to provide insights on contributing factors and outcomes from an implementation science perspective.
We searched 4 databases from their inception to May 20, 2022: MEDLINE (through PubMed), Embase, Scopus, and CINAHL. Articles that reported implementation outcomes and/or contextual determinants under the Reach, Effectiveness, Adoption, Implementation Maintenance (RE-AIM)/Practical, Robust, Implementation, and Sustainability Model (PRISM) framework were included. Characteristics of studies, models, and results of the RE-AIM/PRISM domains were summarized narratively.
Thirty-six reports on 31 implementations were included. The most common prediction models implemented were early warning scores. The most common implementation strategies used were training stakeholders, infrastructural changes, and using evaluative or iterative strategies. Only one report examined ED patients’ perspectives, whereas the rest were focused on the experience of health care workers or organizational stakeholders. Key determinants of successful implementation include strong stakeholder engagement, codevelopment of workflows and implementation strategies, education, and usability.
Examining ED prediction models from an implementation science perspective can provide valuable insights and help guide future implementations.
A predictive analytics model for designing deep underground foundations using artificial neural networks
2023, Decision Analytics Journal
Artificial neural networks (ANNs) are commonly used to solve nonlinear problems. ANNs are composed of parallel interconnected layers that include processing elements called neurons. These neurons use mathematical functions to process the data, mimicking the human brain. In this study, we use ANNs for designing deep underground supports. We develop a predictive model for the ultimate axial capacity of deep foundations called drilled shafts, commonly used as support for highways, bridges and high-rise buildings. Different single hidden layer ANN architectures are utilized to predict the ultimate axial capacity of drilled shafts. The data used in this study are load tests collected from the extended version of the Nevada Deep Foundation Load Test Database. This study focuses on certain difficulties in foundation engineering problems, such as the variability in the soil or construction method which, conventional methods might not fully address. This can sometimes lead to overestimation or underestimation of the capacity. Consequently, a project can be unsafe or expensive. Therefore, the objective of this study is to improve prediction accuracy which is important and can save design and construction costs and time. A total of 45 load tests were divided into 85% for training, and 15% for validation. The developed ANN model achieved good generalization and prediction accuracy with a Root Mean-Squared Error of 2807.08 Kilopounds force (kips), a Mean Absolute Error of 2380.6 kips, and an R-squared (R2) of 87% on unseen data. The results can be adapted in future studies and the industry as a first-order estimate for the architecture of an ANN predictive model and the ultimate nominal axial capacity of drilled shafts.
Presentation of Coronary Artery Disease in a Chiropractic Clinic: A Report of 2 Cases
2016, Journal of Chiropractic Medicine
Citation Excerpt :
Undifferentiated chest pains from musculoskeletal sources are often overlooked.37,38 A large proportion of patients with chest pain who are admitted to hospital do not turn out to have acute coronary syndrome.39 Distinguishing whether a patient presenting with chest pain has acute coronary syndrome or a nonacute coronary syndrome is at best difficult.
The purpose of this report is to describe 2 patients with coronary artery disease presenting with musculoskeletal symptoms to a chiropractic clinic.
A 48-year-old male new patient had thoracic spine pain aggravated by physical exertion. A 61-year-old man under routine care for low back pain experienced a secondary complaint of acute chest pain during a reevaluation.
In both cases, the patients were strongly encouraged to consult their medical physician and were subsequently diagnosed with coronary artery disease. Following their diagnoses, each patient underwent surgical angioplasty procedures with stenting.
Patients may present for chiropractic care with what appears to be musculoskeletal chest pain when the pain may be generating from coronary artery disease necessitating medical and possibly emergency care.
Neural networks for the diagnosis of acute myocardial infarction
2014, Revista Colombiana de Cardiologia
El infarto agudo de miocardio representa la primera causa de muerte no trasmisible en el mundo. Una de las herramientas que sirven como soporte a las decisiones en su diagnóstico son las redes neuronales, de las cuales se ha demostrado un buen nivel de precisión.
Se realizó el entrenamiento y la prueba de varias redes neuronales, con diferentes arquitecturas para el diagnóstico del infarto, a partir de los datos de la escala de clasificación de la probabilidad de angina de Braunwald en un grupo de pacientes que ingresaron por dolor torácico al servicio de urgencias del Hospital San José de Bogotá.
Se generaron 40 redes que fueron probadas en 5 experimentos de los cuales se obtuvo mayor precisión diagnóstica con el modelo de 5 entradas electrocardiográficas más troponina, aunque el mejor valor predictivo negativo se alcanzó en el modelo con 10 variables clínicas, electrocardiográficas y troponina. Varias de las redes diseñadas tuvieron una sensibilidad y una especificidad del 100%. Se requiere un estudio de validación para comprobar estos hallazgos.
Con los resultados encontrados para las redes neuronales en la literatura y en este estudio se puede considerar el uso de esta estrategia de inteligencia computacional en la práctica.
Myocardial infarction represents the leading cause of death by a noncommunicable disease worldwide; one of the tools that serve as decision support for establishing a diagnosis are neural networks. They have been shown to have a good level of accuracy.
Training and testing of several neural networks was performed with different architectures for the diagnosis of the myocardial infarction in a group of patients admitted with chest pain emergency room in the Hospital de San José, Bogotá. This was carried out according to data from the incidence scale of Braunwald's classification of unstable angina.
Forty networks were generated and tested in five experiments obtaining an accurate diagnostic with the electrocardiographic pattern of five entries and troponin. The negative predictive value was 100% in the model with ten clinical variables, electrocardiogram and troponin. Some of the designed networks had a sensitivity and specificity of 100%. A validation study to verify these findings is required.
With the results found for neural networks in the literature and in the present study, we should consider the practical use of this computational intelligence strategy in daily practice.
Modeling daily patient arrivals at Emergency Department and quantifying the relative importance of contributing variables using artificial neural network
2013, Decision Support Systems
Citation Excerpt :
Thus, the structure of ANN is problem-specific. Thus far, ANN has appeared in many healthcare studies, such as diagnosing specific diseases by taking clinically relevant variables into account [8,9,26,27]. The “black box” characteristic of ANN gives rise to difficulty in quantifying the RI of input variables into output variables.
Emergency Department (ED) plays a critical role in healthcare systems by providing emergency care to patients in need. The quality of ED services, measured by waiting time and length of stay, is significantly affected by patient arrivals. Increased patient arrivals could undermine service timeliness, thus putting patients in severe conditions at risk. These factors lead to the following research questions that have rarely been studied before: What are the variables directly associated with patient arrivals in the ED? What is the nature of association between these variables and patient arrivals? Which variable is the most influential and why? To address the above questions, we proposed a three-stage method in this paper. First, a data-driven method is used to identify contributing variables directly correlated with the daily arrivals of Categories 3 and 4 patients (i.e., non-critical patients). Second, the association between contributing variables and daily patient arrival is modeled by using artificial neural network (ANN), and the modeling ability is compared with that of nonlinear least square regression (NLLSR) and multiple linear regression (MLR) in terms of mean average percentage error (MAPE). Third, four types of relative importance (RI) of input variables based on ANN are compared, and their statistical reliability is tested by the MLR-based RI. We applied this three-stage method to one year of data of patient arrivals at a local ED. The contribution of this paper is twofold. Theoretically, this paper emphasizes the importance of using data-driven selection of variables for complex system modeling, and then provides a comprehensive comparison of RI using different computational methods. Practically, this work is a novel attempt of applying ANN to model patient arrivals, and the result can be used to aid in strategic decision-making on ED resource planning in response to predictable arrival variations.
Design of the rule out myocardial ischemia/infarction using computer assisted tomography: A multicenter randomized comparative effectiveness trial of cardiac computed tomography versus alternative triage strategies in patients with acute chest pain in the emergency department
2012, American Heart Journal
Although early cardiac computed tomographic angiography (CCTA) might improve the management of emergency department (ED) patients with acute chest pain, it could also result in increased testing, costs, and radiation exposure. ROMICAT II was a randomized comparative effectiveness trial enrolling patients 40 to 74 years old without known coronary artery disease who presented to the ED with chest pain but without ischemic electrocardiographic (ECG) changes or elevated initial troponin and who required further risk stratification. Overall, 1000 patients at 9 sites within the United States were randomized to either CCTA as the first diagnostic test following serial biomarkers or to standard of care, which included no testing or functional testing such as exercise ECG, stress radionuclide imaging, or stress echocardiography. Test results were provided to ED physicians, yet patient management was not driven by a study protocol in either arm. Data were collected on diagnostic testing, cardiac events, and cost of medical care for the index hospitalization and during the following 28 days. The primary end point was length of hospital stay. Secondary end points were cumulative radiation exposure, resource utilization, and costs of competing strategies. Tertiary end points were institutional, physician, and patient characteristics associated with primary and secondary outcomes. Rate of missed acute coronary syndrome within 28 days was the safety end point. The ROMICAT II will provide rigorous data on whether CCTA is more efficient than standard of care in the management of patients with acute chest pain at intermediate risk for acute coronary syndrome.

View all citing articles on Scopus

^☆: Author contributions: JEH designed the study, developed the data collection instrument, oversaw all study-related procedures, wrote the first draft of the paper, and assisted with data analysis. KLS and DMS assisted with coordination of the data collection, performed data entry, and oversaw the day-to-day operations of the study. KLS, DMS, FDS, FSS, and WGB critically reviewed the manuscript. FDS, FSS, and WGB assisted with study design. FDS assisted with study design and development of the data collection instrument, coordinated the data collection and data entry, and performed patient follow-up in the hospital. FSS performed the statistical analysis. WGB designed the neural network and wrote sections of the manuscript. JEH takes responsibility for the paper as a whole.

^☆☆: The authors report this study did not receive any outside funding or support.

^★: Reprints not available from the authors.

View full text

CardiologyEffects of neural network feedback to physicians on admit/discharge decision for emergency department patients with chest pain☆,☆☆,★

Abstract

Study objective

Methods

Results

Conclusion

Introduction

Section snippets

Study design

Setting

Results

Limitations

Discussion

Ann Emerg Med

Ann Emerg Med

Ann Emerg Med

Lancet

Comp Methods Prog Biomed

Comput Biomed Res

The continuing search to identify the very-low-risk chest pain patient

Acad Emerg Med

Missed diagnosis of acute cardiac ischemia in the emergency department

N Engl J Med

A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain

N Engl J Med

A computer protocol to predict myocardial infarction in emergency department patients with chest pain

N Engl J Med

Prediction of the need for intensive care in patients who come to emergency departments with acute chest pain

N Engl J Med

A predictive instrument to improve coronary care unit admission practices in acute ischemic heart disease

N Engl J Med

Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia

Ann Intern Med

An evaluation of technologies for identifying acute cardiac ischemia in the emergency department: a report from the National Heart Attack Alert Program Working Group

Ann Emerg Med

Cardiology
Effects of neural network feedback to physicians on admit/discharge decision for emergency department patients with chest pain☆,☆☆,★