Nav: Home

Artificial intelligence may fall short when analyzing data across multiple health systems

November 06, 2018

(New York - November 6, 2018) Artificial intelligence (AI) tools trained to detect pneumonia on chest X-rays suffered significant decreases in performance when tested on data from outside health systems, according to a study conducted at the Icahn School of Medicine at Mount and published in a special issue of PLOS Medicine on machine learning and health care. These findings suggest that artificial intelligence in the medical space must be carefully tested for performance across a wide range of populations; otherwise, the deep learning models may not perform as accurately as expected.

As interest in the use of computer system frameworks called convolutional neural networks (CNN) to analyze medical imaging and provide a computer-aided diagnosis grows, recent studies have suggested that AI image classification may not generalize to new data as well as commonly portrayed.

Researchers at the Icahn School of Medicine at Mount Sinai assessed how AI models identified pneumonia in 158,000 chest X-rays across three medical institutions: the National Institutes of Health; The Mount Sinai Hospital; and Indiana University Hospital. Researchers chose to study the diagnosis of pneumonia on chest X-rays for its common occurrence, clinical significance, and prevalence in the research community.

In three out of five comparisons, CNNs' performance in diagnosing diseases on X-rays from hospitals outside of its own network was significantly lower than on X-rays from the original health system. However, CNNs were able to detect the hospital system where an X-ray was acquired with a high-degree of accuracy, and cheated at their predictive task based on the prevalence of pneumonia at the training institution. Researchers found that the difficulty of using deep learning models in medicine is that they use a massive number of parameters, making it challenging to identify specific variables driving predictions, such as the types of CT scanners used at a hospital and the resolution quality of imaging.

"Our findings should give pause to those considering rapid deployment of artificial intelligence platforms without rigorously assessing their performance in real-world clinical settings reflective of where they are being deployed," says senior author Eric Oermann, MD, Instructor in Neurosurgery at the Icahn School of Medicine at Mount Sinai. "Deep learning models trained to perform medical diagnosis can generalize well, but this cannot be taken for granted since patient populations and imaging techniques differ significantly across institutions."

"If CNN systems are to be used for medical diagnosis, they must be tailored to carefully consider clinical questions, tested for a variety of real-world scenarios, and carefully assessed to determine how they impact accurate diagnosis," says first author John Zech, a medical student at the Icahn School of Medicine at Mount Sinai.

This research builds on papers published earlier this year in the journals Radiology and Nature Medicine, which laid the framework for applying computer vision and deep learning techniques, including natural language processing algorithms, for identifying clinical concepts in radiology reports for CT scans.
-end-
About Mount Sinai Health System

The Mount Sinai Health System is New York City's largest integrated delivery system encompassing seven hospital campuses, a leading medical school, and a vast network of ambulatory practices throughout the greater New York region. Mount Sinai's vision is to produce the safest care, the highest quality, the highest satisfaction, the best access and the best value of any health system in the nation. The System includes approximately 6,600 primary and specialty care physicians; 11 joint-venture ambulatory surgery centers; more than 140 ambulatory practices throughout the five boroughs of New York City, Westchester, Long Island, and Florida; and 31 affiliated community health centers. The Icahn School of Medicine is one of three medical schools that have earned distinction by multiple indicators: ranked in the top 20 by U.S. News & World Report's "Best Medical Schools", aligned with a U.S. News & World Report's "Honor Roll" Hospital, it is ranked as a leading medical school for National Institutes of Health funding, and among the top 10 most innovative research institutions as ranked by the journal Nature in its Nature Innovation Index. This reflects a special level of excellence in education, clinical practice, and research. The Mount Sinai Hospital is ranked No. 18 on U.S. News & World Report's "Honor Roll" of top U.S. hospitals; it is one of the nation's top 20 hospitals in Cardiology/Heart Surgery, Gastroenterology/GI Surgery, Geriatrics, Nephrology, and Neurology/Neurosurgery, and in the top 50 in six other specialties in the 2018-2019 "Best Hospitals" issue. Mount Sinai's Kravis Children's Hospital also is ranked nationally in five out of ten pediatric specialties by U.S. News & World Report. The New York Eye and Ear Infirmary of Mount Sinai is ranked 11th nationally for Ophthalmology and 44th for Ear, Nose, and Throat, while Mount Sinai Beth Israel, Mount Sinai St. Luke's and Mount Sinai West are ranked regionally. For more information, visit http://www.mountsinai.org/, or find Mount Sinai on Facebook, Twitter and YouTube.

The Mount Sinai Hospital / Mount Sinai School of Medicine

Related Health Care Articles:

Mental health of health care workers in china in hospitals with patients with COVID-19
This survey study of almost 1,300 health care workers in China at 34 hospitals equipped with fever clinics or wards for patients with COVID-19 reports on their mental health outcomes, including symptoms of depression, anxiety, insomnia and distress.
Large federal program aimed at providing better health care underfunds primary care
Despite a mandate to help patients make better-informed health care decisions, a ten-year research program established under the Affordable Care Act has funded a relatively small number of studies that examine primary care, the setting where the majority of patients in the US receive treatment.
International medical graduates care for Medicare patients with greater health care needs
A study by a Massachusetts General Hospital research team indicates that internal medicine physicians who are graduates of medical schools outside the US care for Medicare patients with more complex medical needs than those cared for by graduates of American medical schools.
The Lancet Global Health: Improved access to care not sufficient to improve health, as epidemic of poor quality care revealed
Of the 8.6 million deaths from conditions treatable by health care, poor-quality care is responsible for an estimated 5 million deaths per year -- more than deaths due to insufficient access to care (3.6 million) .
Under Affordable Care Act, Americans have had more preventive care for heart health
By reducing out-of-pocket costs for preventive treatment, the Affordable Care Act appears to have encouraged more people to have health screenings related to their cardiovascular health.
High-deductible health care plans curb both cost and usage, including preventive care
A team of researchers based at IUPUI has conducted the first systematic review of studies examining the relationship between high-deductible health care plans and the use of health care services.
Health insurance changes, access to care by patients' mental health status
A research letter published by JAMA Psychiatry examined access to care before the Patient Protection and Affordable Care Act (ACA) and after the ACA for patients grouped by mental health status using a scale to assess mental illness in epidemiologic studies.
Medical expenditures rise in most categories except primary care physicians and home health care
This article was published in the July/August 2017 issue of Annals of Family Medicine research journal.
Care management program reduced health care costs in Partners Pioneer ACO
Pesearchers at Partners HealthCare published a study showing that Partners Pioneer ACO not only reduces spending growth, but does this by reducing avoidable hospitalizations for patients with elevated but modifiable risks.
Health care leaders predict patients will lose under President Trump's health care plans
According to a newly released NEJM Catalyst Insights Report, health care executives and industry insiders expect patients -- more than any other stakeholder -- to be the big losers of any comprehensive health care plan from the Trump administration.
More Health Care News and Health Care Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Climate Mindset
In the past few months, human beings have come together to fight a global threat. This hour, TED speakers explore how our response can be the catalyst to fight another global crisis: climate change. Guests include political strategist Tom Rivett-Carnac, diplomat Christiana Figueres, climate justice activist Xiye Bastida, and writer, illustrator, and artist Oliver Jeffers.
Now Playing: Science for the People

#562 Superbug to Bedside
By now we're all good and scared about antibiotic resistance, one of the many things coming to get us all. But there's good news, sort of. News antibiotics are coming out! How do they get tested? What does that kind of a trial look like and how does it happen? Host Bethany Brookeshire talks with Matt McCarthy, author of "Superbugs: The Race to Stop an Epidemic", about the ins and outs of testing a new antibiotic in the hospital.
Now Playing: Radiolab

Speedy Beet
There are few musical moments more well-worn than the first four notes of Beethoven's Fifth Symphony. But in this short, we find out that Beethoven might have made a last-ditch effort to keep his music from ever feeling familiar, to keep pushing his listeners to a kind of psychological limit. Big thanks to our Brooklyn Philharmonic musicians: Deborah Buck and Suzy Perelman on violin, Arash Amini on cello, and Ah Ling Neu on viola. And check out The First Four Notes, Matthew Guerrieri's book on Beethoven's Fifth. Support Radiolab today at Radiolab.org/donate.