Nav: Home

Concerns over 'exaggerated' study claims of AI outperforming doctors

March 25, 2020

Many studies claiming that artificial intelligence is as good as (or better than) human experts at interpreting medical images are of poor quality and are arguably exaggerated, posing a risk for the safety of 'millions of patients' warn researchers in The BMJ today.

Their findings raise concerns about the quality of evidence underpinning many of these studies, and highlight the need to improve their design and reporting standards.

Artificial intelligence (AI) is an innovative and fast moving field with the potential to improve patient care and relieve overburdened health services. Deep learning is a branch of AI that has shown particular promise in medical imaging.

The volume of published research on deep learning is growing, and some media headlines that claim superior performance to doctors have fuelled hype for rapid implementation. But the methods and risk of bias of studies behind these headlines have not been examined in detail.

To address this, a team of researchers reviewed the results of published studies over the past 10 years, comparing the performance of a deep learning algorithm in medical imaging with expert clinicians.

They found just two eligible randomised clinical trials and 81 non-randomised studies.

Of the non-randomised studies, only nine were prospective (tracking and collecting information about individuals over time) and just six were tested in a 'real world' clinical setting.

The average number of human experts in the comparator group was just four, while access to raw data and code (to allow independent scrutiny of results) was severely limited.

More than two thirds (58 of 81) studies were judged to be at high risk of bias (problems in study design that can influence results), and adherence to recognised reporting standards was often poor.

Three quarters (61 studies) stated that performance of AI was at least comparable to (or better than) that of clinicians, and only 31 (38%) stated that further prospective studies or trials were needed.

The researchers point to some limitations, such as the possibility of missed studies and the focus on deep learning medical imaging studies so results may not apply to other types of AI.

Nevertheless, they say that at present, "many arguably exaggerated claims exist about equivalence with (or superiority over) clinicians, which presents a potential risk for patient safety and population health at the societal level."

Overpromising language "leaves studies susceptible to being misinterpreted by the media and the public, and as a result the possible provision of inappropriate care that does not necessarily align with patients' best interests," they warn.

"Maximising patient safety will be best served by ensuring that we develop a high quality and transparently reported evidence base moving forward," they conclude.
-end-
Peer-reviewed? Yes
Evidence type: Systematic review
Subjects: Deep learning studies

BMJ

Related Medical Imaging Articles:

AI techniques in medical imaging may lead to incorrect diagnoses
Machine learning and AI are highly unstable in medical image reconstruction, and may lead to false positives and false negatives, a new study suggests.
Tiny devices promise new horizon for security screening and medical imaging
Miniature devices that could be developed into safe, high-resolution imaging technology, with uses such as helping doctors identify potentially deadly cancers and treat them early, have been created in research involving the University of Strathclyde.
Advanced medical imaging combined with genomic analysis could help treat cancer patients
Melding the genetic and cellular analysis of tumors with how they appear in medical images could give physicians new insights into how to best treat patients, especially those with brain cancer, according to a new study led by TGen.
Low doses of radiation used in medical imaging lead to mutations in cell cultures
Common medical imaging procedures use low doses of radiation that are believed to be safe.
Use of medical imaging
This observational study looked at patterns of use for computed tomography (CT), magnetic resonance imaging (MRI), ultrasound and nuclear medicine imaging in the United States and in Ontario, Canada, from 2000 to 2016.
Medical imaging rates continue to rise despite push to reduce their use
The rates of use of CT, MRI and other scans have continued to increase in both the US and Ontario, Canada, according to a new study of more than 135 million imaging exams conducted by researchers at UC Davis, UC San Francisco and Kaiser Permanente.
Two-in-one contrast agent for medical imaging
Magnetic resonance imaging (MRI) visualizes internal body structures, often with the help of contrast agents to enhance sensitivity.
Medical imaging rates during pregnancy
Researchers looked at rates of medical imaging (CT, MRI, conventional x-rays, angiography, fluoroscopy and nuclear medicine) during pregnancy in this observational study that included nearly 3.5 million pregnant women in the United States and Canada from 1996 to 2016.
Scientists discover new method for developing tracers used for medical imaging
University of North Carolina researchers discovered a method for creating radioactive tracers to better track pharmaceuticals in the body as well as image diseases, such as cancer, and other medical conditions.
Radiology publishes roadmap for AI in medical imaging
In August 2018, a workshop was held at the National Institutes of Health (NIH) in Bethesda, Md., to explore the future of artificial intelligence (AI) in medical imaging.
More Medical Imaging News and Medical Imaging Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Biology Of Sex
Original broadcast date: May 8, 2020. Many of us were taught biological sex is a question of female or male, XX or XY ... but it's far more complicated. This hour, TED speakers explore what determines our sex. Guests on the show include artist Emily Quinn, journalist Molly Webster, neuroscientist Lisa Mosconi, and structural biologist Karissa Sanbonmatsu.
Now Playing: Science for the People

#569 Facing Fear
What do you fear? I mean really fear? Well, ok, maybe right now that's tough. We're living in a new age and definition of fear. But what do we do about it? Eva Holland has faced her fears, including trauma and phobia. She lived to tell the tale and write a book: "Nerve: Adventures in the Science of Fear".
Now Playing: Radiolab

The Wubi Effect
When we think of China today, we think of a technological superpower. From Huweai and 5G to TikTok and viral social media, China is stride for stride with the United States in the world of computing. However, China's technological renaissance almost didn't happen. And for one very basic reason: The Chinese language, with its 70,000 plus characters, couldn't fit on a keyboard.  Today, we tell the story of Professor Wang Yongmin, a hard headed computer programmer who solved this puzzle and laid the foundation for the China we know today. This episode was reported and produced by Simon Adler with reporting assistance from Yang Yang. Special thanks to Martin Howard. You can view his renowned collection of typewriters at: antiquetypewriters.com Support Radiolab today at Radiolab.org/donate.