Machine learning shows similar performance to traditional risk prediction models

November 04, 2020

Some claim that machine learning technology has the potential to transform healthcare systems, but a study published by The BMJ finds that machine learning models have similar performance to traditional statistical models and share similar uncertainty in making risk predictions for individual patients.

The NHS has invested £250m ($323m; €275m) to embed machine learning in healthcare, but researchers say the level of consistency (stability) within and between models should be assessed before they are used to make treatment decisions for individual patients.

Risk prediction models are widely used in clinical practice. They use statistical techniques alongside information about people, such as their age and ethnicity, to identify those at high risk of developing an illness and make decisions about their care.

Previous research has found that a traditional risk prediction model such as QRISK3 has very good model performance at the population level, but has considerable uncertainty on individual risk prediction.

Some studies claim that machine learning models can outperform traditional models, while others argue that they cannot provide explainable reasons behind their predictions, potentially leading to inappropriate actions.

What's more, machine learning models often ignore censoring - when patients are lost (either by error or by being unreachable) during a study and the model assumes they are disease free, leading to biased predictions.

To explore these issues further, researchers in the UK, China and the Netherlands set out to assess the consistency of machine learning and statistical techniques in predicting individual level and population level risks of cardiovascular disease and the effects of censoring on risk predictions.

They assessed 19 different prediction techniques (12 machine learning models and seven statistical models) using data from 3.6 million patients registered at 391 general practices in England between 1998 and 2018.

Data from general practices, hospital admission and mortality records were used to test each model's performance against actual events.

All 19 models yielded similar population level performance. However, cardiovascular disease risk predictions for the same patients varied substantially between models, especially in patients with higher risks.

For example, a patient with a cardiovascular disease risk of 9.5-10.5% predicted by the traditional QRISK3 model had a risk of 2.9-9.2% and 2.4-7.2% predicted by other models.

Models that ignored censoring (including commonly used machine learning models) substantially underestimated risk of cardiovascular disease.

Of the 223,815 patients with a cardiovascular disease risk above 7.5% with QRISK3 (a model that does consider censoring), 57.8% would be reclassified below 7.5% when using another type of model, explain the researchers.

The researchers acknowledge some limitations in comparing the different models, such as the fact that more predictors could have been considered. However, they point out that their results remained similar after more detailed analyses, suggesting that they withstand scrutiny.

"A variety of models predicted risks for the same patients very differently despite similar model performances," they write. "Consequently, different treatment decisions could be made by arbitrarily selecting another modelling technique."

As such, they suggest these models "should not be directly applied to the prediction of long term risks without considering censoring" and that the level of consistency within and between models "should be routinely assessed before they are used to inform clinical decision making."
-end-
Peer reviewed? Yes
Evidence type: Observational
Subjects: Risk prediction models

BMJ

Related Cardiovascular Disease Articles from Brightsurf:

Changes by income level in cardiovascular disease in US
Researchers examined changes in how common cardiovascular disease was in the highest-income earners compared with the rest of the population in the United States between 1999 and 2016.

Fighting cardiovascular disease with acne drug
Researchers from the European Molecular Biology Laboratory (EMBL) in Heidelberg and Stanford University have found the cause of dilated cardiomyopathy - a leading cause of heart failure - and identified a potential treatment for it: a drug already used to treat acne.

A talk with your GP may prevent cardiovascular disease
Having a general practitioner (GP) who is trained in motivational interviewing may reduce your risk of getting cardiovascular disease.

Dilemma of COVID-19, aging and cardiovascular disease
Whether individuals should continue to take angiotensin-converting enzyme inhibitors and angiotensin receptor blockers in the context of coronavirus disease 2019 (COVID-19) is discussed in this article.

Air pollution linked to dementia and cardiovascular disease
People continuously exposed to air pollution are at increased risk of dementia, especially if they also suffer from cardiovascular diseases, according to a study at Karolinska Institutet in Sweden published in the journal JAMA Neurology.

New insights into the effect of aging on cardiovascular disease
Aging adults are more likely to have - and die from - cardiovascular disease than their younger counterparts.

Premature death from cardiovascular disease
National data were used to examine changes from 2000 to 2015 in premature death (ages 25 to 64) from cardiovascular disease in the United States.

Ultrasound: The potential power for cardiovascular disease therapy
In the current issue of Cardiovascular Innovations and Applications volume 4, issue 2, pp.

Despite the ACA, millions of Americans with cardiovascular disease still can't get care
Cardiovascular disease (CVD) is the leading cause of death for Americans, yet millions with CVD or cardiovascular risk factors (CVRF) still can't access the care they need, even years after the implementation of the Affordable Care Act (ACA).

Excess weight and body fat cause cardiovascular disease
In the first Mendelian randomization study to look at this, researchers have found evidence that excess weight and body fat cause a range of heart and blood vessel diseases (rather than just being associated with it).

Read More: Cardiovascular Disease News and Cardiovascular Disease Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.