Algorithms 'consistently' more accurate than people in predicting recidivism, study says

February 14, 2020

In a study with potentially far-reaching implications for criminal justice in the United States, a team of California researchers has found that algorithms are significantly more accurate than humans in predicting which defendants will later be arrested for a new crime.

When assessing just a handful of variables in a controlled environment, even untrained humans can match the predictive skill of sophisticated risk-assessment instruments, says the new study by scholars at Stanford University and the University of California, Berkeley.

But real-world criminal justice settings are often far more complex, and when a larger number of factors are useful for predicting recidivism, the algorithm-based tools performed far better than people. In some tests, the tools approached 90% accuracy in predicting which defendants might be arrested again, compared to about 60% for human prediction.

"Risk assessment has long been a part of decision-making in the criminal justice system," said Jennifer Skeem, a psychologist who specializes in criminal justice at UC Berkeley. "Although recent debate has raised important questions about algorithm-based tools, our research shows that in contexts resembling real criminal justice settings, risk assessments are often more accurate than human judgment in predicting recidivism. That's consistent with a long line of research comparing humans to statistical tools."

"Validated risk-assessment instruments can help justice professionals make more informed decisions," said Sharad Goel, a computational social scientist at Stanford University. "For example, these tools can help judges identify and potentially release people who pose little risk to public safety. But, like any tools, risk assessment instruments must be coupled with sound policy and human oversight to support fair and effective criminal justice reform."

The paper -- "The limits of human predictions of recidivism" -- was slated for publication Feb. 14, 2020, in Science Advances. Skeem presented the research on Feb. 13 in a news briefing at the annual meeting of the American Association for the Advancement of Science (AAAS) in Seattle, Wash. Joining her were two co-authors: Ph.D. graduate Jongbin Jung and Ph.D. candidate Zhiyuan "Jerry" Lin, who both studied computational social science at Stanford.

The research findings are important as the United States debates how to balance the needs communities have for security while reducing incarceration rates that are the highest of any nation in the world--and disproportionately affect African Americans and communities of color.

If the use of advanced risk assessment tools continues and improves, that could refine critically important decisions that justice professionals make daily: Which individuals can be rehabilitated in the community, rather than in prison? Which could go to low-security prisons, and which to high-security sites? And which prisoners can safely be released to the community on parole?

Assessment tools driven by algorithms are widely used in the United States, in areas as diverse as medical care, banking and university admissions. They have long been used in criminal justice, helping judges and others to weigh data in making their decisions.

But in 2018, researchers at Dartmouth University raised questions about the accuracy of such tools in a criminal justice framework. In a study, they assembled 1,000 short vignettes of criminal defendants, with information drawn from a widely used risk assessment called the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS).

The vignettes each included five risk factors for recidivism: the individual's sex, age, current criminal charge, and the number of previous adult and juvenile offenses. The researchers then used Amazon's Mechanical Turk platform to recruit 400 volunteers to read the vignettes and assess whether each defendant would commit another crime within two years. After reviewing each vignette, the volunteers were told whether their evaluation accurately predicted the subject's recidivism.

Both the people and the algorithm were accurate slightly less than two-thirds of the time.

These results, the Dartmouth authors concluded, cast doubt on the value of risk-assessment instruments and algorithmic prediction.

The study generated high-profile news coverage--and sent a wave of doubt through the U.S. criminal justice reform community. If sophisticated tools were no better than people in predicting which defendants would re-offend, some said, then there was little point in using the algorithms, which might only reinforce racial bias in sentencing. Some argued such profound decisions should be made by people, not computers.

Grappling with "noise" in complex decisions

But when the authors of the new California study evaluated additional data sets and more factors, they concluded that that risk assessment tools can be much more accurate than people in assessing potential for recidivism.

The study replicated the Dartmouth findings that had been based on a limited number of factors. However, the information available in justice settings is far more rich -- and often more ambiguous.

"Pre-sentence investigation reports, attorney and victim impact statements, and an individual's demeanor all add complex, inconsistent, risk-irrelevant, and potentially biasing information," the new study explains.

The authors' hypothesis: If research evaluations operate in a real-world framework, where risk-related information is complex and "noisy," then advanced risk assessment tools would be more effective than humans at predicting which criminals would re-offend.

To test the hypothesis, they expanded their study beyond COMPAS to include other data sets. In addition to the five risk factors used in the Dartmouth study, they added 10 more, including employment status, substance use and mental health. They also expanded the methodology: Unlike the Dartmouth study, in some cases the volunteers would not be told after each evaluation whether their predictions were accurate. Such feedback is not available to judges and others in the court system.

The outcome: Humans performed "consistently worse" than the risk assessment tool on complex cases when they didn't have immediate feedback to guide future decisions.

For example, the COMPAS correctly predicted recidivism 89% of the time, compared to 60% for humans who were not provided case-by-case feedback on their decisions. When multiple risk factors were provided and predictive, another risk assessment tool accurately predicted recidivism over 80% of the time, compared to less than 60% for humans.

The findings appear to support continued use and future improvement of risk assessment algorithms. But, as Skeem noted, these tools typically have a support role. Ultimate authority rests with judges, probation officers, clinicians, parole commissioners and others who shape decisions in the criminal justice system.
-end-
ON-SITE PRESS BRIEFING AND LIVE-STREAMING WEBCAST

An embargoed news briefing related to the forthcoming Science Advances paper, "The Limits of Human Predictions of Recidivism," will take place at the 2020 AAAS Annual Meeting. This event, open to credentialed Annual Meeting press registrants only, will take place at 10:00 a.m. U.S. Pacific Time or 1:00 p.m. U.S. Eastern Time on Thursday, 13 February, in room 204, the AAAS Newsroom Briefing Room, on the 2nd Floor of The Washington State Convention Center in Seattle, Washington.

The embargo on the paper by Lin et al. will lift at 2:00 p.m. U.S. Eastern Time on Friday, 14 February.

Reporters who are unable to attend the Annual Meeting can also participate in the live webcast of the briefing. (You must be registered with EurekAlert! to view the webcast.) Please visit the Virtual Newsroom (https://www.eurekalert.org/aaasnewsroom/2020/briefings) for further information. (At that link, you also can see the schedule for all AAAS Annual Meeting news briefing webcasts, which will be archived.)

Speakers at this event will include Jen Skeem, professor of social welfare and public policy at Berkeley; Jongbin (Chongbin) Jung, data scientist and recent PhD graduate from Stanford; and Jerry Lin, PhD student in the Stanford Computer Science Department.

The press briefing and related webcast are being made possible through the generosity of the nonprofit American Association for the Advancement of Science (AAAS). Thanks for citing the journal Science Advances (@ScienceAdvances) as well as the AAAS Annual Meeting (#AAASmtg) in coverage of this story.

Reporters interested in joining this briefing are asked to send an e-mail to vancepak@aaas.org requesting pre-registration.

University of California - Berkeley

Related Risk Factors Articles from Brightsurf:

Insomnia identified as a new risk factor for type 2 diabetes in new study which also confirms many other risk and protective factors
A new 'global atlas' study published in Diabetologia (the journal of the European Association for the Study of Diabetes [EASD]) is the first to identify insomnia as a risk factor associated with increased risk of developing type 2 diabetes (T2D).

Lung-specific risk factors may increase hip fracture risk in individuals who smoke
Smoking has been linked to a higher risk of bone fractures.

Factors associated with firearm suicide risk
Researchers compared the risk of suicide by firearm based on sociodemographic characteristics of US adults.

Modifiable risk factors contribute to gout
Elevated urate in the blood (hyperuricemia) is a precursor of gout, which is the most common form of inflammatory arthritis worldwide.

Social risk factors and readmission penalties
New research shows that US safety net hospitals could benefit substantially from a new model that accounts for social risk factors like poverty and living in a disadvantaged neighborhood in determining how the federal government penalizes hospitals financially for their readmission rates.

AI study of risk factors in type 1 diabetes
In combination with conventional statistical methods, artificial intelligence (AI) has now been used in a study of risk factors in type 1 diabetes.

Stroke risk factors on the rise in Native-Americans
Stroke risk factors such as high blood pressure, diabetes, heart disease and smoking are common and on the rise among Native-Americans with clot-caused stroke.

Discovery of the first common genetic risk factors for ADHD
A global team of researchers has found the first common genetic risk factors associated with attention deficit hyperactivity disorder (ADHD), a complex condition affecting around one in 20 children.

Military risk factors for dementia
In recent years, there has been growing discussion to better understand the pathophysiological mechanisms of traumatic brain injury and post-traumatic stress disorder and how they may be linked to an increased risk of neurodegenerative diseases including Alzheimer's disease in veterans.

Researchers outline risk factors for facial gangrene
Noma, a rare disease found predominantly in underserved areas, causes progressive destruction, or gangrene, of the tissues of the face and jaw within just the span of one week.

Read More: Risk Factors News and Risk Factors Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.