Nav: Home

Algorithms 'consistently' more accurate than people in predicting recidivism, study says

February 14, 2020

In a study with potentially far-reaching implications for criminal justice in the United States, a team of California researchers has found that algorithms are significantly more accurate than humans in predicting which defendants will later be arrested for a new crime.

When assessing just a handful of variables in a controlled environment, even untrained humans can match the predictive skill of sophisticated risk-assessment instruments, says the new study by scholars at Stanford University and the University of California, Berkeley.

But real-world criminal justice settings are often far more complex, and when a larger number of factors are useful for predicting recidivism, the algorithm-based tools performed far better than people. In some tests, the tools approached 90% accuracy in predicting which defendants might be arrested again, compared to about 60% for human prediction.

"Risk assessment has long been a part of decision-making in the criminal justice system," said Jennifer Skeem, a psychologist who specializes in criminal justice at UC Berkeley. "Although recent debate has raised important questions about algorithm-based tools, our research shows that in contexts resembling real criminal justice settings, risk assessments are often more accurate than human judgment in predicting recidivism. That's consistent with a long line of research comparing humans to statistical tools."

"Validated risk-assessment instruments can help justice professionals make more informed decisions," said Sharad Goel, a computational social scientist at Stanford University. "For example, these tools can help judges identify and potentially release people who pose little risk to public safety. But, like any tools, risk assessment instruments must be coupled with sound policy and human oversight to support fair and effective criminal justice reform."

The paper -- "The limits of human predictions of recidivism" -- was slated for publication Feb. 14, 2020, in Science Advances. Skeem presented the research on Feb. 13 in a news briefing at the annual meeting of the American Association for the Advancement of Science (AAAS) in Seattle, Wash. Joining her were two co-authors: Ph.D. graduate Jongbin Jung and Ph.D. candidate Zhiyuan "Jerry" Lin, who both studied computational social science at Stanford.

The research findings are important as the United States debates how to balance the needs communities have for security while reducing incarceration rates that are the highest of any nation in the world--and disproportionately affect African Americans and communities of color.

If the use of advanced risk assessment tools continues and improves, that could refine critically important decisions that justice professionals make daily: Which individuals can be rehabilitated in the community, rather than in prison? Which could go to low-security prisons, and which to high-security sites? And which prisoners can safely be released to the community on parole?

Assessment tools driven by algorithms are widely used in the United States, in areas as diverse as medical care, banking and university admissions. They have long been used in criminal justice, helping judges and others to weigh data in making their decisions.

But in 2018, researchers at Dartmouth University raised questions about the accuracy of such tools in a criminal justice framework. In a study, they assembled 1,000 short vignettes of criminal defendants, with information drawn from a widely used risk assessment called the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS).

The vignettes each included five risk factors for recidivism: the individual's sex, age, current criminal charge, and the number of previous adult and juvenile offenses. The researchers then used Amazon's Mechanical Turk platform to recruit 400 volunteers to read the vignettes and assess whether each defendant would commit another crime within two years. After reviewing each vignette, the volunteers were told whether their evaluation accurately predicted the subject's recidivism.

Both the people and the algorithm were accurate slightly less than two-thirds of the time.

These results, the Dartmouth authors concluded, cast doubt on the value of risk-assessment instruments and algorithmic prediction.

The study generated high-profile news coverage--and sent a wave of doubt through the U.S. criminal justice reform community. If sophisticated tools were no better than people in predicting which defendants would re-offend, some said, then there was little point in using the algorithms, which might only reinforce racial bias in sentencing. Some argued such profound decisions should be made by people, not computers.

Grappling with "noise" in complex decisions

But when the authors of the new California study evaluated additional data sets and more factors, they concluded that that risk assessment tools can be much more accurate than people in assessing potential for recidivism.

The study replicated the Dartmouth findings that had been based on a limited number of factors. However, the information available in justice settings is far more rich -- and often more ambiguous.

"Pre-sentence investigation reports, attorney and victim impact statements, and an individual's demeanor all add complex, inconsistent, risk-irrelevant, and potentially biasing information," the new study explains.

The authors' hypothesis: If research evaluations operate in a real-world framework, where risk-related information is complex and "noisy," then advanced risk assessment tools would be more effective than humans at predicting which criminals would re-offend.

To test the hypothesis, they expanded their study beyond COMPAS to include other data sets. In addition to the five risk factors used in the Dartmouth study, they added 10 more, including employment status, substance use and mental health. They also expanded the methodology: Unlike the Dartmouth study, in some cases the volunteers would not be told after each evaluation whether their predictions were accurate. Such feedback is not available to judges and others in the court system.

The outcome: Humans performed "consistently worse" than the risk assessment tool on complex cases when they didn't have immediate feedback to guide future decisions.

For example, the COMPAS correctly predicted recidivism 89% of the time, compared to 60% for humans who were not provided case-by-case feedback on their decisions. When multiple risk factors were provided and predictive, another risk assessment tool accurately predicted recidivism over 80% of the time, compared to less than 60% for humans.

The findings appear to support continued use and future improvement of risk assessment algorithms. But, as Skeem noted, these tools typically have a support role. Ultimate authority rests with judges, probation officers, clinicians, parole commissioners and others who shape decisions in the criminal justice system.
-end-
ON-SITE PRESS BRIEFING AND LIVE-STREAMING WEBCAST

An embargoed news briefing related to the forthcoming Science Advances paper, "The Limits of Human Predictions of Recidivism," will take place at the 2020 AAAS Annual Meeting. This event, open to credentialed Annual Meeting press registrants only, will take place at 10:00 a.m. U.S. Pacific Time or 1:00 p.m. U.S. Eastern Time on Thursday, 13 February, in room 204, the AAAS Newsroom Briefing Room, on the 2nd Floor of The Washington State Convention Center in Seattle, Washington.

The embargo on the paper by Lin et al. will lift at 2:00 p.m. U.S. Eastern Time on Friday, 14 February.

Reporters who are unable to attend the Annual Meeting can also participate in the live webcast of the briefing. (You must be registered with EurekAlert! to view the webcast.) Please visit the Virtual Newsroom (https://www.eurekalert.org/aaasnewsroom/2020/briefings) for further information. (At that link, you also can see the schedule for all AAAS Annual Meeting news briefing webcasts, which will be archived.)

Speakers at this event will include Jen Skeem, professor of social welfare and public policy at Berkeley; Jongbin (Chongbin) Jung, data scientist and recent PhD graduate from Stanford; and Jerry Lin, PhD student in the Stanford Computer Science Department.

The press briefing and related webcast are being made possible through the generosity of the nonprofit American Association for the Advancement of Science (AAAS). Thanks for citing the journal Science Advances (@ScienceAdvances) as well as the AAAS Annual Meeting (#AAASmtg) in coverage of this story.

Reporters interested in joining this briefing are asked to send an e-mail to vancepak@aaas.org requesting pre-registration.

University of California - Berkeley

Related Risk Factors Articles:

Factors associated with firearm suicide risk
Researchers compared the risk of suicide by firearm based on sociodemographic characteristics of US adults.
Modifiable risk factors contribute to gout
Elevated urate in the blood (hyperuricemia) is a precursor of gout, which is the most common form of inflammatory arthritis worldwide.
Social risk factors and readmission penalties
New research shows that US safety net hospitals could benefit substantially from a new model that accounts for social risk factors like poverty and living in a disadvantaged neighborhood in determining how the federal government penalizes hospitals financially for their readmission rates.
AI study of risk factors in type 1 diabetes
In combination with conventional statistical methods, artificial intelligence (AI) has now been used in a study of risk factors in type 1 diabetes.
Stroke risk factors on the rise in Native-Americans
Stroke risk factors such as high blood pressure, diabetes, heart disease and smoking are common and on the rise among Native-Americans with clot-caused stroke.
Discovery of the first common genetic risk factors for ADHD
A global team of researchers has found the first common genetic risk factors associated with attention deficit hyperactivity disorder (ADHD), a complex condition affecting around one in 20 children.
Military risk factors for dementia
In recent years, there has been growing discussion to better understand the pathophysiological mechanisms of traumatic brain injury and post-traumatic stress disorder and how they may be linked to an increased risk of neurodegenerative diseases including Alzheimer's disease in veterans.
Researchers outline risk factors for facial gangrene
Noma, a rare disease found predominantly in underserved areas, causes progressive destruction, or gangrene, of the tissues of the face and jaw within just the span of one week.
Study examines for risk factors associated with initiation of substance use
Not all individuals who initiate use of a substance such as nicotine, alcohol, cannabis, and cocaine eventually develop a substance use disorder, indicating that the risk factors for substance use and for substance use disorder (SUD) differ to some extent.
What midlife risk factors are associated with late onset of epilepsy?
Potentially changeable lifestyle and vascular risk factors in midlife were associated with onset later in life of epilepsy, a neurological disorder with higher risk in older age.
More Risk Factors News and Risk Factors Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Biology Of Sex
Original broadcast date: May 8, 2020. Many of us were taught biological sex is a question of female or male, XX or XY ... but it's far more complicated. This hour, TED speakers explore what determines our sex. Guests on the show include artist Emily Quinn, journalist Molly Webster, neuroscientist Lisa Mosconi, and structural biologist Karissa Sanbonmatsu.
Now Playing: Science for the People

#569 Facing Fear
What do you fear? I mean really fear? Well, ok, maybe right now that's tough. We're living in a new age and definition of fear. But what do we do about it? Eva Holland has faced her fears, including trauma and phobia. She lived to tell the tale and write a book: "Nerve: Adventures in the Science of Fear".
Now Playing: Radiolab

The Wubi Effect
When we think of China today, we think of a technological superpower. From Huweai and 5G to TikTok and viral social media, China is stride for stride with the United States in the world of computing. However, China's technological renaissance almost didn't happen. And for one very basic reason: The Chinese language, with its 70,000 plus characters, couldn't fit on a keyboard.  Today, we tell the story of Professor Wang Yongmin, a hard headed computer programmer who solved this puzzle and laid the foundation for the China we know today. This episode was reported and produced by Simon Adler with reporting assistance from Yang Yang. Special thanks to Martin Howard. You can view his renowned collection of typewriters at: antiquetypewriters.com Support Radiolab today at Radiolab.org/donate.