A cautionary tale about measuring racial bias in policing

January 20, 2020

Racial bias and policing made headlines last year after a study examining records of fatal police shootings claimed white officers were no more likely to shoot racial minorities than nonwhite officers. There was one problem: The study was based on a logical fallacy.

The original research counted the numbers of fatal shootings, but never considered how often civilians encounter police officers, an essential ingredient to justifying its central claim.

The findings sparked a fiery debate among other academics, including two professors from Princeton University, who raised mathematical concerns about the study's approach. Today, they published their critique as a letter in the Proceedings of the National Academy of Sciences (PNAS).

The pair -- Dean Knox, assistant professor of politics, and Jonathan Mummolo, assistant professor of politics and public affairs-- outline a number of serious flaws in the original study, which was featured in PNAS on Aug. 6, 2019.

For the original study, researchers from Michigan State University and the University of Maryland compiled data on 900 fatal U.S. police shootings from crowdsourced databases. They then contacted each police department, gathering information about the race of the police officers responsible for each fatality.

The researchers then used the shootings data to predict the race of victims. Specifically, they showed that when the shooting officer was black, the civilian who was shot was more likely to be black than white. And controlling for attributes of the county in which shootings occurred, "the relationship between officer and civilian race was attenuated or eliminated." The authors interpreted these results as evidence that white officers are not biased against black civilians.

Yet, Knox and Mummolo show that the authors' conclusions hinge on the assumption that black and white officers encounter black and white civilians in equal numbers. Knox and Mummolo show this formally, but a simple thought experiment also illustrates the conceptual problem.

Imagine a white officer encounters 90 white civilians and 10 black, while a black officer encounters 90 black civilians and 10 white, both under identical circumstances. If both officers shot five black and nine white civilians, the results would -- according to the reasoning of the original study -- appear to show no racial bias.

However, once encounter rates are taken into account, one would see the white officer shot 50% of the black civilians he or she saw while the black officer shot 5.6%. Therefore, failing to incorporate information on encounter rates masks racial bias.

The data from the original study also only includes records of shootings, ignoring all other police-civilian encounters. And it doesn't take into account that all police officers -- white and nonwhite -- could, in theory, be biased in shooting black men.

These critiques have a number of implications on the way data is collected for research and the benchmarks used for analysis.

"New data on police behavior are coming online all the time, and that is great from a research standpoint," Mummolo said. "But all the data in the world do not negate the need to adhere to basic tenets of statistical theory and causal inference. Studies of racial bias demand the utmost rigor, and when blatant mistakes are made, they need to be quickly corrected. To allow provably false results to stand unchallenged risks confusing the public and lawmakers on one of the most pressing policy issues of our time."

After their critique was initially rejected by PNAS, Mummolo published a Twitter thread highlighting the mathematical problems associated with the original study in August 2019. The team also posted their analysis on the preprint server SSRN.

Responding to the critique, the authors of the original paper released a formal response, stating their claim about the relative probability of white and black officers shooting racial minorities was not supported, but adding that the original findings, "as described in that manuscript, largely stand unchanged."

Knox and Mummolo then appealed the rejection at PNAS, and their critique was accepted.
The letter, "Making inferences about racial disparities in political violence," appeared in PNAS on Jan. 20.

Princeton University, Woodrow Wilson School of Public and International Affairs

Related Racial Bias Articles from Brightsurf:

Racial bias worse in police killings of older, mentally ill, unarmed men
While young men still bear the brunt of police killings, a new study in the journal Annals of Epidemiology found that police are five times more likely to shoot and kill unarmed Black men over age 54 than unarmed white men the same age.

Study: Despite training, Vermont police departments still show widespread racial bias
New research conducted in Vermont shows that, while anti-bias police trainings resulted in small improvements in some police departments in the state, they did not by and large alter police behavior.

Racial and LGBT bias persists in ridesharing drivers despite mitigation efforts
Despite efforts by ridesharing companies to eliminate or reduce discrimination, research from the Indiana University Kelley School of Business finds that racial and LGBT bias persists among drivers.

Context reduces racial bias in hate speech detection algorithms
When it comes to accurately flagging hate speech on social media, context matters, says a new USC study aimed at reducing errors that could amplify racial bias.

Gender bias kept alive by people who think it's dead
Workplace gender bias is being kept alive by people who think it's no longer an issue, new research suggests.

Sex bias in pain research
Most pain research remains overwhelmingly based on the study of male rodents, continuing to test hypotheses derived from earlier experiments on males.

New research finds racial bias in rideshare platforms
New research to be published in the INFORMS journal Management Science has found popular rideshare platforms exhibit racial and other biases that penalize under-represented minorities and others seeking to use their services.

Research finds teachers just as likely to have racial bias as non-teachers
Research released today challenges the notion that teachers might be uniquely equipped to instill positive racial attitudes in children or bring about racial justice, without additional support or training from schools.

Scientists can see the bias in your brain
The strength of alpha brain waves reveals if you are about to make a biased decision, according to research recently published in JNeurosci.

A cautionary tale about measuring racial bias in policing
Racial bias and policing made headlines last year after a study examining records of fatal police shootings claimed white officers were no more likely to shoot racial minorities than nonwhite officers.

Read More: Racial Bias News and Racial Bias Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.