Nav: Home

A cautionary tale about measuring racial bias in policing

January 20, 2020

Racial bias and policing made headlines last year after a study examining records of fatal police shootings claimed white officers were no more likely to shoot racial minorities than nonwhite officers. There was one problem: The study was based on a logical fallacy.

The original research counted the numbers of fatal shootings, but never considered how often civilians encounter police officers, an essential ingredient to justifying its central claim.

The findings sparked a fiery debate among other academics, including two professors from Princeton University, who raised mathematical concerns about the study's approach. Today, they published their critique as a letter in the Proceedings of the National Academy of Sciences (PNAS).

The pair -- Dean Knox, assistant professor of politics, and Jonathan Mummolo, assistant professor of politics and public affairs-- outline a number of serious flaws in the original study, which was featured in PNAS on Aug. 6, 2019.

For the original study, researchers from Michigan State University and the University of Maryland compiled data on 900 fatal U.S. police shootings from crowdsourced databases. They then contacted each police department, gathering information about the race of the police officers responsible for each fatality.

The researchers then used the shootings data to predict the race of victims. Specifically, they showed that when the shooting officer was black, the civilian who was shot was more likely to be black than white. And controlling for attributes of the county in which shootings occurred, "the relationship between officer and civilian race was attenuated or eliminated." The authors interpreted these results as evidence that white officers are not biased against black civilians.

Yet, Knox and Mummolo show that the authors' conclusions hinge on the assumption that black and white officers encounter black and white civilians in equal numbers. Knox and Mummolo show this formally, but a simple thought experiment also illustrates the conceptual problem.

Imagine a white officer encounters 90 white civilians and 10 black, while a black officer encounters 90 black civilians and 10 white, both under identical circumstances. If both officers shot five black and nine white civilians, the results would -- according to the reasoning of the original study -- appear to show no racial bias.

However, once encounter rates are taken into account, one would see the white officer shot 50% of the black civilians he or she saw while the black officer shot 5.6%. Therefore, failing to incorporate information on encounter rates masks racial bias.

The data from the original study also only includes records of shootings, ignoring all other police-civilian encounters. And it doesn't take into account that all police officers -- white and nonwhite -- could, in theory, be biased in shooting black men.

These critiques have a number of implications on the way data is collected for research and the benchmarks used for analysis.

"New data on police behavior are coming online all the time, and that is great from a research standpoint," Mummolo said. "But all the data in the world do not negate the need to adhere to basic tenets of statistical theory and causal inference. Studies of racial bias demand the utmost rigor, and when blatant mistakes are made, they need to be quickly corrected. To allow provably false results to stand unchallenged risks confusing the public and lawmakers on one of the most pressing policy issues of our time."

After their critique was initially rejected by PNAS, Mummolo published a Twitter thread highlighting the mathematical problems associated with the original study in August 2019. The team also posted their analysis on the preprint server SSRN.

Responding to the critique, the authors of the original paper released a formal response, stating their claim about the relative probability of white and black officers shooting racial minorities was not supported, but adding that the original findings, "as described in that manuscript, largely stand unchanged."

Knox and Mummolo then appealed the rejection at PNAS, and their critique was accepted.
The letter, "Making inferences about racial disparities in political violence," appeared in PNAS on Jan. 20.

Princeton University, Woodrow Wilson School of Public and International Affairs

Related Racial Bias Articles:

Racial bias worse in police killings of older, mentally ill, unarmed men
While young men still bear the brunt of police killings, a new study in the journal Annals of Epidemiology found that police are five times more likely to shoot and kill unarmed Black men over age 54 than unarmed white men the same age.
Study: Despite training, Vermont police departments still show widespread racial bias
New research conducted in Vermont shows that, while anti-bias police trainings resulted in small improvements in some police departments in the state, they did not by and large alter police behavior.
Racial and LGBT bias persists in ridesharing drivers despite mitigation efforts
Despite efforts by ridesharing companies to eliminate or reduce discrimination, research from the Indiana University Kelley School of Business finds that racial and LGBT bias persists among drivers.
Context reduces racial bias in hate speech detection algorithms
When it comes to accurately flagging hate speech on social media, context matters, says a new USC study aimed at reducing errors that could amplify racial bias.
Gender bias kept alive by people who think it's dead
Workplace gender bias is being kept alive by people who think it's no longer an issue, new research suggests.
Sex bias in pain research
Most pain research remains overwhelmingly based on the study of male rodents, continuing to test hypotheses derived from earlier experiments on males.
New research finds racial bias in rideshare platforms
New research to be published in the INFORMS journal Management Science has found popular rideshare platforms exhibit racial and other biases that penalize under-represented minorities and others seeking to use their services.
Research finds teachers just as likely to have racial bias as non-teachers
Research released today challenges the notion that teachers might be uniquely equipped to instill positive racial attitudes in children or bring about racial justice, without additional support or training from schools.
Scientists can see the bias in your brain
The strength of alpha brain waves reveals if you are about to make a biased decision, according to research recently published in JNeurosci.
A cautionary tale about measuring racial bias in policing
Racial bias and policing made headlines last year after a study examining records of fatal police shootings claimed white officers were no more likely to shoot racial minorities than nonwhite officers.
More Racial Bias News and Racial Bias Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Power Of Spaces
How do spaces shape the human experience? In what ways do our rooms, homes, and buildings give us meaning and purpose? This hour, TED speakers explore the power of the spaces we make and inhabit. Guests include architect Michael Murphy, musician David Byrne, artist Es Devlin, and architect Siamak Hariri.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at     You can read The Transition Integrity Project's report here.