Nav: Home

AI for #MeToo: Training algorithms to spot online trolls

January 08, 2020

Researchers at Caltech have demonstrated that machine-learning algorithms can monitor online social media conversations as they evolve, which could one day lead to an effective and automated way to spot online trolling.

The project unites the labs of artificial intelligence (AI) researcher Anima Anandkumar, Bren Professor of Computing and Mathematical Sciences, and Michael Alvarez, professor of political science. Their work was presented on December 14 at the AI for Social Good workshop at the 2019 Conference on Neural Information Processing Systems in Vancouver, Canada. Their research team includes Anqi Liu, postdoctoral scholar; Maya Srikanth, a junior at Caltech; and Nicholas Adams-Cohen (MS '16, PhD '19) of Stanford University.

"This is one of the things I love about Caltech: the ability to bridge boundaries, developing synergies between social science and, in this case, computer science," Alvarez says.

Prevention of online harassment requires rapid detection of offensive, harassing, and negative social media posts, which in turn requires monitoring online interactions. Current methods to obtain such social media data are either fully automated and not interpretable or rely on a static set of keywords, which can quickly become outdated. Neither method is very effective, according to Srikanth.

"It isn't scalable to have humans try to do this work by hand, and those humans are potentially biased," she says. "On the other hand, keyword searching suffers from the speed at which online conversations evolve. New terms crop up and old terms change meaning, so a keyword that was used sincerely one day might be meant sarcastically the next."

Instead, the team used a GloVe (Global Vectors for Word Representation) model to discover new and relevant keywords. GloVe is a word-embedding model, meaning that it represents words in a vector space, where the "distance" between two words is a measure of their linguistic or semantic similarity. Starting with one keyword, this model can be used to find others that are closely related to that word to reveal clusters of relevant terms that are actually in use. For example, searching Twitter for uses of "MeToo" in conversations yielded clusters of related hashtags like "SupportSurvivors," "ImWithHer," and "NotSilent." This approach gives researchers a dynamic and ever-evolving keyword set to search.

But it is not enough just to know whether a certain conversation is related to the topic of interest; context matters. For that, GloVe shows the extent to which certain keywords are related, providing input on how they are being used. For example, in an online Reddit forum dedicated to misogyny, the word "female" was used in close association with the words "sexual," "negative," and "intercourse." In Twitter posts about the #MeToo movement, the word "female" was more likely to be associated with the terms "companies," "desire," and "victims."

The project was a proof-of-concept aimed at one day giving social media platforms a more powerful tool to spot online harassment. Anandkumar's interest in the topic was intensified by her involvement in the campaign to change the shorthand name of the Neural Information Processing Systems conference from its original acronym, "NIPS," to "NeurIPS."

"The field of AI research is becoming more inclusive, but there are always people who resist change," says Anandkumar, who in 2018 found herself the target of harassment and threats online because of her successful effort to switch to an acronym without potentially offensive connotations. "It was an eye-opening experience about just how ugly trolling can get. Hopefully, the tools we're developing now will help fight all kinds of harassment in the future."
Their study is titled "Finding Social Media Trolls: Dynamic Keyword Selection Methods for Rapidly-Evolving Online Debates." This research was supported by funding from John Randolph Haynes and Dora Haynes Foundation, Microsoft, Google, Adobe, and the Defense Advanced Research Projects Agency. Liu is a PIMCO postdoctoral fellow at Caltech.

The study can be found online at

California Institute of Technology

Related Harassment Articles:

LGBTQ military service members at higher risk of sexual harassment, assault, stalking
A recent study found that LGBTQ service members face an elevated risk of sexual victimization including harassment, assault and stalking while in the military than their non-LGBTQ counterparts.
Study explores sexual harassment at AADR conferences
The research assessed perceptions and experiences related to sexual, gender-based and nongender-based harassment among registrants at AADR annual meetings from 2015 to 2018 (n=10,495); examined demographic factors associated with reported experiences; and identified facilitators and potential solutions concerning these types of harassment.
Women in leadership positions face more sexual harassment
Power in the workplace does not stop women's exposure to sexual harassment.
Sexual harassment may be reduced at fun work events, study finds
The office holiday party loses its luster in light of new study findings from researchers at Penn State and Ohio State demonstrating that incidences of unwanted sexual attention are increased at these and other ''fun'' work events.
Study finds flirting among coworkers can reduce stress
Casual flirting with colleagues at work is relatively harmless and can even be beneficial, according to the study led by Washington State University Assistant Professor Leah Sheppard.
Most physicians and other faculty in large medical center experienced sexual harassment
A new study has shown that the majority of women (82.5%) and men (65.1%) working at an academic medical center reported at least one incident of sexual harassment by staff, students, and faculty during the previous year.
Nearly half of accused harassers can return to work
What happens behind the scenes when employees are accused of harassment?
Scientists' panel urges vigorous prevention of sexual harassment and bias in labs
A diverse group of scientists including Nilanjana Dasgupta, professor of social psychology at UMass Amherst and the campus's director of faculty equity and inclusion, report their findings recently and recommendations on how institutions and funding agencies can address and prevent sexual harassment and gender bias in the STEM workforce.
Scientists take action to prevent sexual harassment and bias
In a policy paper published in the journal Science, scientists from a variety of fields highlight key ways institutions and funding agencies can help address sexual harassment and gender bias in the STEM workplace.
Empathy for perpetrators helps explain victim blaming in sexual harassment
Men's empathy for other men who sexually harass women may help explain why they are more likely to blame victims, new research suggests.
More Harassment News and Harassment Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Our Relationship With Water
We need water to live. But with rising seas and so many lacking clean water – water is in crisis and so are we. This hour, TED speakers explore ideas around restoring our relationship with water. Guests on the show include legal scholar Kelsey Leonard, artist LaToya Ruby Frazier, and community organizer Colette Pichon Battle.
Now Playing: Science for the People

#568 Poker Face Psychology
Anyone who's seen pop culture depictions of poker might think statistics and math is the only way to get ahead. But no, there's psychology too. Author Maria Konnikova took her Ph.D. in psychology to the poker table, and turned out to be good. So good, she went pro in poker, and learned all about her own biases on the way. We're talking about her new book "The Biggest Bluff: How I Learned to Pay Attention, Master Myself, and Win".
Now Playing: Radiolab

First things first: our very own Latif Nasser has an exciting new show on Netflix. He talks to Jad about the hidden forces of the world that connect us all. Then, with an eye on the upcoming election, we take a look back: at two pieces from More Perfect Season 3 about Constitutional amendments that determine who gets to vote. Former Radiolab producer Julia Longoria takes us to Washington, D.C. The capital is at the heart of our democracy, but it's not a state, and it wasn't until the 23rd Amendment that its people got the right to vote for president. But that still left DC without full representation in Congress; D.C. sends a "non-voting delegate" to the House. Julia profiles that delegate, Congresswoman Eleanor Holmes Norton, and her unique approach to fighting for power in a virtually powerless role. Second, Radiolab producer Sarah Qari looks at a current fight to lower the US voting age to 16 that harkens back to the fight for the 26th Amendment in the 1960s. Eighteen-year-olds at the time argued that if they were old enough to be drafted to fight in the War, they were old enough to have a voice in our democracy. But what about today, when even younger Americans are finding themselves at the center of national political debates? Does it mean we should lower the voting age even further? This episode was reported and produced by Julia Longoria and Sarah Qari. Check out Latif Nasser's new Netflix show Connected here. Support Radiolab today at