Why some Wikipedia disputes go unresolved

November 06, 2018

Wikipedia has enabled large-scale, open collaboration on the internet's largest general-reference resource. But, as with many collaborative writing projects, crafting the content can be a contentious subject.

Often, multiple Wikipedia editors will disagree on certain changes to articles or policies. One of the main ways to officially resolve such disputes is the Requests for Comment (RfC) process. Quarreling editors will publicize their deliberation on a forum, where other Wikipedia editors will chime in and a neutral editor will make a final decision.

Ideally, this should solve all issues. But a novel study by MIT researchers finds debilitating factors -- such as excessive bickering and poorly worded arguments -- have led to about one-third of RfCs going unresolved.

For the study, the researchers compiled and analyzed the first-ever comprehensive dataset of RfC conversations, captured over an eight-year period, and conducted interviews with editors who frequently close RfCs, to understand why they don't find a resolution. They also developed a machine-learning model that leverages that dataset to predict when RfCs may go stale. And, they recommend digital tools that could make deliberation and resolution more effective.

"It was surprising to see a full third of the discussions were not closed," says Amy X. Zhang, a PhD candidate in MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and co-author on the paper, which is being presented at this week's ACM Conference on Computer-Supported Cooperative Work and Social Computing. "On Wikipedia, everyone's a volunteer. People are putting in the work, and they have interest ... and editors may be waiting on someone to close so they can get back to editing. We know, looking through the discussions, the job of reading through and resolving a big deliberation is hard, especially with back and forth and contentiousness. [We hope to] help that person do that work."

The paper's co-authors are: first author Jane Im, a graduate student at the University of Michigan's School of Information; Christopher J. Schilling of the Wikimedia Foundation; and David Karger, a professor of computer science and CSAIL researcher.

(Not) finding closure

Wikipedia offers several channels to solve editorial disputes, which involve two editors hashing out their problems, putting ideas to a simple majority vote from the community, or bringing the debate to a panel of moderators. Some previous Wikipedia research has delved into those channels and back-and-forth "edit wars" between contributors. "But RfCs are interesting, because there's much less of a voting mentality," Zhang says. "With other processes, at the end of day you'll vote and see what happens. [RfC participants] do vote sometimes, but it's more about finding a consensus. What's important is what's actually happening in a discussion."

To file an RfC, an editor drafts a template proposal, based on a content dispute that wasn't resolved in an article's basic "talk" page, and invites comment by the broader community. Proposals run the gamut, from minor disagreements about a celebrity's background information to changes to Wikipedia's policies. Any editor can initiate an RfC and any editor -- usually, more experienced ones -- who didn't participate in the discussion and is considered neutral, may close a discussion. After 30 days, a bot automatically removes the RfC template, with or without resolution. RfCs can close formally with a summary statement by the closer, informally due to overwhelming agreement by participants, or be left stale, meaning removed without resolution.

For their study, the researchers compiled a database consisting of about 7,000 RfC conversations from the English-language Wikipedia from 2011 to 2017, which included closing statements, author account information, and general reply structure. They also conducted interviews with 10 of Wikipedia's most frequent closers to better understand their motivations and considerations when resolving a dispute.

Analyzing the dataset, the researchers found that about 57 percent of RfCs were formally closed. Of the remaining 43 percent, 78 percent (or around 2,300) were left stale without informal resolution -- or, about 33 percent of all the RfCs studied. Combining dataset analysis with the interviews, the researchers then fleshed out the major causes of resolution failure. Major issues include poorly articulated initial arguments, where the initiator is unclear about the issue or writes a deliberately biased proposal; excessive bickering during discussions that lead to more complicated, longer, argumentative threads that are difficult to fully examine; and simple lack of interest from third-party editors because topics may be too esoteric, among other factors.

Helpful tools

The team then developed a machine-learning model to predict whether a given RfC would close (formally or informally) or go stale, by analyzing more than 60 features of the text, Wikipedia page, and editor account information. The model achieved a 75 percent accuracy for predicting failure or success within one week after discussion started. Some more informative features for prediction, they found, include the length of the discussion, number of participants and replies, number of revisions to the article, popularity of and interest in the topic, experience of the discussion participants, and the level of vulgarity, negativity, and general aggression in the comments.

The model could one day be used by RfC initiators to monitor a discussion as it's unfolding. "We think it could be useful for editors to know how to a target their interventions," Zhang says. "They could post [the RfC] to more [Wikipedia forums] or invite more people, if it looks like it's in danger of not being resolved."

The researchers suggest Wikipedia could develop tools to help closers organize lengthy discussions, flag persuasive arguments and opinion changes within a thread, and encourage collaborative closing of RfCs.

In the future, the model and proposed tools could potentially be used for other community platforms that involve large-scale discussions and deliberations. Zhang points to online city-and community-planning forums, where citizens weigh in on proposals. "People are discussing [the proposals] and voting on them, so the tools can help communities better understand the discussions ... and would [also] be useful for the implementers of the proposals."

Zhang, Im, and other researchers have now built an external website for editors of all levels of expertise to come together to learn from one another, and more easily monitor and close discussions. "The work of closer is pretty tough," Zhang says, "so there's a shortage of people looking to close these discussions, especially difficult, longer, and more consequential ones. This could help reduce the barrier to entry [for editors to become closers] and help them collaborate to close RfCs."
Related links

ARCHIVE: Cutting down the clutter in online conversations http://news.mit.edu/2017/cutting-down-clutter-in-online-conversations-0309

ARCHIVE: Using friends to fight online harassment http://news.mit.edu/2018/using-friends-to-fight-online-harassment-0405

ARCHIVE: Democratizing databases http://news.mit.edu/2016/spreadsheet-databases-0708

ARCHIVE: Browsing in public http://news.mit.edu/2016/share-browsing-history-friends-researchers-0307

Massachusetts Institute of Technology

Related Wikipedia Articles from Brightsurf:

Wikipedia visits to disease outbreak pages show impact of news media on public attention
During the 2016 Zika outbreak, news exposure appears to have had a far bigger impact than local disease risk on the number of times people visited Zika-related Wikipedia pages in the U.S.

Automated system can rewrite outdated sentences in Wikipedia articles
A system created by MIT researchers could be used to automatically update factual inconsistencies in Wikipedia articles, reducing time and effort spent by human editors who now do the task manually.

Wikipedia, a source of information on natural disasters biased towards rich countries
This is the result of a study led by Valerio Lorini, a PhD student on the ICT programme, led by Carlos Castillo, coordinator of the Web Science and Social Computing group, with Javier Rando, a student at UPF doing the bachelor's degree in Mathematical Engineering in Data Science, focusing on flooding as a case study.

Rise of the bots: Stevens team completes first census of Wikipedia bots
Researchers at Stevens Institute of Technology, in Hoboken, N.J., have completed the first analysis of all 1,601 of Wikipedia's bots, using computer algorithms to classify them by function and shed light on the ways that machine intelligences and human users work together to improve and expand the world's largest digital encyclopedia.

Secretome of pleural effusions associated with non-small cell lung cancer (NSCLC) and malignant meso
Cryopreserved cell-free PE fluid from 101 NSCLC patients, 8 mesothelioma and 13 with benign PE was assayed for a panel of 40 cytokines/chemokines using the Luminex system.

Anatomy of a cosmic seagull
Colourful and wispy, this intriguing collection of objects is known as the Seagull Nebula, named for its resemblance to a gull in flight.

The Wikipedia gender gap
In a recent University of Washington study, researchers interviewed women 'Wikipedians' to examine the lack of female and non-binary editors in Wikipedia.

Dermatology students improve Wikipedia entries on skin disease
A group of medical students recruited to improve Wikipedia articles on skin-related diseases, saw millions more views of those stories following their editing, highlighting the value of expert input on the popular web encyclopedia.

Could internet activity provide accurate in plant and animal conservation?
More than a quarter of the species in their dataset showed seasonal interest.

Analysis of billions of Wikipedia searches reveals biodiversity secrets
An international team of researchers from the University of Oxford, the University of Birmingham and Ben-Gurion University of the Negev have found that the way in which people use the internet is closely tied to patterns and rhythms in the natural world.

Read More: Wikipedia News and Wikipedia Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.