Nav: Home

MIT scholars show new method of harvesting crowd wisdom

January 25, 2017

Magnuviator Fossil with Sketch The wisdom of crowds is not always perfect. But two scholars at MIT's Sloan Neuroeconomics Lab, along with a colleague at Princeton University, have found a way to make it better.

Their method, explained in a newly published paper, uses a technique the researchers call the "surprisingly popular" algorithm to better extract correct answers from large groups of people. As such, it could refine wisdom-of-crowds surveys, which are used in political and economic forecasting, as well as many other collective activities, from pricing artworks to grading scientific research proposals.

The new method is simple. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer.

"In situations where there is enough information in the crowd to determine the correct answer to a question, that answer will be the one [that] most outperforms expectations," says paper co-author Drazen Prelec, a professor at the MIT Sloan School of Management as well as the Department of Economics and the Department of Brain and Cognitive Sciences.

The paper is built on both theoretical and empirical work. The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates.

Across all these areas, the researchers found that the "surprisingly popular" algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers). And it reduced errors by 22.2 percent compared to another kind of confidence-weighted votes, those taking the answers with the highest average confidence levels.

The paper, "A solution to the single-question crowd wisdom problem," is being published today in Nature. The authors are Prelec; John McCoy, a doctoral student in the MIT Department of Brain and Cognitive Sciences; and H. Sebastian Seung, a professor of neuroscience and computer science at Princeton University and a former MIT faculty member. Prelec and McCoy are also researchers in the MIT Neuroeconomics Laboratory, where Prelec is the principal investigator.

A capital idea

To see how the algorithm works in practice, consider a case the researchers tested. A group of people were asked a yes-or-no question: Is Philadelphia the capital of Pennsylvania? They were also asked to predict the prevalence of "yes" votes.

Philadelphia is not the capital of Pennsylvania; the correct answer is Harrisburg. But most people believe Philadelphia is the capital because it is a "large, historically significant city." Moreover, the people who mistakenly thought Philadelphia is the state capital largely thought other people would answer the same way. So they predicted that a very high percentage of people would answer "yes."

Meanwhile, a certain number of respondents knew that Harrisburg is the correct answer. However, a large portion of those people also anticipated that many other people would incorrectly think the capital is Philadelphia. So the people who themselves answered "no" still expected a very high percentage of "yes" answers.

That means the answer to the two questions -- Is Philadelphia the capital? Will other people think so? -- diverged. Almost everyone expected other people to answer "yes." But the actual percentage of people who answered "yes" was significantly lower. For this reason, the "no" answer was the "surprisingly popular" one, since it deviated from expectations of what the answer would be.

And since the "surprisingly popular" answer differed in the "no" direction, that tells us the correct answer: No, Philadelphia is not the capital.

The same principle applies no matter which direction responses deviate from expectations. When people were asked if Columbia is the capital of South Carolina, the opposite happened: More people answered "yes," compared to their expectations of how many people would say "yes." So the surprisingly popular answer was, correctly: Yes, Columbia is the capital.

The wisdom of subsets of crowds

In this sense, the "surprisingly popular" principle is not simply derived from the wisdom of crowds. Instead, it uses the knowledge of a well-informed subgroup of people within the larger crowd as a diagnostically powerful tool that points to the right answer.

"A lot of crowd wisdom weights people equally," McCoy explains. "But some people have more specialized knowledge." And those people -- if they have both correct information and a correct sense of public perception -- make a big difference.

This is the case across scenarios that the researchers tested. Consider art. The researchers asked art professionals to guess the price range for different contemporary artworks. Individually, art experts selected price ranges that were typically too low, perhaps because selecting a lower range is a reasonable, safe answer for an artwork that the expert does not recognize. Collectively, this makes the majority opinion of an expert panel even more biased in the direction of low prices.

And this is where the "surprisingly popular" principle makes a difference, since it does not depend on an absolute majority of expert opinion. Instead, suppose a relatively small number of experts believe a piece sold for $100,000, while anticipating that most other people will think it sold for less. In that case, the evaluations of those experts will lead the "surprisingly popular" answer to be that the artwork was more expensive than most people thought.

"The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention," Prelec says.

Recovering truth

The scholars recognize that the "surprisingly popular" algorithm is not theoretically foolproof in practice. It is at least conceivable that people could anticipate a "surprisingly popular" opinion and try to subvert it, although that would be very hard to execute. It is also the case, as they write in the Nature paper, that "These claims are theoretical and do not guarantee success in practice, as actual respondents will fall short of ideal."

Still, the researchers themselves hope their work will be tested in a variety of settings. In the paper they express confidence that the "surprisingly popular" principle will prove durable, asserting: "Such knowledge can be exploited to recover truth even when traditional voting methods fail."
Additional background

Views you can use? How online ratings affect your judgment

My connectome, myself

Wisdom of crowds

Massachusetts Institute of Technology

Related Algorithm Articles:

QUT algorithm could quash Twitter abuse of women
Online abuse targeting women, including threats of harm or sexual violence, has proliferated across all social media platforms but QUT researchers have developed a sophisticated statistical model to identify misogynistic content and help drum it out of the Twittersphere.
New learning algorithm should significantly expand the possible applications of AI
The e-prop learning method developed at Graz University of Technology forms the basis for drastically more energy-efficient hardware implementations of Artificial Intelligence.
Algorithm predicts risk for PTSD after traumatic injury
With high precision, a new algorithm predicts which patients treated for traumatic injuries in the emergency department will later develop posttraumatic stress disorder.
New algorithm uses artificial intelligence to help manage type 1 diabetes
Researchers and physicians at Oregon Health & Science University have designed a method to help people with type 1 diabetes better manage their glucose levels.
A new algorithm predicts the difficulty in fighting fire
The tool completes previous studies with new variables and could improve the ability to respond to forest fires.
New algorithm predicts optimal materials among all possible compounds
Skoltech researchers have offered a solution to the problem of searching for materials with required properties among all possible combinations of chemical elements.
New algorithm to help process biological images
Skoltech researchers have presented a new biological image processing method that accurately picks out specific biological objects in complex images.
Skoltech scientists break Google's quantum algorithm
In the near term, Google has devised new quantum enhanced algorithms that operate in the presence of realistic noise.
The most human algorithm
A team from the research group SEES:lab of the Department of Chemical Engineering of the Universitat Rovira I Virgili and ICREA has made a breakthrough with the development of a new algorithm that makes more accurate predictions and generates mathematical models that also make it possible to understand these predictions.
Algorithm turns cancer gene discovery on its head
Prediction method could help personalize cancer treatments and reveal new drug targets.
More Algorithm News and Algorithm Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Debbie Millman: Designing Our Lives
From prehistoric cave art to today's social media feeds, to design is to be human. This hour, designer Debbie Millman guides us through a world made and remade–and helps us design our own paths.
Now Playing: Science for the People

#574 State of the Heart
This week we focus on heart disease, heart failure, what blood pressure is and why it's bad when it's high. Host Rachelle Saunders talks with physician, clinical researcher, and writer Haider Warraich about his book "State of the Heart: Exploring the History, Science, and Future of Cardiac Disease" and the ails of our hearts.
Now Playing: Radiolab

Insomnia Line
Coronasomnia is a not-so-surprising side-effect of the global pandemic. More and more of us are having trouble falling asleep. We wanted to find a way to get inside that nighttime world, to see why people are awake and what they are thinking about. So what'd Radiolab decide to do?  Open up the phone lines and talk to you. We created an insomnia hotline and on this week's experimental episode, we stayed up all night, taking hundreds of calls, spilling secrets, and at long last, watching the sunrise peek through.   This episode was produced by Lulu Miller with Rachael Cusick, Tracie Hunte, Tobin Low, Sarah Qari, Molly Webster, Pat Walters, Shima Oliaee, and Jonny Moens. Want more Radiolab in your life? Sign up for our newsletter! We share our latest favorites: articles, tv shows, funny Youtube videos, chocolate chip cookie recipes, and more. Support Radiolab by becoming a member today at