Nav: Home

MIT scholars show new method of harvesting crowd wisdom

January 25, 2017

Magnuviator Fossil with Sketch The wisdom of crowds is not always perfect. But two scholars at MIT's Sloan Neuroeconomics Lab, along with a colleague at Princeton University, have found a way to make it better.

Their method, explained in a newly published paper, uses a technique the researchers call the "surprisingly popular" algorithm to better extract correct answers from large groups of people. As such, it could refine wisdom-of-crowds surveys, which are used in political and economic forecasting, as well as many other collective activities, from pricing artworks to grading scientific research proposals.

The new method is simple. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer.

"In situations where there is enough information in the crowd to determine the correct answer to a question, that answer will be the one [that] most outperforms expectations," says paper co-author Drazen Prelec, a professor at the MIT Sloan School of Management as well as the Department of Economics and the Department of Brain and Cognitive Sciences.

The paper is built on both theoretical and empirical work. The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates.

Across all these areas, the researchers found that the "surprisingly popular" algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers). And it reduced errors by 22.2 percent compared to another kind of confidence-weighted votes, those taking the answers with the highest average confidence levels.

The paper, "A solution to the single-question crowd wisdom problem," is being published today in Nature. The authors are Prelec; John McCoy, a doctoral student in the MIT Department of Brain and Cognitive Sciences; and H. Sebastian Seung, a professor of neuroscience and computer science at Princeton University and a former MIT faculty member. Prelec and McCoy are also researchers in the MIT Neuroeconomics Laboratory, where Prelec is the principal investigator.

A capital idea

To see how the algorithm works in practice, consider a case the researchers tested. A group of people were asked a yes-or-no question: Is Philadelphia the capital of Pennsylvania? They were also asked to predict the prevalence of "yes" votes.

Philadelphia is not the capital of Pennsylvania; the correct answer is Harrisburg. But most people believe Philadelphia is the capital because it is a "large, historically significant city." Moreover, the people who mistakenly thought Philadelphia is the state capital largely thought other people would answer the same way. So they predicted that a very high percentage of people would answer "yes."

Meanwhile, a certain number of respondents knew that Harrisburg is the correct answer. However, a large portion of those people also anticipated that many other people would incorrectly think the capital is Philadelphia. So the people who themselves answered "no" still expected a very high percentage of "yes" answers.

That means the answer to the two questions -- Is Philadelphia the capital? Will other people think so? -- diverged. Almost everyone expected other people to answer "yes." But the actual percentage of people who answered "yes" was significantly lower. For this reason, the "no" answer was the "surprisingly popular" one, since it deviated from expectations of what the answer would be.

And since the "surprisingly popular" answer differed in the "no" direction, that tells us the correct answer: No, Philadelphia is not the capital.

The same principle applies no matter which direction responses deviate from expectations. When people were asked if Columbia is the capital of South Carolina, the opposite happened: More people answered "yes," compared to their expectations of how many people would say "yes." So the surprisingly popular answer was, correctly: Yes, Columbia is the capital.

The wisdom of subsets of crowds

In this sense, the "surprisingly popular" principle is not simply derived from the wisdom of crowds. Instead, it uses the knowledge of a well-informed subgroup of people within the larger crowd as a diagnostically powerful tool that points to the right answer.

"A lot of crowd wisdom weights people equally," McCoy explains. "But some people have more specialized knowledge." And those people -- if they have both correct information and a correct sense of public perception -- make a big difference.

This is the case across scenarios that the researchers tested. Consider art. The researchers asked art professionals to guess the price range for different contemporary artworks. Individually, art experts selected price ranges that were typically too low, perhaps because selecting a lower range is a reasonable, safe answer for an artwork that the expert does not recognize. Collectively, this makes the majority opinion of an expert panel even more biased in the direction of low prices.

And this is where the "surprisingly popular" principle makes a difference, since it does not depend on an absolute majority of expert opinion. Instead, suppose a relatively small number of experts believe a piece sold for $100,000, while anticipating that most other people will think it sold for less. In that case, the evaluations of those experts will lead the "surprisingly popular" answer to be that the artwork was more expensive than most people thought.

"The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention," Prelec says.

Recovering truth

The scholars recognize that the "surprisingly popular" algorithm is not theoretically foolproof in practice. It is at least conceivable that people could anticipate a "surprisingly popular" opinion and try to subvert it, although that would be very hard to execute. It is also the case, as they write in the Nature paper, that "These claims are theoretical and do not guarantee success in practice, as actual respondents will fall short of ideal."

Still, the researchers themselves hope their work will be tested in a variety of settings. In the paper they express confidence that the "surprisingly popular" principle will prove durable, asserting: "Such knowledge can be exploited to recover truth even when traditional voting methods fail."
-end-
Additional background

ARCHIVE:
Views you can use? How online ratings affect your judgment

ARCHIVE:
My connectome, myself

ARCHIVE:
Wisdom of crowds

Massachusetts Institute of Technology

Related Algorithm Articles:

Algorithm personalizes which cancer mutations are best targets for immunotherapy
As tumor cells multiply, they often spawn tens of thousands of genetic mutations.
Universal algorithm set to boost microscopes
EPFL scientists have developed an algorithm that can determine whether a super-resolution microscope is operating at maximum resolution based on a single image.
Algorithm designed to map universe, solve mysteries
Cornell University researchers have developed an algorithm designed to visualize models of the universe in order to solve some of physics' greatest mysteries.
Algorithm tells robots where nearby humans are headed
A new tool for predicting a person's movement trajectory may help humans and robots work together in close proximity.
Algorithm to transform investment banking with higher returns
A University of Bath researcher has created an algorithm which aims to remove the elements of chance, bias or emotion from investment banking decisions, a development which has the potential to reduce errors in financial decision making and improve financial returns in global markets.
Algorithm provides customized caffeine strategy for alertness
A web-based caffeine optimization tool successfully designs effective strategies to maximize alertness while avoiding excessive caffeine consumption, according to preliminary results from a new study.
New algorithm optimizes quantum computing problem-solving
Tohoku University researchers have developed an algorithm that enhances the ability of a Canadian-designed quantum computer to more efficiently find the best solution for complicated problems, according to a study published in the journal Scientific Reports.
Machine learning algorithm helps in the search for new drugs
Researchers have designed a machine learning algorithm for drug discovery which has been shown to be twice as efficient as the industry standard, which could accelerate the process of developing new treatments for disease.
Researchers create algorithm to predict PEDV outbreaks
Researchers from North Carolina State University have developed an algorithm that could give pig farms advance notice of porcine epidemic diarrhea virus (PEDV) outbreaks.
New algorithm provides a more detailed look at urban heat islands
Urban areas are warmer than the adjacent undeveloped land, a phenomenon known as the urban heat island effect.
More Algorithm News and Algorithm Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Rethinking Anger
Anger is universal and complex: it can be quiet, festering, justified, vengeful, and destructive. This hour, TED speakers explore the many sides of anger, why we need it, and who's allowed to feel it. Guests include psychologists Ryan Martin and Russell Kolts, writer Soraya Chemaly, former talk radio host Lisa Fritsch, and business professor Dan Moshavi.
Now Playing: Science for the People

#538 Nobels and Astrophysics
This week we start with this year's physics Nobel Prize awarded to Jim Peebles, Michel Mayor, and Didier Queloz and finish with a discussion of the Nobel Prizes as a way to award and highlight important science. Are they still relevant? When science breakthroughs are built on the backs of hundreds -- and sometimes thousands -- of people's hard work, how do you pick just three to highlight? Join host Rachelle Saunders and astrophysicist, author, and science communicator Ethan Siegel for their chat about astrophysics and Nobel Prizes.