Nav: Home

MIT scholars show new method of harvesting crowd wisdom

January 25, 2017

Magnuviator Fossil with Sketch The wisdom of crowds is not always perfect. But two scholars at MIT's Sloan Neuroeconomics Lab, along with a colleague at Princeton University, have found a way to make it better.

Their method, explained in a newly published paper, uses a technique the researchers call the "surprisingly popular" algorithm to better extract correct answers from large groups of people. As such, it could refine wisdom-of-crowds surveys, which are used in political and economic forecasting, as well as many other collective activities, from pricing artworks to grading scientific research proposals.

The new method is simple. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer.

"In situations where there is enough information in the crowd to determine the correct answer to a question, that answer will be the one [that] most outperforms expectations," says paper co-author Drazen Prelec, a professor at the MIT Sloan School of Management as well as the Department of Economics and the Department of Brain and Cognitive Sciences.

The paper is built on both theoretical and empirical work. The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates.

Across all these areas, the researchers found that the "surprisingly popular" algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers). And it reduced errors by 22.2 percent compared to another kind of confidence-weighted votes, those taking the answers with the highest average confidence levels.

The paper, "A solution to the single-question crowd wisdom problem," is being published today in Nature. The authors are Prelec; John McCoy, a doctoral student in the MIT Department of Brain and Cognitive Sciences; and H. Sebastian Seung, a professor of neuroscience and computer science at Princeton University and a former MIT faculty member. Prelec and McCoy are also researchers in the MIT Neuroeconomics Laboratory, where Prelec is the principal investigator.

A capital idea

To see how the algorithm works in practice, consider a case the researchers tested. A group of people were asked a yes-or-no question: Is Philadelphia the capital of Pennsylvania? They were also asked to predict the prevalence of "yes" votes.

Philadelphia is not the capital of Pennsylvania; the correct answer is Harrisburg. But most people believe Philadelphia is the capital because it is a "large, historically significant city." Moreover, the people who mistakenly thought Philadelphia is the state capital largely thought other people would answer the same way. So they predicted that a very high percentage of people would answer "yes."

Meanwhile, a certain number of respondents knew that Harrisburg is the correct answer. However, a large portion of those people also anticipated that many other people would incorrectly think the capital is Philadelphia. So the people who themselves answered "no" still expected a very high percentage of "yes" answers.

That means the answer to the two questions -- Is Philadelphia the capital? Will other people think so? -- diverged. Almost everyone expected other people to answer "yes." But the actual percentage of people who answered "yes" was significantly lower. For this reason, the "no" answer was the "surprisingly popular" one, since it deviated from expectations of what the answer would be.

And since the "surprisingly popular" answer differed in the "no" direction, that tells us the correct answer: No, Philadelphia is not the capital.

The same principle applies no matter which direction responses deviate from expectations. When people were asked if Columbia is the capital of South Carolina, the opposite happened: More people answered "yes," compared to their expectations of how many people would say "yes." So the surprisingly popular answer was, correctly: Yes, Columbia is the capital.

The wisdom of subsets of crowds

In this sense, the "surprisingly popular" principle is not simply derived from the wisdom of crowds. Instead, it uses the knowledge of a well-informed subgroup of people within the larger crowd as a diagnostically powerful tool that points to the right answer.

"A lot of crowd wisdom weights people equally," McCoy explains. "But some people have more specialized knowledge." And those people -- if they have both correct information and a correct sense of public perception -- make a big difference.

This is the case across scenarios that the researchers tested. Consider art. The researchers asked art professionals to guess the price range for different contemporary artworks. Individually, art experts selected price ranges that were typically too low, perhaps because selecting a lower range is a reasonable, safe answer for an artwork that the expert does not recognize. Collectively, this makes the majority opinion of an expert panel even more biased in the direction of low prices.

And this is where the "surprisingly popular" principle makes a difference, since it does not depend on an absolute majority of expert opinion. Instead, suppose a relatively small number of experts believe a piece sold for $100,000, while anticipating that most other people will think it sold for less. In that case, the evaluations of those experts will lead the "surprisingly popular" answer to be that the artwork was more expensive than most people thought.

"The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention," Prelec says.

Recovering truth

The scholars recognize that the "surprisingly popular" algorithm is not theoretically foolproof in practice. It is at least conceivable that people could anticipate a "surprisingly popular" opinion and try to subvert it, although that would be very hard to execute. It is also the case, as they write in the Nature paper, that "These claims are theoretical and do not guarantee success in practice, as actual respondents will fall short of ideal."

Still, the researchers themselves hope their work will be tested in a variety of settings. In the paper they express confidence that the "surprisingly popular" principle will prove durable, asserting: "Such knowledge can be exploited to recover truth even when traditional voting methods fail."
-end-
Additional background

ARCHIVE:
Views you can use? How online ratings affect your judgment

ARCHIVE:
My connectome, myself

ARCHIVE:
Wisdom of crowds

Massachusetts Institute of Technology

Related Algorithm Articles:

New algorithm to help process biological images
Skoltech researchers have presented a new biological image processing method that accurately picks out specific biological objects in complex images.
Skoltech scientists break Google's quantum algorithm
In the near term, Google has devised new quantum enhanced algorithms that operate in the presence of realistic noise.
The most human algorithm
A team from the research group SEES:lab of the Department of Chemical Engineering of the Universitat Rovira I Virgili and ICREA has made a breakthrough with the development of a new algorithm that makes more accurate predictions and generates mathematical models that also make it possible to understand these predictions.
Algorithm turns cancer gene discovery on its head
Prediction method could help personalize cancer treatments and reveal new drug targets.
New algorithm predicts gestational diabetes
Timely prediction may help prevent the condition using nutritional and lifestyle changes.
New algorithm could mean more efficient, accurate equipment for Army
Researchers working on an Army-funded project have developed an algorithm to simulate how electromagnetic waves interact with materials in devices to create equipment more efficiently and accurately.
Universal algorithm set to boost microscopes
EPFL scientists have developed an algorithm that can determine whether a super-resolution microscope is operating at maximum resolution based on a single image.
Algorithm designed to map universe, solve mysteries
Cornell University researchers have developed an algorithm designed to visualize models of the universe in order to solve some of physics' greatest mysteries.
Algorithm tells robots where nearby humans are headed
A new tool for predicting a person's movement trajectory may help humans and robots work together in close proximity.
Algorithm to transform investment banking with higher returns
A University of Bath researcher has created an algorithm which aims to remove the elements of chance, bias or emotion from investment banking decisions, a development which has the potential to reduce errors in financial decision making and improve financial returns in global markets.
More Algorithm News and Algorithm Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Climate Mindset
In the past few months, human beings have come together to fight a global threat. This hour, TED speakers explore how our response can be the catalyst to fight another global crisis: climate change. Guests include political strategist Tom Rivett-Carnac, diplomat Christiana Figueres, climate justice activist Xiye Bastida, and writer, illustrator, and artist Oliver Jeffers.
Now Playing: Science for the People

#562 Superbug to Bedside
By now we're all good and scared about antibiotic resistance, one of the many things coming to get us all. But there's good news, sort of. News antibiotics are coming out! How do they get tested? What does that kind of a trial look like and how does it happen? Host Bethany Brookeshire talks with Matt McCarthy, author of "Superbugs: The Race to Stop an Epidemic", about the ins and outs of testing a new antibiotic in the hospital.
Now Playing: Radiolab

Speedy Beet
There are few musical moments more well-worn than the first four notes of Beethoven's Fifth Symphony. But in this short, we find out that Beethoven might have made a last-ditch effort to keep his music from ever feeling familiar, to keep pushing his listeners to a kind of psychological limit. Big thanks to our Brooklyn Philharmonic musicians: Deborah Buck and Suzy Perelman on violin, Arash Amini on cello, and Ah Ling Neu on viola. And check out The First Four Notes, Matthew Guerrieri's book on Beethoven's Fifth. Support Radiolab today at Radiolab.org/donate.