Best of the best: Who makes the most accurate decisions in expert groups?

November 20, 2019

Experts don't always agree with one another when making predictions or diagnoses. So how can we find out which expert in a group makes the best and most accurate decisions? An interdisciplinary team of researchers at the Max Planck Institute for Human Development and the Leibniz Institute of Freshwater Ecology and Inland Fisheries has developed a simple method for identifying the most accurate experts and tested it successfully in various groups. Their findings have been published in Science Advances.

Does a mass on a mammogram indicate breast cancer? Will Serbia be a member of the EU by 2025? Will there be more floods in Germany in five years' time? The diagnoses and predictions made by doctors, scientists, and experts often have far-reaching consequences. And in many cases, it is only years later that it is possible to say which expert made the right call most often.

An interdisciplinary research team from the Max Planck Institute for Human Development and the Leibniz Institute of Freshwater Ecology and Inland Fisheries has developed a simple new method that can be used to identify the best decision-makers from a group of experts without having to know whether their decisions--past or present--are correct or incorrect. "Providing that at least half of all decisions made within the group are correct--which is typically the case in expert groups--and that each person has made about 20 yes/no decisions, this method has proved to work very well," says Max Wolf, researcher at the Leibniz Institute of Freshwater Ecology and Inland Fisheries and co-author of the study.

The method was developed on the basis of insights into collective intelligence. It rests on a simple assumption: Those individuals in a group of experts who make decisions that are most similar to the decisions of others also make the best decisions. For yes/no decisions, this assumption is easily confirmed by means of mathematical modeling. To test whether the method also works in real groups, the researchers analyzed published predictions and diagnoses made by various groups in different fields.

For example, the researchers examined the diagnoses made by 100 radiologists in the USA. In the early 2000s, the radiologists interpreted the mammograms of 155 women to determine whether or not they had breast cancer. The research team analyzed the data to identify the radiologists whose decisions were, on average, most similar to the decisions of the others. As they had access to follow-up information on the health status of the 155 women screened, the researchers were also able to determine which radiologists made the most accurate and thus best diagnoses. They were the same radiologists as those identified using the new statistical method.

"It has been shown time and again that experts who are good in their field are good in a similar way, whereas poor performers are bad in very different ways. Working on the basis of this observation, we developed this new method and tested it in various areas," says Ralf Kurvers, lead author and researcher at the Center for Adaptive Rationality at the Max Planck Institute for Human Development.

In addition to radiologists' diagnoses, the research team analyzed skin cancer diagnoses made by 40 Italian dermatologists; geopolitical predictions made by 90 forecasters on the online platform Good Judgment Project; and the results of a simple general knowledge test, in which 100 participants were asked to identify the larger of two American cities.

"We believe that the relationship between similarity and accuracy of decisions can be an effective tool for practice. The method can be used to improve collective and individual decision-making processes in medical diagnostics, environmental risk analyses, and the business world," says co-author Stefan Herzog, also a researcher at the Center for Adaptive Rationality.
-end-
Original Publication

Kurvers, R., Herzog, S. M., Hertwig, R., Krause, J., Moussaid, M., Argenziano, G., ... Wolf, M. (2019). How to detect high-performing individuals and groups: Decision similarity predicts accuracy. Science Advances.

Max-Planck-Gesellschaft

Related Breast Cancer Articles from Brightsurf:

Oncotarget: IGF2 expression in breast cancer tumors and in breast cancer cells
The Oncotarget authors propose that methylation of DVDMR represents a novel epigenetic biomarker that determines the levels of IGF2 protein expression in breast cancer.

Breast cancer: AI predicts which pre-malignant breast lesions will progress to advanced cancer
New research at Case Western Reserve University in Cleveland, Ohio, could help better determine which patients diagnosed with the pre-malignant breast cancer commonly as stage 0 are likely to progress to invasive breast cancer and therefore might benefit from additional therapy over and above surgery alone.

Partial breast irradiation effective treatment option for low-risk breast cancer
Partial breast irradiation produces similar long-term survival rates and risk for recurrence compared with whole breast irradiation for many women with low-risk, early stage breast cancer, according to new clinical data from a national clinical trial involving researchers from The Ohio State University Comprehensive Cancer Center - Arthur G.

Breast screening linked to 60 per cent lower risk of breast cancer death in first 10 years
Women who take part in breast screening have a significantly greater benefit from treatments than those who are not screened, according to a study of more than 50,000 women.

More clues revealed in link between normal breast changes and invasive breast cancer
A research team, led by investigators from Georgetown Lombardi Comprehensive Cancer Center, details how a natural and dramatic process -- changes in mammary glands to accommodate breastfeeding -- uses a molecular process believed to contribute to survival of pre-malignant breast cells.

Breast tissue tumor suppressor PTEN: A potential Achilles heel for breast cancer cells
A highly collaborative team of researchers at the Medical University of South Carolina and Ohio State University report in Nature Communications that they have identified a novel pathway for connective tissue PTEN in breast cancer cell response to radiotherapy.

Computers equal radiologists in assessing breast density and associated breast cancer risk
Automated breast-density evaluation was just as accurate in predicting women's risk of breast cancer, found and not found by mammography, as subjective evaluation done by radiologists, in a study led by researchers at UC San Francisco and Mayo Clinic.

Blood test can effectively rule out breast cancer, regardless of breast density
A new study published in PLOS ONE demonstrates that Videssa® Breast, a multi-protein biomarker blood test for breast cancer, is unaffected by breast density and can reliably rule out breast cancer in women with both dense and non-dense breast tissue.

Study shows influence of surgeons on likelihood of removal of healthy breast after breast cancer dia
Attending surgeons can have a strong influence on whether a patient undergoes contralateral prophylactic mastectomy after a diagnosis of breast cancer, according to a study published by JAMA Surgery.

Young breast cancer patients undergoing breast conserving surgery see improved prognosis
A new analysis indicates that breast cancer prognoses have improved over time in young women treated with breast conserving surgery.

Read More: Breast Cancer News and Breast Cancer Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.