Machine learning has a flaw; it's gullible

June 23, 2020

Artificial intelligence and machine learning technologies are poised to supercharge productivity in the knowledge economy, transforming the future of work. But they're far from perfect.

Machine learning (ML) - technology in which algorithms "learn" from existing patterns in data to conduct statistically driven predictions and facilitate decisions - has been found in multiple contexts to reveal bias. An example is coming under fire for a hiring algorithm that revealed gender and racial bias. Such biases often result from slanted training data or skewed algorithms.

And in other business contexts, there's another potential source of bias. It comes when outside individuals stand to benefit from bias predictions, and work to strategically alter the inputs. In other words, they're gaming the ML systems. A couple of the most common contexts are perhaps job applicants and people making a claim against their insurance. ML algorithms are built for these contexts. They can review resumes faster than any recruiter and comb through insurance claims faster than any human processor. But people who submit resumes and insurance claims have a strategic interest in getting positive outcomes - and some of them know how to outthink the algorithm.

Rajshree Agarwal and Evan Starr, researchers at the University of Maryland's Robert H. Smith School of Business worked with Prithwiraj Choudhury at Harvard Business School to set out to answer "Can ML correct for such strategic behavior?"

They found that two critical attributes of humans serve as important complements to machine learning in correcting for these biases. The more obvious one is vintage-specific skills, this ensures that humans have the ability to properly interface with machine learning so they can guide it appropriately. The other important attribute is domain-specific expertise, the knowledge that humans can provide machines on how to correct for the incompleteness of inputs.

Their work is forthcoming in Strategic Management Journal as "Machine Learning and Human Capital Complementarities: Experimental Evidence on Bias Mitigation."

Prior research in so-called "adversarial" ML looked closely at attempts to "trick" ML technologies, and generally concluded that it's extremely challenging to prepare the ML technology to account for every possible input and manipulation. In other words, ML is trickable.

What should firms do about it? Can they limit ML prediction bias? And, is there a role for humans to work with ML to do so? Agarwal, Choudhury and Starr honed their focus on patent examination, a context rife with potential trickery. "Patent examiners face a time-consuming challenge of accurately determining the novelty and nonobviousness of a patent application by sifting through ever-expanding amounts of 'prior art,'" or inventions that have come before, the researchers explain. It's challenging work.

Compounding the challenge: patent applicants are permitted by law to create hyphenated words and assign new meaning to existing words to describe their inventions. It's an opportunity, the researchers explain, for applicants to strategically write their applications in a strategic, ML-targeting way. The U.S. Patent and Trademark Office is generally wise to this. It has invited in ML technology that "reads" the text of applications, with the goal of spotting the most relevant prior art quicker and leading to more accurate decisions. "Although it is theoretically feasible for ML algorithms to continually learn and correct for ways that patent applicants attempt to manipulate the algorithm, the potential for patent applicants to dynamically update their writing strategies makes it practically impossible to train an ML algorithm to correct for this behavior," the researchers write.

In its study, the team conducted observational and experimental research. They found that patent language changes over time, making it highly challenging for any ML tool to operate perfectly on its own. The ML benefited strongly, they found, from human collaboration.

People with skills and knowledge accumulated through prior learning within a domain complement ML in mitigating bias stemming from applicant manipulation, the researchers found, because domain experts bring relevant outside information to correct for strategically altered inputs. And individuals with vintage-specific skills - skills and knowledge accumulated through prior familiarity of tasks with the technology - are better able to handle the complexities in ML technology interfaces.

They caution that although the provision of expert advice and vintage-specific human capital increases initial productivity, it remains unclear whether constant exposure and learning-by-doing by workers would cause the relative differences between the groups to grow or shrink over time. They encourage further research into the evolution in the productivity of all ML technologies, and their contingencies.
The research drew the 2019 Best Conference Paper Award from the Strategic Management Society and won the Best Interdisciplinary Paper Award from the Strategic Human Capital Interest Group at Strategic Management Society 2019.

University of Maryland

Related Learning Articles from Brightsurf:

Learning the language of sugars
We're told not to eat too much sugar, but in reality, all of our cells are covered in sugar molecules called glycans.

When learning on your own is not enough
We make decisions based on not only our own learning experience, but also learning from others.

Learning more about particle collisions with machine learning
A team of Argonne scientists has devised a machine learning algorithm that calculates, with low computational time, how the ATLAS detector in the Large Hadron Collider would respond to the ten times more data expected with a planned upgrade in 2027.

Getting kids moving, and learning
Children are set to move more, improve their skills, and come up with their own creative tennis games with the launch of HomeCourtTennis, a new initiative to assist teachers and coaches with keeping kids active while at home.

How expectations influence learning
During learning, the brain is a prediction engine that continually makes theories about our environment and accurately registers whether an assumption is true or not.

Technology in higher education: learning with it instead of from it
Technology has shifted the way that professors teach students in higher education.

Learning is optimized when we fail 15% of the time
If you're always scoring 100%, you're probably not learning anything new.

School spending cuts triggered by great recession linked to sizable learning losses for learning losses for students in hardest hit areas
Substantial school spending cuts triggered by the Great Recession were associated with sizable losses in academic achievement for students living in counties most affected by the economic downturn, according to a new study published today in AERA Open, a peer-reviewed journal of the American Educational Research Association.

Lessons in learning
A new Harvard study shows that, though students felt like they learned more from traditional lectures, they actually learned more when taking part in active learning classrooms.

Learning to look
A team led by JGI scientists has overhauled the perception of inovirus diversity.

Read More: Learning News and Learning Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to