Nav: Home

Gestures improve communication -- even with robots

April 04, 2016

In the world of robot communication, it seems actions speak louder than words. Scientists in the UK have discovered that by getting robot avatars to "talk with their hands," we understand them as well as we do our fellow human beings.

Avatars have been in existence since the 1980s and today are used by millions of people across the globe. They are big business too: from artificial intelligence to social media and psychotherapy to high-end video games, they are used to sell things, to solve problems, to teach us and to entertain us. As avatars become more sophisticated, and their use in society grows, research is focusing on how to improve communication with them. Getting your message across with your avatar is more important than ever, and learning how to improve this communication is a big deal.

Scientists Paul Bremner and Ute Leonards took on this challenge in a recent study published in Frontiers in Psychology. They built their study around the hypothesis that if avatars were to use "iconic" hand gestures together with speech, we would understand them more easily. Iconic gestures have a distinct meaning, like opening a door or a book, and using gestures together with speech is known as "multi-modal communication." The aim of the study was to discover if people could understand avatars performing multi-modal communication as well as they could a human actor. The study also investigated if multi-modal communication by an avatar was more understandable than speech alone.

To test their theory, the scientists filmed an actor reading out a series of phrases whilst performing specific iconic gestures. They then filmed an avatar using these recorded phrases and mimicking the gestures. Films of both the actor and avatar were then shown to the experiment participants, who had to identify what the human and avatar were trying to communicate. The research was a success: the scientists were able to prove that multi-modal communication by avatars is indeed more understandable than speech alone. Not only that, but when using multi-modal communication, we understand them as well as we do humans.

Getting avatars to talk with their hands in the same way that humans do was a challenge in itself. Whilst performing the gestures, the actor used state-of-the-art technology. His movements were tracked using a Microsoft Kinect sensor so that his arm gestures could be recorded as data. The avatar used this data to mimic his gestures. The equipment did have some limitations however; the avatar does not have the same hand shape or degree of movement available as a human - something the pair plans to work on in the future.

Despite the limitations, the scientists' research showed that their method of translating human gestures to an avatar was successful. More importantly, they are confident that the avatar gestures, when used with speech, are as easily understood as from a human. Now that this is established, the pair plans to carry out more research in the field. Future work will involve looking at more types of gestures and in different settings, plus how to make the translation of gestures from human to avatar more efficient. There will be plenty of work to keep them going - they have yet to take on different cultures, and Italy, a nation of people famed for expressive hand gestures, is still on the horizon.
-end-


Frontiers

Related Speech Articles:

Speech and language deficits in children with autism may not cause tantrums
Speech or language impairments may not be the cause of more frequent tantrums in children with autism, according to Penn State College of Medicine researchers.
What's coming next? Scientists identify how the brain predicts speech
A new study, publishing on April 25 in the open access journal PLOS Biology, has shed light on how the brain helps us to predict what is coming next in speech.
Whether our speech is fast or slow, we say about the same
Fast talkers tend to convey less information with each word and syntactic structure than slower-paced speakers, meaning that no matter our pace, we all say just about as much in a given time, a new study finds.
Do dogs of all ages respond equally to dog-directed speech?
People tend to talk to dogs as though they are human babies.
New approach may open up speech recognition to more languages
At the Neural Information Processing Systems conference this week, researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) are presenting a new approach to training speech-recognition systems that doesn't depend on transcription.
Preschoolers' expectations shape how they interpret speech
When we listen to people speak, we aren't just hearing the sounds they're making, we're also actively trying to infer what they're going to say.
Genes for speech may not be limited to humans
Mice use language but not speech, which is thought to need biological functions particular to people.
Hearing with your eyes -- a Western style of speech perception
Which parts of a person's face do you look at when you listen them speak?
When do speech difficulties in children matter for literacy?
A new study found that speech difficulties are linked with difficulties in learning to read when children first start school, but these effects are no longer apparent at 8 years of age.
Male mice model human speech defect
Male mice carrying a mutation in the Foxp2 gene have difficulty putting the syllables of their ultrasonic wooing song into proper order.

Related Speech Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Anthropomorphic
Do animals grieve? Do they have language or consciousness? For a long time, scientists resisted the urge to look for human qualities in animals. This hour, TED speakers explore how that is changing. Guests include biological anthropologist Barbara King, dolphin researcher Denise Herzing, primatologist Frans de Waal, and ecologist Carl Safina.
Now Playing: Science for the People

#SB2 2019 Science Birthday Minisode: Mary Golda Ross
Our second annual Science Birthday is here, and this year we celebrate the wonderful Mary Golda Ross, born 9 August 1908. She died in 2008 at age 99, but left a lasting mark on the science of rocketry and space exploration as an early woman in engineering, and one of the first Native Americans in engineering. Join Rachelle and Bethany for this very special birthday minisode celebrating Mary and her achievements. Thanks to our Patreons who make this show possible! Read more about Mary G. Ross: Interview with Mary Ross on Lash Publications International, by Laurel Sheppard Meet Mary Golda...