Nav: Home

Do as i say: Translating language into movement

September 10, 2019

PITTSBURGH--Researchers at Carnegie Mellon University have developed a computer model that can translate text describing physical movements directly into simple computer-generated animations, a first step toward someday generating movies directly from scripts.

Scientists have made tremendous leaps in getting computers to understand natural language, as well as in generating a series of physical poses to create realistic animations. These capabilities might as well exist in separate worlds, however, because the link between natural language and physical poses has been missing.

Louis-Philippe Morency, associate professor in the Language Technologies Institute (LTI), and Chaitanya Ahuja, an LTI Ph.D. student, are working to bring those worlds together using a neural architecture they call Joint Language-to-Pose, or JL2P. The JL2P model enables sentences and physical motions to be jointly embedded, so it can learn how language is related to action, gestures and movement.

"I think we're in an early stage of this research, but from a modeling, artificial intelligence and theory perspective, it's a very exciting moment," Morency said. "Right now, we're talking about animating virtual characters. Eventually, this link between language and gestures could be applied to robots; we might be able to simply tell a personal assistant robot what we want it to do.

"We also could eventually go the other way -- using this link between language and animation so a computer could describe what is happening in a video," he added.

Ahuja will present JL2P on Sept. 19 at the International Conference on 3D Vision in Quebec City, Canada.

To create JL2P, Ahuja used a curriculum-learning approach that focuses on the model first learning short, easy sequences -- "A person walks forward" -- and then longer, harder sequences - "A person steps forward, then turns around and steps forward again," or "A person jumps over an obstacle while running."

Verbs and adverbs describe the action and speed/acceleration of the action, while nouns and adjectives describe locations and directions. The ultimate goal is to animate complex sequences with multiple actions happening either simultaneously or in sequence, Ahuja said.

For now, the animations are for stick figures.

Making it more complicated is the fact that lots of things are happening at the same time, even in simple sequences, Morency explained.

"Synchrony between body parts is very important," Morency said. "Every time you move your legs, you also move your arms, your torso and possibly your head. The body animations need to coordinate these different components, while at the same time achieving complex actions. Bringing language narrative within this complex animation environment is both challenging and exciting. This is a path toward better understanding of speech and gestures."
-end-


Carnegie Mellon University

Related Language Articles:

Human language most likely evolved gradually
One of the most controversial hypotheses for the origin of human language faculty is the evolutionary conjecture that language arose instantaneously in humans through a single gene mutation.
'She' goes missing from presidential language
MIT researchers have found that although a significant percentage of the American public believed the winner of the November 2016 presidential election would be a woman, people rarely used the pronoun 'she' when referring to the next president before the election.
How does language emerge?
How did the almost 6000 languages of the world come into being?
New research quantifies how much speakers' first language affects learning a new language
Linguistic research suggests that accents are strongly shaped by the speaker's first language they learned growing up.
Why the language-ready brain is so complex
In a review article published in Science, Peter Hagoort, professor of Cognitive Neuroscience at Radboud University and director of the Max Planck Institute for Psycholinguistics, argues for a new model of language, involving the interaction of multiple brain networks.
Do as i say: Translating language into movement
Researchers at Carnegie Mellon University have developed a computer model that can translate text describing physical movements directly into simple computer-generated animations, a first step toward someday generating movies directly from scripts.
Learning language
When it comes to learning a language, the left side of the brain has traditionally been considered the hub of language processing.
Learning a second alphabet for a first language
A part of the brain that maps letters to sounds can acquire a second, visually distinct alphabet for the same language, according to a study of English speakers published in eNeuro.
Sign language reveals the hidden logical structure, and limitations, of spoken language
Sign languages can help reveal hidden aspects of the logical structure of spoken language, but they also highlight its limitations because speech lacks the rich iconic resources that sign language uses on top of its sophisticated grammar.
Lying in a foreign language is easier
It is not easy to tell when someone is lying.
More Language News and Language Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Teaching For Better Humans 2.0
More than test scores or good grades–what do kids need for the future? This hour, TED speakers explore how to help children grow into better humans, both during and after this time of crisis. Guests include educators Richard Culatta and Liz Kleinrock, psychologist Thomas Curran, and writer Jacqueline Woodson.
Now Playing: Science for the People

#556 The Power of Friendship
It's 2020 and times are tough. Maybe some of us are learning about social distancing the hard way. Maybe we just are all a little anxious. No matter what, we could probably use a friend. But what is a friend, exactly? And why do we need them so much? This week host Bethany Brookshire speaks with Lydia Denworth, author of the new book "Friendship: The Evolution, Biology, and Extraordinary Power of Life's Fundamental Bond". This episode is hosted by Bethany Brookshire, science writer from Science News.
Now Playing: Radiolab

Dispatch 3: Shared Immunity
More than a million people have caught Covid-19, and tens of thousands have died. But thousands more have survived and recovered. A week or so ago (aka, what feels like ten years in corona time) producer Molly Webster learned that many of those survivors possess a kind of superpower: antibodies trained to fight the virus. Not only that, they might be able to pass this power on to the people who are sick with corona, and still in the fight. Today we have the story of an experimental treatment that's popping up all over the country: convalescent plasma transfusion, a century-old procedure that some say may become one of our best weapons against this devastating, new disease.   If you have recovered from Covid-19 and want to donate plasma, national and local donation registries are gearing up to collect blood.  To sign up with the American Red Cross, a national organization that works in local communities, head here.  To find out more about the The National COVID-19 Convalescent Plasma Project, which we spoke about in our episode, including information on clinical trials or plasma donation projects in your community, go here.  And if you are in the greater New York City area, and want to donate convalescent plasma, head over to the New York Blood Center to sign up. Or, register with specific NYC hospitals here.   If you are sick with Covid-19, and are interested in participating in a clinical trial, or are looking for a plasma donor match, check in with your local hospital, university, or blood center for more; you can also find more information on trials at The National COVID-19 Convalescent Plasma Project. And lastly, Tatiana Prowell's tweet that tipped us off is here. This episode was reported by Molly Webster and produced by Pat Walters. Special thanks to Drs. Evan Bloch and Tim Byun, as well as the Albert Einstein College of Medicine.  Support Radiolab today at Radiolab.org/donate.