Nav: Home

Voice-driven games: Dialog Box supports collaborative gaming in multilingual environment

March 01, 2016

Language technologists have created the basis for a new kind of realistic communication in computer games. The Dialog Box supports collaboration and interaction between players speaking different languages. Using a space adventure game as a model, the research team led by Dietrich Klakow at Saarland University has developed the Dialog Box in cooperation with other project partners. Players can pose intuitive questions to the computer, issue commands and receive information. The Dialog Box acts as a mediating agent that enables the computer to understand spoken words and to manage the mission by linking the game being played by German-, French- and English-speaking players and sharing knowledge between them. The research project has received about three million euros in funding from the EU research initiative EUREKA, with half a million euros awarded to the research work carried out in Saarland.

Demo video of the computer game 'Sonar Silence': https://www.lsv.uni-saarland.de/index.php?id=71

In a space station that is clearly in deep trouble, Player One wakes from his cryogenic sleep. According to 'Uri', the all-knowing onboard computer, the oxygen level is sinking fast. Uri, short for 'User recognition interface', helps the player to locate and identify the source of the problem, to solve it, and thus save the station and its crew. And it all happens without a joystick or a mouse, simply by using spoken questions and commands. Uri provides detailed answers in a pleasant voice and in the same language as that spoken by the player. By executing the spoken commands it receives, Uri brings Player One into contact with the other members of the crew. The crew members then work as a team to repair the station, with each team member speaking to Uri in their own language.

'For our showcase application "Sonar Silence", which is the name our model game goes by, we've created a novel architecture for dialogue systems. The new architecture can be quickly and easily integrated into computer games, but can also be used for other multilingual conversational applications in which several people work collaboratively to solve a problem," explains Professor Dietrich Klakow. The computational linguist from Saarland University was part of the Dialog Box project in which he and his team worked together with the Austrian games company and project coordinator Mi'pu'mi Games, the Idiap Research Institute, the Swiss company Koemei, the Belgian company Acapela and the German firm Sikom Software.

The Dialog Box multilingual agent supports the players by using higher-level knowledge to coordinate the overall game. By sharing important information among the different players, the individual games can be merged seamlessly into a single game. The Dialog Box understands and speaks the languages German, English and French. 'Training the system in other languages is not a problem,' says computer linguist Anna Schmidt, a PhD student in Klakow's research group and recipient of a Google Anita Borg Memorial Scholarship.

The Saarbrücken language technologists developed the automatic language understanding module for the Dialog Box. Specifically, they taught the computer to understand the players and the situations that the players find themselves in during the game. As a result, Uri knows what an airlock is and can draw appropriate conclusions regarding it, such as whether the airlock is open or closed. To achieve this, the research team put together a vast array of data and information regarding context, environment and background. 'We used a prototype to determine the typical questions asked, which involved carrying out tests with volunteers,' explains Thomas Kleinbauer, a researcher in Dietrich Klakow's group and a research scientist within the Cluster of Excellence 'Multimodal Computing and Interaction'. Everything was then transcribed and translated so that it could be understood by the computer. The researchers were able to build on the results obtained in earlier projects. With this knowledge base available, Uri is in a position to understand questions and commands, recognize relationships and respond accordingly, for example by opening an airlock when requested.

The research group led by Dietrich Klakow is working on numerous other projects to expand and improve automatic language understanding systems. The group has already received a number of awards for its work, including a Google Research Award for Professor Klakow.
-end-
Background:

The conversational interaction with computers that is possible today has been decisively influenced by the work of computer linguists in Saarbrücken. Saarland University's Saarbrücken campus is one of the major centres of language technology. Many of the threads linking international research projects in the field of language technology converge in Saarbrücken. A new collaborative research centre is examining the role of information density in language use. One of the goals of the Saarbrücken-based Cluster of Excellence 'Multimodal Computing and Interaction' (MMCI) is to construct computer systems that enable users to interact with them in a manner similar to how humans interact with one another. The computer linguists work closely with the German Research Center for Artificial Intelligence and the two Max Planck Institutes for Informatics and for Software Systems, all of which are located on the Saarbrücken campus.

Project funding is provided by the EU EUREKA programme https://www.eurostars-eureka.eu/project//id/7152

Contact:

Prof. Dietrich Klakow: Tel.: +49 (0)681 302-58122, E-Mail: dietrich.klakow@lsv.uni-saarland.de
Dr. Thomas Kleinbauer: Tel.: +49 (0)681 302-58130; E-Mail: thomas.kleinbauer@lsv.uni-saarland.de
Anna Schmidt/ Martin Gropp: Tel.: +49 (0)681 302-4348, E-Mail: aschmidt@lsv.uni-saarland.de

Press photographs are available at http://www.uni-saarland.de/Pressefotos and can be used free of charge. Please read and comply with the conditions of use.

Press releases from 2015 on other areas of research conducted in the Klakow group:

Sprachforscher machen das Fliegen sicherer: Assistenzsystem versteht Gespräche von Lotse und Pilot [Computer linguists are making flying safer: New support system understands spoken interactions between pilots and air traffic controllers - available in German only] https://idw-online.de/de/news635391

Sprachforscher bringen dem Computer bei, zwischen den Zeilen zu lesen [Linguists teach computers to read between the lines - available in German only] https://idw-online.de/de/news626293

Note for radio journalists: Studio-quality telephone interviews can be conducted using broadcast audio IP codec technology (IP direct dial or via the ARD node 106813020001). Contact: +49 (0)681 302-2601 or -64091.

Saarland University

Related Language Articles:

How effective are language learning apps?
Researchers from Michigan State University recently conducted a study focusing on Babbel, a popular subscription-based language learning app and e-learning platform, to see if it really worked at teaching a new language.
Chinese to rise as a global language
With the continuing rise of China as a global economic and trading power, there is no barrier to prevent Chinese from becoming a global language like English, according to Flinders University academic Dr Jeffrey Gil.
'She' goes missing from presidential language
MIT researchers have found that although a significant percentage of the American public believed the winner of the November 2016 presidential election would be a woman, people rarely used the pronoun 'she' when referring to the next president before the election.
How does language emerge?
How did the almost 6000 languages of the world come into being?
New research quantifies how much speakers' first language affects learning a new language
Linguistic research suggests that accents are strongly shaped by the speaker's first language they learned growing up.
Why the language-ready brain is so complex
In a review article published in Science, Peter Hagoort, professor of Cognitive Neuroscience at Radboud University and director of the Max Planck Institute for Psycholinguistics, argues for a new model of language, involving the interaction of multiple brain networks.
Do as i say: Translating language into movement
Researchers at Carnegie Mellon University have developed a computer model that can translate text describing physical movements directly into simple computer-generated animations, a first step toward someday generating movies directly from scripts.
Learning language
When it comes to learning a language, the left side of the brain has traditionally been considered the hub of language processing.
Learning a second alphabet for a first language
A part of the brain that maps letters to sounds can acquire a second, visually distinct alphabet for the same language, according to a study of English speakers published in eNeuro.
Sign language reveals the hidden logical structure, and limitations, of spoken language
Sign languages can help reveal hidden aspects of the logical structure of spoken language, but they also highlight its limitations because speech lacks the rich iconic resources that sign language uses on top of its sophisticated grammar.
More Language News and Language Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Warped Reality
False information on the internet makes it harder and harder to know what's true, and the consequences have been devastating. This hour, TED speakers explore ideas around technology and deception. Guests include law professor Danielle Citron, journalist Andrew Marantz, and computer scientist Joy Buolamwini.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

How to Win Friends and Influence Baboons
Baboon troops. We all know they're hierarchical. There's the big brutish alpha male who rules with a hairy iron fist, and then there's everybody else. Which is what Meg Crofoot thought too, before she used GPS collars to track the movements of a troop of baboons for a whole month. What she and her team learned from this data gave them a whole new understanding of baboon troop dynamics, and, moment to moment, who really has the power.  This episode was reported and produced by Annie McEwen. Support Radiolab by becoming a member today at Radiolab.org/donate.