New technology uses mouth gestures to interact in virtual reality

October 05, 2017

BINGHAMTON, NY - Researchers at Binghamton University, State University of New York have developed a new technology that allows users to interact in a virtual reality environment using only mouth gestures.

The proliferation of affordable virtual reality head-mounted displays provides users with realistic immersive visual experiences. However, head-mounted displays occlude the upper half of a user's face and prevent facial action recognition from the entire face. To combat this issue, Binghamton University Professor of Computer Science Lijun Yin and his team created a new framework that interprets mouth gestures as a medium for interaction within virtual reality in real-time.

Yin's team tested the application on a group of graduate students. Once a user put on a head-mounted display, they were presented with a simplistic game; the objective of the game was to guide the player's avatar around a forest and eat as many cakes as possible. Players had to select their movement direction using head rotation, move using mouth gestures and could only eat the cake by smiling. The system was able to describe and classify the user's mouth movements, and it achieved high correct recognition rates. The system has also been demonstrated and validated through a real-time virtual reality application.

"We hope to make this applicable to more than one person, maybe two. Think Skype interviews and communication," said Yin. "Imagine if it felt like you were in the same geometric space, face to face, and the computer program can efficiently depict your facial expressions and replicate them so it looks real."

Though the tech is still in the prototype phase, Yin believes his technology is applicable to a plethora of fields.

"The virtual world isn't only for entertainment. For instance, health care uses VR to help disabled patients," said Yin. "Medical professionals or even military personal can go through training exercises that may not be possible to experience in real life. This technology allows the experience to be more realistic."

Students Umur Aybars Ciftci and Xing Zhang contributed to this research.

The paper, "Partially occluded facial action recognition and interaction in virtual reality applications," was presented at the 2017 IEEE International Conference on Multimedia and Expo.
-end-


Binghamton University

Related Health Care Articles from Brightsurf:

Study evaluates new World Health Organization Labor Care Guide for maternity care providers
The World Health Organization developed the new Labor Care Guide to support clinicians in providing good quality, women-centered care during labor and childbirth.

Six ways primary care "medical homes" are lowering health care spending
New analysis of 394 U.S. primary care practices identifies the aspects of care delivery that are associated with lower health care spending and lower utilization of emergency care and hospital admissions.

Modifiable health risks linked to more than $730 billion in US health care costs
Modifiable health risks, such as obesity, high blood pressure, and smoking, were linked to over $730 billion in health care spending in the US in 2016, according to a study published in The Lancet Public Health.

Spending on primary care vs. other US health care expenditures
National health care survey data were used to assess the amount of money spent on primary care relative to other areas of health care spending in the US from 2002 to 2016.

MU Health Care neurologist publishes guidance related to COVID-19 and stroke care
A University of Missouri Health Care neurologist has published more than 40 new recommendations for evaluating and treating stroke patients based on international research examining the link between stroke and novel coronavirus (COVID-19).

Large federal program aimed at providing better health care underfunds primary care
Despite a mandate to help patients make better-informed health care decisions, a ten-year research program established under the Affordable Care Act has funded a relatively small number of studies that examine primary care, the setting where the majority of patients in the US receive treatment.

International medical graduates care for Medicare patients with greater health care needs
A study by a Massachusetts General Hospital research team indicates that internal medicine physicians who are graduates of medical schools outside the US care for Medicare patients with more complex medical needs than those cared for by graduates of American medical schools.

The Lancet Global Health: Improved access to care not sufficient to improve health, as epidemic of poor quality care revealed
Of the 8.6 million deaths from conditions treatable by health care, poor-quality care is responsible for an estimated 5 million deaths per year -- more than deaths due to insufficient access to care (3.6 million) .

Under Affordable Care Act, Americans have had more preventive care for heart health
By reducing out-of-pocket costs for preventive treatment, the Affordable Care Act appears to have encouraged more people to have health screenings related to their cardiovascular health.

High-deductible health care plans curb both cost and usage, including preventive care
A team of researchers based at IUPUI has conducted the first systematic review of studies examining the relationship between high-deductible health care plans and the use of health care services.

Read More: Health Care News and Health Care Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.