Smartphone understands hand gestures

October 08, 2014

It does seem slightly odd at first: you hold the phone in one hand, and move the other in the air above its built-in camera making gestures that resemble sign language. Sometimes you move your index finger to the left, sometimes to the right. You can spread out your fingers, or imitate a pair of pliers or the firing of a pistol. These gestures are not, however, intended for communicating with deaf people; they are for controlling your smartphone.

By mimicking the firing of a pistol, for example, a user can switch to another browser tab, change the map's view from satellite to standard, or shoot down enemy planes in a game. Spreading out your fingers magnifies a section of a map or scrolls the page of a book forwards.

All this gesturing wizardry is made possible by a new type of algorithm developed by Jie Song, a Master's student in the working group headed by by Otmar Hilliges, Professor of Computer Science. The researchers presented the app to an audience of industry professionals at the UIST symposium in Honolulu, Hawaii.

Intelligent programming uses computer memory efficiently

The program uses the smartphone's built-in camera to register its environment. It does not evaluate depth or colour. The information it does register - the shape of the gesture, the parts of the hand - is reduced to a simple outline that is classified according to stored gestures. The program then executes the command associated with the gesture it observes. The program also recognises the hand's distance from the camera and warns the user when the hand is either too close or too far away.

"Many movement-recognition programs need plenty of processor and memory power", explains Hilliges, adding that their new algorithm uses a far smaller portion of computer memory and is thus ideal for smartphones. He believes the application is the first of its kind that can run on a smartphone. The app's minimal processing footprint means it could also run on smart watches or in augmented-reality glasses.

More control

The program currently recognises six different gestures and executes their corresponding commands. Although the researchers have tested 16 outlines, this is not the app's theoretical limit. What matters is that gestures generate unambiguous outlines. Gestures that resemble others are not suitable for this application. "To expand its functionality, we're going to add further classification schemes to the program", says the ETH researcher.

He is convinced that this new way of operating smartphones greatly increases the range of interactivity. The researcher's objective is to keep the gestures as simple as possible, so that users can operate their smartphone effortlessly.

But will smartphone users want to adapt to this new style of interaction? Otmar Hilliges is confident they will. Gesture control will not replace touchscreen control, but supplement it. "People got used to operating computer games with their movements." Touchscreens, Hilliges reminds us, also required a very long adjustment period before making a big impact in consumers' lives. He is therefore certain that this application - or at least parts of it - will find its way onto the market.
-end-
Reference

Song J, Sörös G, Pece F, Fanello S, Izadi S, Keskin C, Hilliges O: In-air Gestures Around Unmodified Mobile Devices, ACM User Interface Software and Technology Symposium, Honolulu, Hawaii, 7 October 2014

ETH Zurich

Related Smartphone Articles from Brightsurf:

Mobile smartphone technology is associated with better clinical outcomes for OHCA
Mobile smartphone technology can accelerate first responder dispatch and may be instrumental to improving out?of?hospital cardiac arrest (OCHA) survival.

New tool can diagnose strokes with a smartphone
A new tool created by researchers at Penn State and Houston Methodist Hospital could diagnose a stroke based on abnormalities in a patient's speech ability and facial muscular movements, and with the accuracy of an emergency room physician -- all within minutes from an interaction with a smartphone.

App analyzes coronavirus genome on a smartphone
A team led by Garvan's Dr Ira Deveson developed the app 'Genopo' that can analyse the coronavirus genome on a portable Android device.

Smartphone accelerometers could help in resistance workouts and rehabilitation protocols
Smartphone accelerometers are effective tools to measure key time-under-tension indicators of muscle training -- and could help in resistance-based workouts and rehabilitation protocols.

Parents' smartphone use does not harm parent/child relationships
Contrary to popular views, parental smartphone use is rarely associated with poor parenting, and more often than not, tends to be associated with warm and attached parenting.

The effects of smartphone use on parenting
Parents may worry that spending time on their smartphones has a negative impact on their relationships with their children.

Inexpensive retinal diagnostics via smartphone
Retinal damage due to diabetes is now considered the most common cause of blindness in working-age adults.

Nanosensor can alert a smartphone when plants are stressed
MIT engineers can closely track how plants respond to stresses such as injury, infection, and light damage using sensors made of carbon nanotubes.

Smartphone apps not accurate enough to spot all skin cancers
Smartphone apps that assess the risk of suspicious moles cannot be relied upon to detect all cases of skin cancer, finds a review of the evidence published by The BMJ today.

Detecting mental and physical stress via smartphone
The team led by Professor Enrico Caiani of the Department of Electronics, Information and Bioengineering at Politecnico di Milano, Italy, has shown that it is possible to use our smartphones without any other peripherals or wearables to accurately extract vital parameters, such as heart beat rate and stress level.

Read More: Smartphone News and Smartphone Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.