Nav: Home

Operating smart devices from the space on and above the back of your hand

May 03, 2017

It relies on a depth sensor that tracks movements of the thumb and index finger on and above the back of the hand. In this way, not only can smartwatches be controlled, but also smartphones, smart TVs and devices for augmented and virtual reality.

They're called the "Apple Watch Series 2", "LG Watch", "Samsung GEAR S3" or "Moto 360 2nd Gen" but they all have the same problem. "Every new product generation has better screens, better processors, better cameras, and new sensors, but regarding input, the limitations remain," explains Srinath Sridhar, a researcher in the Graphics, Vision and Video group at the Max Planck Institute for Informatics.

Together with Christian Theobalt, head of the Graphics, Vision and Video group at MPI, Anders Markussen and Sebastian Boring at the University of Copenhagen and Antti Oulasvirta at Aalto University in Finland, Srinath Sridhar has therefore developed an input method that requires only a small camera to track fingertips in mid-air, and touch and position of the fingers on the back of the hand. This combination enables more expressive interactions than any previous sensing technique.

Regarding hardware, the prototype, which the researchers have named "WatchSense", requires only a depth sensor, a much smaller version of the well-known "Kinect" game controller from the Xbox 360 video game console. With WatchSense, the depth sensor is worn on the user's forearm, about 20cm from the watch. As a sort of 3D camera, it captures the movements of the thumb and index finger, not only on the back of the hand but also in the space over and above it. The software developed by the researchers recognizes the position and movement of the fingers within the 3D image, allowing the user to control apps on smartphones or other devices. "The currently available depth sensors do not fit inside a smartwatch, but from the trend it's clear that in the near future, smaller depth sensors will be integrated into smartwatches," Sridhar says.

But this is not all that's required. According to Sridhar, with their software system the scientists also had to solve the challenges of handling the unevenness of the back of the hand and the fact that the fingers can occlude each other when they are moved. "The most important thing is that we can not only recognize the fingers, but also distinguish between them," explains Sridhar, "which nobody else had managed to do before in a wearable form factor. We can now do this even in real time." The software recognizes the exact positions of the thumb and index finger in the 3D image from the depth sensor, because the researchers trained it to do this via machine learning. In addition, the researchers have successfully tested their prototype in combination with several mobile devices and in various scenarios. "Smartphones can be operated with one or more fingers on the display, but they do not use the space above it. If both are combined, this enables previously impossible forms of interaction," explains Sridhar. He and his colleagues were able to show that with WatchSense, in a music program, the volume could be adjusted and a new song selected more quickly than was possible with a smartphone's Android app. The researchers also tested WatchSense for tasks in virtual and augmented reality, in a map application, and used it to control a large external screen. Preliminary studies showed that WatchSense was more satisfactory for each case than conventional touch-sensitive displays. Sridhar is confident that "we need something like WatchSense whenever we want to be productive while moving. WatchSense is the first to enable expressive input for devices while on the move."

From May 6, the researchers will present WatchSense at the renowned "Conference on Human Factors in Computing," or CHI for short, which this time takes place in the city of Denver in the US.
Further Information:

Video and Paper

Send questions to:

Dr. Srinath Sridhar
Max Planck Institute for Informatics
Saarland Informatics Campus
Tel.: +49 681 9325 4057

Prof. Dr. Christian Theobalt
Max Planck Institute for Informatics
Saarland Informatics Campus
Tel.: +49 681 9325 4028

Gordon Bolduan
Competence Center Computer Science Saarland
Tel: +49 681 302 70741

Saarland University

Related Smartphones Articles:

Next-gen smartphones to keep their cool
Multilayered carbon material could be the perfect fit for heat management in electronic devices.
With digital phenotyping, smartphones may play a role in assessing severe mental illness
Digital phenotyping approaches that collect and analyze Smartphone-user data on locations, activities, and even feelings - combined with machine learning to recognize patterns and make predictions from the data - have emerged as promising tools for monitoring patients with psychosis spectrum illnesses, according to a report in the September/October issue of Harvard Review of Psychiatry.
Smartphones can predict brain function associated with anxiety and depression
Phone data such as social activity, screen time and location can predict connectivity between regions of the brain that are responsible for emotion.
Smartphones can tell when you're drunk by analyzing your walk
Your smartphone can tell when you've had too much to drink by detecting changes in the way you walk, according to a new study published in the Journal of Studies on Alcohol and Drugs.
Smartphones are lowering student's grades, study finds
The ease of finding information on the internet is hurting students' long-term retention and resulting in lower grades on exams, according to a Rutgers University-New Brunswick study.
Smartphones may help detect diabetes
Researchers at UC San Francisco have developed a ''digital biomarker'' that would use a smartphone's built-in camera to detect Type 2 diabetes - one of the world's top causes of disease and death - potentially providing a low-cost, in-home alternative to blood draws and clinic-based screening tools.
Smartphones prove to be time-saving analytical tools
Scientists use a smartphone camera to easily measure soil density -- a key metric for analyzing our soils
Why smartphones are digital truth serum
People are more willing to reveal personal information about themselves online using their smartphones compared to desktop computers.
Which comes first: Smartphone dependency or depression?
New research suggests a person's reliance on his or her smartphone predicts greater loneliness and depressive symptoms, as opposed to the other way around.
New solution to elderly falls: drones, smartphones and sensors
Drones, smartphones and sensors could provide a lifeline to the world's growing elderly population at risk of falls, helping to cut global hospital costs.
More Smartphones News and Smartphones Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Power Of Spaces
How do spaces shape the human experience? In what ways do our rooms, homes, and buildings give us meaning and purpose? This hour, TED speakers explore the power of the spaces we make and inhabit. Guests include architect Michael Murphy, musician David Byrne, artist Es Devlin, and architect Siamak Hariri.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at     You can read The Transition Integrity Project's report here.