3D hand-sensing wristband signals future of wearable tech

July 20, 2020

ITHACA, N.Y. - In a potential breakthrough in wearable sensing technology, researchers from Cornell University and the University of Wisconsin, Madison, have designed a wrist-mounted device that continuously tracks the entire human hand in 3D.

The bracelet, called FingerTrak, can sense and translate into 3D the many positions of the human hand, including 20 finger joint positions, using three or four miniature, low-resolution thermal cameras that read contours on the wrist. The device could be used in sign language translation, virtual reality, mobile health, human-robot interaction and other areas, the researchers said.

"This was a major discovery by our team - that by looking at your wrist contours, the technology could reconstruct in 3D, with keen accuracy, where your fingers are," said Cheng Zhang, assistant professor of information science and director of Cornell's new SciFi Lab, where FingerTrak was developed. "It's the first system to reconstruct your full hand posture based on the contours of the wrist."

Past wrist-mounted cameras have been considered too bulky and obtrusive for everyday use, and most could reconstruct only a few discrete hand gestures.

FingerTrak's breakthrough is a lightweight bracelet, allowing for free movement. Instead of using cameras to directly capture the position of the fingers, the focus of most prior research, FingerTrak uses a combination of thermal imaging and machine learning to virtually reconstruct the hand. The bracelet's four miniature, thermal cameras - each about the size of a pea - snap multiple "silhouette" images to form an outline of the hand.

A deep neural network then stitches these silhouette images together and reconstructs the virtual hand in 3D. Through this method, Zhang and his fellow researchers were able to capture the entire hand pose, even when the hand is holding an object.

While the technology has a wide range of possible uses, Zhang said the most promising is its potential application in sign language translation.

"Current sign language translation technology requires the user to either wear a glove or have a camera in the environment, both of which are cumbersome," he said. "This could really push the current technology into new areas."

FingerTrak could also have an impact on health care applications, specifically in monitoring disorders that affect fine-motor skills, said Yin Li, assistant professor of biostatistics and medical informatics at the University of Wisconsin, Madison School of Medicine and Public Health, who contributed to the software behind FingerTrak.

"How we move our hands and fingers often tells about our health condition," Li said. "A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson's and Alzheimer's."

"FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist," was published in the Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies. It also will be presented at the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing, taking place virtually Sept. 12-16.
-end-


Cornell University

Related Technology Articles from Brightsurf:

December issue SLAS Technology features 'advances in technology to address COVID-19'
The December issue of SLAS Technology is a special collection featuring the cover article, ''Advances in Technology to Address COVID-19'' by editors Edward Kai-Hua Chow, Ph.D., (National University of Singapore), Pak Kin Wong, Ph.D., (The Pennsylvania State University, PA, USA) and Xianting Ding, Ph.D., (Shanghai Jiao Tong University, Shanghai, China).

October issue SLAS Technology now available
The October issue of SLAS Technology features the cover article, 'Role of Digital Microfl-uidics in Enabling Access to Laboratory Automation and Making Biology Programmable' by Varun B.

Robot technology for everyone or only for the average person?
Robot technology is being used more and more in health rehabilitation and in working life.

Novel biomarker technology for cancer diagnostics
A new way of identifying cancer biomarkers has been developed by researchers at Lund University in Sweden.

Technology innovation for neurology
TU Graz researcher Francesco Greco has developed ultra-light tattoo electrodes that are hardly noticeable on the skin and make long-term measurements of brain activity cheaper and easier.

April's SLAS Technology is now available
April's Edition of SLAS Technology Features Cover Article, 'CURATE.AI: Optimizing Personalized Medicine with Artificial Intelligence'.

Technology in higher education: learning with it instead of from it
Technology has shifted the way that professors teach students in higher education.

Post-lithium technology
Next-generation batteries will probably see the replacement of lithium ions by more abundant and environmentally benign alkali metal or multivalent ions.

Rethinking the role of technology in the classroom
Introducing tablets and laptops to the classroom has certain educational virtues, according to Annahita Ball, an assistant professor in the University at Buffalo School of Social Work, but her research suggests that tech has its limitations as well.

The science and technology of FAST
The Five hundred-meter Aperture Spherical radio Telescope (FAST), located in a radio quiet zone, with the targets (e.g., radio pulsars and neutron stars, galactic and extragalactic 21-cm HI emission).

Read More: Technology News and Technology Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.