​NTU Singapore scientists develop artificial intelligence system for high precision recognition of hand gestures

August 13, 2020

Scientists from Nanyang Technological University, Singapore (NTU Singapore) have developed an Artificial Intelligence (AI) system that recognises hand gestures by combining skin-like electronics with computer vision.

The recognition of human hand gestures by AI systems has been a valuable development over the last decade and has been adopted in high-precision surgical robots, health monitoring equipment and in gaming systems.

AI gesture recognition systems that were initially visual-only have been improved upon by integrating inputs from wearable sensors, an approach known as 'data fusion'. The wearable sensors recreate the skin's sensing ability, one of which is known as 'somatosensory'.

However, gesture recognition precision is still hampered by the low quality of data arriving from wearable sensors, typically due to their bulkiness and poor contact with the user, and the effects of visually blocked objects and poor lighting. Further challenges arise from the integration of visual and sensory data as they represent mismatched datasets that must be processed separately and then merged at the end, which is inefficient and leads to slower response times.

To tackle these challenges, the NTU team created a 'bioinspired' data fusion system that uses skin-like stretchable strain sensors made from single-walled carbon nanotubes, and an AI approach that resembles the way that the skin senses and vision are handled together in the brain.

The NTU scientists developed their bio-inspired AI system by combining three neural network approaches in one system: they used a 'convolutional neural network', which is a machine learning method for early visual processing, a multilayer neural network for early somatosensory information processing, and a 'sparse neural network' to 'fuse' the visual and somatosensory information together.

The result is a system that can recognise human gestures more accurately and efficiently than existing methods.

Lead author of the study, Professor Chen Xiaodong, from the School of Materials Science and Engineering at NTU, said, "Our data fusion architecture has its own unique bioinspired features which include a man-made system resembling the somatosensory-visual fusion hierarchy in the brain. We believe such features make our architecture unique to existing approaches."

"Compared to rigid wearable sensors that do not form an intimate enough contact with the user for accurate data collection, our innovation uses stretchable strain sensors that comfortably attaches onto the human skin. This allows for high-quality signal acquisition, which is vital to high-precision recognition tasks," added Prof Chen, who is also Director of the Innovative Centre for Flexible Devices (iFLEX) at NTU.

The team comprising scientists from NTU Singapore and the University of Technology Sydney (UTS) published their findings in the scientific journal Nature Electronics in June.

High recognition accuracy even in poor environmental conditions

To capture reliable sensory data from hand gestures, the research team fabricated a transparent, stretchable strain sensor that adheres to the skin but cannot be seen in camera images.

As a proof of concept, the team tested their bio-inspired AI system using a robot controlled through hand gestures and guided it through a maze.

Results showed that hand gesture recognition powered by the bio-inspired AI system was able to guide the robot through the maze with zero errors, compared to six recognition errors made by a visual-based recognition system.

High accuracy was also maintained when the new AI system was tested under poor conditions including noise and unfavourable lighting. The AI system worked effectively in the dark, achieving a recognition accuracy of over 96.7 per cent.

First author of the study, Dr Wang Ming from the School of Materials Science & Engineering at NTU Singapore, said, "The secret behind the high accuracy in our architecture lies in the fact that the visual and somatosensory information can interact and complement each other at an early stage before carrying out complex interpretation. As a result, the system can rationally collect coherent information with less redundant data and less perceptual ambiguity, resulting in better accuracy".

Providing an independent view, Professor Markus Antonietti, Director of Max Planck Institute of Colloids and Interfaces in Germany said, "The findings from this paper bring us another step forward to a smarter and more machine-supported world. Much like the invention of the smartphone which has revolutionised society, this work gives us hope that we could one day physically control all of our surrounding world with great reliability and precision through a gesture."

"There are simply endless applications for such technology in the marketplace to support this future. For example, from a remote robot control over smart workplaces to exoskeletons for the elderly."

The NTU research team is now looking to build a VR and AR system based on the AI system developed, for use in areas where high-precision recognition and control are desired, such as entertainment technologies and rehabilitation in the home.

Nanyang Technological University

Related Artificial Intelligence Articles from Brightsurf:

Physics can assist with key challenges in artificial intelligence
Two challenges in the field of artificial intelligence have been solved by adopting a physical concept introduced a century ago to describe the formation of a magnet during a process of iron bulk cooling.

A survey on artificial intelligence in chest imaging of COVID-19
Announcing a new article publication for BIO Integration journal. In this review article the authors consider the application of artificial intelligence imaging analysis methods for COVID-19 clinical diagnosis.

Using artificial intelligence can improve pregnant women's health
Disorders such as congenital heart birth defects or macrosomia, gestational diabetes and preterm birth can be detected earlier when artificial intelligence is used.

Artificial intelligence (AI)-aided disease prediction
Artificial Intelligence (AI)-aided Disease Prediction https://doi.org/10.15212/bioi-2020-0017 Announcing a new article publication for BIO Integration journal.

Artificial intelligence dives into thousands of WW2 photographs
In a new international cross disciplinary study, researchers from Aarhus University, Denmark and Tampere University, Finland have used artificial intelligence to analyse large amounts of historical photos from WW2.

Applying artificial intelligence to science education
A new review published in the Journal of Research in Science Teaching highlights the potential of machine learning--a subset of artificial intelligence--in science education.

New roles for clinicians in the age of artificial intelligence
New Roles for Clinicians in the Age of Artificial Intelligence https://doi.org/10.15212/bioi-2020-0014 Announcing a new article publication for BIO Integration journal.

Artificial intelligence aids gene activation discovery
Scientists have long known that human genes are activated through instructions delivered by the precise order of our DNA.

Artificial intelligence recognizes deteriorating photoreceptors
A software based on artificial intelligence (AI), which was developed by researchers at the Eye Clinic of the University Hospital Bonn, Stanford University and University of Utah, enables the precise assessment of the progression of geographic atrophy (GA), a disease of the light sensitive retina caused by age-related macular degeneration (AMD).

Classifying galaxies with artificial intelligence
Astronomers have applied artificial intelligence (AI) to ultra-wide field-of-view images of the distant Universe captured by the Subaru Telescope, and have achieved a very high accuracy for finding and classifying spiral galaxies in those images.

Read More: Artificial Intelligence News and Artificial Intelligence Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.