Augmented reality (AR) and mixed reality (MR) headsets let us see the world around us with virtual elements superimposed on top. For example, many modern AR/MR headsets use hand-tracking cameras to detect hand gestures in the air, which allows users to type on a virtual keyboard that appears to be floating in front of the user. As exciting as this sounds initially, the approach often needs handheld controllers, and keeping your hands in the air for a long time can cause arm fatigue. These air gestures also lack physical feedback, which can make interaction less comfortable.
Searching for an innovative way to make typing more comfortable while using AR/MR technology, researchers at Tohoku University found a method that transforms regular surfaces into keyboards - no extra equipment required.
The key to this innovation is a natural reaction our body has when we apply pressure to our fingertips. If you try pressing down on a hard surface like a desk, you will notice a flush of white. This color change is called the blanching phenomenon. By analyzing images from the headset camera with an AI model, the system can detect when a fingertip touches a surface.
"This research means that ordinary surfaces all around us - walls, desks, or partitions - can be used as a touch input area," said Guanghan Zhao, who led the study. "Moreover, this method doesn't require special sensors, markers, or additional devices. Anyone can use it easily.
To make this vision a reality, the research team trained an AI model to reliably recognize the blanching phenomenon in fingertip images captured by a standard camera on the headset. In addition, a supporting interaction system was developed to provide easy-to-access user experience.
User studies showed that the system reliably detected fingertip contact across several common surface materials. Participants were able to perform interaction tasks with stable input accuracy. The surface-supported interaction also allowed users to rest their fingers on the surface during operation.
"Our primary objective was to develop a technology that allows touch input on everyday physical surfaces for AR and MR without the need for special hardware, making these devices easier to use and promoting the wider adoption of AR and MR technologies for everyday use," said Guanghan.
The findings were presented at the 33rd IEEE Conference on Virtual Reality and 3D User Interfaces, held in South Korea from March 21 - 25, 2026. The paper has also been accepted for future publication in IEEE Computer Society Digital Library.