During the COVID-19 pandemic, face masks have become common, posing challenges to facial recognition systems used for security and identification. Current techniques to digitally recreate the covered parts of faces often miss important details and identity traits. To overcome this, Mr. Akatsuka and Prof. Sei and their colleagues have introduced a unique method that uses voice data to assist in revealing the covered parts of faces.
Their significant discovery is the inherent connection between speech-related vocal tract structures and certain facial features. Therefore, vocal characteristics can offer valuable insights for rebuilding facial geometry in a combined voice and image approach.
Their research, using a vast collection of matched voice and face data, demonstrates significant improvements in recreating critical facial areas like the mouth, nose, and overall face shape. Both human evaluations and technical accuracy measurements highlight the effectiveness of including voice data in restoring hidden facial features. Additionally, their approach shows notable advancements over traditional methods that do not use voice data.
This inventive use of voice in facial image reconstruction signals exciting new directions in this developing area of research. This research was awarded the IEEE Computer Society Japan Chapter JAWS Young Researcher Award.
Conceptual Diagram of the Proposed Method: The 3D Face Model (3DMM) is created using a masked face image and voice data. The model then assists in digitally painting in the unmasked face.
Authors:
Tetsumaru Akatsuka
-- The University of Electro-Communications, Master student
Ryohei Orihara
-- The University of Electro-Communications, Visiting Professor
Yuichi Sei
-- The University of Electro-Communications, Professor
Yasuyuki Tahara
-- The University of Electro-Communications, Associate Professor
Akihiko Ohsuga
Computational simulation/modeling
People
Estimation of Unmasked Face Images Based on Voice and 3DMM
27-Nov-2023
The authors declare no competing interests.