Surgeons may use hand gestures to manipulate MRI images in OR

January 10, 2013

WEST LAFAYETTE, Ind. -- Doctors may soon be using a system in the operating room that recognizes hand gestures as commands to tell a computer to browse and display medical images of the patient during a surgery.

Surgeons routinely need to review medical images and records during surgery, but stepping away from the operating table and touching a keyboard and mouse can delay the procedure and increase the risk of spreading infection-causing bacteria, said Juan Pablo Wachs, an assistant professor of industrial engineering at Purdue University.

"One of the most ubiquitous pieces of equipment in U.S. surgical units is the computer workstation, which allows access to medical images before and during surgery," he said. "However, computers and their peripherals are difficult to sterilize, and keyboards and mice have been found to be a source of contamination. Also, when nurses or assistants operate the keyboard for the surgeon, the process of conveying information accurately has proven cumbersome and inefficient since spoken dialogue can be time-consuming and leads to frustration and delays in the surgery."

Researchers are creating a system that uses depth-sensing cameras and specialized algorithms to recognize hand gestures as commands to manipulate MRI images on a large display. Recent research to develop the algorithms has been led by doctoral student Mithun George Jacob.

Findings from the research were detailed in a paper published in December in the Journal of the American Medical Informatics Association. The paper was written by Jacob, Wachs and Rebecca A. Packer, an associate professor of neurology and neurosurgery in Purdue's College of Veterinary Medicine.

The researchers validated the system, working with veterinary surgeons to collect a set of gestures natural for clinicians and surgeons. The surgeons were asked to specify functions they perform with MRI images in typical surgeries and to suggest gestures for commands. Ten gestures were chosen: rotate clockwise and counterclockwise; browse left and right; up and down; increase and decrease brightness; and zoom in and out.

Critical to the system's accuracy is the use of "contextual information" in the operating room -- cameras observe the surgeon's torso and head -- to determine and continuously monitor what the surgeon wants to do.

"A major challenge is to endow computers with the ability to understand the context in which gestures are made and to discriminate between intended gestures versus unintended gestures," Wachs said. "Surgeons will make many gestures during the course of a surgery to communicate with other doctors and nurses. The main challenge is to create algorithms capable of understanding the difference between these gestures and those specifically intended as commands to browse the image-viewing system. We can determine context by looking at the position of the torso and the orientation of the surgeon's gaze. Based on the direction of the gaze and the torso position we can assess whether the surgeon wants to access medical images."

The hand-gesture recognition system uses a camera developed by Microsoft, called Kinect, which senses three-dimensional space. The camera, found in consumer electronics games that can track a person's hands, maps the surgeon's body in 3-D. Findings showed that integrating context allows the algorithms to accurately distinguish image-browsing commands from unrelated gestures, reducing false positives from 20.8 percent to 2.3 percent.

"If you are getting false alarms 20 percent of the time, that's a big drawback," Wachs said. "So we've been able to greatly improve accuracy in distinguishing commands from other gestures."

The system also has been shown to have a mean accuracy of about 93 percent in translating gestures into specific commands, such as rotating and browsing images.

The algorithm takes into account what phase the surgery is in, which aids in determining the proper context for interpreting the gestures and reducing the browsing time.

"By observing the progress of the surgery we can tell what is the most likely image the surgeon will want to see next," Wachs said.

The researchers also are exploring context using a mock brain biopsy needle that can be tracked in the brain.

"The needle's location provides context, allowing the system to anticipate which images the surgeon will need to see next and reducing the number of gestures needed," Wachs said. "So instead of taking five minutes to browse, the surgeon gets there faster."

Sensors in the surgical needle reveal the position of its tip.
-end-
The research was supported by the Agency for Healthcare Research and Quality, grant number R03HS019837.

Writer: Emil Venere, 765-494-4709, venere@purdue.edu

Sources: Juan Pablo Wachs, 765 496-7380, jpwachs@purdue.edu

Related Web sites:

Juan Pablo Wachs: http://web.ics.purdue.edu/~jpwachs/

IMAGE CAPTION:

This table shows hand gestures surgeons might use in the operating room to browse and display medical images of the patient during an operation. Surgeons routinely need to review medical images and records during surgery, but stepping away from the operating table and touching a keyboard and mouse can delay the surgery and increase the risk of spreading infection-causing bacteria (Purdue University photo)

A publication-quality photo is available at http://www.purdue.edu/uns/images/2013/gestures-table.jpg

ABSTRACT

HAND-GESTURE-BASED STERILE INTERFACE FOR THE OPERATING ROOM USING CONTEXTUAL CUES FOR THE NAVIGATION OF RADIOLOGICAL IMAGES

Mithun George Jacob1, Juan Pablo Wachs1, Rebecca A Packer2

1School of Industrial Engineering, Purdue University

2Departments of Basic Medical Sciences and Veterinary Clinical Sciences, College of Veterinary Medicine, Purdue University

This paper presents a method to improve the navigation and manipulation of radiological images through a sterile hand gesture recognition interface based on attentional contextual cues. Computer vision algorithms were developed to extract intention and attention cues from the surgeon's behavior and combine them with sensory data from a commodity depth camera. The developed interface was tested in a usability experiment to assess the effectiveness of the new interface. An image navigation and manipulation task was performed, and the gesture recognition accuracy, false positives and task completion times were computed to evaluate system performance. Experimental results show that gesture interaction and surgeon behavior analysis can be used to accurately navigate, manipulate and access MRI images, and therefore this modality could replace the use of keyboard and mice-based interfaces.

Note to Journalists: A video related to the research at available at http://youtu.be/jfgX3KGJdsk, and the research paper is available by contacting Emil Venere, 765-494-4709, venere@purdue.edu

Purdue University

Related Gestures Articles from Brightsurf:

Guiding light: Skoltech technology puts a light-painting drone at your fingertips
Skoltech researchers have designed and developed an interface that allows a user to direct a small drone to light-paint patterns or letters through hand gestures.

​NTU Singapore scientists develop artificial intelligence system for high precision recognition of hand gestures
Scientists from Nanyang Technological University, Singapore (NTU Singapore) have developed an Artificial Intelligence (AI) system that recognises hand gestures by combining skin-like electronics with computer vision.

Children improve their narrative performance with the help of rhythmic gestures
Gesture is an integral part of language development. Recent studies carried out by the same authors in collaboration with other members of the Prosodic Studies Group (GrEP) coordinated by Pilar Prieto, ICREA research professor Department of Translation and Language Sciences at UPF, have shown that when the speaker accompanies oral communication with rhythmic gesture, preschool children are observed to better understand the message and improve their oral skills.

Gestures heard as well as seen
Gesturing with the hands while speaking is a common human behavior, but no one knows why we do it.

Oink, oink makes the pig
In a new study, neuroscientists at TU Dresden demonstrated that the use of gestures and pictures makes foreign language teaching in primary schools more effective and sustainable.

New dog, old tricks? Stray dogs can understand human cues
Pet dogs are highly receptive to commands from their owners.

Sport-related concussions
Concussions are a regular occurrence in sport but more so in contact sports such as American football, ice hockey or soccer.

Economists find mixed values of 'thoughts and prayers'
Christians who suffer from natural and human-caused disasters value thoughts and prayers from religious strangers, while atheists and agnostics believe they are worse off from such gestures.

Do as i say: Translating language into movement
Researchers at Carnegie Mellon University have developed a computer model that can translate text describing physical movements directly into simple computer-generated animations, a first step toward someday generating movies directly from scripts.

Gestures and visual animations reveal cognitive origins of linguistic meaning
Gestures and visual animations can help reveal the cognitive origins of meaning, indicating that our minds can assign a linguistic structure to new informational content 'on the fly' -- even if it is not linguistic in nature.

Read More: Gestures News and Gestures Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.