Nav: Home

Brown undergraduate researcher teaches robots handwriting and drawing

May 15, 2019

PROVIDENCE, RI [Brown University] -- An algorithm developed by Brown University computer scientists enables robots to put pen to paper, writing words using stroke patterns similar to human handwriting. It's a step, the researchers say, toward robots that are able to communicate more fluently with human co-workers and collaborators.

"Just by looking at a target image of a word or sketch, the robot can reproduce each stroke as one continuous action," said Atsunobu Kotani, an undergraduate student at Brown who led the algorithm's development. "That makes it hard for people to distinguish if it was written by the robot or actually written by a human."

The algorithm makes use of deep learning networks that analyze images of handwritten words or sketches and can deduce the likely series of pen strokes that created them. The robot can then reproduce the words or sketches using the pen strokes it learned. In a paper to be presented at this month's International Conference on Robotics and Automation, the researchers demonstrate a robot that was able to write "hello" in 10 languages that employ different character sets. The robot was also able to reproduce rough sketches, including one of the Mona Lisa.

Stefanie Tellex, an assistant professor of computer science at Brown and Kotani's advisor, says that what makes this work unique is the ability of the robot to learn stroke order from scratch.

"A lot of the existing work in this area requires the robot to have information about the stroke order in advance," Tellex said. "If you wanted the robot to write something, somebody would have to program the stroke orders each time. With what Atsu has done, you can draw whatever you want and the robot can reproduce it. It doesn't always do the perfect stroke order, but it gets pretty close."

Another remarkable aspect of the work, Tellex says, is how the algorithm was able to generalize its ability to reproduce strokes. Kotani trained his deep learning algorithm using a set of Japanese characters, and showed that it could reproduce the characters and the strokes that created them with around 93 percent accuracy. But much to the researchers' surprise, the algorithm wound up being able to reproduce very different character types it had never seen before -- English print and cursive, for example.

"We would have been happy if it had only learned the Japanese characters," Tellex said. "But once it started working on English, we were amazed. Then we decided to see how far we could take it."

Tellex and Kotani asked everyone who works in Tellex's Humans to Robots lab to write "hello" in their native languages, which included Greek, Hindi, Urdu, Chinese and Yiddish among others. The robot was able to reproduce them all with reasonable stroke accuracy.

"I feel like there's something really beautiful about the robot writing in so many different languages," Tellex said. "I thought that was really cool."

But the system's masterwork may be its copy of Kotani's Mona Lisa sketch. He drew his sketch on a dry erase board in Tellex's lab, and then allowed the robot to copy it -- fairly faithfully -- on the same board just below Kotani's original.

"It was early morning that our robot finally drew the Mona Lisa on the whiteboard," Kotani said. "When I came back to the lab, everybody was standing around the whiteboard looking at the Mona Lisa and asking me if [the robot] drew this. They couldn't believe it."

It was a big moment for Kotani because "it was the moment that our robot defined what's beyond mere printing." An ink jet printer can recreate an image, but it does so with a print head that goes back in forth building the image line by line. But this was the robot creating an image with human-like strokes, which to Kotani is "something much more humane and expressive."

Key to making the system work, Kotani says, is that the algorithm uses two distinct models of the image it's trying to reproduce. Using a global model that considers the image as a whole, the algorithm identifies a likely starting point for making the first stroke. Once that stroke has begun, the algorithm zooms in, looking at the image pixel by pixel to determine where that stroke should go and how long it should be. When it reaches the end of the stroke, the algorithm again calls the global model to determine where the next stroke should start, then it's back to the zoomed-in model. This process is repeated until the image is complete.

Both Kotani and Tellex say the work is a step toward better communication between people and robots. Ultimately, they envision robots that can leave Post-it Notes, take dictation or sketch diagrams for their human coworkers and collaborators.

"I want a robot to be able to do everything a person can do," Tellex said. "I'm particularly interested in a robot that can use language. Writing is a way that people use language, so we thought we should try this."
-end-
Video: https://youtu.be/xX_LX6qeiYY

Brown University

Related Stroke Articles:

Retraining the brain to see after stroke
A new study out today in Neurology, provides the first evidence that rigorous visual training restores rudimentary sight in patients who went partially blind after suffering a stroke, while patients who did not train continued to get progressively worse.
Catheter ablations reduce risks of stroke in heart patients with stroke history, study finds
Atrial fibrillation patients with a prior history of stroke who undergo catheter ablation to treat the abnormal heart rhythm lower their long-term risk of a recurrent stroke by 50 percent, according to new research from the Intermountain Medical Center Heart Institute.
Imaging stroke risk in 4-D
A new MRI technique developed at Northwestern University detects blood flow velocity to identify who is most at risk for stroke, so they can be treated accordingly.
Biomarkers may help better predict who will have a stroke
People with high levels of four biomarkers in the blood may be more likely to develop a stroke than people with low levels of the biomarkers, according to a study published in the Aug.
Pre-stroke risk factors influence long-term future stroke, dementia risk
If you had heart disease risk factors, such as high blood pressure, before your first stoke, your risk of suffering subsequent strokes and dementia long after your initial stroke may be higher.
More Stroke News and Stroke Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Teaching For Better Humans
More than test scores or good grades — what do kids need to prepare them for the future? This hour, guest host Manoush Zomorodi and TED speakers explore how to help children grow into better humans, in and out of the classroom. Guests include educators Olympia Della Flora and Liz Kleinrock, psychologist Thomas Curran, and writer Jacqueline Woodson.
Now Playing: Science for the People

#535 Superior
Apologies for the delay getting this week's episode out! A technical glitch slowed us down, but all is once again well. This week, we look at the often troubling intertwining of science and race: its long history, its ability to persist even during periods of disrepute, and the current forms it takes as it resurfaces, leveraging the internet and nationalism to buoy itself. We speak with Angela Saini, independent journalist and author of the new book "Superior: The Return of Race Science", about where race science went and how it's coming back.