Nav: Home

Brown undergraduate researcher teaches robots handwriting and drawing

May 15, 2019

PROVIDENCE, RI [Brown University] -- An algorithm developed by Brown University computer scientists enables robots to put pen to paper, writing words using stroke patterns similar to human handwriting. It's a step, the researchers say, toward robots that are able to communicate more fluently with human co-workers and collaborators.

"Just by looking at a target image of a word or sketch, the robot can reproduce each stroke as one continuous action," said Atsunobu Kotani, an undergraduate student at Brown who led the algorithm's development. "That makes it hard for people to distinguish if it was written by the robot or actually written by a human."

The algorithm makes use of deep learning networks that analyze images of handwritten words or sketches and can deduce the likely series of pen strokes that created them. The robot can then reproduce the words or sketches using the pen strokes it learned. In a paper to be presented at this month's International Conference on Robotics and Automation, the researchers demonstrate a robot that was able to write "hello" in 10 languages that employ different character sets. The robot was also able to reproduce rough sketches, including one of the Mona Lisa.

Stefanie Tellex, an assistant professor of computer science at Brown and Kotani's advisor, says that what makes this work unique is the ability of the robot to learn stroke order from scratch.

"A lot of the existing work in this area requires the robot to have information about the stroke order in advance," Tellex said. "If you wanted the robot to write something, somebody would have to program the stroke orders each time. With what Atsu has done, you can draw whatever you want and the robot can reproduce it. It doesn't always do the perfect stroke order, but it gets pretty close."

Another remarkable aspect of the work, Tellex says, is how the algorithm was able to generalize its ability to reproduce strokes. Kotani trained his deep learning algorithm using a set of Japanese characters, and showed that it could reproduce the characters and the strokes that created them with around 93 percent accuracy. But much to the researchers' surprise, the algorithm wound up being able to reproduce very different character types it had never seen before -- English print and cursive, for example.

"We would have been happy if it had only learned the Japanese characters," Tellex said. "But once it started working on English, we were amazed. Then we decided to see how far we could take it."

Tellex and Kotani asked everyone who works in Tellex's Humans to Robots lab to write "hello" in their native languages, which included Greek, Hindi, Urdu, Chinese and Yiddish among others. The robot was able to reproduce them all with reasonable stroke accuracy.

"I feel like there's something really beautiful about the robot writing in so many different languages," Tellex said. "I thought that was really cool."

But the system's masterwork may be its copy of Kotani's Mona Lisa sketch. He drew his sketch on a dry erase board in Tellex's lab, and then allowed the robot to copy it -- fairly faithfully -- on the same board just below Kotani's original.

"It was early morning that our robot finally drew the Mona Lisa on the whiteboard," Kotani said. "When I came back to the lab, everybody was standing around the whiteboard looking at the Mona Lisa and asking me if [the robot] drew this. They couldn't believe it."

It was a big moment for Kotani because "it was the moment that our robot defined what's beyond mere printing." An ink jet printer can recreate an image, but it does so with a print head that goes back in forth building the image line by line. But this was the robot creating an image with human-like strokes, which to Kotani is "something much more humane and expressive."

Key to making the system work, Kotani says, is that the algorithm uses two distinct models of the image it's trying to reproduce. Using a global model that considers the image as a whole, the algorithm identifies a likely starting point for making the first stroke. Once that stroke has begun, the algorithm zooms in, looking at the image pixel by pixel to determine where that stroke should go and how long it should be. When it reaches the end of the stroke, the algorithm again calls the global model to determine where the next stroke should start, then it's back to the zoomed-in model. This process is repeated until the image is complete.

Both Kotani and Tellex say the work is a step toward better communication between people and robots. Ultimately, they envision robots that can leave Post-it Notes, take dictation or sketch diagrams for their human coworkers and collaborators.

"I want a robot to be able to do everything a person can do," Tellex said. "I'm particularly interested in a robot that can use language. Writing is a way that people use language, so we thought we should try this."
-end-
Video: https://youtu.be/xX_LX6qeiYY

Brown University

Related Stroke Articles:

Retraining the brain to see after stroke
A new study out today in Neurology, provides the first evidence that rigorous visual training restores rudimentary sight in patients who went partially blind after suffering a stroke, while patients who did not train continued to get progressively worse.
Catheter ablations reduce risks of stroke in heart patients with stroke history, study finds
Atrial fibrillation patients with a prior history of stroke who undergo catheter ablation to treat the abnormal heart rhythm lower their long-term risk of a recurrent stroke by 50 percent, according to new research from the Intermountain Medical Center Heart Institute.
Imaging stroke risk in 4-D
A new MRI technique developed at Northwestern University detects blood flow velocity to identify who is most at risk for stroke, so they can be treated accordingly.
Biomarkers may help better predict who will have a stroke
People with high levels of four biomarkers in the blood may be more likely to develop a stroke than people with low levels of the biomarkers, according to a study published in the Aug.
Pre-stroke risk factors influence long-term future stroke, dementia risk
If you had heart disease risk factors, such as high blood pressure, before your first stoke, your risk of suffering subsequent strokes and dementia long after your initial stroke may be higher.
Intervention methods of stroke need to focus on prevention for blacks to reduce stroke mortality
Blacks are four times more likely than their white counterparts to die from stroke at age 45.
Study shows area undamaged by stroke remains so, regardless of time stroke is left untreated
A study led by Achala Vagal, M.D., associate professor at the University of Cincinnati College of Medicine and a UC Health radiologist, looked at a group of untreated acute stroke patients and found that there was no evidence of time dependence on damage outcomes for the penumbra, or tissue that is at risk of progressing to dead tissue but is still salvageable if blood flow is returned in a stroke, but rather an association with collateral flow -- or rerouting of blood through clear vessels.
Immediate aspirin after mini-stroke substantially reduces risk of major stroke
Using aspirin urgently could substantially reduce the risk of major strokes in patients who have minor 'warning' events.
SAGE launches the European Stroke Journal with the European Stroke Organisation
SAGE, a world leading independent and academic publisher, is delighted to announce the launch of the European Stroke Journal, the flagship journal of the European Stroke Organisation.
The S-stroke or I-stroke?
The year 2016 is an Olympic year. Developments in high-performance swimwear for swimming continue to advance, along with other areas of scientific research.

Related Stroke Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Jumpstarting Creativity
Our greatest breakthroughs and triumphs have one thing in common: creativity. But how do you ignite it? And how do you rekindle it? This hour, TED speakers explore ideas on jumpstarting creativity. Guests include economist Tim Harford, producer Helen Marriage, artificial intelligence researcher Steve Engels, and behavioral scientist Marily Oppezzo.
Now Playing: Science for the People

#524 The Human Network
What does a network of humans look like and how does it work? How does information spread? How do decisions and opinions spread? What gets distorted as it moves through the network and why? This week we dig into the ins and outs of human networks with Matthew Jackson, Professor of Economics at Stanford University and author of the book "The Human Network: How Your Social Position Determines Your Power, Beliefs, and Behaviours".