Nav: Home

Computer program looks five minutes into the future

June 13, 2018

Computer scientists from the University of Bonn have developed software that can look a few minutes into the future: The program first learns the typical sequence of actions, such as cooking, from video sequences. Based on this knowledge, it can then accurately predict in new situations what the chef will do at which point in time. Researchers will present their findings at the world's largest Conference on Computer Vision and Pattern Recognition, which will be held June 19-21 in Salt Lake City, USA.

The perfect butler, as every fan of British social drama knows, has a special ability: He senses his employer's wishes before they have even been uttered. The working group of Prof. Dr. Jürgen Gall wants to teach computers something similar: "We want to predict the timing and duration of activities - minutes or even hours before they happen", he explains.

A kitchen robot, for example, could then pass the ingredients as soon as they are needed, pre-heat the oven in time - and in the meantime warn the chef if he is about to forget a preparation step. The automatic vacuum cleaner meanwhile knows that it has no business in the kitchen at that time, and instead takes care of the living room.

We humans are very good at anticipating the actions of others. For computers however, this discipline is still in its infancy. The researchers at the Institute of Computer Science at the University of Bonn are now able to announce a first success: They have developed self-learning software that can estimate the timing and duration of future activities with astonishing accuracy for periods of several minutes.

Training data: four hours of salad videos

The training data used by the scientists included 40 videos in which performers prepare different salads. Each of the recordings was around 6 minutes long and contained an average of 20 different actions. The videos also contained precise details of what time the action started and how long it took.

The computer "watched" these salad videos totaling around four hours. This way, the algorithm learned which actions typically follow each other during this task and how long they last. This is by no means trivial: After all, every chef has his own approach. Additionally, the sequence may vary depending on the recipe.

"Then we tested how successful the learning process was", explains Gall. "For this we confronted the software with videos that it had not seen before." At least the new short films fit into the context: They also showed the preparation of a salad. For the test, the computer was told what is shown in the first 20 or 30 percent of one of the new videos. On this basis it then had to predict what would happen during the rest of the film.

That worked amazingly well. Gall: "Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future." For activities that were more than three minutes in the future, the computer was still right in 15 percent of cases. However, the prognosis was only considered correct if both the activity and its timing were correctly predicted.

Gall and his colleagues want the study to be understood only as a first step into the new field of activity prediction. Especially since the algorithm performs noticeably worse if it has to recognize on its own what happens in the first part of the video, instead of being told. Because this analysis is never 100 percent correct - Gall speaks of "noisy" data. "Our process does work with it", he says. "But unfortunately nowhere near as well."
-end-
The study was developed as part of a research group dedicated to the prediction of human behavior and financially supported by the German Research Foundation (DFG).

Publication: Yazan Abu Farha, Alexander Richard and Jürgen Gall: When will you do what? - Anticipating Temporal Occurrences of Activities. IEEE Conference on Computer Vision and Pattern Recognition 2018; http://pages.iai.uni-bonn.de/gall_juergen/download/jgall_anticipation_cvpr18.pdf

Sample test videos and predictions derived from them are available at https://www.youtube.com/watch?v=xMNYRcVH_oI

Contact:

Prof. Dr. Jürgen Gall
Institute of Computer Science
University of Bonn
Tel. +49(0)228/7369600
E-mail: gall@informatik.uni-bonn.de

University of Bonn

Related Algorithm Articles:

Scientists use algorithm to peer through opaque brains
A new algorithm helps scientists record the activity of individual neurons within a volume of brain tissue.
Algorithm generates origami folding patterns for any shape
A new algorithm generates practical paper-folding patterns to produce any 3-D structure.
New algorithm tracks neurons in bendy brain of freely crawling worm
Scientists at Princeton University have developed a new algorithm to track neurons in the brain of the worm Caenorhabditis elegans while it crawls.
Does my algorithm work? There's no shortcut for community detection
Community detection is an important tool for scientists studying networks, but a new paper published in Science Advances calls into question the common practice of using metadata for ground truth validation.
'Cyclops' algorithm spots daily rhythms in cells
Humans, like virtually all other complex organisms on Earth, have adapted to their planet's 24-hour cycle of sunlight and darkness.
An algorithm that knows when you'll get bored with your favorite mobile game
Researchers from the Tokyo-based company Silicon Studio, led by Spanish data scientist África Periáñez, have developed a new algorithm that predicts when a user will leave a mobile game.
Algorithm identified Trump as 'not-married'
Scientists from Russia and Singapore created an algorithm that predicts user marital status with 86% precision using data from three social networks instead of one.
A novel positioning algorithm based on self-adaptive algorithm
Much attention has been paid to the Taylor series expansion (TSE) method these years, which has been extensively used for solving nonlinear equations for its good robustness and accuracy of positioning.
Algorithm can create a bridge between Clinton and Trump supporters
The article that received the best student-paper award in the Tenth International Conference on Web Search and Data Mining (WSDM 2017) builds algorithmic techniques to mitigate the rising polarization by connecting people with opposing views -- and evaluates them on Twitter.
Deep learning algorithm does as well as dermatologists in identifying skin cancer
In hopes of creating better access to medical care, Stanford researchers have trained an algorithm to diagnose skin cancer.

Related Algorithm Reading:

Introduction to Algorithms, 3rd Edition (The MIT Press)
by Thomas H. Cormen (Author), Charles E. Leiserson (Author), Ronald L. Rivest (Author), Clifford Stein (Author)

Algorithms (4th Edition)
by Robert Sedgewick (Author), Kevin Wayne (Author)

Grokking Algorithms: An illustrated guide for programmers and other curious people
by Aditya Bhargava (Author)

The Algorithm Design Manual
by Steven S Skiena (Author)

Algorithms
by Sanjoy Dasgupta Algorithms (Author), Christos H. Papadimitriou Algorithms (Author), Umesh Vazirani Algorithms (Author)

Data Structures and Algorithms in Java
by Michael T. Goodrich (Author), Roberto Tamassia (Author), Michael H. Goldwasser (Author)

Algorithm Design
by Jon Kleinberg (Author), Éva Tardos (Author)

Bioinformatics Algorithms: An Active Learning Approach
by Active Learning Publishers (Publisher)

Understanding Machine Learning: From Theory to Algorithms
by Shai Shalev-Shwartz (Author), Shai Ben-David (Author)

Algorithms of Oppression: How Search Engines Reinforce Racism
by Safiya Umoja Noble (Author)

Best Science Podcasts 2018

We have hand picked the best science podcasts for 2018. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Dying Well
Is there a way to talk about death candidly, without fear ... and even with humor? How can we best prepare for it with those we love? This hour, TED speakers explore the beauty of life ... and death. Guests include lawyer Jason Rosenthal, humorist Emily Levine, banker and travel blogger Michelle Knox, mortician Caitlin Doughty, and entrepreneur Lux Narayan.
Now Playing: Science for the People

#492 Flint Water Crisis
This week we dig into the Flint water crisis: what happened, how it got so bad, what turned the tide, what's still left to do, and the mix of science, politics, and activism that are still needed to finish pulling Flint out of the crisis. We spend the hour with Dr Mona Hanna-Attisha, a physician, scientist, activist, the founder and director of the Pediatric Public Health Initiative, and author of the book "What the Eyes Don't See: A Story of Crisis, Resistance, and Hope in an American City".