Reactive Video playback that you control with your body

December 10, 2020

Computer scientists have developed an entirely new way of interacting with video content that adapts to, and is controlled by, your body movement.

Fitness videos and other instructional content that aims to teach viewers new martial arts skills, exercises or yoga positions have been popular since VHS in the 80s and are abundant on Internet platforms like YouTube.

However, these traditional forms of instructional videos can lead to frustration, and even the potential for physical strain, as novice viewers, or those with limited physical mobility, struggle to keep up and mimic the movements of the expert instructors.

Now an international team of researchers from Lancaster University, Stanford University and FXPAL, have created a solution that dynamically adapts to mirror the position of the viewer's body and matches the speed of video playback to the viewer's movements.

The system, called 'Reactive Video', uses a Microsoft Kinect sensor, the latest in skeleton-tracking software, and probabilistic algorithms to identify the position, and movement of joints and limbs - such as elbows, knees, arms, hands, hips and legs. By working out the viewer's movements it can match and compare this with the movement of the instructor in the video footage. It then estimates the time the user will take to perform a movement and adjusts playback of the video to the correct position, and pace, of the viewer.

As well as providing a more immersive experience, Reactive Video also helps users to more accurately mimic and learn new movements.

The researchers tested the system on study participants performing tai chi and radio exercises - a form of callisthenics popular in Japan. The results from the study showed that both systems could adapt to the users' movements.

Dr Christopher Clarke, researcher from Lancaster University and co-author on the paper, said: "Since the 1980s, and especially now with the Internet, videos have helped people stay active and have offered a cheaper, more convenient alternative to gym memberships and personal trainers. However, traditional video players do have limitations - they can't provide feedback, or adapt the pace and intensity of the physical movement to the user.

"We know performing movements in slow motion is beneficial for learning by providing opportunities to analyse your movements, and developing timing. We also know it can result in less physical strain for inexperienced users.

"For some people, keeping pace can be tricky - especially when learning something new, and for older people or those with impaired movement. Also, constantly reaching for a remote to pause, rewind and replay, can be frustrating and breaks the immersion.

"Our system overcomes these issues by having the video automatically adjust itself to play back at the user's speed, which is less stressful and more beneficial for learning."

Don Kimber, co-author of the research, said: "Reactive Video acts and feels like a magic mirror where as you move the video mirrors your movement, but with a cleaned-up version of the procedure, or position, performed correctly by an expert for the user to mimic and learn from."

An additional benefit of Reactive Video, and something that sets it apart from exercise content developed for game consoles, is that it can be applied to existing footage of appropriate video content removing the need to create specially produced bespoke content.

"By using this system we can post-process existing instructional video content and enhance it to dynamically adapt to users providing a fundamental shift in how we can potentially interact with videos," said Dr Clarke.

The team believe that with further research this kind of adaptive technology could be developed for sports and activities such as learning dance routines or honing golf swings.
The Reactive Video system was presented at UIST2020, a leading academic conference for the field of Human Computer Interaction.

It is detailed in the paper 'Reactive Video: Adaptive Video Playback Based on User Motion for Supporting Physical Activity'.

The study's authors are Christopher Clarke, of Lancaster University; Doga Cavdir of Stanford University; and Patrick Chiu, Laurent Denoue and Don Kimber, of FXPAL.

Lancaster University

Related Learning Articles from Brightsurf:

Learning the language of sugars
We're told not to eat too much sugar, but in reality, all of our cells are covered in sugar molecules called glycans.

When learning on your own is not enough
We make decisions based on not only our own learning experience, but also learning from others.

Learning more about particle collisions with machine learning
A team of Argonne scientists has devised a machine learning algorithm that calculates, with low computational time, how the ATLAS detector in the Large Hadron Collider would respond to the ten times more data expected with a planned upgrade in 2027.

Getting kids moving, and learning
Children are set to move more, improve their skills, and come up with their own creative tennis games with the launch of HomeCourtTennis, a new initiative to assist teachers and coaches with keeping kids active while at home.

How expectations influence learning
During learning, the brain is a prediction engine that continually makes theories about our environment and accurately registers whether an assumption is true or not.

Technology in higher education: learning with it instead of from it
Technology has shifted the way that professors teach students in higher education.

Learning is optimized when we fail 15% of the time
If you're always scoring 100%, you're probably not learning anything new.

School spending cuts triggered by great recession linked to sizable learning losses for learning losses for students in hardest hit areas
Substantial school spending cuts triggered by the Great Recession were associated with sizable losses in academic achievement for students living in counties most affected by the economic downturn, according to a new study published today in AERA Open, a peer-reviewed journal of the American Educational Research Association.

Lessons in learning
A new Harvard study shows that, though students felt like they learned more from traditional lectures, they actually learned more when taking part in active learning classrooms.

Learning to look
A team led by JGI scientists has overhauled the perception of inovirus diversity.

Read More: Learning News and Learning Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to