A deep learning-augmented smart mirror to enhance fitness training

In recent years, engineers and computer scientists have created a wide range of technological tools that can enhance fitness training experiences, including smart watches, fitness trackers, sweat-resistant earphones or headphones, smart home gym equipment and smartphone applications. New state-of-the-art computational models, particularly deep learning algorithms, have the potential to improve these tools further, so that they can better meet the needs of individual users.

Researchers at University of Brescia in Italy have recently developed a computer vision system for a smart mirror that could improve the effectiveness of fitness training both in home and gym environments. This system, introduced in a paper published by the International Society of Biomechanics in Sports, is based on a deep learning algorithm trained to recognize human gestures in video recordings.

“Our commercial partner ABHorizon invented the concept of a product that can guide and teach you during your personal fitness training,” Bernardo Lanza, one of the researchers who carried out the study, told TechXplore. “This device can show you the best way to train based on your specific needs. To develop this device further, they asked us to investigate the viability of an integrated vision system for exercise evaluation.”

The low-cost computer vision system developed by Lanza and his colleagues utilizes a skeletonization algorithm (i.e., a deep learning algorithm that can attain skeletons from images), running on an embedded Nvidia Jetson Nano device with two fisheye cameras. As part of their study, the researchers trained this system to process and detect human movements in the video footage captured by the two fisheye cameras.

“A vision system, like the one we developed, can extract information from images by means of an AI algorithm,” Lanza said. “Our most recent paper demonstrates the accuracy of our system in measuring arm movements in simple fitness exercises, such as biceps curls.”

In one of their previous studies, the researchers presented a software design that could be used to create a comprehensive prototype of the smart fitness mirror envisioned by AB-Horizon. Their goal was to produce a device with production costs, a high performance, and a low energy consumption.

“The main advantage of our system is the absence of objects in contact with the user,” Lanza explained. “With cameras and AI applications, we understand and assess body motion, detect postural errors, and analyze simple fitness exercises. Nowadays our system analysis is based on simple body variables (elbow angle, hand position…) but we are working to improve the evaluation capability of the machine.”

The smart mirror that Lanza and his colleagues are helping to design would ideally be able to evaluate fitness exercises similarly to human personal trainers or in even more comprehensive ways. For instance, it could allow users to keep count of repetitions they performed for specific exercises, while also detecting the fundamental motion (e.g., traction, flection, rotation, etc.) of different body parts.

All fitness related information detected and calculated by the mirror is displayed on it, changing in real-time, so that users can keep track of it during workouts or use it to improve their training performance. Lanza and his colleagues evaluated their computer vision system in a series of tests, particularly focusing on its ability to track and make fitness predictions while users were performing biceps curls.

“We evaluated the accuracy of the vision system in understanding the different phases of an exercise,” Lanza said. “In traditional biomechanical analyses, the specific accuracy of our measurements is not acceptable, but we analyze a whole time series of body kinematics. This approach allows us to detect and understand fitness exercises and their peculiarities.”

The researchers found that with well-designed and calibrated software, their low-cost vision system could offer valuable fitness-related data while users performed simple fitness exercises. When integrated into the smart mirror created by AB-Horizon, the new system could significantly help users who are training without a supervising coach in both home and gym environments.

So far, the Lanza and his colleagues primarily evaluated their system’s performance on its own. However, they are now creating a prototype that would display the results of their system’s analyses on a smart mirror screen integrated into a motorized gymnastics machine.

“For this project, we collaborated with AB-Horizon, our commercial partner,” Lanza added. “In addition to designing the gym machinery, our partner will integrate the vision system with their prototype. Their experience in the fitness industry allows us to develop our software using athletic principles and a personal trainer from the company also guides us through the testing process. As part of future developments, an intelligent evaluator will detect the exercise.”

The intelligent evaluator that Lanza and his colleagues are working on should be able to interpret qualitative information by analyzing raw body kinematic data. To train this model, therefore, the researchers will first be collecting large amounts of data during fitness tests with both athletes and less-experienced fitness trainees.

© 2022 Science X Network