Fitness correction using deep learning model Posenet from Tensorflow.js
The pose estimation task involves using an ML model to detect the body joints and form a skeleton based on the person’s pose. Tensorflow.js offers a Posenet model that can be utilized for this purpose, and it can be seamlessly integrated with a live camera feed to detect poses.
To enhance the user experience beyond tracking the human pose, it would be beneficial to provide exercise or workout suggestions. These suggestions should be not only realistic but also responsive to hand and leg movements. As a starting point, I attempted to compare two images to adjust the position of the hands or legs. To achieve this functionality, I use a reference image containing a pose that you want to try.
Tensorflow.js Posenet model provides the skeleton for one or more people in real time. The model tracks 17 body joints and we consider 8 joints for hand and leg movements. These pixel values are used to detect the slope of reference image and compared with the camera feed. For example, the values of the reference image between left elbow and left wrist body points are taken and slope is calculated. This slope is compared with the camera feed. We compare the skeletons and results the correction if any as shown in Figure.
Issues: There is a small issue with this approach. Since this is 2D pose estimation from the image, certain yoga poses like
cannot not be used as the model results incorrect values. This is still a developing field, there are 3D pose estimation methods available which can be utilized for better results.
For more information, you can access the paper by following this link.
