Possible to use coordinates of pose estimation with kineticstoolkit?

[gabriead]

Hi folks,
I am completely new to Biomechanics and have been looking at it only from the perspective of pose estimation. Is it possible to use the data of a pose estimation model (e.g. MediaPipe) directly with pyomeca? How would I transfer the joint coordinates in a format that pyomeca can use (e.g. c3d)? Thank's a lot!

[felixchenier]

Good morning @gabriead

I never tried MediaPipe, but I see that's it's aimed to real-time applications, while Kinetics Toolkit is more aimed to offline movement/force timeseries analysis. But if you use MediaPipe offline, I guess you could transpose the pose coordinates to Kinetics Toolkit.

For what I understand from https://google.github.io/mediapipe/solutions/pose.html:

Output

Naming style may differ slightly across platforms/languages.

pose_landmarks

A list of pose landmarks. Each landmark consists of the following:

    x and y: Landmark coordinates normalized to [0.0, 1.0] by the image width and height respectively.
    z: Represents the landmark depth with the depth at the midpoint of hips being the origin, and the smaller the value the closer the landmark is to the camera. The magnitude of z uses roughly the same scale as x.

I understand that MediaPipe gives you a list of objects that have coordinates in it. I would suggest to try converting this list to a Pandas DataFrame, with the DataFrame index being the time in seconds. Then, you can convert it as a Kinetics Toolkit TimeSeries as explained here: https://kineticstoolkit.uqam.ca/doc/timeseries_dataframes.html

Don't hesitate to ask for more help, and it'd be nice if you can post your success if you make it work 😃

Have a happy new year 🎉

---

PS Concerning Pyomeca, this is another project by another team. It may be confusing since we're both in Montreal and both projects have related objectives but with different approaches. If you prefer object-oriented programming, Pyomeca may be your choice and I'm pretty sure you could also load coordinates from MediaPipe without problem since it is built around xarray which is compatible with Pandas DataFrames. You could ask them here: https://pyomeca.github.io/#bug-reports-questions

[gabriead]

Good Morning @felixchenier, thank you for your great response! That was exactly the information I needed. It will be a few more days until I can try your approach, but I will let you know how it went!

[felixchenier]

@gabriead No problem! Another, probably more efficient and maybe easier solution would be to directly construct the TimeSeries. You create it from scratch:

ts = ktk.TimeSeries()

you assign its time vector of length N:

ts.time = np.array(...)

and you assign its data of length Nx4 ([x, y, z, 1.0]):

ts.data["ShoulderL"] = np.array(...)
ts.data["ShoulderR"] = np.array(...)
etc.

based on MediaPipe's data format. Maybe it's easier and faster that way.