Hello, I am trying to use ZED body tracking in the mujoco simulation engine. I am animating a humanoid in mujoco with zed body tracking data, and encountered some problems.
Similar to other 3D software, a rigged human model is needed. The root joint (Hip) should be supplied with global position (x,y,z) and global orientation quaternion (w,x,y,z). The other joints need local joint orientations expressed in euler angles.
The model I used is as below.
The global coordinate frame is on the ground with RGB indicating x, y, z. All the joints use the same coordinate frames as the global frame, which is left: x, forward: y, up: z.
I modified the zed body tracking example
to obtain local orientations per joint, the global root orientation and global root position (the first keypoint).
Mujoco uses a right-hand z-up coordinate system, so I change line 43 to use RIGHT_HANDED_Z_UP. I change line 68 to use POSE_34, and use the HUMAN_BODY_MEDIUM accuracy level. After that, I performed joint mapping, quaternion xyzw->wxyz reordering, and quaternion->euler conversions before feeding into mujoco.
The results and problems
Clips featuring walking towards and away from the ZED camera seems to get converted correctly. However, clips with the humanoid walking from left to right (and right to left) have some problems. Sometimes the humanoid faces wrongly to the front, instead of facing right.
Also, sometimes the humanoid walks backwards and turns around incorrectly, which does not exhibit in the ZED unity plugin. Finally, sometimes the legs penetrate each other.
Did I do something wrong? The ZED documentation has a guide on how to place an avatar model in the unreal engine, but I need to place the model in a right-handed z-up coordinate system. Also, is there a way to reduce the leg penetration? Look forward for your answer and advice, thank you!