Using Body34 to animate characters in Unity livelink

Hello! I am trying to write custom logic in ZED livelink for unity, streaming Body34 skeleton data to display a pre-recorded scenario. I am able to stream the data, but I am not sure if Body34 skeleton data is fully supported? This line in SkeletonHandler.cs indicates that Body_34 is supported, while this line does not(?).

As a current hacky solution I have updated UpdateAvatarControl in ZedBodyTrackingManager.cs to read a new line of my json file and display the skeleton data in there. I have also enforced sl.BODY_FORMAT.BODY_34 here.

Must be something going wrong though, as my character is walking backwards and got some crazy rotations. For reference, here is my custom UpdateAvatarControl (and yeah, I am no C# developer :joy:):

	private void UpdateAvatarControl(SkeletonHandler handler, sl.BodyData data)
    {
        Dictionary<string, object> obj = jsonData[jsonKeyList[jsonCounter]];
        jsonCounter += 1;

        // If object_list is not a key in obj return null
        if (obj.ContainsKey("object_list")) {
            List<Dictionary<string, object>> object_list = ((JArray)obj["object_list"]).ToObject<List<Dictionary<string, object>>>();

            // Iterate through list: object_list
            foreach (Dictionary<string, object> person_dict in object_list) {
                // Create lists of quaternions and vectors from the data - something like:
                List<Quaternion> quaternions = new List<Quaternion>();
                foreach (List<float> values in ((JArray)person_dict["local_orientation_per_joint"]).ToObject<List<List<float>>>())
                {
                    float x = values[0];
                    float y = values[1];
                    float z = values[2];
                    float w = values[3];

                    Quaternion quaternion = new Quaternion(x, y, z, w);

                    quaternions.Add(quaternion);
                }

                Quaternion[] quaternionsArray = quaternions.ToArray();

                // Do the same for keypoint_confidence and keypoint
                Debug.Log(person_dict["keypoint_confidence"].GetType());
                
                float[] keypoint_confidence = ((JArray)person_dict["keypoint_confidence"]).Select(x => (float)x).ToArray();
                Vector3[] keypoint = ((JArray)person_dict["keypoint"]).Select(x => new Vector3((float)x[0], (float)x[1], (float)x[2])).ToArray();
                JArray jarr_glob_root_orientation = (JArray)person_dict["global_root_orientation"];
                Quaternion global_root_orientation = new Quaternion((float)jarr_glob_root_orientation[0], (float)jarr_glob_root_orientation[1], (float)jarr_glob_root_orientation[2], (float)jarr_glob_root_orientation[3]);

                handler.SetConfidences(keypoint_confidence);
                handler.SetControlWithJointPosition(
                    keypoint,
                    quaternionsArray, 
                    global_root_orientation,
                    enableAvatar, 
                    mirrorMode
                ); 
            }
        }
    }

Here’s a video of the current result:

Hi,

The body 34 format should be supported as well as 38 and 70.

In this sample we are also reading data from a json (the message sent by UDP is basically a json file), you can take a look at how we converted the Json data to usable data if you are curious.

But, I don’t any anything wrong in your code snippet.

My first idea is your data is in the wrong coordinate system. Unity is using a Left Handed Y Up system, and our plugin is expected the data to be in this reference frame. In the ZED SDK, the coordinate system used is an init parameter (see : https://github.com/stereolabs/zed-unity-livelink/blob/main/zed-unity-livelink-mono/src/main.cpp#L52).

If the data you recorded is not in this format, you need to convert it.

Stereolabs Support

@BenjaminV I believe the json file was recorded using Right hand Y up coordinate system. Looking here Coordinate Frames | Stereolabs it seems like I just need to reverse the Z axis to the conversion? Simply adding a “-” in front of the z-axis still gives a wonkey movement though:

Vector3[] keypoint = ((JArray)person_dict["keypoint"]).Select(x => new Vector3((float)x[0], (float)x[1], -(float)x[2])).ToArray();
JArray jarr_glob_root_orientation = (JArray)person_dict["global_root_orientation"];
Quaternion global_root_orientation = new Quaternion((float)jarr_glob_root_orientation[0], (float)jarr_glob_root_orientation[1], -(float)jarr_glob_root_orientation[2], (float)jarr_glob_root_orientation[3]);

What are the steps to transfrom from right handed y up to left handed y up?

Hi,

There is a function in the ZED SDK that gives you the Transform matrix to apply to change the coordinate system : sl Namespace Reference | API Reference | Stereolabs

In your case the matrix should be :

tf_matrix =

1.000000 0.000000 0.000000 0.000000
 -0.000000 1.000000 0.000000 -0.000000
0.000000 0.000000 -1.000000 0.000000
0.000000 0.000000 0.000000 1.000000

with the following formula :

pt_coord_dst = tf_matrix * pt_coord_src

Edit : You can’t directly apply this formula to a quaternion. So what you have to do is creating a matrix using Unity - Scripting API: Matrix4x4.SetTRS that contains the position and roation you want to convert.
And then, multiply this matrix with the transform matrix shown above.
Finally, you can extract the position and rotation from the resulting matrix and it should be in the correct coordinate system.

@BenjaminV But what position do I use for each of the joints? Unity is only using local rotation and I want to do that as well. I tried the following, but it did not work at all:

        List<Quaternion> quaternions = new List<Quaternion>();
        foreach (List<float> values in ((JArray)person_dict["local_orientation_per_joint"]).ToObject<List<List<float>>>())
        {
            float x = values[0];
            float y = values[1];
            float z = values[2];
            float w = values[3];

            Quaternion quaternion = new Quaternion(x, y, z, w);

            // Perform transformation to get proper coordinate system: Convert to rotation matrix -> multiply by transformation matrix -> convert back to quaternion
            Matrix4x4 transformationMatrix = Matrix4x4.TRS(Vector3.zero, quaternion, Vector3.one);

            // Create the identity matrix with a -1 in the third row
            Matrix4x4 invertedZMatrix = Matrix4x4.identity;
            invertedZMatrix[2,2] = -1;

            // Multiply the transformation matrix with the invertedZMatrix
            Matrix4x4 resultMatrix = invertedZMatrix * transformationMatrix;

            // Extract the forward and up vectors from the result matrix
            Vector3 forward = resultMatrix.GetColumn(2);
            Vector3 up = resultMatrix.GetColumn(1);

            // Convert the forward and up vectors to a quaternion
            Quaternion newQuaternion = Quaternion.LookRotation(forward, up);

            quaternions.Add(newQuaternion);
        }

Hi,

You don’t need to apply the keypoints positions from the SDK to the avatars, but only the rotations.

you can directly extract the rotation using : Unity - Scripting API: Matrix4x4.rotation

Actually would it be possible for you to share a json with the data you recorded and your modified project so I can test on my side? You can either share it here or at support@stereolabs.com if you don’t want to share it in public.

@BenjaminV Sure, here are the updated Scripts folder: Scripts.zip - Google Drive

It contains the json file “me-single-person.json”. The only additional change I have made is to remove the ZedStreamingClient script from the project (FusionManager object).

Hi,

That’s my fault, I’m sorry. I gave you a wrong formula.

You actually need to do :

Matrix4x4 resultMatrix = invertedZMatrix * transformationMatrix * invertedZMatrix.inverse;
Quaternion newQuaternion = Quaternion.LookRotation(resultMatrix.GetColumn(2), 
             resultMatrix.GetColumn(1));

Same for the global orientation.

@BenjaminV It works, thanks for the help. It seems like your Unreal sample may have more added functionality to smooth the data and make the avatars move more accurately. Is that the case?

Could you kindly help me in making the same changes to the Unreal live link sample, displaying avatars from a similar .JSON file? Thanks :slight_smile: