Error in Tracking State

CristianViSa · July 10, 2025, 3:46pm

Hi,

I have a ZED 2i camera mounted on an e-scooter driving through sidewalks. I am using the cloud points to calculate widths and slopes of the sidewalks.

Everything is fine with the widths, but with the slopes I keep getting errors, I think it’s due to the rotation of my point clouds. As the camera is not completely aligned with the ground, as it is placed looking a bit downwards, it has a certain initial angle that can be corrected by rotating the point cloud.
Well, I use the positional_tracking_parameters in order to rotate this point clouds, i am using this:

	positional_tracking_parameters = sl.PositionalTrackingParameters()

positional_tracking_parameters.set_gravity_as_origin = True
positional_tracking_parameters.enable_imu_fusion = True
positional_tracking_parameters.mode = sl.POSITIONAL_TRACKING_MODE.GEN_2
self.zed. enable_positional_tracking(positional_tracking_parameters)
Then, every frame i rotate with
pose = sensors_data.get_imu_data().get_pose()
rotation = pose.get_rotation_matrix()
rotation_matrix = rotation.r
rotated_points = np.dot(xyz.reshape(-1, 3), rotation_matrix.T).reshape(xyz.shape)

However, I constantly see high vibration alerts on the accelerometer and so on, due to the scooter bouncing on the pavement, and I think this is causing my point cloud to not get straightened properly, so I get false slopes due to the cloud being incorrectly straightened. I also see that the position_track_state sometimes gets lost.

Am I rotating the cloud wrong or is it because of these high vibrations that I am not able to get my slopes right?

Myzhar · July 11, 2025, 9:01am

Hi @CristianViSa

Normally, you do not need to rotate the point cloud because you enabled set_gravity_as_origin, which uses the IMU to automatically rotate it to align to the gravity vector.

CristianViSa · July 11, 2025, 10:35am

Thanks,

What about the high vibrations? They make the IMU to provide wrong measures? Can the IMU recover its state or the measurements are wrong since that moment?

Myzhar · July 11, 2025, 10:44am

I need more information to reply to this.

Not necessarily, the Positional Tracking will recover a good status as soon as the camera can see a “known” location

CristianViSa · July 21, 2025, 8:16am

Sorry for the late reply.

I am constantly getting this warnigns:
[2025-07-21 08:09:52 UTC][ZED][WARNING] IMU data issue detected : IMU: HIGH_VIBRATION_ACC

I think my IMU is not providing correct measures in order to orient my cloud point, therefore, i am obtaining wrong values for my slopes.

Myzhar · July 21, 2025, 12:46pm

Hi @CristianViSa
are you using the latest ZED SDK v5.0.3?

CristianViSa · July 21, 2025, 12:49pm

I have my SDK updated now, but those videos were recorded using 4.2.5

Myzhar · July 21, 2025, 1:01pm

Please place the camera on a stable surface, then open ZED Sensor Viewer and check the stability of the Accelerometer and Gyroscope sensor data.
Please take and send me a few screenshots as a double check.

CristianViSa · July 21, 2025, 1:14pm

Seems to be OK
Accelerometer

Gyroscope

It normally happens to my vehicle when the sidewalk is a bit more bumped than usual.

Myzhar · July 21, 2025, 4:06pm

This is a piece of relevant information; the warning is expected in this case.

CristianViSa · July 22, 2025, 6:20am

It makes sense, but how can I be sure that the sensor readings, from that moment on, are correct? Sometimes I see that the tracking_state is lost and I don’t know if the readings are corrupted after that.

Myzhar · July 22, 2025, 8:35am

If the tracking is lost, it’s not a problem with the IMU, but with the visual features for Visual Odometry.
A slow “turn in place” could allow the robot to see “past feature” and recover the tracking.

CristianViSa · July 23, 2025, 11:16am

I have made test, in order too see what is the problem, and I noticed that the values of the pointcloud does not change if i use set_gravity_as_origin and if don’t use any positional tracking parameters.
See this screenshot i get this values for both cases:

Is this expected? You told me that is not neccesary to rotate my pointcloud using the set_gravity_as_origin=True, but i can see the same exact values.
What I am trying to correct is the fact that my camera is not straight, it is facing a bit towards the ground and therefore the point clouds are not well oriented and my slopes have this error in the angle, but I am not being able to do it at all. See this picture to illustrate how I mount my zed 2i on a scooter, I have to correct this initial tilt.

Myzhar · July 23, 2025, 12:52pm

Please explain this with other words adding details and eventually the configuration file.

Please send screenshots and videos to show the problem as it is.

CristianViSa · July 23, 2025, 1:16pm

Sorry, let me explain my problem.

I mount my camera like this, note the tilt of the camera, is facing the ground.

Because of this tilt, the camera generates the point clouds wrongly, they are wrongly oriented due to this angle and have to be rotated. If they are not rotated, the slopes measured are unrealistic as the cloud is not straightened.
To do this, what I do is to obtain two points, calculate the distance between them and the difference in height between them, in order to calculate the slope.
For this to work, the point cloud has to be correctly oriented, using the slope angles returned by the IMU.
I am using this script: SlopeCalculation (m84xy5cs) - PasteCode.io

What i said is that you mentioned this:

Blockquote Normally, you do not need to rotate the point cloud because you enabled set_gravity_as_origin, which uses the IMU to automatically rotate it to align to the gravity vector.

But i noticed that if i just comment this part, to not use any positional tracking parameter:

    positional_tracking_parameters = sl.PositionalTrackingParameters()
    positional_tracking_parameters.set_gravity_as_origin = True
    positional_tracking_parameters.enable_imu_fusion = True
    positional_tracking_parameters.mode = sl.POSITIONAL_TRACKING_MODE.GEN_2
    self.zed.enable_positional_tracking(positional_tracking_parameters)

The values obtained in my point cloud is the same as enabling the positional_tracking_parameters.set_gravity_as_origin = True

So basically, it is not rotating anything.
For example, if i set gravity_as_origin = True i obtain this values for both points:
p1: px=(1376, 1070), X=0.35, Y=1.41, Z= -0.58
p2: px=(1423, 791), X=0.75, Y=2.55, >=-0.38

And if i disable the positional tracking parameters, the values are the same:

p1: px=(1376, 1070), X=0.35, Y=1.41, Z= -0.58
p2: px=(1423, 791), X=0.75, Y=2.55, >=-0.38

As mentioned in my first message, I have also tried to rotate my point cloud with this piece of code:
pose = sensors_data.get_imu_data().get_pose()
rotation = pose.get_rotation_matrix()
rotation_matrix = rotation.r
rotated_points = np.dot(xyz.reshape(-1, 3), rotation_matrix.T).reshape(xyz.shape)

But the values for the slope are not correct either. I dont know what i am missing to solve this problem.

RodolphePerrin · July 23, 2025, 2:36pm

Hi Cristian,

Not too sure I’m getting the issue exactly but wanted to share some tips. Having your camera tilted shouldn’t be an issue.

Looking briefly at your code, it seems that you may be getting the pointcloud in the camera frame instead of the world frame. World frame origin is based on the Positional Tracking parameters that you set. It may explain why you’re not seeing any differences at the moment since results are probably given in the camera frame.

To change that behavior, add this when initializing your system :

runtime_params = sl.RuntimeParameters()
runtime_params.measure3D_reference_frame = sl.REFERENCE_FRAME.WORLD

More info can be found here in the API.

Please note that with our latest SDK, we recommend now using GEN_3 as mode for Positional Tracking since it is providing more robust results.

CristianViSa · July 23, 2025, 3:11pm

The main problem is that i am measuring slopes and due to some point cloud orientation problem I keep getting wrong values.

I used this frame because I don’t really understand how the world frame works. In the first frame, the values are coherent, for example these in the screenshot

but as the video progresses, the values become very strange, in this case there are some very low negative values, I guess it’s because of the starting point, it’s the difference from the starting point.

See the new snippet with the world frame. WorldFrame (rub0yx7e) - PasteCode.io
I am pretty sure i am missing something on this point cloud orientation, but I cant see what am i doing wrong.

RodolphePerrin · July 23, 2025, 3:54pm

Are you able to share what the pointclouds look like or the SVO ? I don’t get what you mean by “due to some point cloud orientation problem”.

CristianViSa · July 24, 2025, 6:22am

Sure, here is a cut of my SVO2 file. I know that the slope of that sidewalk is 4%, topographically measured. slope.svo2

Let me explain what i mean by point cloud orientation problem. Lets use the Depth Viewer, if you dont mark the “IMU ORIENTATION” box, the point cloud look like this:

You can see that is not true, this is due to the camera tilt, so you need to orient the point cloud, if we mark the IMU ORIENTATION box, the point cloud goes like this:

This is more realistic, as the tilt is corrected.
What I think is happening is that this correction is not being done properly, because the slopes I am getting are not correct. If we understand slope as the difference in height / distance travelled, with the values of this point cloud oriented with the IMU I get a value of 11%, which is far from the reality.
See the point cloud in las format pointcloud_rotated_starting_0.las

Another thing i have noticed is that, the same frame (frame 1044 of the SVO) has different values if you use zed.set_svo_postion(1044), see screenshot:

And if you just go through the video from frame 0 to frame 1044, the values are so different, see this screenshot:

See its point cloud in here:
pointcloud_rotated_starting_0.las

How is this possible? I think the problem is clearly the angles used for the orientation.

This is my snippet, i am using world rotation now SlopeWORLDFrame (fzwq2hjf) - PasteCode.io

CristianViSa · July 28, 2025, 8:48am

Hi,

Did you have time to check it?