Relocalization zed camera pose on point cloud mapping

Software C++
1

Hi, I retrieved fusedPointCloud map and stored point cloud data in PLY file. I want to use the PLY data and then I want to locate the static zed camera on the map. Is there a way to relocalization the zed camera to have the camera’s pose align with the mapping?

2

Hi @nctran
it is not possible to relocate the camera by using an existing point cloud map with the ZED SDK.
You can consider the “Area Mapping” feature of the Positional Tracking module for relocation:

3

Thanks that help. but when I try to use it, I load the .area file and got the pose. but when I test it by move my camera in different location in the room and print out the position. It only print out the original position where I am capture the map, not the new location.

4

@nctran can you explain better the problem with an example? It’s not clear

5

Hi.
For example, let’s say I developed a program that can digitally map my entire room and save the resulting map as a “ply” file, along with an associated “.area” file. Afterwards, I designed another program to load both the map and the .area file, with the purpose of positioning my camera within the map base on the link you give me Area Memory and Relocalization | Stereolabs. Unfortunately, I encountered an issue where moving the camera to a different location with the hope of the camera will recognize it pose base on the map. However, it gave me the same position as when it initially captured the map. In the second program, I just load both file base on the documentation and I am not sure how the camera remember the map and how does it work to relocalization.

6

Hi @nctran
the Area Memory allows you to correctly relocate the camera in the correct position of the environment with respect to the initial position of the camera that you used to create the Area Memory file.
If you start the positional tracking with a valid area memory file, as soon as a known position is detected the pose of the camera is modified according to that.

7

that is weird! in my case, the pose of the camera is not modified according to initial position even I changed the camera location. when I set enable area memory to TRUE and I get [PositionalTracking][ERROR] INCOMPSTIBLE AREA FILE but I am pretty sure I use same Depth Mode for both of my program.

8

It is possible that you did not correctly save the Area Memory file or that it’s not good enough to be used.
Be sure to call the saveAreaMap map before closing the camera:
https://www.stereolabs.com/docs/api/classsl_1_1Camera.html#afe82ca1a5a374233a0f28bf7ba747674

9

Hi,
I pretty sure I save before close the camera. Also, if I change set_as_static to true then I do not have the error.

zed.disablePositionalTracking(“zedArea.area”);
zed.close();

10

The error INCOMPATIBLE AREA FILE is returned when the size of the Area Map file is not big enough.
This happens in two cases:

  • the file has not been correctly saved when updated
  • the camera was not moved while creating the Area Map file for relocation
11

Hi,
I just mapping again to get new mapping but still same result.

  • the file has not been correctly saved when updated.

I saved the area file only at the end of the program right before I close the zed camera. do I have to save it every time the camera grap the map?

zed.disablePositionalTracking(“zedArea.area”);
zed.close();

  • the camera was not moved while creating the Area Map file for relocation.

I pretty sure That I moved my camera around the room to map and the area file 40 MB or above every time I map.

12

Hi @nctran
please explicitly call the saveAreaMap function before disabling the positional tracking and check the return type for errors.

Please also record and share an SVO with ZED Explorer in the same way you performed the room scan for generating your Area Map file. We can use the SVO to try to replicate this behavior.

Finally please also share the InitParameters, and the PositionalTrackingParameters, that you used to initialize the camera and the positional tracking module.

13

Hi,
I did what you suggested but I still have same result.
I used this to save the area file.

zed.saveAreaMap(“zedArea.area”);
auto export_state = sl::AREA_EXPORTING_STATE::RUNNING;
while (export_state == sl::AREA_EXPORTING_STATE::RUNNING) {
export_state = zed.getAreaExportState();
sl::sleep_ns(5);
}
cout <<export_state << endl;

the output is SUCCESS.
I also record and share an SVO with ZED Explorer and it has normal behavior. And, I saved both InitParameters, and PositionalTrackingParameters in yml files so I can read them on the second program. I have to set the PositionalTrackingParameters.set_as_static = true to remove this error [PositionalTracking][ERROR] INCOMPSTIBLE AREA FILE.
Also, the camera always at the initial pose where it starts to capture images/map and did not change to its current position.

14

In the beginning, you wrote that the ZED is static when you are trying to have it find its current position. Is it possible for the ZED to relocalize without the camera moving? I’m curious. I’ll be working on relocalization very soon. Thanks.

15

No, Area Mapping requires a few different images to initialize.
Here the problem is conceptual: relocalization means that something moved… if the camera is static the relocalization is nonsense.

1 Like
16

LOCALIZE MULTIPLE DEVICES IN SPACE
When multiple cameras are successfully relocalized against the same map, the devices can then share a common reference frame and real-world positions during a session.

The devices will localize themselves in the World Frame recorded in the Area file. A user can specify the position of this reference frame using an Initial World Transform, otherwise the World Frame will be located at the initial position of the camera when tracking with spatial memorization was started. Without area memory, different devices will not be able to understand their absolute position in the world and will track their position with respect to their own starting point.

Thanks for your answer, base on what I read from the Documentation, Area Memory and Relocalization - Stereolabs
What I understand that each camera will have their own localization in their own map space but when join multiple camera into same map space, then each camera should be able to relocalized and know its localization in the same map. Meaning, relocalization doesn’t means that something moved. it means to localize the camera localization base on the new map or environment.
if you said the camera use area file to relocalize when it moved. it is not make sense to me.

17

This process requires that you acquire the working space with a camera by manual mapping.
You move the camera all around the scene to grab all the visual features and provide a detailed virtualization of the environment.
Then you can use static cameras that localize themself in this virtual map by using the information stored in the Area Mapping file. The origin of this reference system is the initial position (instant of the calling of enablePositionalTracking) of the camera used to acquire the information.

18

Hi can you clarify more about this?

19

When you start the positional tracking with enablePositionalTracking and you pass the path of an empty area mapping file, the positional tracking module will fix the origin of the reference system (world) in the point where the camera is placed.

You can change this behavior by setting the initial_world_transform parameter. In that case the origin of the reference system is placed in the location set by the inverse of the initial_world_transform with respect to the position of the camera.

20

Thanks,
I actually did what you said by initialize the camera transform by set initial_world_transform to initial_position.setTranslation(Translation(0,170,0))