Potential problems using the SDK with a simulated ZED 2i with a ZED X serial number?

[aidan073]

Hello,

I’ve made some modifications on a fork of the zed-isaac-sim repository that allows for basic ZED 2i compatibility (mostly just adding the proper intrinsics). However, upon attempting to use the SN of an actual ZED 2i, the is_sn_valid function loaded from the following code within the extension is causing issues:
hLibrary = LoadLib(zed_lib_path.c_str());

is_sn_valid = (IsSNValidFunc)GetFunc(hLibrary, "is_sn_valid");
I did try bypassing the check, but the initialization of the streamer fails, presumably due to some internal validation. For the time being, I am simply using the ZED X SNs provided within the extension’s code for the 2i as well. While this does appear to work, I’m concerned I will run into issues where something within the SDK will make assumptions based on the SN belonging to a ZED X, leading to calibration issues or other errors. I’ve seen so far that the DepthViewer downloads the .conf file based on the SN, so this is a problem. My question primarily is: will there be other problems with the SDK if I continue to use the ZED X SN for the 2i? Is there something else I should be doing instead of using the ZED X SN, to make the 2i compatible with the isaac sim extension?

Thanks.

[Myzhar]

Hi @aidan073
Welcome to the Stereolabs community.

@aidan073

I’ve made some modifications on a fork of the zed-isaac-sim repository that allows for basic ZED 2i compatibility (mostly just adding the proper intrinsics). However, upon attempting to use the SN of an actual ZED 2i, the is_sn_valid function loaded from the following code within the extension is causing issues:
hLibrary = LoadLib(zed_lib_path.c_str());

is_sn_valid = (IsSNValidFunc)GetFunc(hLibrary, "is_sn_valid");

What you made is not allowed. You cannot simulate “generic cameras”, but only cameras created to be used for simulation.
The ZED SDK won’t accept any serial number that has not been validated for simulation purposes.

Why do you need a simulated ZED 2i? What’s your goal?

I recommend you continue using the ZED X as a simulated camera.
It won’t make any difference from a software point of view.

[aidan073]

Thanks for the quick response,

I’m using the ZED X .usd file, but applying the 2i intrinsics to the actual camera prims. I’m using the video feed downstream in an ML pipeline, and since I have a 2i, I want to use its intrinsics in simulation to avoid sim-to-real issues. If the SN doesn’t make any difference on the software end, then this works fine. However, I already noticed that the DepthViewer downloads a .conf file based on the SN. So in my case, it is defaulting to the ZED X intrinsics, which I assume are needed for pixel-matching. If this is the only problem I will encounter, then it should be a quick fix by replacing the .conf file with the proper intrinsics. I just wanted to verify that similar problems won’t happen due to the SN not matching a 2i.

[Myzhar]

@aidan073

I’m using the ZED X .usd file, but applying the 2i intrinsics to the actual camera prims.

You cannot do this. The parameters would not match the simulated parameters.

@aidan073

I’m using the video feed downstream in an ML pipeline, and since I have a 2i, I want to use its intrinsics in simulation to avoid sim-to-real issues.

This is not required because the ZED SDK operates on rectified images.

@aidan073

However, I already noticed that the DepthViewer downloads a .conf file based on the SN. So in my case, it is defaulting to the ZED X intrinsics, which I assume are needed for pixel-matching.

Yes, this is correct.

@aidan073

If this is the only problem I will encounter, then it should be a quick fix by replacing the .conf file with the proper intrinsics.

No, you cannot; the ZED SDK will block this operation.

@aidan073

I just wanted to verify that similar problems won’t happen due to the SN not matching a 2i.

They will.

Just to confirm: you cannot simulate cameras with custom parameters, the ZED SDK blocks this kind of customization.

[aidan073]

Thanks for responding to each piece,

would you be able to elaborate on a few of these points:

@Myzhar

You cannot do this. The parameters would not match the simulated parameters.

Which parameters would not match the simulated parameters? I added 2i intrinsics as an option in the extension, so at the time of the transport layer it should be correct. Is it the parameters determined by the SDK that wouldn’t match?

@Myzhar

This is not required because the ZED SDK operates on rectified images.

Given that this camera has an intentional and fixed location on a robot, I do believe the custom intrinsics would matter a little bit in this scenario despite rectification due to the FOV change.

@Myzhar

No, you cannot; the ZED SDK will block this operation.

Would you be able to give any insight about what in the SDK blocks this? It seems like replacing this file with the .conf for the physical 2i would be my only option if I want to use the SDK for this use case. The SDK using the ‘default’ ZED X intrinsics for rectification while the simulated cameras are using the ‘true’ intrinsic based on my 2i will certainly cause some problems.

[Myzhar]

@aidan073

Would you be able to give any insight about what in the SDK blocks this?

This is a security block. We do not allow ingesting data into the ZED SDK pipeline from not certified input sources.

A simulated camera with custom parameters is not a certified source, even if the parameters come from a real camera.

[aidan073]

I think this just about answers everything, thanks.

Just as one final follow-up, do you have any recommendation on how I could proceed with simulating the 2i, if at all? You mentioned using the ZED X as a simulated camera in its place, but the FOV concern makes me hesitant. Is there any plan to add approved 2i SNs for simulation?