Isaac Sim and ZEDX

Software ROS2
1

Hi there,
I am trying to work with Isaac Sim and ZEDx utilizing ROS2, for this purpose I am using official demo available here: Using ZED with ROS 2 and Isaac Sim - Stereolabs

I am facing issue of no file available at this stage (ss attached)

image
image986×784 87.6 KB

Let me please post the details.

Error#1: When I run this command (ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zedx sim_mode:=true use_sim_time:=true)

image
image1385×943 301 KB

Error#2: Rviz2 error too, here, when I run (ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zedx sim_mode:=true use_sim_time:=true)

image
image895×263 53.3 KB

Additionally, I am lost, if I successfully start viewing my virtual environment (from within Isaac Sim), how can I further utilize this viewable data for other ML/LLM/CV purposes?

2

@Myzhar
Sir, any solution to it?

3

Hi @iamshakeelsindhu,

Sorry for the wait.

After following the instructions from the Isaac Sim documentation to run a virtual camera, are you able to use retrieve the camera’s stream from one of our tools such as ZED_Explorer?

This will allow me to know if it is a problem related to the configuration of your setup, or specific to ROS.

4

Thank you for your reply, Sir.

I can see the virtual environment (Isaac Sim Environment) on ZED_Explorer, once I run the simulation in Isaac Sim.

Let me show you and explain my workspace.

image
image926×500 98.4 KB

In this photo, I am running ROS2 services to control my robot (this is the simplest environment for testing purposes with some objects placed in front of the manipulator). As mentioned in the documentation of ZED integrating with Isaac Sim, I am using ZED_X in Isaac Assets and running it through Action Graphs. I can move my robot and also can see the live data streaming out of ZED Camera on ZED_Explorer. However, my purpose is to get this data in a format which will be able to go into Computer Vision pipeline (right now, I am working on YOLO-World and later, I need to feed my virtual environment in a VLM).
I am kinda lost, how to get the suitable data (for the VLM as on my real robot, I am using ZED2i, so both the data on real scenario and virtual should be same).

If you need further details, please let me know.

5

@Myzhar @mattrouss

Is there any solution to it? I have been trying to get my sim data from within Isaac Sim to fed it into a CV model. Is there any way to it?

Thanks

6

Hi @iamshakeelsindhu,

Thank you for sharing the output in ZED_Explorer, this means that there is an issue with the ROS 2 wrapper retrieving the stream, and not the streaming from Isaac Sim itself.

Can you set the following parameter as such in the ros 2 configuration file for your camera:

        pos_tracking: # Only stereo cameras
            pos_tracking_enabled: true # True to enable positional tracking from start
            pos_tracking_mode: "GEN_1" # Matches the ZED SDK setting: 'GEN_1', 'GEN_2'

Tracking GEN_2 is currently not available with Isaac Sim, this is currently in our roadmap.