Using sl::MEM::GPU in ZED ROS2 Wrapper with Isaac ROS – Is Direct GPU Integration Possible?

Software ROS2
1

Hello everyone,

I am working with NVIDIA Jetson AGX Orin (JetPack 6.1, Ubuntu 22.04) and integrating the ZED ROS2 Wrapper with Isaac ROS. I am using a ZED X camera and want to optimize image retrieval for GPU-based processing.

By default, the ZED ROS2 Wrapper retrieves images into CPU memory (sl::MEM::CPU), but since I am using Isaac ROS for GPU-accelerated processing, I would like to retrieve images directly into GPU memory (sl::MEM::GPU) to eliminate unnecessary CPU-GPU transfers.

I have modified the retrieveImage function in zed_wrapper_node.cpp as follows:

if (mRgbSubCount + mLeftSubCount + mStereoSubCount > 0) {
    retrieved |=
      sl::ERROR_CODE::SUCCESS == 
      mZed->retrieveImage(mMatLeft, sl::VIEW::LEFT, sl::MEM::GPU, mMatResol);
    mSdkGrabTS = mMatLeft.timestamp;
    mRgbSubscribed = true;
    DEBUG_VD("Left image retrieved");
}

My Questions:

  1. Is this modification sufficient to ensure that Isaac ROS nodes (such as those using TensorRT or GXF) can directly consume images in GPU memory?
  2. Would integrating Isaac ROS Argus Camera provide any advantage, or is it unnecessary since I am using the ZED SDK?

I would appreciate any insights from those who have experience optimizing ROS2 image pipelines for GPU processing.

Thanks in advance!

2

Hi @ba2sakal

No, this is not enough. ROS 2 publishing does not allow GPU memory data transport unless you integrate NVIDIA Nitros.
The ZED ROS2 Wrapper does not allow this yet.

You will lose the ZED SDK features, first of all the factory camera frame rectification.

3

Thanks for the answer @Myzhar ! Indeed ZED SDK is very useful with the great Depth Engine, wouldn’t like to lose it.

Just to clarify, if I integrate NVIDIA NITROS into the ZED ROS2 Wrapper, would that be enough to retrieve images directly into GPU memory and transport them to other Isaac ROS packages without going through the CPU?

Thanks again!

4

Yes, that should be the case.
We have not yet tested it internally, so I cannot be 100% sure that it will be straightforward to integrate it

1 Like
5

Perfect, I will try and would post my results here. Thanks a lot!