ZED X One Stereo, images steaming and image IMU data recording

Software
1

Hi,

I am struggling with understanding the image streaming and data recording functionalities for the ZED X One Stereo pairs and would greatly appreciate your assistance.

1. Image Streaming

I have successfully configured the ZED X One camera pairs in stereo mode, and the ZED_Media_Server software operates flawlessly. As I understand, to initiate image streaming, I must first open the software to set up the virtual camera S/N and start streaming. The software can then be closed as a background service will continue running. However, I plan to use the NVIDIA Jetson in headless setup, i.e. without monitors, keyboards, which necessitats that image streaming starts automatically upon power-up. Are there any APIs available that could automate this setup process instead of requiring manual configuration through the software?

2. Timestamp for Images

The following Python code snippet from image_capture.py captures images but I’m unclear about the timestamp logic:

while i < 50:
    if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
        zed.retrieve_image(image, sl.VIEW.LEFT)
        timestamp = zed.get_timestamp(sl.TIME_REFERENCE.CURRENT)  
        print("Image resolution: {0} x {1} || Image timestamp: {2}\n".format(image.get_width(), image.get_height(), timestamp.get_milliseconds()))

Questions:

  1. Why does the official example use TIME_REFERENCE.CURRENT instead of TIME_REFERENCE.IMAGE for left image timestamps?
  2. What is the practical difference between these timestamp references?
  3. For hardware-synchronized stereo pairs via capture card:
  • Should both images share identical TIME_REFERENCE.IMAGE timestamps?
  • Does TIME_REFERENCE.IMAGE represent the exact sensor capture moment?
  1. If I would like to record both left and right images, is the code snippet below correct?
while i < 50:
    if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
        zed.retrieve_image(image_left, sl.VIEW.LEFT)
        zed.retrieve_image(image_right, sl.VIEW.LEFT)
        timestamp = zed.get_timestamp(sl.TIME_REFERENCE.IMAGE)

3. Image and IMU Data Recording

Here is another script from sensor_data.py that I attempted to combine with image recording, but only images are being recorded successfully:

while time.time() - time_0 < 5:
    if zed.get_sensors_data(sensors_data, sl.TIME_REFERENCE.CURRENT) == sl.ERROR_CODE.SUCCESS:
        if ts_handler.is_new(sensors_data.get_imu_data()):
            quaternion = sensors_data.get_imu_data().get_pose().get_orientation().get()
            print(f"\tOrientation: [ Ox: {quaternion[0]}, Oy: {quaternion[1]}, Oz: {quaternion[2]}, Ow: {quaternion[3]} ]")
            linear_acceleration = sensors_data.get_imu_data().get_linear_acceleration()
            print(f"\tAcceleration: [ {linear_acceleration[0]} {linear_acceleration[1]} {linear_acceleration[2]} ] [m/sec^2]")
            angular_velocity = sensors_data.get_imu_data().get_angular_velocity()
            print(f"\tAngular Velocities: [ {angular_velocity[0]} {angular_velocity[1]} {angular_velocity[2]} ] [deg/sec]")
    if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
        zed.retrieve_image(image_left, sl.VIEW.LEFT)
        zed.retrieve_image(image_right, sl.VIEW.RIGHT)
        timestamp = zed.get_timestamp(sl.TIME_REFERENCE.IMAGE)
        print(f"Left Image resolution: {image_left.get_width()} x {image_left.get_height()} || Image timestamp: {timestamp.get_milliseconds()}\n")

Is it feasible to record both IMU and image data in a single script without missing any data? If so, how can this be achieved effectively?

Thanks for your reply in advance.

2

Hi @syc,

Sorry for the late reply.

  1. The samples use TIME_REFERENCE.CURRENT (which uses the host timestamp when calling retrieve) as an example, TIME_REFERENCE.IMAGE can also be used
  2. TIME_REFERENCE.CURRENT uses the host timestamp when calling retrieve, whereas TIME_REFERENCE.IMAGE is the timestamp at image capture
  3. You can find information on time synchronization for ZED X One cameras on this post: https://support.stereolabs.com/hc/en-us/articles/28554661445911-Why-frames-from-different-ZED-X-GMSL2-cameras-connected-to-the-same-capture-card-have-different-timestamps . In your case the timestamps will not necessarily be the same, but we guarantee that both images are captured at the same moment with microsecond precision
  4. You can indeed retrieve the images as such, and save them either with the sl.Mat.write() method, or using another library such as OpenCV

The most effective way to save data from the ZED camera is to use the SVO recording format, more information on this here: Video Recording - Stereolabs

The recording includes video and sensor data, which can then be replayed and analyzed with the ZED SDK, just as if you would be using a live camera feed.

3

Hello @mattrouss ,

thanks for your reply.

Could you also explain the how does the image streaming without the GUI ZED_Media_Server work as I asked in the section Image Streaming?

And also the question about recording images and IMU data within a single python script as I asked in the section Image and IMU Data Recording.

Thanks for your reply in advance.

4

Hi @syc,

The setup process of calibrating the cameras is required to run the ZED X One stereo streaming. However once you have generated your calibration file once for your configuration, it can be used by ZED_Media_Server in CLI to start streaming. This can be done with the following command:

ZED_Media_Server --cli /path/to/config/file

When saving images and IMU data of a live camera, there are many constraints to ensure that the data saving does not slow down the capture rate to save all data without any data loss. A common way to do this is to have the saving done in parallel of the capture thread.

You can implement this yourself, or use our SVO format that has been optimized to write with the least amount of dropped frames or IMU data, given the disk writing speeds and CPU load at the time of capture.