# ZED SDK v4 is out| **URL:** \ **Category:** Announcement\ **Tags:** patch, release, sdk\ **Created:** [April 7, 2023, 4:54pm UTC](https://community.stereolabs.com/t/zed-sdk-v4-is-out/2401 "2023-04-07T16:54:37Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Myzhar](https://community.stereolabs.com/user_avatar/community.stereolabs.com/myzhar/32/11_2.png) [@Myzhar](https://community.stereolabs.com/u/Myzhar)\ **Post date:** [April 7, 2023, 4:54pm UTC](https://community.stereolabs.com/t/zed-sdk-v4-is-out/2401/1 "2023-04-07T16:54:37Z")
# **[ZED SDK v4.0.1](https://www.stereolabs.com/developers/release/)** ### What’s New 2023-03-28 We are excited to announce the release of ZED SDK 4.0, which introduces a range of new features and enhancements to our ZED cameras. Our latest update supports the ZED X and ZED X Mini cameras, designed specifically for autonomous mobile robots in indoor and outdoor environments. We are also introducing an improved NEURAL depth mode, which offers even more accurate depth maps in challenging situations such as low-light environments and textureless surfaces. We are proud to introduce the new **multi-camera Fusion API** , which makes it easier than ever to fuse data coming from multiple cameras. This module handles time synchronization and geometric calibration issues, along with 360° data fusion with noisy data coming from multiple cameras and sensor sources. We believe that these updates will unlock even more potential for our users to create innovative applications that push the boundaries of what is possible with depth-sensing technology. Here’s a closer look at some of the new features in ZED SDK 4.0: ## 4.0.0 (Early Access) #### New Features - ZED X - Added support for the new **ZED X** and **ZED X Mini** cameras. - Introduced new VIDEO\_SETTINGS, RESOLUTION, and sl::InputType parameters to enable users to use the same code for both GMSL or USB 3.0 cameras without any added complexity. - Multi-Camera Fusion - Introduced new Multi-Camera Fusion API. The new module allows seamless synchronization and integration of data from multiple cameras in real time, providing more accurate and reliable information than a single camera setup. Additionally, the Fusion module offers redundancy in case of camera failure or occlusions, making it a reliable solution for critical applications. - The new API can be found in the header sl/Fusion.hpp as sl::Fusion API. In 4.0 EA (Early Access), multi-camera fusion supports multi-camera capture, calibration, and body tracking fusion. All SDK capabilities will be added to the Fusion API in the final 4.0 release. - Geo-tracking - Introduced new Geo-tracking API for accurate global location tracking. During a geo-tracking session, the API can constantly update the device’s position in the real world by combining data from an external GPS and ZED camera odometry as it moves, delivering latitude and longitude with centimeter-level accuracy. - By fusing visual odometry with GPS data, we can compensate for GPS dropouts in challenging outdoor environments and provide more accurate and reliable positioning information in real time. - Depth Perception - Improved NEURAL depth mode which is now more robust to challenging situations such as low-light, heavy compression, noise, and textureless areas such as interior plain walls, overexposed areas, and exterior sky. - Our new ZEDnet Gen 2 AI model now powers the depth sensing module for stereo depth perception, providing enhanced performance. The Neural depth in 4.0 offers a glimpse of what’s to come, as we plan to roll out an even more robust model later in the year. - Body Tracking - Introduced new Body Tracking Gen 2 module. The new module employs ML to infer up to 70 landmarks of a body from a single frame. This new module goes beyond the existing pose models that are trained on only 17 key points and enable localization of a new topology of 38 and 70 human body key points, making it ideal for advanced body tracking use cases. - Added two new Body Tracking models: - Object Detection - Introduced Object Detection as a standalone module, separate from Body Tracking. - Added Concurrent execution of different AI models. Users can now run multiple instances of Object Detection and/or Body Tracking simultaneously. For example, body tracking could be run concurrently with a ball object detection model. The ZED SDK’s object detection model could also run in parallel with a user-defined custom object detection model. There are no limits on the number or type of models that can be run. A new instance\_id parameter has been added for every object detection function. - The new concurrent execution of Object Detection and Body Tracking should enable the creation of accurate digital twins of real-world environments. - Platform Support - Added support for CUDA 11.8. Support for CUDA 12.0 will follow soon. - Added support for Python 3.11. - Added support for the new JetPack 5.1 (L4T 35.2). Support for JetPack 5.1.1 (L4T 35.3) will follow soon. - Dropped support for older JetPack versions (L4T 32.3 to 32.6). ## 4.0.1 #### Bug Fixes - Fixed body tracking fusion with BODY\_FORMAT::BODY\_34. - Fixed inconsistent behavior of BodyTrackingFusionRuntimeParameters::enable\_body\_fitting. - Fixed BodyTrackingFusionRuntimeParameters::skeleton\_smoothing to be applied to BODY\_FORMAT::BODY\_38 and BODY\_FORMAT::BODY\_70. - Camera::getCameraSettings and Camera::setCameraSettings now return ERROR\_CODE for non available settings (particularly with ZED X cameras). - Fixed logger file creation without write permission. #### Support - Jetson running on Jetpack 5.1 (L4T 35.2) now supports ZED X cameras. - Added ZED SDK support for Jetpack 5.1.1 (L4T 35.3), ZED X support coming soon. - Added CUDA 12.0 partial support, CUDA 12.1 general support coming soon. #### Tools - Fixed Calibration tool on Windows. #### Samples - Added Geo Tracking samples. #### Known issues - Calibration tool does not support new ZED Mini with fisheye lenses. ## Major 4.0 API Changes ### Video - Added new camera models MODEL::ZED\_X and MODEL::ZED\_XM. - Added new input type INPUT\_TYPE::GMSL field to sl::DeviceProperties for GMSL cameras (support for GMSL cameras is only available on hosts with Nvidia Jetson SoC). - Added INPUT\_TYPE input\_type field to sl::DeviceProperties struct. - Added Camera reboot non-blocking behavior. This is enabled using the new InitParameters::async\_grab\_camera\_recovery parameter. When the asynchronous automatic camera recovery is enabled and there’s an issue with the communication with the camera, the grab() will exit after a short period of time, and return the new ERROR\_CODE::CAMERA\_REBOOTING error code. The recovery will run in the background until proper communication is restored. The default and previous ZED SDK behavior is synchronous, the grab() function is blocking and will return only once the camera communication is restored or the timeout has been reached. - setFromCameraID and setFromCameraSerialNumber methods from sl::InputType now take an optional BUS\_TYPE to choose between USB or GMSL cameras. When unspecified, the method searches for available USB cameras _first_, then searches for GMSL. - Added new methods getInputType(), getConfiguration(), and isInit() for sl::InputType. - Added a scale utility method to sl::CameraParameters to easily convert camera intrinsic parameters for a given resolution. - Added RESOLUTION:HD1200 (1920x1200) and RESOLUTION::SVGA (960x600) resolutions for ZED X and ZED X Mini cameras. - Added RESOLUTION:AUTO resolution which sets RESOLUTION::HD720 resolution for USB cameras, and RESOLUTION::HD1200 resolution for GMSL cameras. - Added new parameters in VIDEO\_SETTINGS for GMSL cameras only: - EXPOSURE\_TIME: Image sensor exposure time in ms - ANALOG\_GAIN: Analog sensor gain in dB - DIGITAL\_GAIN: Digital ISP gain in dB - AUTO\_EXPOSURE\_TIME\_RANGE: Defines the range of the exposure time in automatic control - AUTO\_ANALOG\_GAIN\_RANGE: Defines the range of sensor gain in automatic control. Min/Max range between [1000 - 16000] mdB. - AUTO\_DIGITAL\_GAIN\_RANGE: Defines the range of digital ISP gain in automatic control. - EXPOSURE\_COMPENSATION: Exposure target compensation made after AE. Reduces the overall illumination by a factor of F-stops. values range is [0 - 100] (mapped between [-2.0, 2.0]). - DENOISING: Defines the level of denoising applied on both left and right images. values range is [0 - 100]. - Camera::getCameraSettings now returns an ERROR\_CODE and uses a reference to retrieve the value. - Camera::setCameraSettings now returns an ERROR\_CODE instead of void. ### Depth - SENSING\_MODE has been removed from the API. To get the same behavior, use the new sl::RuntimeParameters.enable\_fill\_mode parameter. - Improved Fused Point Cloud extraction time by 2.5x. - Improved Positional Tracking loop closure time. - Improved Windows AI model download error handling: when there’s no internet access or if the server can’t be reached the exact error is now displayed. It will also auto-retry an alternative link if the primary server can’t be reached. ### Multi-Camera Fusion API - Added sl::Fusion API in a new header sl/Fusion.hpp. In 4.0 EA (Early Access), multi-camera fusion supports multi-camera capture, calibration, and body tracking fusion. All SDK capabilities will be added to the Fusion API in the final 4.0 release. - The Fusion API introduces _publishers_ and _subscribers_ to exchange data between source cameras and the fusion module. - A _publisher_ will perform its computations and _publish_ its data using the startPublishing() method. - A subscriber will use the subscribe() method from the sl::Fusion class to connect to all of the publishers and start retrieving data. The sl::Fusion.process() method takes care of all the heavy lifting of synchronizing and fusing the data, and produces the enhanced body tracking information in real-time. - Local workflow: sl::Fusion is available for a setup with multiple cameras on a single host, using inter-process communication. The _publishers_ and the single _subscriber_ run on the same machine in this configuration. - Distributed workflow: sl::Fusion is available for a setup with multiple cameras on multiple hosts, communicating on the local network, enabled by [ZED Hub](https://www.stereolabs.com/zed-hub/). Each _publisher_ and the single subscriber run on separate machines such as ZED Boxes, in order to distribute computations with devices on the edge. ### Geo-tracking API - The Geo-tracking API is a new module that brings global-scale location tracking to the ZED SDK. It improves the ZED camera’s positional tracking capabilities by incorporating additional GNSS data to achieve global positioning. This is done by utilizing the new sl::Fusion.ingestGNSSdata method, which integrates the GNSS data with the existing tracking data to create a fused position. The resulting position can be accessed both locally and globally using the getPosition and getGeoPosemethods, respectively. With this capability, users can achieve highly accurate positioning information that can be used in a variety of applications. - Added a GeoPose structure containing the tracked position in global world reference. - Added a GNSSData structure containing information about GNSS input data. - Added structures ECEF, LatLng, and UTM for different GPS/GNSS coordinate systems. - Added getGeoPose and getPosition methods in sl::Fusion that return the fused position in the local and global world frames. - Added Geo2Camera and Camera2Geo methods in sl::Fusion to convert from a position in the global world reference (world map) frame to a position in the local world frame, and vice-versa. ### Body Tracking - Body Tracking is now a standalone module, with its own specific parameters and functions. - The new BODY\_TRACKING\_MODEL enum contains the following models: - HUMAN\_BODY\_FAST: Keypoints based, specific to human skeleton, real-time performance even on Jetson or low-end GPU cards. - HUMAN\_BODY\_MEDIUM: Keypoints based, specific to human skeletons, compromise between accuracy and speed. - HUMAN\_BODY\_ACCURATE: Keypoints based, specific to human skeletons, state-of-the-art accuracy, requires powerful GPU. model now depends both on the accuracy wanted both also the BODY\_FORMAT (see below) - Changed BODY\_FORMAT namings from POSE\_XX to BODY\_XX. The POSE\_18 and POSE\_34 formats have been renamed to BODY\_18 and BODY\_34 respectively. - Compared to ZED SDK 3.8, the HUMAN\_BODY\_FAST and HUMAN\_BODY\_ACCURATE models when using BODY\_18 and BODY\_34 are slower but more precise. Up to 50% more precise for the fast model with a runtime difference of around 2 ms on Jetson Orin. - Added two new human body formats in BODY\_FORMAT: - BODY\_38: Adds additional key points to BODY\_34, specifically on the feet and hands. The body fitting now provides accurate orientations for the feet and hands. - BODY\_70: Adds full hand key points to BODY\_38, 4 key points per finger. - BODY\_18 and BODY\_34: The previous body models are still available - Added the possibility to use multiple body tracking models at once using the instance\_id parameter in sl::BodyTrackingParameters (and in related methods such as Camera::getBodyTrackingParameters, Camera::disableBodyTracking, …). - Added a BODY\_KEYPOINTS\_SELECTION enum to filter key points for specific use cases. Currently available options include BODY\_KEYPOINTS\_SELECTION::FULL and BODY\_KEYPOINTS\_SELECTION::UPPER\_BODY. - Analogous to object detection, body-tracking objects now have their own methods and parameters exposed separately: - Added a new sl::BodyData class which contains detection and tracking information on detected human bodies, such as keypoint, joint, and skeleton root information. - Added a new Camera::retrieveBodies method to retrieve human body detection data after a Camera::grab call, similar to Camera::retrieveObjects. - Added a new sl::Bodies class which is returned from Camera::retrieveBodies, similar to sl::Objects. - Added a new sl::BodyTrackingParameters and sl::BodyTrackingRuntimeParameters classes which define all of the parameters related to body tracking. - Renamed BODY\_PARTS to BODY\_18\_PARTS. - Renamed BODY\_PARTS\_POSE\_34 to BODY\_34\_PARTS. - Added BODY\_38\_PARTS and BODY\_70\_PARTS which list all key points for new body formats. - Renamed BODY\_BONES to BODY\_18\_BONES. - Renamed BODY\_BONES\_POSE\_34 to BODY\_34\_BONES. - Added BODY\_38\_BONES and BODY\_70\_BONES bones containers for new body formats. ### Object Detection - Object detection models have been separated into _object detection models_ and _human body detection models_, in two separate classes: OBJECT\_DETECTION\_MODEL and BODY\_TRACKING\_MODEL respectively. - The new OBJECT\_DETECTION\_MODEL enum contains the following models: - MULTI\_CLASS\_BOX\_FAST: Any objects, bounding box based. - MULTI\_CLASS\_BOX\_MEDIUM: Any objects, bounding box based, a compromise between accuracy and speed. - MULTI\_CLASS\_BOX\_ACCURATE: Any objects, bounding box based, more accurate but slower than the base model. - PERSON\_HEAD\_BOX\_FAST: Bounding Box detector specialized in the person’s head, particularly well suited for crowded environments, the person localization is also improved. - PERSON\_HEAD\_BOX\_ACCURATE: Bounding Box detector specialized in the person’s head, particularly well suited for crowded environments, the person localization is also improved, more accurate but slower than the base model. - CUSTOM\_BOX\_OBJECTS: For external inference, by using your own custom model and/or frameworks. This mode disables the internal inference engine, the 2D bounding box detection must be provided. - Added the possibility to use multiple object detection models instances at once using the instance\_id parameter in sl::ObjectDetectionParameters (and in related methods such as Camera::getObjectDetectionParameters, Camera::disableObjectDetection, …). - Renamed MULTI\_CLASS\_BOX to MULTI\_CLASS\_BOX\_FAST in OBJECT\_DETECTION\_MODEL. - Renamed PERSON\_HEAD\_BOX to PERSON\_HEAD\_BOX\_FAST in OBJECT\_DETECTION\_MODEL. - Renamed enable\_mask\_output to enable\_segmentation in sl::ObjectDetectionParameters. - Removed keypoint\_2d, keypoint, keypoint\_confidence, local\_position\_per\_joint, local\_orientation\_per\_joint, and global\_root\_orientation from sl::ObjectData. These attributes have been moved to the sl::BodyData class for Body Tracking. ### Samples - Added Multi Camera recording sample. - Added new Fusion samples to get started with the new Fusion API and fuse body tracking from multiple ZED Cameras. - Added new Geo-tracking samples, a recording, and a playback sample for offline data recording and visualization. - Updated Body Tracking samples with new AI models. All samples are available in your ZED installation folder and on our Github. ### Integrations #### Python, C#, and C wrappers - Updated wrappers with ZED SDK 4.0’s new features and improvements. - Updated [documentation](https://www.stereolabs.com/docs/tutorials/), [tutorials](https://github.com/stereolabs/zed-examples/tree/master/tutorials), and [samples](https://github.com/stereolabs/zed-examples) accordingly. #### Unity - Added support for ZED SDK 4.0. - Added support of BODY\_38 and BODY\_70 in the Body Tracking scene. - The Fusion API is not available in the Unity plugin for the moment. - Use Live Link for Unity to fuse skeletons from multiple cameras in Unity. - Get started with our [Unity plugin](https://github.com/stereolabs/zed-unity) alongside its [documentation](https://www.stereolabs.com/docs/unity/) #### Live Link for Unity - Introducing a new sample that sends Body tracking data from a C++ sample to Unity via UDP. This sample supports single-camera setups as well as multiple-camera setups using the Fusion API. - Get started with our [Live Link for Unity sample](https://github.com/stereolabs/zed-unity) alongside its [documentation](https://www.stereolabs.com/docs/livelink/livelink-unity/). #### Unreal Engine - Added support for ZED SDK 4.0. - Added support of BODY\_38 and BODY\_70 in the Body Tracking level. - The Fusion API is not available in the UE5 plugin for the moment. - Use Live Link for Unreal Engine 5 to fuse skeletons from multiple cameras in UE5. - Get started with our [Unreal Engine 5 plugin](https://github.com/stereolabs/zed-UE5) alongside its [documentation](https://www.stereolabs.com/docs/ue5/) - Deprecated the custom Unreal Engine 4 build integration. It will not support ZED SDK 4.0. #### Live Link for Unreal Engine 5 - Added support for ZED SDK 4.0. - Updated the Live link sample to support both single-camera setups and multiple-camera setups using the Fusion API. - Get started with our [Live Link for Unreal Engine 5 sample](https://github.com/stereolabs/zed-livelink-plugin) alongside its [documentation](https://www.stereolabs.com/docs/livelink/). #### ROS 2 - Added support for ZED X and ZED X Mini. - Added support for ZED SDK 4.0. Fusion API integration coming soon with native multi-camera support. - Improved Positional Tracking module. - Added the parameter pos\_tracking.set\_as\_static for applications with a static camera monitoring a robotics environment. - New message on topic ~/pose/status with the current status of the pose from the ZED SDK. - New message on topic ~/odom/status with the current status of the odometry from the ZED SDK. - Added support for the Geo-tracking API fusion module. - Added support for the new Body Tracking module. - The RVIZ2 plugin supports the new BODY\_38 and BODY\_70 skeleton models. - Improved Object Detection module. - Added the parameter object\_detection.allow\_reduced\_precision\_inference to allow inference to run at a lower precision to improve runtime and memory usage. - Added the parameter object\_detection.max\_range to define an upper-depth range for detections. - Added examples of Dockerfile to create Docker images for ROS 2 Humble on Desktop PC and NVIDIA Jetson devices. - New examples and tutorials. - Other new features and improvements. Full changelog [available on GitHub](https://github.com/stereolabs/zed-ros2-wrapper/blob/master/CHANGELOG.rst). ### Known issues - Maxwell and Kepler GPUs (compute capabilities 35, 50, and 52) are not supported using CUDA 11.x, only with CUDA 10.2 - Windows installers are not signed due to certificate expiry, this should be resolved for the next releases.