I am currently working with a ZED 2i camera and ZED Box Mini for a robotic environmental sensing project. My goal is to use the ZED 2i to support localisation and 3D mapping, then combine this with external sensor data such as CO₂, NO₂, O₃, humidity, temperature, and later methane gas readings.
I understand that the ZED 2i can provide positional tracking, where the camera trajectory can be estimated using visual-inertial odometry and IMU data. I have also seen examples showing camera path tracking, point cloud generation, 3D perception, indoor localisation, and occupancy mapping.
I would like to ask more specifically about the relationship between the ZED point cloud and localisation data:
Does the point cloud generated by the ZED SDK contain real 3D coordinate information?
If I record the camera pose over time using positional tracking, can I use this pose information to align external sensor readings with the point cloud/map?
For example, if my gas sensor records CO₂ or humidity at a specific timestamp, can I match that timestamp with the ZED camera pose, then place the sensor reading as a coloured 3D point inside the reconstructed environment?
Is there a recommended workflow or example for exporting or accessing both the point cloud/map and the camera trajectory together?
My goal is to create a 3D environmental sensing map with colour-coded gas concentration points, as shown in the picture below.
I understand from a previous reply that ZEDfu does not directly support overlaying external sensor data onto the reconstructed 3D map. However, I would like to know whether the ZED SDK provides enough point cloud coordinate and camera pose information for me to build this custom overlay pipeline myself.
Any advice, recommended API functions, example projects, or suggested export formats would be very helpful.
Hi @Justinmars
Thanks for the detailed write-up — your use case is a great fit for the ZED SDK, and yes, everything you want to do is achievable with the data the SDK already exposes.
Yes. Camera::retrieveMeasure(point_cloud, MEASURE::XYZRGBA) returns a 4-channel float Mat where each pixel stores (X, Y, Z, color) in metric units (millimeters by default, configurable via InitParameters::coordinate_units). The values are real metric 3D coordinates, not normalized or relative. The coordinate system is also configurable via InitParameters::coordinate_system (e.g. RIGHT_HANDED_Y_UP for Meshlab-style viewing, or RIGHT_HANDED_Z_UP / IMAGE depending on your downstream tools).
By default the point cloud is expressed in the camera reference frame (origin at the left lens). If you want it directly in the world/map frame, enable positional tracking first and the SDK will keep the relationship consistent — but for your use case it’s usually cleaner to retrieve the cloud in CAMERA frame and transform it explicitly with the pose (see below), so you have full control over the world frame definition.
Yes — this is exactly what positional tracking is for.
The pose gives you the rigid transform T_world_camera at the exact image timestamp. If your gas sensor is rigidly mounted on the same platform, the workflow is:
Measure the static extrinsic T_camera_gasSensor (offset/orientation of the sensor relative to the ZED left lens). Just an offset if the sensor is point-like, no orientation needed.
The gas sample position in world frame is the translation part of that result.
Fully doable. Two things to be aware of:
Pose::timestamp is the image capture timestamp in nanoseconds, on the same monotonic clock the SDK uses internally. To compare against your gas sensor’s clock, either timestamp gas readings with the same system clock, or do a one-time offset calibration. If your sensor pipeline is in ROS 2 (which I see you already use), use a message_filters::ApproximateTime synchronizer between the pose topic and the gas-sensor topic — that handles the matching for you with a configurable slop window.
Gas concentration fields are slow-varying compared to camera frame rate, so nearest-neighbor in time within ~100 ms is usually fine. Don’t over-engineer the sync.
Once a reading is matched to a pose, you produce one coloured point: position from the composed transform above, color from a colormap applied to the concentration value (jet/viridis/turbo — viridis is the safest perceptually). Accumulate these in a separate “sensor layer” cloud (a pcl::PointCloud<pcl::PointXYZRGB> or a ROS 2 sensor_msgs/PointCloud2) — keep it logically separate from the structural ZED point cloud so you can toggle/recolor it independently.
For your goal (one consolidated 3D environmental map with gas overlay), I’d recommend the Spatial Mapping module rather than dumping raw per-frame point clouds — it gives you a clean, deduplicated map of the environment, fused across the trajectory.
SpatialMappingParameters params;
params.map_type = SpatialMappingParameters::SPATIAL_MAP_TYPE::FUSED_POINT_CLOUD;
params.resolution_meter = 0.05f; // tune to your environment
zed.enablePositionalTracking();
zed.enableSpatialMapping(params);
// ... grab() loop, the SDK ingests frames automatically ...
sl::FusedPointCloud fpc;
zed.extractWholeSpatialMap(fpc);
fpc.save("environment.ply", MESH_FILE_FORMAT::PLY);
Use FUSED_POINT_CLOUD (not MESH) — points are what you’ll be overlaying gas data onto, and a mesh just gets in the way. Save as PLY rather than OBJ; PLY round-trips cleanly with FusedPointCloud::load() if you ever need to reload, and OBJ has known issues there.
For the camera trajectory, log pose + timestamp every frame you call getPosition() to a CSV or a TUM-format trajectory file. Then your offline pipeline becomes:
Offline script: time-interpolate trajectory at each gas timestamp (SLERP for orientation, linear for translation), apply static extrinsic, color-code, append to a second PLY (gas_overlay.ply).
Load both PLYs in CloudCompare / Meshlab / Open3D / RViz2 to visualize together.
This decouples acquisition from visualization and lets you re-render the gas overlay with different colormaps or thresholds without re-recording.
ROS 2 note (since you’re already there)
The zed_ros2_wrapper publishes everything you need out of the box:
/zed/zed_node/point_cloud/cloud_registered — live point cloud in the configured frame
/zed/zed_node/pose and /tf (map → camera_link) — pose with synchronized stamps
/zed/zed_node/mapping/fused_cloud — the fused spatial map
Publish your gas data on a separate topic with proper header.stamp values, set up a TF from base_link to gas_sensor_link, and you can compose the world-frame gas points with a tiny node that subscribes to /tf + gas topic. Record the whole thing as an mcap rosbag for offline replay and overlay generation.
Useful sample code references
zed-sdk/spatial mapping/spatial mapping/ (C++/Python) — full fused point cloud sample
zed-sdk/recording/export/ — point cloud and mesh export examples
zed-pcl — direct PCL interop if you prefer building the overlay there
