Hi There,
We are working on calibrating ZED360 with two or more ZED X One 4K cameras. All of these cameras are in a stereo setup, using the suggested checkerboard methods for calibration.
For each pair of stereo cameras, we connect them to a ZED Box Orin and use the box’s ZED_Media_Server to stream to another centralized machine for depth handling and body parsing. We also tested with the ZED_Depth_Viewer and the body tracking demo from the SDK examples on the centralized machine with streaming, and everything looks good. We can obtain bounding boxes, positions, keypoints, etc.
The problem occurs when we run multiple instances of the body tracking code (attached) with different stereo cameras. We use start_publishing() on the local network to send the skeleton information to ZED360. However, ZED360 only recognizes one skeleton (or possibly multiple identical skeletons in the same place) instead of multiple skeletons from each camera at different locations, and the calibration never starts. Please refer to the attached screenshot and sample video. I checked with Wireshark, and the UDP connection seems good.
There is also a screen capture video of ZED360 for preview.
We attempted to use the exact same code but switched the camera reading procedure from streaming to a USB connection with two ZED 2i cameras, and everything worked correctly. The only difference is that the testing area for the ZED X One 4K is larger.
I am wondering if there is anything specific to the ZED X One 4K cameras for setting up ZED360. We have tried different depth models, body tracking parameters, and runtime parameters, and we are sure they are not the cause of the issue.
The SDK version we tested is 5.0.5.
import pyzed.sl as sl
import argparse
from config import camera_configs, serial_numbers
def main():
# Set up argument parser for camera index
parser = argparse.ArgumentParser()
parser.add_argument('--index', type=int, choices=[0, 1, 2, 3], required=True,
help='Camera index (0, 1, 2, or 3)')
args = parser.parse_args()
# Use the provided index instead of a loop
index = args.index
if index >= len(camera_configs):
print(f"Error: Camera index {index} out of range. Only {len(camera_configs)} cameras configured.")
return
init_params = sl.InitParameters()
cam = sl.Camera()
# Parse IP and port from config for the selected camera
ip = camera_configs[index].split(':')[0]
port = int(camera_configs[index].split(':')[1])
init_params.camera_resolution = sl.RESOLUTION.HD2K # Use HD1080 video mode
init_params.coordinate_units = sl.UNIT.METER # Set coordinate units
init_params.depth_mode = sl.DEPTH_MODE.NEURAL
init_params.coordinate_system = sl.COORDINATE_SYSTEM.RIGHT_HANDED_Y_UP
#init_params.depth_maximum_distance = 5.0
#init_params.depth_minimum_distance = 0.5
init_params.depth_stabilization = 75
# Set camera to stream from the IP:port in config
#init_params.set_from_stream(ip, port)
print(f"{serial_numbers[index]}")
init_params.set_from_serial_number(serial_numbers[index], bus_type=sl.BUS_TYPE.USB)
status = cam.open(init_params)
if status != sl.ERROR_CODE.SUCCESS:
print(f"Camera {index + 1} open failed with error code: {repr(status)}")
return
communication_parameters = sl.CommunicationParameters()
communication_parameters.set_for_local_network(30000+10*index, "0.0.0.0")
cam.start_publishing(communication_parameters)
positional_tracking_parameters = sl.PositionalTrackingParameters()
# If the camera is static, uncomment the following line to have better performances
positional_tracking_parameters.set_as_static = True
cam.enable_positional_tracking(positional_tracking_parameters)
body_param = sl.BodyTrackingParameters()
body_param.enable_tracking = True # Track people across images flow
body_param.enable_body_fitting = True # Smooth skeleton move
body_param.detection_model = sl.BODY_TRACKING_MODEL.HUMAN_BODY_ACCURATE
body_param.body_format = sl.BODY_FORMAT.BODY_18 # Choose the BODY_FORMAT you wish to use
cam.enable_body_tracking(body_param)
runtime_params = sl.RuntimeParameters()
#runtime_params.enable_fill_mode = True
#runtime_params.confidence_threshold = 95
#runtime_params.remove_saturated_areas = True
body_runtime_param = sl.BodyTrackingRuntimeParameters()
body_runtime_param.detection_confidence_threshold = 50
image = sl.Mat()
bodies = sl.Bodies()
try:
while True:
# Capture frames from the camera
if cam.grab(runtime_params) == sl.ERROR_CODE.SUCCESS:
#cam.retrieve_image(image, sl.VIEW.LEFT)
cam.retrieve_bodies(bodies, body_runtime_param)
timestamp = cam.get_timestamp(sl.TIME_REFERENCE.CURRENT)
#print(f"Camera {index + 1} - Image resolution: {image.get_width()} x {image.get_height()} || "
#f"Image timestamp: {timestamp.get_milliseconds()}")
print(f"Camera {index + 1} - Number of bodies detected: {bodies.body_list} || "
f"Timestamp: {timestamp.get_milliseconds()} ms")
except KeyboardInterrupt:
pass
# Stop publishing and close the camera
cam.stop_publishing()
cam.close()
if __name__ == "__main__":
main()
