Quantifying ZED depth offset against solvePnP — looking for reduction strategies

Hi,

In the future i need to manually combine multiple Zed Cameras, so for starters i though id check the PCD accuracies and also try to find some calibration tool → cv2 checkboard seems useful here

so now i wanted to compare the clean cv2 checkboard → solve PnP position with the physical cornerpoints on the pointcloud and then compare the cornerpoints

in general the script works, but i can see always residuals between the cv2 position and the estimation from depth/pointcloud

Example: at ~1m away there is an offset of around 6mm in Z direction
Trying different depth modes did not change much
And I already tried to do the Zed Calibration but also no better results

Is there anything else i can do to reduce this inaccuracy?