Quantifying ZED depth offset against solvePnP — looking for reduction strategies

Hardware ZED2
,
1

Hi,

In the future i need to manually combine multiple Zed Cameras, so for starters i though id check the PCD accuracies and also try to find some calibration tool → cv2 checkboard seems useful here

so now i wanted to compare the clean cv2 checkboard → solve PnP position with the physical cornerpoints on the pointcloud and then compare the cornerpoints

in general the script works, but i can see always residuals between the cv2 position and the estimation from depth/pointcloud

Example: at ~1m away there is an offset of around 6mm in Z direction
Trying different depth modes did not change much
And I already tried to do the Zed Calibration but also no better results

Is there anything else i can do to reduce this inaccuracy?

depth_resiudals
depth_resiudals1920×818 192 KB

2

Hi @rokl
Welcome to the StereoLabs community.

The checker applications in the zed-opencv-calibration GitHub repository performs something close to what you want to achieve.

I recommend you analyze this code:

3

Hello there,

Thx for the link
But if i understand it right, this method you proposed ist trying to fit the calibration for a “minimal reprojection error”
Also the checker can only tell the quality
Respective to the 3D Data, my goal is to reduce inaccuracies in depth/pointcloud (noise compensation as much as possible and also systematic bias)
I dont need a quality check of the results, i need a way to improve them
As you can see in my images, there is a gap in 3d between expected checkboard points and the physical points there in the pointcloud and thats what i need to reduce

Currently my only idea is to use RBF correction or mabye a polynom approach, but if there is a better way (best case some simple hidden setting i dont know about yet) i would appreciate it
Or do you think this is already the best i can get out from the Zed2i?

Thx in advance!

4

It’s impossible to say if this is a limitation of the camera without more information concerning the algorithm that you used to perform your point projection.

Can you provide more information?

5

Sorry for the late reply

The dataflow here is:
Checkerboard image → solvePnP → ideal 3D points
Checkerboard image + ZED PCD→ sample Points @ HW idxs → measured 3D points
PNP 3D vs. measured 3D → residual metrics
Visualize (Live or on Trigger)

Here is the Code:

"""
ZED Checkerboard Depth Validation
==================================
Compares the "ideal" board pose (from solvePnP) with the raw depth point cloud
from the ZED camera. Two quality metrics are displayed:

  mean_residual         — mean 3-D distance between PnP corners and ZED cloud (mm)
  mean_residual_debiased — same, after removing the mean systematic offset (mm)

LIVE_MODE = True   → runs continuously, ~2 fps, useful for real-time monitoring
LIVE_MODE = False  → "trigger" mode: grab one frame, show it, wait for the
                     user to close the window, then capture the next frame
"""

import pyzed.sl as sl
import cv2
import numpy as np
import matplotlib.pyplot as plt
import time

# ─────────────────────────────────────────────
# Configuration
# ─────────────────────────────────────────────

LIVE_MODE = True          # True = continuous loop | False = close-window trigger

CHECKERBOARD  = (8, 5)    # inner corner count (cols, rows)
SQUARE_SIZE   = 236.0 / 9.0  # physical square size in mm

# Median-window half-size: samples a (2W+1)² patch around each corner
DEPTH_PATCH_HALF_SIZE = 1         # → 3×3 patch

# Only used in LIVE_MODE to cap CPU usage (~2 fps)
LIVE_SLEEP = 0.08      # seconds between frames


# ─────────────────────────────────────────────
# ZED camera initialisation
# ─────────────────────────────────────────────

zed = sl.Camera()

init_params                     = sl.InitParameters()
init_params.camera_resolution   = sl.RESOLUTION.HD2K
init_params.depth_mode          = sl.DEPTH_MODE.NEURAL
init_params.coordinate_units    = sl.UNIT.MILLIMETER
init_params.depth_stabilization = 0  

if zed.open(init_params) != sl.ERROR_CODE.SUCCESS:
    raise RuntimeError("Failed to open ZED camera")

runtime = sl.RuntimeParameters()
if hasattr(sl, "SENSING_MODE"):
    runtime.sensing_mode = sl.SENSING_MODE.STANDARD


# ─────────────────────────────────────────────
# Camera intrinsics (left eye)
# ─────────────────────────────────────────────

calib  = zed.get_camera_information().camera_configuration.calibration_parameters
fx, fy = calib.left_cam.fx, calib.left_cam.fy
cx, cy = calib.left_cam.cx, calib.left_cam.cy

print(f"ZED intrinsics  fx={fx:.2f}  fy={fy:.2f}  cx={cx:.2f}  cy={cy:.2f}")

camera_matrix = np.array([[fx, 0, cx],
                           [0, fy, cy],
                           [0,  0,  1]], dtype=np.float64)

distortion_coeffs = np.array(calib.left_cam.disto) if hasattr(calib.left_cam, "disto") else None


# ─────────────────────────────────────────────
# ZED data containers
# ─────────────────────────────────────────────

zed_image       = sl.Mat()
zed_point_cloud = sl.Mat()


# ─────────────────────────────────────────────
# 3-D object points for the checkerboard model (mm)
# Layout: row-major grid, Z=0 (flat board assumption)
# ─────────────────────────────────────────────

board_model_pts = np.zeros((CHECKERBOARD[0] * CHECKERBOARD[1], 3), np.float32)
board_model_pts[:, :2] = np.mgrid[0:CHECKERBOARD[0],
                                   0:CHECKERBOARD[1]].T.reshape(-1, 2)
board_model_pts *= SQUARE_SIZE


# ─────────────────────────────────────────────
# Matplotlib figure setup
# ─────────────────────────────────────────────

# Dark background gives better contrast for the coloured scatter points
plt.style.use("dark_background")

plt.ion()
fig = plt.figure(figsize=(14, 6))
fig.suptitle("ZED · Checkerboard Depth Validation", fontsize=13,
             color="#e0e0e0", fontweight="bold")

ax_image  = fig.add_subplot(1, 2, 1)
ax_scatter3d = fig.add_subplot(1, 2, 2, projection="3d")

fig.tight_layout(rect=[0, 0, 1, 0.95])


# ─────────────────────────────────────────────
# Helper: robust depth sampling
# ─────────────────────────────────────────────

def sample_depth_patch(point_cloud_data, pixel_x, pixel_y, half_window):
    """
    Return the median (X, Y, Z) in a (2·half_window+1)² patch around pixel (pixel_x, pixel_y).
    Ignores pixels with non-finite or non-positive depth.
    Returns None when no valid sample is found in the patch.
    """
    img_h, img_w = point_cloud_data.shape[:2]
    patch_x_vals, patch_y_vals, patch_z_vals = [], [], []

    for row_offset in range(-half_window, half_window + 1):
        for col_offset in range(-half_window, half_window + 1):
            sample_x = pixel_x + col_offset
            sample_y = pixel_y + row_offset
            if 0 <= sample_x < img_w and 0 <= sample_y < img_h:
                point = point_cloud_data[sample_y, sample_x]
                if np.isfinite(point[2]) and point[2] > 0:
                    patch_x_vals.append(point[0])
                    patch_y_vals.append(point[1])
                    patch_z_vals.append(point[2])

    if not patch_z_vals:
        return None
    return [np.median(patch_x_vals), np.median(patch_y_vals), np.median(patch_z_vals)]


# ─────────────────────────────────────────────
# Main loop
# ─────────────────────────────────────────────

while True:

    # ── Grab frame ──────────────────────────────
    if zed.grab(runtime) != sl.ERROR_CODE.SUCCESS:
        continue

    zed.retrieve_image(zed_image, sl.VIEW.LEFT)
    zed.retrieve_measure(zed_point_cloud, sl.MEASURE.XYZ)

    raw_frame        = zed_image.get_data().copy()
    point_cloud_data = zed_point_cloud.get_data().copy()

    frame_bgr  = cv2.cvtColor(raw_frame, cv2.COLOR_BGRA2BGR)
    frame_gray = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2GRAY)


    # ── Checkerboard detection ───────────────────
    board_found, detected_corners = cv2.findChessboardCorners(
        frame_gray,
        CHECKERBOARD,
        flags=cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE,
    )

    zed_corner_pts   = []
    pnp_corner_pts   = None
    mean_residual         = None
    mean_residual_debiased = None

    if board_found:
        # Sub-pixel refinement for corner accuracy
        detected_corners = cv2.cornerSubPix(
            frame_gray, detected_corners, (11, 11), (-1, -1),
            (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001),
        )

        # ── solvePnP → ideal board pose ─────────────
        _, rotation_vec, translation_vec = cv2.solvePnP(
            board_model_pts, detected_corners, camera_matrix, distortion_coeffs
        )
        rotation_mat, _ = cv2.Rodrigues(rotation_vec)

        # Transform flat board model corners into camera space
        pnp_corner_pts = (rotation_mat @ board_model_pts.T).T + translation_vec.reshape(1, 3)

        # ── ZED point cloud sampling ─────────────────
        # For every detected corner: take the median of a small patch to
        # suppress depth noise and stereo matching artefacts
        for (corner_x, corner_y) in detected_corners.reshape(-1, 2):
            sampled_pt = sample_depth_patch(
                point_cloud_data, int(corner_x), int(corner_y), DEPTH_PATCH_HALF_SIZE
            )
            if sampled_pt is not None:
                zed_corner_pts.append(sampled_pt)

        zed_corner_pts = np.array(zed_corner_pts)

        # ── Quality metrics ──────────────────────────
        if len(zed_corner_pts) == len(pnp_corner_pts):
            per_corner_residuals = zed_corner_pts - pnp_corner_pts
            per_corner_distances = np.linalg.norm(per_corner_residuals, axis=1)

            # mean_residual — raw mean 3-D distance; captures offset + shape error
            mean_residual = np.mean(per_corner_distances)

            # mean_residual_debiased — removes the mean systematic offset first,
            # so only shape/warp error remains (ignores a global depth shift)
            systematic_offset = np.mean(per_corner_residuals, axis=0)
            debiased_residuals = per_corner_residuals - systematic_offset
            mean_residual_debiased = np.mean(np.linalg.norm(debiased_residuals, axis=1))


    # ── 2-D overlay ─────────────────────────────
    display_frame = frame_bgr.copy()

    if board_found:
        status_text  = f"CHECKERBOARD FOUND  ({len(detected_corners)} pts)"
        status_color = (80, 220, 80)
        for (corner_x, corner_y) in detected_corners.reshape(-1, 2):
            cv2.circle(display_frame, (int(corner_x), int(corner_y)), 6, (80, 220, 80), -1)
    else:
        status_text  = "CHECKERBOARD NOT FOUND"
        status_color = (60, 60, 220)

    cv2.putText(display_frame, status_text, (20, 45),
                cv2.FONT_HERSHEY_SIMPLEX, 1.1, status_color, 2, cv2.LINE_AA)

    if mean_residual is not None:
        mode_tag = "LIVE" if LIVE_MODE else "TRIGGER"
        cv2.putText(
            display_frame,
            f"residual: {mean_residual:.2f} mm   debiased: {mean_residual_debiased:.2f} mm   [{mode_tag}]",
            (20, 90),
            cv2.FONT_HERSHEY_SIMPLEX, 0.85, (240, 220, 50), 2, cv2.LINE_AA,
        )


    # ── Matplotlib update ────────────────────────

    # Left panel — annotated camera image
    ax_image.clear()
    ax_image.imshow(cv2.cvtColor(display_frame, cv2.COLOR_BGR2RGB))
    ax_image.set_title("Camera view  (left eye)", color="#c8c8c8", pad=8)
    ax_image.axis("off")

    # Right panel — 3-D scatter comparison
    ax_scatter3d.clear()
    ax_scatter3d.set_facecolor("#111111")

    if len(zed_corner_pts) > 0:
        ax_scatter3d.scatter(
            zed_corner_pts[:, 0], zed_corner_pts[:, 1], zed_corner_pts[:, 2],
            c="#4fc3f7", s=40, label="ZED depth  (median patch)",
            depthshade=True, alpha=0.9,
        )

    if pnp_corner_pts is not None:
        ax_scatter3d.scatter(
            pnp_corner_pts[:, 0], pnp_corner_pts[:, 1], pnp_corner_pts[:, 2],
            c="#a5d6a7", s=40, marker="^", label="PnP board  (ideal)",
            depthshade=True, alpha=0.9,
        )

    ax_scatter3d.set_xlabel("X (mm)", labelpad=6, color="#aaaaaa")
    ax_scatter3d.set_ylabel("Y (mm)", labelpad=6, color="#aaaaaa")
    ax_scatter3d.set_zlabel("Z (mm)", labelpad=6, color="#aaaaaa")
    ax_scatter3d.tick_params(colors="#888888")
    ax_scatter3d.set_title("3-D comparison", color="#c8c8c8", pad=8)

    legend = ax_scatter3d.legend(loc="upper left", fontsize=8,
                                 framealpha=0.3, edgecolor="#555555")
    for legend_label in legend.get_texts():
        legend_label.set_color("#dddddd")

    # Metric annotation inside the 3-D panel
    if mean_residual is not None:
        ax_scatter3d.text2D(
            0.02, 0.06,
            f"residual: {mean_residual:.2f} mm\ndebiased: {mean_residual_debiased:.2f} mm",
            transform=ax_scatter3d.transAxes,
            color="#ffe082", fontsize=9, va="bottom",
            bbox=dict(boxstyle="round,pad=0.3", fc="#1a1a1a", ec="#555555", alpha=0.8),
        )

    plt.pause(0.001)


    # ── Frame-rate / trigger control ─────────────
    if LIVE_MODE:
        # Continuous mode: just throttle to ~2 fps
        time.sleep(LIVE_SLEEP)
    else:
        # Trigger mode: block until the user closes the figure window,
        # then immediately re-open it for the next capture.
        plt.ioff()
        plt.show(block=True)   # blocks here; user closes window to advance
        plt.ion()
        # Recreate the figure so the next frame has a fresh window
        fig = plt.figure(figsize=(14, 6))
        fig.suptitle("ZED · Checkerboard Depth Validation", fontsize=13,
                     color="#e0e0e0", fontweight="bold")
        ax_image     = fig.add_subplot(1, 2, 1)
        ax_scatter3d = fig.add_subplot(1, 2, 2, projection="3d")
        fig.tight_layout(rect=[0, 0, 1, 0.95])


# ─────────────────────────────────────────────
# Cleanup
# ─────────────────────────────────────────────
zed.close()
6

Hi @rokl,

Thanks for the detailed dataflow — that’s exactly what we needed. The key thing to flag is that your script is comparing two independent estimates and implicitly treating solvePnP as ground truth. It isn’t, and that’s most likely where your 6 mm comes
from.

Why PnP can’t be your reference for absolute depth

For a roughly fronto-parallel board, solvePnP’s distance is essentially:

t_z ≈ f · S / s

where f is the focal length (px), s is the corner pitch the camera actually measured (px), and S is the square size you typed in (mm). The pixel pitch is real data — the only inputs are the focal length and your square size. So PnP’s absolute scale is
locked to those two numbers:

PnP_Z_error / Z ≈ (S_assumed − S_true)/S_true + (f_assumed − f_true)/f_true

A 0.6% error in either reproduces your 6 mm at 1 m, with zero contribution from the depth sensor. Your SQUARE_SIZE = 236/9 = 26.22 mm looks like a single tape measurement: ±1 mm over that span is already 0.42% → ~4 mm at 1 m, before printer scaling and
paper flatness. And 6 mm / 1000 mm = 0.6%, which is inside the ZED 2i’s depth-accuracy envelope at this range.

This is also why an RBF/polynomial correction is premature: you’d be fitting depth to a reference that is itself uncertain at the ~0.6% level — “improving” the residual without improving the depth.

What we’d recommend, in order (stop when the offset is explained):

  1. Measure the square size with calipers (across many squares, then divide) and re-run. This is the highest-leverage step — PnP distance scales 1:1 with it.
  2. Add a real reference: place the board at a tape/laser-measured distance and compare both PnP-Z and ZED-Z to that number. This tells you which side actually owns the offset.
  3. Use a plane-to-plane metric, not corner-to-corner. Checkerboard corners are the worst place to sample stereo depth (high-contrast discontinuity). Fit a plane to the ZED cloud over the board — sampling square centres — and compare it to the PnP
    plane: signed offset along the normal + tilt angle. This removes corner-localization noise and isolates true systematic bias.
  4. Only if a curved, distance-dependent bias remains (sweep 0.5/1.0/1.5/2.0 m) is a per-camera correction justified — and it should be fit against the measured reference, never against PnP.
    Our expectation: once the square size is verified and you compare against a measured distance, most of the 6 mm will resolve to board/scale, and the residual depth bias should land near spec — at which point a correction is optional rather than
    necessary.
7

Hi @TristanLed

Thanks for the detailed reply.

I did fuhter tests and here is the results:

1) Checkerboard & PnP:

The square size was measured with calipers → 236.26 (Software Ideal = 236.28)
→ since this size affects the accuracy directly, i validated this by iterating through the options:

## symbolic code:
best_bias = 999999
best_size = 0
for size in create_linspace(233, 239, step 0.01):
    local_bias = []
    for n in range(5):
       bias = capture_and_calculate()
       local_bias.append(bias)
    b = np.mean(sorted(local_bias)[-3:])
    if b > best_bias :
      best_bias = b
      best_size size

Putting 236.28 in did in fact alreadey make already a differece - avg diff is now ~2.5

So to better understand the possible influence of corner localization on the solvePnP result, I also estimated the metric size of one image pixel at a working distance of approximately 1 meter:

since ZED2i intrinsics (fx/fy i susually in the 1900 range @ HD2K) the scale is approximately
fx = 1900 px → 1000 mm / 1900 ≈ 0.53 mm per pixel

This means that even a 1-pixel localization error corresponds to only about 0.5 mm in object space at 1 meter. Typical checkerboard corner localization after sub-pixel refinement is considerably better than one pixel, so the expected contribution of image localization to the final pose estimate should normally be well below 1 mm.
I think Im on the right rack, but there still seems to besomething off a little.

2) Physical ground-truth distance

I like the idea of comparing both solvePnP and the depth map against an external ground-truth distance. The practical difficulty however is defining that reference with precision.

Since the camera origin is an internal point of the optical system, there is no physically accessible location on the camera housing from which the exact optical distance can be measured. Even if I use a laser distance meter or calipers, I do not know the precise offset between the outside of the camera enclosure and the actual camera coordinate origin used by the calibration model. At millimeter level accuracy, that uncertainty already becomes significant.

Instead, I tried to validate the depth measurements independently of the camera origin. I measured several objects with known physical dimensions at different working distances. For example, objects at approximately 400–600 mm as well as a box around ~3 meters away with a known length of roughly 1.5 meters. In both cases the measured dimensions were within approximately 0.2% of the real values, which gives me confidence that the ZED measurements are generally well scaled.

3) Plane Fit

Based on your recommendation, I updated my validation script to compare the overall geometry rather than only the raw corner correspondences.
The script now fits a plane.

  • After alignment, the residual distance between the fitted and the measured 3D is typically below 1 mm.
  • The indiviudal point to fitted offsets are <= 1mm
  • We Still have a bias of 2 mm between the planes

4) Polynomial depth correction

I also did an additional experiment to investigate whether the observed offset behaves like a systematic and could be globally reduced, so I recorded approximately 50 checkerboard poses over different distances and angles.
Using all this Data I fitted a second-order polynomial correction model for each camera

def phi(P: np.ndarray) -> np.ndarray:
    x, y, z = P[:, 0], P[:, 1], P[:, 2]

    return np.column_stack([
        np.ones_like(x),   # bias
        x, y, z,           # linear
        x*x, y*y, z*z,     # quadratic
        x*y, x*z, y*z      # cross
    ])

def fit_polynomial(P_fit: np.ndarray, P_pnp: np.ndarray) -> np.ndarray:
    """
    Learns correction:
        P_target ≈ P_src + φ(P_src) @ W
    """
    Delta = P_pnp[:, :3] - P_fit[:, :3]
    W, *_ = np.linalg.lstsq(
        phi(P_fit[:, :3]),
        Delta,
        rcond=None
    )
    return W

def apply_correction(P, W, scale=1.0):
    W = W.reshape((10, 3))
    P = P.copy()
    P_xyz = P[:, :3] / scale
    xyz_corr = phi(P_xyz) @ W

    P_corr = P.copy()
    P_corr[:, :3] /= scale
    P_corr[:, :3] += xyz_corr
    P_corr[:, :3] *= scale
    return P_corr

Long Story Short:
after new measurement → bettter results
after applying Polynom correction → bias went down singificant
So at least we know, its not any combination of random tolerances or just noise, its defenitly some systematic value - either im alredy hitting the general accuracy limits here or the whole checkerboard-PNP thing is really that sensitive
I’d prefer here to have at least a solid board wiht precision (eg CNC produced - but i guess thats a little to expensive atm for just experimenting)

Since my final goal is to combine multiple Zed Pointclouds - what should i aim for here?
The whloe reason for these tests here ist that I wanted to check the start condition to estimate if i will get a problem and to reduce that impact where possible

So the worst case would be to have two cams, and in their overlap areas the objects in the pointcloud are projected with different “bias”, leading to have the physical “smooth” object to have visible levelwise edges in the capture when i combine all the points:

zed_overlap
zed_overlap1128×919 28.6 KB

Also this alone would then indicate the question: if 2 cameras see different things - how good is the frame in general?
So the calibration here is somthing i should be really careful about.