Global Localization (GNSS Fusion) fails with DEPTH_MODE.NONE even though GEN_3

Software Python
, ,
1

First of all, thank you very much for the great SDK and for maintaining this community — it’s been a pleasure to build with.

Note: part of the experimentation/verification below was carried out with the help of Claude Code, and this post itself was translated/drafted from my original question with AI assistance. Sorry if any part reads a bit awkwardly — please feel free to ask me to clarify anything.

Summary

I’m trying to minimize GPU/CPU load by disabling depth computation (DEPTH_MODE.NONE) while still using POSITIONAL_TRACKING_MODE.GEN_3, which works perfectly on its own (see “Test 1” below). However, when I add the Fusion module for GNSS-based Global Localization on top of the same camera, Camera::startPublishing() fails outright as soon as depth is disabled, with the internal message:

[ZED][ERROR] [Grab]  This feature requires an active depth map, but depth computation is disabled (MODE_NONE). Enable depth computation in the initialization parameters.

I could not find this requirement documented anywhere (Global Localization Overview, Troubleshooting page, Fusion Overview, or the Camera.hpp/Fusion.hpp header comments only show sample code that happens to use DEPTH_MODE::ULTRA, without stating that NONE is unsupported).

My questions

  1. Is it fundamentally impossible to run Global Localization / GNSS Fusion (Camera::startPublishing() + Fusion::subscribe() + Fusion::ingestGNSSdata()) with DEPTH_MODE::NONE? Or is there a workaround/parameter I’m missing to make it work without an active depth map?
  2. If it is indeed impossible, why? What internal data does startPublishing()/the Fusion pipeline actually need from the depth module, given that GEN_3 positional tracking itself performs its own feature-based triangulation independently of the DEPTH_MODE-configured dense depth map (confirmed working with DEPTH_MODE::NONE — see Test 1)?
  3. If depth data is indeed being consumed internally by the Fusion/Global Localization pipeline, does the choice of depth mode (PERFORMANCE / QUALITY / ULTRA / NEURAL_LIGHT / NEURAL / NEURAL_PLUS) have a measurable effect on Global Localization accuracy? i.e., is it worth spending the extra compute on ULTRA/NEURAL_PLUS for better geo-pose accuracy, or is any active depth mode functionally equivalent from Fusion’s point of view (just needed to be “on”)?

Test environment

  • ZED SDK: 5.3.0
  • Camera: ZED 2i (serial 33004600, FW 1523), connected via USB3 (confirmed 5000 Mbps enumeration)
  • Host: NVIDIA Jetson (Linux 5.15.148-tegra, aarch64), Python 3.10, pyzed bindings
  • Positional tracking mode: sl.POSITIONAL_TRACKING_MODE.GEN_3
  • GNSS receiver: u-blox GNSS receiver over USB (/dev/ttyACM0), NMEA parsed directly with pyserial + pynmea2 (no gpsd running on this box), confirmed valid 3D fix (12 satellites, fix quality 1, HDOP 0.87) at real-world coordinates in Tokyo, Japan (lat ≈ 35.6955°N, lon ≈ 139.5958°E)

What I tried and observed

Test 1 — GEN_3 Positional Tracking alone, DEPTH_MODE.NONE (no Fusion)

init_params = sl.InitParameters()
init_params.depth_mode = sl.DEPTH_MODE.NONE
zed.open(init_params)

pt_params = sl.PositionalTrackingParameters()
pt_params.mode = sl.POSITIONAL_TRACKING_MODE.GEN_3
zed.enable_positional_tracking(pt_params)   # -> SUCCESS

Result: works as expected. enable_positional_tracking() succeeds, get_position() returns POSITIONAL_TRACKING_STATE.OK on every frame over a 15s run, and the reported pose visibly tracks real hand movement of the camera (up to ~0.46 m translation). retrieve_measure(sl.MEASURE.DEPTH) correctly returns INVALID FUNCTION PARAMETERS, confirming depth truly isn’t being computed.

Test 2 — Add Fusion / GNSS Global Localization on top, still DEPTH_MODE.NONE

init_params.depth_mode = sl.DEPTH_MODE.NONE
zed.open(init_params)
zed.enable_positional_tracking(pt_params)   # mode=GEN_3, -> SUCCESS

fusion = sl.Fusion()
fusion.init(sl.InitFusionParameters())      # -> SUCCESS

communication_parameters = sl.CommunicationParameters()  # default INTRA_PROCESS
pub_status = zed.start_publishing(communication_parameters)
# -> ERROR_CODE.INVALID_FUNCTION_CALL

uuid = sl.CameraIdentifier(zed.get_camera_information().serial_number)
sub_status = fusion.subscribe(uuid, communication_parameters, sl.Transform())
# -> FUSION_ERROR_CODE.MEMORY_ALREADY_USED   (looks like a cascading failure from the failed publish)

positional_tracking_fusion_parameters = sl.PositionalTrackingFusionParameters()
positional_tracking_fusion_parameters.enable_GNSS_fusion = True
fusion.enable_positionnal_tracking(positional_tracking_fusion_parameters)
# -> SUCCESS (misleadingly, since publishing/subscribing already failed)

...
fusion.ingest_gnss_data(gnss_data)
# -> FUSION_ERROR_CODE.MODULE_NOT_ENABLED, on every single frame for 30s

SDK log at the moment start_publishing() is called:

[ZED][ERROR] [Grab]  This feature requires an active depth map, but depth computation is disabled (MODE_NONE). Enable depth computation in the initialization parameters.

Reproduced identically on 2 separate runs.

Test 3 — Control test: same code, DEPTH_MODE.PERFORMANCE instead of NONE

Only the depth mode was changed, everything else identical:

init_params.depth_mode = sl.DEPTH_MODE.PERFORMANCE

Result: start_publishing() → SUCCESS, fusion.subscribe() → SUCCESS, fusion.ingest_gnss_data() → SUCCESS for 16/~30 GNSS samples over the same 30s window (the remaining ones were rejected with GNSS_DATA_COVARIANCE_MUST_VARY, which is a separate, expected issue on my end since my NMEA reader currently reports a fixed covariance value rather than a varying one — not related to depth). The real GNSS coordinates (Tokyo area) were successfully ingested.

What I’ve already checked

  • Global Localization Overview — sample code always sets a real depth_mode (e.g. ULTRA), but never states NONE is disallowed.
  • Global Localization Troubleshooting — covers MODULE_NOT_ENABLED, INVALID_COVARIANCE, INVALID_TIMESTAMP, GNSS_DATA_NEED_FIX, GNSS_DATA_COVARIANCE_MUST_VARY, but nothing about depth mode.
  • Fusion Overview — describes startPublishing() conceptually but doesn’t mention any depth precondition.
  • The Doxygen comments for Camera::startPublishing() in Camera.hpp don’t mention depth either.
  • Searched the forums for the exact error string and for MODULE_NOT_ENABLED + depth — found one thread with the same NOT_ENABLE ingest error, but that case was caused by the user simply forgetting to call start_publishing(), unrelated to depth mode.

Any clarification from the SDK team on the three questions above would be greatly appreciated. Thanks!

2

Thank you for reporting this. The behavior is not expected.

Before I forward the details to the ZED SDK team, can you please upgrade to the latest ZED SDK v5.4.1 and verify if it has been already fixed with it?

3

Sorry, I accidentally replied to a different topic by mistake just before this — I’ve already deleted that reply.

Thanks for the quick response!

I upgraded to 5.4.1 to check as requested, but hit a bigger blocker: even the official hello_zed.py tutorial sample segfaults on this Jetson, on the most basic call — zed.get_camera_information(). Tested with a completely untouched, unmodified copy of the sample (no edits, default InitParameters). Same result.

Error log / backtrace (gdb):

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
sl::String::set(char const*)
 <- sl::String::operator=(sl::String const&)
 <- sl::Matrix4f::operator=(sl::Matrix4f&&)
 <- sl::Transform::operator=(sl::Transform&&)
 <- CameraInformation.__cinit__(...)   [pyzed]

I bisected across SDK versions on the same camera/Jetson (JetPack 6 / L4T R36.4, ZED SDK Python API):

Version get_camera_information()
5.2.3 OK
5.3.0 OK
5.3.1 OK
5.4.0 Segfault (same backtrace)
5.4.1 Segfault (same backtrace, persists after a reboot)

Happy to share the full repro script / gdb session if useful.

Sorry in advance if I’m misreading something on my end!

4

Hi @ShinKengoAgbee

Thank you for the clean bisect, that is exactly the kind of information we need.

You are not misreading anything, but before we escalate this as an SDK regression I would like to rule out a version mismatch, because the backtrace points strongly in that direction:

sl::String::set(char const*)
 <- sl::Matrix4f::operator=(sl::Matrix4f&&)
 <- sl::Transform::operator=(sl::Transform&&)
 <- CameraInformation.__cinit__(...)   [pyzed]

sl::Transform derives from sl::Matrix4f, which holds an sl::String member, i.e. a wrapper around a heap char*. A SIGSEGV inside String::set() while CameraInformation is being built in the Cython layer is the typical signature of the Python wrapper reading the C++ structures at the wrong offsets. This happens when the installed pyzed wheel was built against a different ZED SDK version than the libsl_zed library actually loaded at runtime. The Python API is not version independent, so it must be reinstalled after every SDK upgrade; a wheel left over from 5.3.1 will keep loading and crash exactly like this.

Could you please run these checks?

1. Verify the version actually seen by Python

python3 -c "import pyzed.sl as sl; print(sl.Camera.get_sdk_version())"
python3 -c "import pyzed.sl as sl; print(sl.__file__)"
pip3 show pyzed

The first command must report 5.4.1. If it reports 5.3.x, the mismatch is confirmed.

2. Force a clean reinstall of the wrapper

pip3 uninstall pyzed          # repeat until nothing is left to remove
rm -rf ~/.cache/pip
python3 /usr/local/zed/get_python_api.py

Please also check that no second copy is shadowing the correct one, for example in ~/.local/lib/python3.10/site-packages while you work inside a virtual environment, and that LD_LIBRARY_PATH is not pointing to an old libsl_zed.so from a previous installation:

echo $LD_LIBRARY_PATH
ldd $(python3 -c "import pyzed.sl as sl; print(sl.__file__)") | grep sl_zed

3. Run the C++ control test

This is the most useful single test: please build and run tutorials/tutorial 1 - hello ZED from the C++ samples on the same machine with 5.4.1. If C++ prints the camera information correctly and only Python crashes, the issue is in the wrapper installation; if C++ crashes too, it is in the SDK core and I will escalate it immediately.

4. Confirm the installer matches your L4T revision

You reported L4T R36.4, so please send me the exact filename of the 5.4.1 installer you used, from the ZED SDK download page. Installing a package built for a different JetPack 6.x point release brings in mismatched CUDA and TensorRT dependencies and can produce crashes at library boundaries.

5

You were exactly right — it was a version mismatch, not an SDK regression. Thank you for the clear diagnosis.

Root cause found: I had two shadowing pyzed installations on this machine:

  • a user-level copy under ~/.local/lib/python3.10/site-packages/pyzed (reporting itself as version 5.3)
  • a root-owned, permission-locked copy under /usr/local/lib/python3.10/dist-packages/pyzed (version 5.4) left over from an earlier sudo-run installer

Python’s import resolution was picking up the stale 5.3-built pyzed.sl module (the actual .so predated my latest 5.4.1 core library install), even though sl.Camera.get_sdk_version() still correctly reported “5.4.1” — that call just forwards to whatever libsl_zed.so is loaded, so it didn’t reveal the mismatch. pip3 show pyzed did reveal it (Version: 5.3), matching your prediction.

What fixed it:

sudo find /usr/local/lib/python3.10/dist-packages/ -iname "*pyzed*" -exec chmod -R a+rwX {} \;
sudo pip3 uninstall -y pyzed
pip3 uninstall -y pyzed          # repeated until nothing left
rm -rf ~/.local/lib/python3.10/site-packages/pyzed*
rm -rf ~/.cache/pip
python3 /usr/local/zed/get_python_api.py

After this, pip3 show pyzed correctly reports Version: 5.4, and the official hello_zed.py sample — as well as every argument variant of get_camera_information() I tried before — now runs cleanly (3/3, 4/4 repeat runs). Confirmed it was the leftover/shadowed install, not the SDK core.

Sorry for the false alarm on the crash — and thanks again for catching it so quickly.

Circling back to the original question now that the environment is clean: on ZED SDK 5.4.1, the original finding still holds —

  • POSITIONAL_TRACKING_MODE.GEN_3 alone works fine with DEPTH_MODE.NONE.
  • Global Localization (Camera::startPublishing() → Fusion::subscribe()) still fails with ERROR_CODE::INVALID_FUNCTION_CALL under DEPTH_MODE.NONE, with the same log line (“This feature requires an active depth map, but depth computation is disabled (MODE_NONE)”). Switching to DEPTH_MODE.PERFORMANCE makes startPublishing/subscribe succeed and GNSS ingestion works as expected. So my original three questions from the top of this thread still stand — any insight appreciated whenever you have time.
6

I reported this problem. It will be fixed with one of the next ZED SDK release.

Note: v5.5.0 will be announced soon, so it won’t include the fix.

1 Like
7

Thanks for taking care of this! Really looking forward to the version with the fix.