Memory leak in GNSS fusion?

I’m working on a python application that uses both object tracking and GNSS fusion. Using the latest SDK i.e. 4.0.6. I’ve noticed my memory keeps increasing at about 1mb/second (though it’s slower to start with). After a lot of messing round with code, I think I’ve found the cause - if your main loop isn’t processing fast enough, memory starts to grow. In my case, object tracking seems to slow it down enough.

For example, in the sample if you run it as-is (with the GUI disabled), the memory starts around 291mb (as reported by psutil.Process().memory_info().rss), and grows slowly - after a minute it’s about 296mb. It’s pretty stable after that - increasing less than 1mb per minute. It may not stop increasing (i.e. still be a memory leak) - I’m not sure. But it’s pretty “good”.

However, as soon as you add a single time.sleep(0.1) in the main while loop, things get weird. Firstly, after about 5s it starts at 305mb (not 291mb). The memory grows slowly at first, but accelerates - within a minute it’s growing at multiple mb per second. I stopped it at 400mb, and it was still growing.

NB: my GPS is spitting out at 10hz, so if you’re testing at normal 1hz you may not see this. time.sleep(1) may help reproduce in this case.

If you comment out fusion.ingest_gnss_data(input_gnss) i.e. so it’s still grabbing the GPS from GPSD but never ingesting it, the problem goes away. I think this makes it clear it’s something to do with the Zed GNSS fusion, and not anything else.

I’m not sure what’s going on here - it seems like a buffer is filling up because the main loop is getting behind the camera? Not sure. Anyway, is there a way to avoid this? In my application I can’t avoid the slow-down as I’m using object detection, which seems to limit the FPS, even in fast mode. It feels like I need a way to clear the buffer even though object detection is running slowly.

Hello and thank you for reaching out to us,

The Fusion module contains a synchronization queue which ingests data and pops data. This queue will indeed grow in memory, however it should not cause the issues you mention. Are you able to share some code that reproduce the issue ?

Thanks for responding. I was using v4.0.6 of this file where I simply added a time.sleep inside the while loop. (I also disabled the viewer as I was running headless.) If you can’t reproduce or don’t have a test rig with GPS, I could create a standalone file for you with simulated GPS data.

Hello @kiwi and thank you for your feedback !

Indeed we have identified a memory leak that occurs during the GNSS/VIO initialization process. The bug-fix for this issue will be included in the upcoming patch. That the memory leak is specific to the initialization phase and should not occur afterwards.
Please note that even with this fix, there may still be a slight increase in memory usage during this phase. This is because we internally need to retain GNSS/VIO data in order to perform the calibration process. However, the increase in memory should not be as significant as what you have previously identified.