Object tracking in 3D

Tetsujinfr · March 8, 2021, 12:28am

Hi
I would like to get some object tracking in 3D coordinates.
Something like what you show for the ZED2, but for the ZED mini.
Do you expose some function in the SDK that would enable me to do that ?
I have an object detector, I got depth info for each object, but I need the obj tracking piece integrated with the camera world coordinate system and its own positional tracking (adjusted with the IMU data).

Can you help/give me some suggestions?

Thanks

Ps: I do want to use the ZED mini, not the ZED2.

Tetsujinfr · March 8, 2021, 2:53pm

so after looking around, I found this code sample which seems a good start for me:

github.com

stereolabs/zed-tensorflow/blob/master/object_detection_zed.py

import numpy as np
import os
import six.moves.urllib as urllib
import sys
import tensorflow as tf
import collections
import statistics
import math
import tarfile
import os.path

from threading import Lock, Thread
from time import sleep

import cv2

# ZED imports
import pyzed.sl as sl

sys.path.append('utils')

This file has been truncated. show original

I am planning to replace the tensorflow detection piece with mine.

Do you agree this is a good direction to go with?

Are there resources in the SDK to help with the object tracking (with ID instantiation)? E.g. passing a table of object corrdinates to a tracking function which would leverage the camera positional tracking as well as some Kalman filter /optical flow type of algo? Kind of a deepsort but leveraging properly the 3rd dimension of the objects as well as the camera positional tracking, as a competitive hedge.

thanks in advance.

Tetsujinfr · March 13, 2021, 11:24am

The depth data processing of the github code is quite slow. Is there a way to leverage the sdk and the gpu to compute the depth xyza data over xy bounding boxes area?