Motion tracking using additional Azimuth and Elevation motors

Hardware ZED X
1

Hi, I’m currently working on a project to track objects, namely drones, at close range for training purposes. We want to use the ZED X 4mm due to its global shutter and greater ranging but that comes with the downside of a narrower FoV. Our solution would be mount the camera on a gimbal with azimuth and elevation motors to keep the target object in view.

I’m reaching out to ask:

  • Has anyone done this previously and could help.
  • If not, can anyone advise if the motor could be driven direct from the ZED Box or if an intermediary is need, e.g. Arduino. If possible do you know what motors can be used, ideally stepper.

We’ve had some success in the past with a similar system using PixyCam but this is a massive step up.

Thanks in advance for the help :folded_hands:

2

Hi @antrossall
Welcome to the StereoLabs community.

thanks for the detailed description of the project, it is an interesting use case.

We do not provide an official pan/tilt reference design, and I am not aware of a public community project doing exactly this with a ZED X. However, all the building blocks you need are available in the ZED SDK, and the architecture is quite standard: detection, bearing computation, servo loop.

An intermediary is definitely recommended, for two reasons:

  1. Electrical: the ZED Box Mini exposes only 2 GPIOs, 1 UART and 1 CAN on the 10-pin expansion header, all at 3.3 V logic level and low current. They are signal pins, they cannot drive a motor in any case. See Connectivity and Using GPIO. The ZED Box Orin does not expose a general purpose GPIO header of this kind.
  2. Timing: generating a STEP/DIR pulse train from Linux user space is not deterministic; you would get visible jitter and lost steps as soon as the CPU is loaded by the depth and AI pipeline. Even with the Real-Time Kernel option, I would not put the step generation on the Jetson.

The clean split is: the ZED Box runs perception and outputs an angular setpoint (or an angular rate) at 30/60 Hz over UART, USB or CAN; a small MCU (Arduino, STM32, or a “smart” driver with an integrated controller) runs the trajectory generation and the current loop.

Steppers are usable, with some care:

  • Use microstepping drivers and a belt or harmonic reduction; the ZED X weighs 240 g, plus the FAKRA cable, which is stiff and adds a non-negligible restoring torque on the elevation axis.
  • Open-loop steppers have no way to detect a lost step. For a target like a drone, I would use closed-loop steppers (encoder on the shaft) or add an absolute encoder per axis, so the controller always knows the true pointing angle.
  • If you can, consider direct-drive BLDC gimbal motors instead. They are backlash free, silent, and much smoother at the small correction angles you will be commanding most of the time; commercial gimbal controllers already implement the rate loop for you.

Mechanically, the ZED X has a 1/4"-20 UNC thread plus 2x M4 on the bottom and 4x M4 on the back, so mounting on a bracket is straightforward. Leave a generous service loop for the GMSL2 cable, or use a slip ring if you need continuous azimuth rotation.

On the vision side, a few points that will save you time:

  • There is no “drone” class in the built-in detector, so you will need Custom Object Detection with your own model. The YOLO integration is the fastest path; the SDK takes care of 3D localization and tracking of your 2D detections. In ROS 2, see Custom Object Detection and Tracking.
  • For the servo loop, work in the camera reference frame, not the world frame. A gimbal produces an almost pure rotation, which is a degenerate case for visual-inertial SLAM, so the translation estimate will not be reliable. Retrieve the objects with sl::REFERENCE_FRAME::CAMERA and compute the pointing error directly:
// obj.position is expressed in the CAMERA frame
float az_err = std::atan2(obj.position.x, obj.position.z);
float el_err = std::atan2(obj.position.y, obj.position.z);
  • Use the object velocity returned by the tracker as a feed-forward term. With a drone, a pure PID on the position error will always lag; the prediction is what keeps the target centered.
  • Force a short exposure time manually via the Camera Controls. The global shutter removes the rolling shutter artifacts, but motion blur still depends on the integration time, and the gimbal itself adds apparent motion.
  • The built-in IMU runs at 200 Hz and gives you the angular rate of the camera, which is useful as a stabilization feed-forward, much faster than the vision loop.
  • Run the camera at 1200p@60 and use a light depth mode; depth runs up to 120 Hz on the ZED X, so the bottleneck will be your detector, not the SDK.

One last note about the lens choice: the ZED X 4 mm has a FoV of 80°(H) x 52°(V) and a depth range starting at 1.0 m, ideal between 1.0 m and 20 m. If “close range” for you means below 1 m, the ZED X Mini 4 mm covers 0.15 m to 12 m with the same global shutter sensor, and it is lighter, which helps the gimbal dynamics.

The full range of cameras and embedded PCs is available in the StereoLabs store, and you can find more information on www.stereolabs.com.

Please keep us updated on the project, I am curious to see the result.