Set detected object based on world reference frame

Krisna · January 12, 2023, 11:58am

Hello,
After running the launch file

roslaunch zed_wrapper zed2.launch

I get the published objects data from the topic /zed2/zed_node/obj_det/objects (after setting the parameters od_enabled=True in the zed_wrapper/params/zed2.yaml) and it sure gives the detected objects position in the world coordinate frame as the default. However, those position are based on camera reference frame.

Since my zed2 camera is attached to the robot that moves in the world, i need the detected object position based on world reference frame.

In python code, you can specify like this to set it to the reference frame world

runtime_parameters = sl.RuntimeParameters()
runtime_parameters.measure3D_reference_frame = sl.REFERENCE_FRAME.WORLD

then you can just use the runtime parameters in the grab function as follows:

zed.grab(runtime_parameters)

But im using the zed2 node which publishes the data based on the parameters of the .yaml files.

My question is, where to specify such parameter? I dont see any parameter which corresponds to that in the zed2.yaml and common.yaml files.

any help is very much appreciated. Thanks!

Myzhar · January 12, 2023, 4:40pm

Hi Krisna,
object positions are in the world (map) reference frame with the origin in the starting position of the robot, they are not relative to the camera frame.

Krisna · January 16, 2023, 9:19am

Hi,
But why are the positions of the detected object changes when I move the camera? Even though the object did not move.

I created a ROS marker object, set the header.frame_id to map, and assign the points of it with the detected object position (in this case person) by the camera.

In RViz, the marker points move as if the object are getting closer to the camera, but what’s happening is the other way around, where the one that actually move is the camera that is getting closer to the object.

Myzhar · January 16, 2023, 9:22am

@Krisna please provide more information (a video could be useful for example) because what you describe is not clear.

Myzhar · January 16, 2023, 9:29am

@Krisna sorry, my fault. The position of the objects is relative to the left camera sensor frame:

github.com

stereolabs/zed-ros-wrapper/blob/master/zed_nodelets/src/zed_nodelet/src/zed_wrapper_nodelet.cpp#L4356


      
          mObjDetPeriodMean_msec->addValue(elapsed_msec);
          old_time = now;
          // <---- Diagnostic information update
          
          
NODELET_DEBUG_STREAM("Detected " << objects.object_list.size() << " objects");
          
          
size_t objCount = objects.object_list.size();
          
          
zed_interfaces::ObjectsStampedPtr objMsg = boost::make_shared<zed_interfaces::ObjectsStamped>();
          objMsg->header.stamp = t;
          objMsg->header.frame_id = mLeftCamFrameId;
          
          
objMsg->objects.resize(objCount);
          
          
size_t idx = 0;
          for (auto data : objects.object_list) {
              objMsg->objects[idx].label = sl::toString(data.label).c_str();
              objMsg->objects[idx].sublabel = sl::toString(data.sublabel).c_str();
              objMsg->objects[idx].label_id = data.id;
              objMsg->objects[idx].confidence = data.confidence;

you can obtain the position of the objects in the map frame by using TF, something like this:

github.com

stereolabs/zed-ros-wrapper/blob/master/zed_nodelets/src/zed_nodelet/src/zed_wrapper_nodelet.cpp#L3754-L3755


      
          geometry_msgs::TransformStamped b2m =
                  mTfBuffer->lookupTransform(mMapFrameId, mBaseFrameId, ros::Time(0));

Tutorials and doc:
http://wiki.ros.org/tf/Overview/Using%20Published%20Transforms
http://wiki.ros.org/tf/Tutorials/Writing%20a%20tf%20listener%20(C%2B%2B)