Games of frames - how do these frame run along?


i want to send obj detection data to another computer over OSC/UDP.
attached is a snippet of the processing loop.

right now an osc message is sent every 200 ms for 80% of the time … sometimes around 250ms, outliers around 3000 and 0. i would like to get it down to a rate of 15 frames.
iḿ running a zed 2i on a ZED BOX Orin NX 8GB. in the end the project needs 2 cameras sending object detectino data to the other computer.
Are the specs of the Orin NX 8 GB good enough? ( also in regard to my code quality ^^)
attached is also a pastebin with the whole code ( OSC bit is line 330 to 370 )

pastebin code

    #if ENABLE_GUI
            gl_viewer_available &&
            !quit && zed.grab(runtime_parameters) == ERROR_CODE::SUCCESS) 
        // update confidence threshold based on TrackBar
        if (detection_parameters_rt.object_class_filter.empty())
            detection_parameters_rt.detection_confidence_threshold = detection_confidence;
        else // if using class filter, set confidence for each class
            for (auto& it : detection_parameters_rt.object_class_filter)
                detection_parameters_rt.object_class_detection_confidence_threshold[it] = detection_confidence;

        returned_state = zed.retrieveObjects(objects, detection_parameters_rt);
        if (returned_state == ERROR_CODE::SUCCESS) 
            //what rate here?
            // Iterate through detected objects and send data via OSC
            for (int i = 0; i < objects.object_list.size(); i++) 
                sl::ObjectData& obj = objects.object_list[i];

                float objVol = calculateVolume(obj.dimensions);
                float velocity = calcTotalVelocity(obj.velocity);
                std::string label = objectClassToString( obj.label );
                std::string state = trackingStateToString(obj.tracking_state);

                //cout << "OSC sending " << endl;

                // Send OSC message (you'll need to replace this with your actual OSC send function)
                sendOSCMessage( "/front",
                                objVol, // size
                                velocity, // summed velocity

                // Right after sending an OSC message:
                auto currentTimePoint = std::chrono::high_resolution_clock::now();

                // Calculating the interval (difference) in milliseconds
                auto interval = std::chrono::duration_cast<std::chrono::milliseconds>(currentTimePoint - previousTimePoint).count();

                // Update previousTimePoint for the next iteration
                previousTimePoint = currentTimePoint;

                // Output the interval
                std::cout << "OSC message sent, interval since last message: " << interval << " ms" << std::endl;

            // store the id of detetected objects
            for(auto &it: objects.object_list)
                id_counter[] = 1;

            // check if bacthed trajectories are available
            std::vector<sl::ObjectsBatch> objectsBatch;
                    std::cout<<"During last batch processing: "<<id_counter.size()<<" Object were detected: ";
                    for(auto it :id_counter) std::cout<<it.first<<" ";
                    std::cout<<"\nWhile "<<objectsBatch.size()<<" different only after reID: ";
                    for(auto it :objectsBatch) std::cout<<<<" ";

            zed.retrieveMeasure(point_cloud, MEASURE::XYZRGBA, MEM::GPU, pc_resolution);
            zed.getPosition(cam_w_pose, REFERENCE_FRAME::WORLD);
            zed.retrieveImage(image_left, VIEW::LEFT, MEM::CPU, display_resolution);
            track_view_generator.generate_view(objects, image_left_ocv,img_scale,cam_w_pose, image_track_ocv, objects.is_tracked);  
            // Tiefenbild
            // viewer.updateData(point_cloud, objects.object_list, cam_w_pose.pose_data);
            // gl_viewer_available = viewer.isAvailable();

            // CONVERT BETWEEN CV_8UC4 AND CV_8UC3
            cv::Mat img_bgra;       // Your CV_8UC4 image
            cv::Mat global_image_c3;        // C3 image
            cv::cvtColor(global_image, global_image_c3, cv::COLOR_BGRA2BGR);

            cout << "frame: " << frameCnt++ << endl;

            // Write the frame into the file

            // as image_left_ocv and image_track_ocv are both ref of global_image, no need to update it
            cv::imshow(window_name, global_image_c3);

            #if ENABLE_REC

            key = cv::waitKey(10);
            if (key == 'i') {
            } else if (key == 'o') {
            } else if (key == 'q') {
                quit = true;
            } else if (key == 'p') {
                //detection_parameters_rt.object_class_detection_confidence_threshold[OBJECT_CLASS::PERSON] = detection_confidence;
                cout << "Person only" << endl;
            } else if (key == 'v') {
                detection_parameters_rt.object_class_detection_confidence_threshold[OBJECT_CLASS::VEHICLE] = detection_confidence;
                cout << "Vehicle only" << endl;
            } else if (key == 'c') {
                cout << "Clear Filters" << endl;

            //cout << "Current Position: " << zed.getSVOPosition() << ", Total Frames: " << zed.getSVONumberOfFrames() << ", Is Playback: " << is_playback << endl;

            //if (is_playback && zed.getSVOPosition() == zed.getSVONumberOfFrames() ) // framerate playblack problem
            if (is_playback && zed.getSVOPosition() == 200 ) // framerate playblack problem
                cout << "setSVOPos 0" << endl;
                //zed.setSVOPosition(0); // LOOP file
                quit = true; 


    } // ENABLE GUI / while loop  -  END

Hey @D300, welcome to the forums!

  • Can you check how many fps you get without the sending part?
  • I notice you don’t set the fps in the init_parameters, you can set camera_fps to 15 to ensure more stable 15 fps, avoiding too many computations in the SDK.
  • To run 2 cameras, I recommend the Orin NX 16GB, otherwise, you will experience decreased performance, especially if running other modules than just depth (object detection for instance).

thanks alot for the fast reply!

  • i checked without the sending part , did not make a difference. suprisingly i had better values today . made something was running in the background yesterday night and i forgot about it. but its still to slow for my use cases, around 180ms sending interval, with and without OSC Sending.
  • i added init_parameters.camera_fps = 15; did not make it better
  • fortunately i ordered exactly an Orin NX 16GB yesterday.

do you have other ideas how i could get a better rate?
whats the workflow of balacing CPU and GPU capacities?
iḿ also wondering if there are unnecessary copy processes…

Hello @D300

I’m surprised there is no difference, it should not be so low.

  • Sorry, when I said without the sending part, I should have said not only without the sending, but pretty much what fps can you get with the “raw” object detection sample? This is to see if the issue lies with the Box/Camera/SDK or with the added code.

Sorry, I can’t really give a review of your code as a whole though. Also, when sharing such a large sample, please remove the if 0’ed parts beforehand (GUI etc).