SVO to mp4 export speed up on Nvidia Jetsons

efan4ik · June 5, 2024, 4:03pm

Hello

I am recording videos (lossy H265) on Jetson Orin NX 16GB

using the following code

stereolabs/zed-sdk/blob/master/recording/recording/mono/python/svo_recording.py

########################################################################
#
# Copyright (c) 2022, STEREOLABS.
#
# All rights reserved.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
########################################################################

This file has been truncated. show original

After that I need to perform extraction to an .avi or .mp4 file while applying rotation and blurring of the faces (anonymization) using built-in body tracking (head position). Unfortunately on Jetson Orin NX it takes enormous amount of time (up to x10-15 of the original video duration).

Based on a basic profiling most of the time takes actual encoding. In the official ZED SDK sample cv2.VideoWriter_fourcc is used. I tried to use ffmpeg and Python pipes to speed up the process but it didn’t end with any significant performance boost.

What are the most optimal params for ffmpeg to use Jetson hardware acceleration for video extraction from .svo files ? Is there an alternative way of doing that ?

works but runs entirely on CPU which is very slow

ffmpeg_process = sp.Popen(shlex.split(
    f'ffmpeg -y -s {rotated_width}x{rotated_height} -pixel_format bgr24 -f rawvideo -r {svo_fps} -i pipe: -vcodec libx265 -pix_fmt yuv420p -crf 26 {deided_mp4_filename}'),
    stdin=sp.PIPE)

doesn’t work

# ffmpeg_process = sp.Popen(shlex.split(
#     f'ffmpeg -hwaccel cuda -y -s {rotated_width}x{rotated_height} -pixel_format bgr24 -f rawvideo -r {svo_fps} -i pipe: -vcodec hevc_nvenc -pix_fmt yuv420p -crf 26 {deided_mp4_filename}'),
#     stdin=sp.PIPE)

Thank you

mattrouss · June 10, 2024, 4:37pm

Hi @efan4ik,

Welcome to the Stereolabs forums

If performance is what you are looking for, have you tried the export sample in C++ and noticed any differences for exporting the videos?

I am not familiar with ffmpeg options in general, but the workflow you are performing is correct, decoding the svo and re-encoding it again is the way to go.

efan4ik · August 9, 2024, 12:20pm

This is the fastest way I found to do that on Jetson

gst_out = f"appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h265enc ! h265parse ! matroskamux ! filesink location={mp4_filename}"
output_deided_mp4 = cv2.VideoWriter(gst_out, cv2.CAP_GSTREAMER, 0, float(svo_fps), (int(rotated_width), int(rotated_height)))

The solution was found on Nvidia Developer Forum