Make optimized models available for download

For various reasons, we need to optimize the models during SDK installation rather than on first use. It really stinks to have to wait 30+ minutes when installing the Zed SDK for models to be optimized. When deploying to multiple machines, it becomes a huge time sink.

Can’t you guys just put the optimized models online and have the SDK installer download them, instead of optimizing them on the spot with every install?

If not, what files does the optimization process generate? I’d like to be able to just copy them between machines rather than optimizing every time. It wouldn’t be ideal but it’d be an improvement over the 30 minute waits.

SDK 4.2.2 CUDA 12.1 Ubuntu 22.04

This would be ideal but is unfortunately not possible as the optimized models depend on the GPU model and all AI libraries stack as well (CUDA, TensorRT, cuDNN + OS) creating a nearly infinite number of possible configurations. If you have strictly identical machines you can just copy the resource folder, however.

The optimized model is the TensorRT engine/plan file. It tries to find the most efficient backend implementation of every function for a given AI model architecture (depending on the operation used in the model and its characteristics) on your specific hardware and software configuration.

It does that by actually running them all and checking the time it took, to choose the fastest. There’s a timing cache in place to avoid doing the full benchmark every time to reuse identical functions’ timing for similar models or operations.

That’s one of the reasons why AI inference is so much faster than standard training frameworks.

1 Like

Hi @JC31

We cannot. AI model optimization depends on the hardware configuration of the machine where they are used.

If you use a fixed hardware configuration you can make the optimization once and use the files in the folder /usr/local/zed/resources/ for every machine.
Please note that the optimization result files are hidden.

An example of the content of the folder:

$ ls -la /usr/local/zed/resources/
total 1476416
drwxrwx---  2 walter zed         4096 Jan  9 13:19 .
drwxrwx--- 10 walter zed         4096 Jan  2 12:05 ..
-rwxrwx---  1 walter zed      5182317 Jan  9 13:19 .8601_timing_cache
-rwxrwx---  1 walter zed      6947700 Dec  9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-dcabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed      6947707 Dec  9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed      6948353 Dec  9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed      6161019 Dec  9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-aaabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed      6163587 Dec  9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed      6161039 Dec  9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-#ggbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed    182429522 Dec  9 15:21 .ffd7b9016982578d870dd2d6d11b192e.model_optimized-aaabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed    182429788 Dec  9 15:21 .ffd7b9016982578d870dd2d6d11b192e.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     19759308 Dec  9 15:21 neural_depth_3.6.model
-rwxrwx---  1 walter zed     11841917 Dec  9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12010-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx---  1 walter zed     11842622 Dec  9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx---  1 walter zed     11841889 Dec  9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx---  1 walter zed     11842233 Dec  9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-0e53-512
-rwxrwx---  1 walter zed     12018406 Dec  9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12010-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx---  1 walter zed     12018796 Dec  9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx---  1 walter zed     12018712 Dec  9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx---  1 walter zed     12018726 Dec  9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx---  1 walter zed    142657713 Dec  9 15:21 objects_accurate_3.2.model
-rwxrwx---  1 walter zed     82234218 Dec  9 15:21 .objects_accurate_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     85526241 Dec  9 15:21 objects_medium_3.2.model
-rwxrwx---  1 walter zed     49281184 Dec  9 15:21 .objects_medium_3.2.model_optimized-ebgbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     37580125 Dec  9 15:21 objects_performance_3.2.model
-rwxrwx---  1 walter zed     21784379 Dec  9 15:21 .objects_performance_3.2.model_optimized-fbcbl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     21783594 Dec  9 15:21 .objects_performance_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rw-rw-r--  1 walter walter  21783330 Jan  9 13:18 .objects_performance_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     56581340 Dec  9 15:21 person_head_accurate_2.4.model
-rwxrwx---  1 walter zed     39964873 Dec  9 15:21 .person_head_accurate_2.4.model_optimized-jgabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     69103733 Dec  9 15:21 person_head_accurate_2.6.model
-rwxrwx---  1 walter zed      4830872 Dec  9 15:21 person_head_performance_2.4.model
-rwxrwx---  1 walter zed      6262534 Dec  9 15:21 person_head_performance_2.6.model
-rwxrwx---  1 walter zed      8357581 Dec  9 15:21 person_reid_1.4.model
-rwxrwx---  1 walter zed     10046926 Dec  9 15:21 .person_reid_1.4.model_optimized-aaabl-8-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx---  1 walter zed     33279104 Dec  9 15:21 positional_tracking_2.2.model
-rwxrwx---  1 walter zed     94709564 Dec  9 15:21 skeleton_body18_3.2.model
-rwxrwx---  1 walter zed     48732047 Dec  9 15:21 .skeleton_body18_3.2.model_optimized-ccebl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-2-224
-rwxrwx---  1 walter zed    100767150 Dec  9 15:21 skeleton_body38_3.5.model
-rw-rw-r--  1 walter walter  51824136 Jan  9 13:19 .skeleton_body38_3.5.model_optimized-dcabl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-2-320
1 Like