For various reasons, we need to optimize the models during SDK installation rather than on first use. It really stinks to have to wait 30+ minutes when installing the Zed SDK for models to be optimized. When deploying to multiple machines, it becomes a huge time sink.
Can’t you guys just put the optimized models online and have the SDK installer download them, instead of optimizing them on the spot with every install?
If not, what files does the optimization process generate? I’d like to be able to just copy them between machines rather than optimizing every time. It wouldn’t be ideal but it’d be an improvement over the 30 minute waits.
This would be ideal but is unfortunately not possible as the optimized models depend on the GPU model and all AI libraries stack as well (CUDA, TensorRT, cuDNN + OS) creating a nearly infinite number of possible configurations. If you have strictly identical machines you can just copy the resource folder, however.
The optimized model is the TensorRT engine/plan file. It tries to find the most efficient backend implementation of every function for a given AI model architecture (depending on the operation used in the model and its characteristics) on your specific hardware and software configuration.
It does that by actually running them all and checking the time it took, to choose the fastest. There’s a timing cache in place to avoid doing the full benchmark every time to reuse identical functions’ timing for similar models or operations.
That’s one of the reasons why AI inference is so much faster than standard training frameworks.
We cannot. AI model optimization depends on the hardware configuration of the machine where they are used.
If you use a fixed hardware configuration you can make the optimization once and use the files in the folder /usr/local/zed/resources/ for every machine.
Please note that the optimization result files are hidden.
An example of the content of the folder:
$ ls -la /usr/local/zed/resources/
total 1476416
drwxrwx--- 2 walter zed 4096 Jan 9 13:19 .
drwxrwx--- 10 walter zed 4096 Jan 2 12:05 ..
-rwxrwx--- 1 walter zed 5182317 Jan 9 13:19 .8601_timing_cache
-rwxrwx--- 1 walter zed 6947700 Dec 9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-dcabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 6947707 Dec 9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 6948353 Dec 9 15:21 .877184d9b5beaccf65b8d55d99f028b6.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 6161019 Dec 9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-aaabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 6163587 Dec 9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 6161039 Dec 9 15:21 .c722bc9b2d2f8cf8cba436d781865b3e.model_optimized-#ggbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 182429522 Dec 9 15:21 .ffd7b9016982578d870dd2d6d11b192e.model_optimized-aaabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 182429788 Dec 9 15:21 .ffd7b9016982578d870dd2d6d11b192e.model_optimized-geabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 19759308 Dec 9 15:21 neural_depth_3.6.model
-rwxrwx--- 1 walter zed 11841917 Dec 9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12010-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx--- 1 walter zed 11842622 Dec 9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx--- 1 walter zed 11841889 Dec 9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-0e53-512
-rwxrwx--- 1 walter zed 11842233 Dec 9 15:21 .neural_depth_3.6.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-0e53-512
-rwxrwx--- 1 walter zed 12018406 Dec 9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12010-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx--- 1 walter zed 12018796 Dec 9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx--- 1 walter zed 12018712 Dec 9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx--- 1 walter zed 12018726 Dec 9 15:21 .neural_depth_3.6.model_optimized-kcebl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-0e53-1024
-rwxrwx--- 1 walter zed 142657713 Dec 9 15:21 objects_accurate_3.2.model
-rwxrwx--- 1 walter zed 82234218 Dec 9 15:21 .objects_accurate_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 85526241 Dec 9 15:21 objects_medium_3.2.model
-rwxrwx--- 1 walter zed 49281184 Dec 9 15:21 .objects_medium_3.2.model_optimized-ebgbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 37580125 Dec 9 15:21 objects_performance_3.2.model
-rwxrwx--- 1 walter zed 21784379 Dec 9 15:21 .objects_performance_3.2.model_optimized-fbcbl-1-89-12020-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 21783594 Dec 9 15:21 .objects_performance_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rw-rw-r-- 1 walter walter 21783330 Jan 9 13:18 .objects_performance_3.2.model_optimized-fbcbl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 56581340 Dec 9 15:21 person_head_accurate_2.4.model
-rwxrwx--- 1 walter zed 39964873 Dec 9 15:21 .person_head_accurate_2.4.model_optimized-jgabl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 69103733 Dec 9 15:21 person_head_accurate_2.6.model
-rwxrwx--- 1 walter zed 4830872 Dec 9 15:21 person_head_performance_2.4.model
-rwxrwx--- 1 walter zed 6262534 Dec 9 15:21 person_head_performance_2.6.model
-rwxrwx--- 1 walter zed 8357581 Dec 9 15:21 person_reid_1.4.model
-rwxrwx--- 1 walter zed 10046926 Dec 9 15:21 .person_reid_1.4.model_optimized-aaabl-8-89-12060-8907-8601-13-192-36864-48-100-512-46-00
-rwxrwx--- 1 walter zed 33279104 Dec 9 15:21 positional_tracking_2.2.model
-rwxrwx--- 1 walter zed 94709564 Dec 9 15:21 skeleton_body18_3.2.model
-rwxrwx--- 1 walter zed 48732047 Dec 9 15:21 .skeleton_body18_3.2.model_optimized-ccebl-1-89-12060-8907-8601-12-192-36864-48-100-512-46-0-2-224
-rwxrwx--- 1 walter zed 100767150 Dec 9 15:21 skeleton_body38_3.5.model
-rw-rw-r-- 1 walter walter 51824136 Jan 9 13:19 .skeleton_body38_3.5.model_optimized-dcabl-1-89-12060-8907-8601-13-192-36864-48-100-512-46-0-2-320