Diagnostic and Unity claim AI Models not optimized even after optimizing them

Software
1

I am trying to use the object detection functionality in Unity, but when I try ZED reports that the ai models must be downloaded/optimized for this to work. It directed me to the ZED Diagnostic app which confirmed the detection models were not optimized. Using the diagnostic app in the terminal, I ran
ZED Diagnostic.exe -aio
After several minutes, most of the installs reported they were done in the terminal.
ZED Diagnostic.exe -ais 7 and ZED Diagnostic.exe -ais 8 picked up the stragglers.

However, the diagnostic still reported that none of the models were optimized.

I tried restarting every program involved and rebooting the computer, but there was no change.

I have since cleared the models with ZED Diagnostic.exe -aic and tried optimizing a single model. This is what I saw in the terminal.

C:\Program Files (x86)\ZED SDK\tools>"ZED Diagnostic.exe" -ais 0

Optimizing: MULTI CLASS DETECTION...
 Downloading objects_performance_2.2  100.0%[========================================] 19.7/19.7 MB              Done
 Optimizing objects_performance_2.2 |  100.0%[=============================>] Done ! in 5min 12s

C:\Program Files (x86)\ZED SDK\tools>

ZED Diagnostic still reports that no models are optimized.

Here is the complete text of the diagnostic file:

{
    "AI Models": {
        "info": [
            "MULTI CLASS DETECTION is not optimized",
            "MULTI CLASS MEDIUM DETECTION is not optimized",
            "MULTI CLASS ACCURATE DETECTION is not optimized",
            "HUMAN BODY FAST DETECTION is not optimized",
            "HUMAN BODY MEDIUM DETECTION is not optimized",
            "HUMAN BODY ACCURATE DETECTION is not optimized",
            "PERSON HEAD DETECTION is not optimized",
            "REID ASSOCIATION is not optimized",
            "NEURAL DEPTH is not optimized"
        ]
    },
    "Camera Test": {
        "InternalDevicesCount": 2,
        "ZEDCount": 1,
        "camera": {
            "Device ID": 1,
            "Firmware": 1523,
            "Init Output": "OK",
            "Model": "ZED 2i",
            "Serial Number": 37285655,
            "valid": [
                "<b>Camera: </b>ZED 2i",
                "<b>Firmware: </b>1523"
            ]
        },
        "resolutions": {
            "HD1080@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD1080@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD2K@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1242,
                "height (output)": 1242,
                "initialization": "OK",
                "status": true,
                "width (input)": 2208,
                "width (output)": 2208
            },
            "HD720@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@60": {
                "dropCount": -1,
                "fps (input)": 60,
                "fps (output)": 60,
                "height (input)": 720,
                "height (output)": 720,
                "imageCount": 502,
                "initialization": "OK",
                "status": true,
                "tearingCount": 0,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "VGA@100": {
                "fps (input)": 100,
                "fps (output)": 91,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@60": {
                "fps (input)": 60,
                "fps (output)": 60,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            }
        },
        "sensors": {
            "Firmware": 777,
            "Init Output": "OK",
            "Model": "ZED 2i (MCU)",
            "Serial Number": 37285655
        }
    },
    "Graphics Card": {
        "deviceCount": 1,
        "deviceDriverVersion": 11070,
        "devices": [
            {
                "arch": "Turing",
                "computeCapability": "7.5",
                "cores": 2048,
                "name": "NVIDIA GeForce GTX 1650 Ti with Max-Q Design",
                "totalMemoryBytes": "4294639616",
                "totalMemoryMB": 4095.6875
            }
        ],
        "initResult": 0,
        "valid": [
            "<b>Graphics card: </b> NVIDIA GeForce GTX 1650 Ti with Max-Q Design"
        ]
    },
    "Processor": {
        "OS": "Windows 10 FallCreators",
        "avxSupported": true,
        "coreCount": "6",
        "cpu": "Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz",
        "datetime": "2022-09-29 10:07:05",
        "hyperThreading": "1",
        "motherboard": "20TLS1UU00, LENOVO",
        "ramCapacity": "31.7259",
        "ramUsage": "0",
        "ramUsed": "11.9348",
        "threadCount": "12",
        "valid": [
            "<b>Processor: </b> Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz",
            "<b>Motherboard: </b> 20TLS1UU00, LENOVO"
        ]
    },
    "USB": {
        "ASMediaFound": false,
        "CorruptedFirmware": false,
        "MCUDetected": true,
        "OC_Cam_Detected": false,
        "OtherUSBDevices": [
        ],
        "USBList": [
            {
                "USBMode": 2,
                "USB_path": "/6/1/1/2",
                "bDescriptorType": 1,
                "bDeviceProtocol": 0,
                "bLength": 18,
                "bMaxPacketSize0": 64,
                "bNumConfigurations": 1,
                "bcdDevice": "3.9",
                "bcdUSB": "2.0",
                "bcdUSBClass": 0,
                "bcdUSBSubClass": 0,
                "busNumber": 2,
                "device": "ZED2i MCU",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 3,
                "idProduct": "0xf881",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/6/1/1",
                "idProduct": "0x2512",
                "idVendor": "0x0424"
            },
            {
                "USBMode": 3,
                "USB_path": "/22/1/1",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED2i",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 4,
                "idProduct": "0xf880",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/22",
                "idProduct": "0x0211",
                "idVendor": "0x2109"
            },
            {
                "USB_path": "/14",
                "idProduct": "0x0026",
                "idVendor": "0x8087"
            },
            {
                "USB_path": "/9",
                "idProduct": "0x00bd",
                "idVendor": "0x06cb"
            },
            {
                "USB_path": "/8",
                "idProduct": "0xb6cb",
                "idVendor": "0x04f2"
            },
            {
                "USB_path": "/22/1",
                "idProduct": "0x0211",
                "idVendor": "0x2109"
            },
            {
                "USB_path": "/1",
                "idProduct": "0x2510",
                "idVendor": "0x093a"
            },
            {
                "USBMode": 3,
                "USB_path": "/22/1/1",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED2i",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 4,
                "idProduct": "0xf880",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/6/1",
                "idProduct": "0x2211",
                "idVendor": "0x2109"
            },
            {
                "USB_path": "/6",
                "idProduct": "0x2211",
                "idVendor": "0x2109"
            }
        ],
        "USBMode": 3,
        "ZED Camera Module Detected": 63616,
        "ZED MCU Module Detected": 63617,
        "ZEDDetected": true,
        "captureDevices": [
        ],
        "controllers": [
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0032",
                "Location": "Port_#0001.Hub_#0006",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-11"
            },
            {
                "CompatibleIDs": "USB\\USB30_HUB",
                "Description": "Generic SuperSpeed USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0027",
                "Location": "Port_#0022.Hub_#0001",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-2"
            },
            {
                "CompatibleIDs": "",
                "Description": "USB Root Hub (USB 3.0)",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0002",
                "Location": "",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-0"
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0010",
                "Location": "Port_#0001.Hub_#0005",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "",
                "Description": "USB Root Hub (USB 3.0)",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0006",
                "Location": "",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB30_HUB",
                "Description": "Generic SuperSpeed USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0021",
                "Location": "Port_#0001.Hub_#0002",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0011",
                "Location": "Port_#0001.Hub_#0008",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0018",
                "Location": "Port_#0004.Hub_#0001",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "",
                "Description": "USB Root Hub (USB 3.0)",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0003",
                "Location": "",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-1"
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0023",
                "Location": "Port_#0001.Hub_#0002",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0008",
                "Location": "Port_#0002.Hub_#0002",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "PCI\\VEN_8086&DEV_06ED&REV_00",
                "Description": "USB xHCI Compliant Host Controller",
                "DeviceName": "Intel(R) USB 3.1 eXtensible Host Controller - 1.10 (Microsoft)",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0000",
                "Location": "PCI bus 0, device 20, function 0",
                "Manufacturer": "Generic USB xHCI Host Controller",
                "Name": "PCI",
                "PhysicalName": "\\Device\\NTPNP_PCI0006"
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0031",
                "Location": "Port_#0001.Hub_#0004",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-8"
            },
            {
                "CompatibleIDs": "USB\\USB30_HUB",
                "Description": "Generic SuperSpeed USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0009",
                "Location": "Port_#0001.Hub_#0004",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB30_HUB",
                "Description": "Generic SuperSpeed USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0024",
                "Location": "Port_#0003.Hub_#0003",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "USB\\USB30_HUB",
                "Description": "Generic SuperSpeed USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0007",
                "Location": "Port_#0004.Hub_#0002",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "PCI\\VEN_8086&DEV_15F0&REV_06",
                "Description": "USB xHCI Compliant Host Controller",
                "DeviceName": "Intel(R) USB 3.1 eXtensible Host Controller - 1.10 (Microsoft)",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0005",
                "Location": "PCI bus 11, device 0, function 0",
                "Manufacturer": "Generic USB xHCI Host Controller",
                "Name": "PCI",
                "PhysicalName": ""
            },
            {
                "CompatibleIDs": "PCI\\VEN_8086&DEV_15EC&REV_06",
                "Description": "USB xHCI Compliant Host Controller",
                "DeviceName": "Intel(R) USB 3.1 eXtensible Host Controller - 1.10 (Microsoft)",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0001",
                "Location": "PCI bus 8, device 0, function 0",
                "Manufacturer": "Generic USB xHCI Host Controller",
                "Name": "PCI",
                "PhysicalName": "\\Device\\NTPNP_PCI0028"
            },
            {
                "CompatibleIDs": "USB\\USB20_HUB",
                "Description": "Generic USB Hub",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0014",
                "Location": "Port_#0003.Hub_#0005",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": ""
            }
        ],
        "valid": [
            "<b>USB Bandwidth: </b> OK"
        ]
    },
    "ZED SDK": {
        "CUDA Toolkit version": "V11.7.64 -->located in C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/bin/nvcc.exe",
        "CUDA loading": false,
        "CUDA_reachable": true,
        "ZED SDK Version (Diag)": "3.7.7",
        "ZED SDK Version (RT)": "3.7.7",
        "ai": {
            "checkAI": true,
            "cudaVersion": 11070,
            "cudnnCudartVersion": 11060,
            "cudnnVersion": 8401,
            "cudnnVersionExpected": 8401,
            "tensorVersion": 8401
        },
        "binFiles": [
            "cudnn64_8.dll",
            "cudnn_adv_infer64_8.dll",
            "cudnn_adv_train64_8.dll",
            "cudnn_cnn_infer64_8.dll",
            "cudnn_cnn_train64_8.dll",
            "cudnn_ops_infer64_8.dll",
            "cudnn_ops_train64_8.dll",
            "nvinfer.dll",
            "nvinfer_builder_resource.dll",
            "nvinfer_plugin.dll",
            "nvonnxparser.dll",
            "nvparsers.dll",
            "sl_ai64.dll",
            "sl_zed64.dll",
            "zlibwapi.dll"
        ],
        "driverVersion": "516.01",
        "drivervDllVersion": "31.0.15.1601",
        "resourcesFiles": [
            "objects_performance_2.2.model",
            "objects_performance_2.2.model_optimized-ebgbw-1-75-11070-8401-8401-5-128-1024-48-64-512-16-0"
        ],
        "resourcesFilesExpected": [
            "objects_performance_2.2"
        ],
        "runtimeDllVersion": 11070,
        "valid": [
            "<b>ZED SDK version:</b> 3.7.7",
            "<b>CUDA version:</b> V11.7.64"
        ]
    }
}

I am at a loss for what is going on. Any ideas?

2

Hi,

Can you try to run a C++ sample using the object detection you optimized and tell me if it’s working ?

The diagnostic tool is giving conflicting information about conflicting informations about whether the AI model is optimized or not .

Best,
Benjamin Vallon

Stereolabs Support

3

Hi,

It seems the detection of optimized models is not working correctly.

You can ignore the error message in Unity for the moment, the detection should work anyway.

We will investigate this issue and fix it in the next release.

Stereolabs Support

4

I’ve started seeing this message in Unity as well:

Object Detection failed to start. (Error: CORRUPTED_SDK_INSTALLATION )
UnityEngine.Debug:Log (object)
ZEDManager/<startObjectDetection>d__319:MoveNext () (at Assets/ZED/SDK/Helpers/Scripts/ZEDManager.cs:2775)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)

I’ve completely uninstalled and reinstalled the ZED SDK and I’m still seeing it.

5

I’ve started noticing this message as well:

Object Detection failed to start. (Error: CORRUPTED_SDK_INSTALLATION )
UnityEngine.Debug:Log (object)
ZEDManager/<startObjectDetection>d__319:MoveNext () (at Assets/ZED/SDK/Helpers/Scripts/ZEDManager.cs:2775)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)

I have completely deleted and reinstalled the latest SDK, but I still get the same error.

6

Hi,

Could you run the ZED Diagnostic tool available in the ZED SDK folder and send us the report at support@stereolabs.com please ?

Thanks.

Best,
Benjamin Vallon

Stereolabs Support

7

This is the output saved by the diagnostic tool. I don’t know if it’s different from the one I posted earlier.

8

Hi,

Could you tell me which Unity version you are using please ?

This issue usually happen when the AI lib can not be loaded but for the moment, I don’t know why, everything seems correctly installed.
Can you also confirm the C++ samples that uses AI models are working correctly ? Body tracking or Object detection for example.

Thanks
Best,
Benjamin Vallon

Stereolabs Support

9

I tried using the Depth Viewer and found a similar pattern. If I try to switch to Neural, I have to click apply, which brings up a prompt saying that the AI is not optimized and offering to set it up. When I clicked OK the first time, it opened a terminal that loaded and optimized the Neural model over about 20 minutes, then the Neural model took effect within that instance of Depth Viewer.

If I close and re-open the Depth Viewer, I once again have to select Neural, and hit apply, which brings up the same window offering to set it up. This time the setup was almost instantaneous, and reached the same result. It’s as if there was no work to do the second time.

ZED Diagnostic still says that NEURAL is not optimized.

It seems like there is an “optimized” flag that isn’t getting set. This can be overcome in the C++ apps because they can internally initiate the optimization process to get past it. However, the Unity version can’t (as far as I know) so the flag prevents anything from being done.

I am using Unity 2022.3.22f1, ZED unity package ZED_Unity_Plugin_v3.7.1, and ZED SDK ZED_SDK_Windows10_cuda11.7_v3.7.7.

Are there any previous versions of any of those you would recommend?

Cheers,
-Michael

10

Hi,

Sorry for the late reply, I was out of the office.

You are right. Their is a bug in this version which prevents the SDK to accurately detect if a model is optimized or not.

However, it should not affect the Unity wrapper at all. The error in Unity is just a “warning” telling you the model is not optimized as the process of optimization is not very well handled in Unity. It is done in the main thread and therefore freezes the editor during the whole optimization.

The 3.8 version should be available this week, in the mean time you can ignore the error message, or even comment it ( here https://github.com/stereolabs/zed-unity/blob/master/ZEDCamera/Assets/ZED/SDK/Helpers/Scripts/ZEDManager.cs#L2698).

I apologize for that, I understand it is not a good solution.

Best,
Benjamin Vallon

Stereolabs Support