[Task]: How to improve OpenVINO Inferece speed?

[glucasol]

What is the motivation for this task?

I am running this Notebook - 501a_training_a_model_with_cubes_from_a_robotic_arm.ipynb and getting ~16 FPS in inference (CPU: Intel(R) Core(TM) i5-9300H CPU @ 2.40GHz; GPU: NVIDIA GeForce GTX 1650 (dGPU)). I want to know if it's possible to improve it's performance and get more FPS in OpenVINO Inference.

Thanks!

Describe the solution you'd like

I would like to improve the inference performance.

Additional context

No response

[samet-akcay]

Hi @glucasol, we'll get back to you asap! Cheers!

[glucasol]

Thank you @samet-akcay!

[samet-akcay]

@glucasol, one potential approach would be to use NNCF that would accelerate the inference. @paularamo was planning on expanding the notebook with NNCF support, but she is currently out of office. To get an idea you could refer to this NNCF notebook.
https://github.com/openvinotoolkit/anomalib/blob/main/notebooks/400_openvino/401_nncf.ipynb

We will hopefully add more to the robotics arm notebook soon.
https://github.com/openvinotoolkit/anomalib/blob/main/notebooks/500_use_cases/501_dobot/501a_training_a_model_with_cubes_from_a_robotic_arm.ipynb

[samet-akcay]

Let's continue this from the discussions section. Cheers!

[glucasol]

@samet-akcay , following your suggestion, I have modified this notebook 501a_training_a_model_with_cubes_from_a_robotic_arm.ipynb in the Callbacks section, to include nncf optimization, so my callbacks list looks like:

from anomalib.post_processing import NormalizationMethod, ThresholdMethod
from anomalib.utils.callbacks import (
    MetricsConfigurationCallback,
    MinMaxNormalizationCallback,
    PostProcessingConfigurationCallback,
)
from anomalib.utils.callbacks.export import ExportCallback, ExportMode
from anomalib.utils.callbacks.nncf.callback import NNCFCallback

callbacks = [
    MetricsConfigurationCallback(
        task=TaskType.CLASSIFICATION,
        image_metrics=["AUROC"],
    ),
    ModelCheckpoint(
        mode="max",
        monitor="image_AUROC",
    ),
    PostProcessingConfigurationCallback(
        normalization_method=NormalizationMethod.MIN_MAX,
        threshold_method=ThresholdMethod.ADAPTIVE,
    ),
    MinMaxNormalizationCallback(),
    ExportCallback(
        input_size=(256, 256),
        dirpath=str(Path.cwd()),
        filename="model",
        export_mode=ExportMode.OPENVINO,
    ),
    NNCFCallback(
        config={
                "apply": True,
                "input_info": {"sample_size": [1, 3, 256, 256]},
                "compression": {
                                "algorithm": "quantization",
                                "preset": "mixed",
                                "initializer": {"range": {"num_init_samples": 250}, "batchnorm_adaptation": {"num_bn_adaptation_samples": 250}},
                                },
                },
        export_dir=str(Path.cwd())
    )
]

and when I run trainer.fit, it returns me the following error:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/gabrieloliveira/anaconda3/envs/anomaly/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:67: UserWarning: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `pytorch_lightning` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default
  warning_cache.warn(
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
/home/gabrieloliveira/anaconda3/envs/anomaly/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `ROC` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)
INFO:nncf:Collecting tensor statistics |██              | 1 / 8
INFO:nncf:Collecting tensor statistics |████            | 2 / 8
INFO:nncf:Collecting tensor statistics |██████          | 3 / 8
INFO:nncf:Scales will be unified for quantizer group:
PadimModel/FeatureExtractor[feature_extractor]/FeatureListNet[feature_extractor]/Sequential[layer1]/BasicBlock[1]/ReLU[act2]/relu__0|OUTPUT
PadimModel/interpolate_0|OUTPUT
PadimModel/interpolate_1|OUTPUT

INFO:nncf:Compiling and loading torch extension: quantized_functions_cpu...
INFO:nncf:Finished loading torch extension: quantized_functions_cpu
INFO:nncf:BatchNorm statistics adaptation |██              | 1 / 8
INFO:nncf:BatchNorm statistics adaptation |████            | 2 / 8
INFO:nncf:BatchNorm statistics adaptation |██████          | 3 / 8
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/gabrieloliveira/anaconda3/envs/anomaly/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py:183: UserWarning: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer
  rank_zero_warn(

  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AnomalyScoreThreshold    | 0     
1 | pixel_threshold       | AnomalyScoreThreshold    | 0     
2 | model                 | PadimModel               | 2.8 M 
3 | image_metrics         | AnomalibMetricCollection | 0     
4 | pixel_metrics         | AnomalibMetricCollection | 0     
5 | normalization_metrics | MinMax                   | 0     
-------------------------------------------------------------------
2.8 M     Trainable params
57        Non-trainable params
2.8 M     Total params
11.141    Total estimated model params size (MB)
/home/gabrieloliveira/anaconda3/envs/anomaly/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (3) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(
Epoch 0:   0%|          | 0/5 [00:00<?, ?it/s] INFO:nncf:Compiling and loading torch extension: quantized_functions_cuda...
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File ~/anaconda3/envs/anomaly/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1893, in _run_ninja_build(build_directory, verbose, error_prefix)
   1892     stdout_fileno = 1
-> 1893     subprocess.run(
   1894         command,
   1895         stdout=stdout_fileno if verbose else subprocess.PIPE,
   1896         stderr=subprocess.STDOUT,
   1897         cwd=build_directory,
   1898         check=True,
   1899         env=env)
   1900 except subprocess.CalledProcessError as e:
   1901     # Python 2 and 3 compatible way of getting the error object.

File ~/anaconda3/envs/anomaly/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
...
     96         "guidance) and that 'nvcc' is available on your system's PATH variable.\n"
     97     ) from e

RuntimeError: CUDA is available for PyTorch, but NNCF could not compile GPU quantization extensions. Make sure that you have installed CUDA development tools (see https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html for guidance) and that 'nvcc' is available on your system's PATH variable.

When I run nvcc -V:

Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

When I run nvidia-smi:

Mon Aug  7 15:31:08 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1650        On  | 00000000:01:00.0 Off |                  N/A |
| N/A   41C    P8               3W /  50W |    103MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1455      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      3125      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      7735      C   ...a/anaconda3/envs/anomaly/bin/python       90MiB |
+---------------------------------------------------------------------------------------+

[samet-akcay]

@AlexanderDokuchaev, do you have any insights here please?

[AlexanderDokuchaev]

Need to check that nvcc is installed and the PATH LD_LIBRARY_PATH variables contain paths to cuda

export PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

$ whereis nvcc
nvcc: /usr/local/cuda-12.2/bin/nvcc

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

[AlexanderDokuchaev]

Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Looks like old nvcc.

[samet-akcay]

Thanks @AlexanderDokuchaev !

[glucasol]

Thank you guys @AlexanderDokuchaev @samet-akcay for your support!
I was able to train with nncf quantization besides both models (quantized or not) has the same size and I see some improvements in benchmark ( 42.08 FPS for non quantized and 72.32 FPS for quantized model).

By the way, Do you know if its possible to run OpenVINO inference on Jetson devices?

[AlexanderDokuchaev]

Jetson Nano supported
https://docs.openvino.ai/2023.0/openvino_docs_OV_UG_supported_plugins_Supported_Devices.html