ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNs

• ECQ refers to Entropy-Constrained (trained) Quantization as described in (Becking et al., 2020) and applied to ternary neural networks by the "EC2T" algorithm (Marban et al., 2020) as well as 4bit hardware-software co-design-aware neural networks running on FPGA and the "FantastIC4" 22nm ASIC (Wiedemann et al., 2021).

• ECQx is an eXplainability-driven version of ECQ which corrects cluster assignments based on their relevances (Becking et al., 2022).

Table of Contents

• Information
• Installation
• ECQx Usage:
  • Reproducibility of paper results
  • Demo
• Citation and Publications
• License

Information

This repository demonstrates ECQx using ResNet architectures to solve CIFAR and PASCAL VOC. How to run the code, reproduce paper results and run the demo is described in the ECQx Usage section.

[TBD: ADD MORE INFO]

Installation

The software utilizes python packages which can be installed using pip. However, core coding technologies of NNC are implemented using C++, which requires a C++ compiler for the installation process.

The software has been tested on different target platforms (mainly Linux and macOS).

Requirements

• python >= 3.6 (recommended versions 3.6, 3.7, 3.8, and 3.9) with working pip
• Windows: Microsoft Visual Studio 2015 Update 3 or later

Package installation

Note: If your NVIDIA GPU does not support CUDA 12, add the following line to the requirements.txt file:

--extra-index-url https://download.pytorch.org/whl/cu118

• On Linux/Mac, running the script create_env.sh sets up a virtual python environment "env" and installs all required packages and the software itself, automatically.

• For Windows, it is recommended to issue from the root of the cloned repository:

  pip install wheel
  pip install -r requirements.txt
  pip install .
  

For activating this environment, issue:

source env/bin/activate

Note: For further information on how to set up a virtual python environment (also on Windows) refer to https://docs.python.org/3/library/venv.html .

ECQx Usage

Before running the code, first create the environment as described in Installation, and activate it.

Execute

python run.py --help

for more detailed help on parser argument descriptions.

Logging results using Weights & Biases

We used Weights & Biases (wandb) for experiment logging. It is enabled by --wandb. If you want to use it, add your --wandb_key and optionally an experiment identifier for the run (--wandb_run_name).

Reproducibility of paper results

CIFAR-10 experiments (w/ ResNet20):

--dataset_path must be specified in accordance with your local data directories. For the CIFAR experiment, the data will be downloaded (< 200MB) to --dataset_path, which defaults to "../data", if the data is not already available there.

• Basic setting with default hyperparameters and parser arguments for running 4bit ECQ (without x) on CIFAR10 with an already pre-trained ResNet20:

  python run.py --model_path=./models/pretrained/resnet20.pt --dataset_path=<YOUR_PATH> --verbose

• Basic setting for running 4bit ECQx on CIFAR10 with an already pre-trained ResNet20:

  python run.py --lrp --model_path=./models/pretrained/resnet20.pt --dataset_path=<YOUR_PATH> --verbose

  The above command generates LRP relevances using the default "resnet" --canonizer, and the "epsilon_plus_flat_bn_pass" --lrp_composite.

  For simple network architectures, e.g., without BatchNorm modules and without residual connections, it is recommended to use "vgg" or "resnetcifar" --canonizers, otherwise "resnet" or "mobilenet" --canonizers.

  Investigating different --lrp_composites can also improve the ECQx performance: For instance, we recommend "epsilon_plus_flat_bn_pass", "epsilon_plus_flat", "alpha2_beta1_flat_bn_pass", and "alpha2_beta1_flat".

Increasing the --Lambda hyperparameter will intensify the entropy constraint and thus lead to a higher sparsity (and thus performance degradation, which can be compensated to a certain extent by ECQx).

Pascal VOC experiments:

[TBD]

Demo

[TBD]

Citation and Publications

If you use ECQx in your work, please cite:

@inproceedings{becking2022ecqx,
  author={Becking, Daniel and Dreyer, Maximilian and Samek, Wojciech and M{\"u}ller, Karsten and Lapuschkin, Sebastian},
  title={{ECQ}$^{\text{X}}$: Explainability-{D}riven {Q}uantization for {L}ow-{B}it and {S}parse {DNN}s},
  booktitle={xxAI - Beyond Explainable AI, Lecture Notes in Computer Science (LNAI Vol. 13200), Springer International Publishing},
  pages={271--296},
  year={2022},
  doi={10.1007/978-3-031-04083-2_14}
}

Publications

• (Becking et al., 2022) - "ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNs", in xxAI - Beyond Explainable AI, Lecture Notes in Computer Science (LNAI Vol. 13200), Springer International Publishing, pp. 271–296, 2022
• (Becking et al., 2023) - "NNCodec: An Open Source Software Implementation of the Neural Network Coding ISO/IEC Standard", 40th International Conference on Machine Learning (ICML), Neural Compression Workshop (Spotlight), 2023
• (Anders et al., 2021 - "Software for dataset-wide XAI: from local explanations to global insights with Zennit, CoRelAy, and ViRelAy", arXiv preprint arXiv:2106.13200, 2021
• (Wiedemann et al., 2021) - "FantastIC4: A Hardware-Software Co-Design Approach for Efficiently Running 4Bit-Compact Multilayer Perceptrons", in IEEE Open Journal of Circuits and Systems, Vol. 2, pp. 407-419, 2021
• (Marban et al., 2020) - "Learning Sparse & Ternary Neural Networks With Entropy-Constrained Trained Ternarization (EC2T)", Proceedings of the IEEE/CVF CVPR Workshops, pp. 722-723, 2020
• (Becking et al., 2020) - "Finding Storage-and Compute-Efficient Convolutional Neural Networks", Master Thesis, Technical University of Berlin, 2020

License

Please see LICENSE.txt file for the terms of the use of the contents of this repository. For Zennit and NNCodec licences please also check the license files in the according subdirectories and the current github repositories:

For more information and bug reports, please contact: daniel.becking@hhi.fraunhofer.de

Copyright (c) 2019-2024, Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.

All rights reserved.