DNNMark is now supporting MIOpen. Right now DNNMark can run on both AMD and Nvidia platform. HCC, HIP, MIOpen and miopengemm are required in order to build MIOpen version of DNNMark.
Bibtex is as below:
@inproceedings{Dong:2017:DDN:3038228.3038239,
author = {Dong, Shi and Kaeli, David},
title = {DNNMark: A Deep Neural Network Benchmark Suite for GPUs},
booktitle = {Proceedings of the General Purpose GPUs},
series = {GPGPU-10},
year = {2017},
isbn = {978-1-4503-4915-4},
location = {Austin, TX, USA},
pages = {63--72},
numpages = {10},
url = {http://doi.acm.org/10.1145/3038228.3038239},
doi = {10.1145/3038228.3038239},
acmid = {3038239},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {Benchmark Suite, Deep Neural Network, GPU, cuDNN},
}
[1] Shi Dong and David Kaeli. 2017. DNNMark: A Deep Neural Network Benchmark Suite for GPUs. In Proceedings of the General Purpose GPUs (GPGPU-10). ACM, New York, NY, USA, 63-72.
[2] Shi Dong, Xiang Gong, Yifan Sun, Trinayan Baruah, and David Kaeli. 2018. Characterizing the Microarchitectural Implications of a Convolutional Neural Network (CNN) Execution on GPUs. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering (ICPE '18). ACM, New York, NY, USA, 96-106. DOI: https://doi.org/10.1145/3184407.3184423 (Best Paper Award)
[3] Yifan Sun, Saoni Mukherjee, Trinayan Baruah, Shi Dong, Julian Gutierrez, Prannoy Mohan, David Kaeli. 2018. Evaluating Performance Tradeoffs on the Radeon Open Compute Platform. ISPASS 2018
[4] Shi Dong, Pu Zhao, Xue Lin, David Kaeli. 2020. Exploring GPU acceleration of Deep Neural Networks using Block Circulant Matrices. Parallel Computing
[5] Shi Dong, Yifan Sun, Nicolas Bohm Agostini, Elmira Karimi, Daniel Lowell, Jing Zhou, Jose Cano, Jose L. Abellan, David Kaeli, "Spartan: A Sparsity-Adaptive Framework to Accelerate Deep Neural Network Training on GPUs," in IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 10, pp. 2448-2463, 1 Oct. 2021
Configurable benchmark suite of Deep Neural Networks
DNNMark is a highly configurable, extensible, and flexible Deep Neural Network benchmark framework. In this benchmark suite, each of DNN primitive workloads can be easily invoked separately, without any sacrifice on configurability. One can specify any type of scenarios for benchmarking on an algorithm-specific level. Second, the actual measurement can be done during the execution of any specific kernel. Our framework allows us to ignore the setup stages and focus on only the training steps. Unlike other deep learning frameworks, attaching a real database for training purposes is not mandatory anymore. This capability will greatly aid the computer architecture community, which is more interested in designing/tuning hardware/software, and less interested in the details or configuration of the deep neural net.
Depending on the specific configuration, deep neural networks can involve combinations of DNN primitives. A model composed of two or more primitive functions may be more desirable in terms of performance evaluation. In such cases, a composed model rather than standalone primitives, are preferred. To provide this capability, DNNmark can be extended to more sophisticated DNN models, where layers are connected to, and dependent upon, each other.
This frame work provides configurability in both general and algorithm specific parameters. Users can do this through a plain-text configuration file. Several examples are provided inside config_example directory.
New DNN models/scenarios can be easily built for benchmarking through configuration files
Designing benchmarks takes little effort thanks to its centralized library
DNNMark contains commonly-used DNN primitives and also provides an easy approach to compose a model
- Configurable
- Provide insights of GPU execution behavior using detailed GPU metrics/events
- Separatable or combined DNN primitives benchmarking
- Convolution forward and backward
- Pooling forward and backward
- LRN forward and backward
- Activation forward and backward
- Fully Connected forward and backward
- Softmax forward and backward
- Batch Normalization forward and backward
- Dropout forward and backward
OS: Ubuntu 16.04
CUDA related library: CUDA tool kit v8.0; CuDNN v5.0
Other Software: CMake v3.5.1; g++ v5.4.0
Google libraries: gflags (sudo apt-get install libgflags-dev); glog(sudo apt-get install libgoogle-glog-dev)
There is one known problem regarding gflags. Sometimes, the compile process complains that ‘gflags’ has not been declared. This could be related to the gflag version used. So the solution could be either downloading a latest one or changing the namespace name from 'gflags' to 'google'
After you download and unzip the DNNMark, you should go to its root directory and edit setup.sh to set up path to cuDNN. And then run ./setup.sh. This will create a build directory and run cmake automatically. To build the code, go to build directory build and run make
To run the benchmarks that have been built, go to the directory build and you will see a directory benchmarks. Go inside and select the benchmark you want to run. Run command ./[name of benchmark] -config [path to config file] -debuginfo [1 or 0] to execute the benchmark.
If you want to include warm up phase in the benchmark, one suggestion is that you mannually add it in the source code and use -warmup [1 or 0] to toggle the usage of the warm up phase. You can find one example the test_alexnet benchmark.
- Fork the repository to your own remote repository.
- Git clone the repository:
git clone git@github.com/your_account_name/DNNMark.git - Add this project as an upstream to your local repository by
git remote add upstream https://github.com/doody1986/DNNMark.git. You can usegit remote -vto view the upstream. - Create your own local feature branch:
git checkout -b your-own-feature-branch develop - Make your own feature branch visible by pushing it to your own remote repository (DO NOT PUSH IT TO THE DEVELOP BRANCH):
git push --set-upstream origin your-own-feature-branch - Develop your own feature branch in your local repository:
git add,git commit, etc.. - After your own branch is completed, make sure to merge the latest change from upstream develop branch to your own origin develop branch: 1)
git checkout develop2)git pull upstream develop3)git push origin develop - Since that you have the latest change in your own origin develop branch from upstream one, now you can update your own feature branch on the your own remote repository by: 1)
git checkout your-own-feature-branch2)git pull origin develop3)git push origin your-own-feature-branch - Make a pull request from your own feature branch on your own remote repository on github to the develop branch of this repository.
- After the pull request is merged, you can delete your own feature branch by 1)
git push origin --delete your-own-feature-branchto delete the remote branch and 2)git branch -d your-own-feature-branchto delete your local branch. - More instructions on using fork can be found here.