Seminar-NAS

1. Topic
2. Papers
   1. Main Papers
   2. Groups of Papers
3. Structure
4. Links

1. Topic

NAS using evolutionary algorithms with RL and goal attainment

Keywords:

• NAS

  • Network Architecture Search
  • Neural Architecture Search

• RL

  • Reinforcement Learning

• Goal Attainment

  • Selecting the fittest individuals

2. Papers

Here, you can find a great database of all relevant publications around NAS.

2.1 Main Papers

Title	Summary
Neural Architecture Search: A Survey Base Paper	O.G. paper on the initial snapshot of NAS research. Cited 1540 times, 2019
A Survey on Evolutionary Neural Architecture Search Base Paper	More recent review/survey, specifically for Evolutionary methods! Cited 71 times, 2021
Large-Scale Evolution of Image Classifiers Evolutionary NAS	One of the first notable evolutionary search space attempt for NAS. Cited 1294 times, 2017
Neural Architecture Search with Reinforcement Learning RL NAS (Origin)	The first notable NAS paper, RL-based NAS. Cited 3979 times, 2017
DARTS Gradient-based NAS	The first notable gradient-based NAS paper, reduced significantly GPU Days down to single digits. Cited 2568 times, 2018
Regularized Evolution for Image Classifier Architecture Search Evolutionary NAS	Tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Cited 1973 times, 2019

2.2 Groups of Papers

Category	Bundle of Papers
TinyML + NAS	1. MCUNet: Tiny Deep Learning on IoT Devices 2. MnasNet: Platform-Aware Neural Architecture Search for Mobile 3. MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers 4. Tiny neural network search and implementation for embedded FPGA: a software-hardware co-design approach 5. Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search
NAS with Cells	1. Learning Transferable Architectures for Scalable Image Recognition 2. Path-Level Network Transformation for Efficient Architecture Search 3. Regularized Evolution for Image Classifier Architecture Search
NAS with Blocks	1. Learning Transferable Architectures for Scalable Image Recognition 2. Auto-creation of Effective Neural Network Architecture by Evolutionary Algorithm and ResNet for Image Classification 3. Efficient Residual Dense Block Search for Image Super-Resolution
Micro-Architecture Optimization	1. Regularized Evolution for Image Classifier Architecture Search
Rich Initialization	1. Deep convolutional networks for human sketches by means of the evolutionary deep learning

3. Paper Structure

@startuml
skinparam componentStyle rectangle

node "Base Papers" {
    A - [ Survey 1 ]
    [ Survey 2 ] - A
 }

node "Important Breakthrough Papers" {
    [ RL ] - B
    B - [ EV ]
    B -down- [ GR ]
 }

node "Evolutionary Papers" {
    C - [ EV1 ]
    [ EV2 ] - C
 }

node "tinyML + NAS Papers" {
    D - [ MicroNets ]
    [ MnasNET ] - D
 }


cloud {
     [ Paper ]
}

A --> [ Paper ]
B --> [ Paper ]
C --> [ Paper ]
D --> [ Paper ]

@enduml

4. Links

• Literature on NAS
  • Database
• Microsoft Research NAS Lecture:
  • Great video on NAS, heavily based on the 2019 Survey Paper!
• Microsoft Research: NAS Directions
  • Great video on possible NAS directions and what tools are used around it

Questions

• Question
  • R: Reply