CNN Model: AlexNet

AlexNet is the name of a convolutional neural network, designed by Alex Krizhevsky,¹ and published with Ilya Sutskever and Krizhevsky’s PhD advisor Geoffrey Hinton,²³ who was originally resistant to the idea of his student.¹⁴

Refer to: 13.2 Build AlexNet and test on CIFAR10

l2_reg = 0

# Initialize model
model = Sequential()

# 1st Conv Layer 
model.add(Conv2D(96, (11, 11), input_shape=x_train.shape[1:],
    padding='same', kernel_regularizer=l2(l2_reg)))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# 2nd Conv Layer 
model.add(Conv2D(256, (5, 5), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Conv Layer 
model.add(ZeroPadding2D((1, 1)))
model.add(Conv2D(512, (3, 3), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# 4th Conv Layer 
model.add(ZeroPadding2D((1, 1)))
model.add(Conv2D(1024, (3, 3), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))

# 5th Conv Layer 
model.add(ZeroPadding2D((1, 1)))
model.add(Conv2D(1024, (3, 3), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# 1st FC Layer
model.add(Flatten())
model.add(Dense(3072))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.5))

# 2nd FC Layer
model.add(Dense(4096))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.5))

# 3rd FC Layer
model.add(Dense(num_classes))
model.add(BatchNormalization())
model.add(Activation('softmax'))

print(model.summary())

model.compile(loss = 'categorical_crossentropy',
              optimizer = keras.optimizers.Adadelta(),
              metrics = ['accuracy'])

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 32, 32, 96)        34944     
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 32, 96)        384       
_________________________________________________________________
activation_1 (Activation)    (None, 32, 32, 96)        0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 96)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 16, 16, 256)       614656    
_________________________________________________________________
batch_normalization_2 (Batch (None, 16, 16, 256)       1024      
_________________________________________________________________
activation_2 (Activation)    (None, 16, 16, 256)       0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 256)         0         
_________________________________________________________________
zero_padding2d_1 (ZeroPaddin (None, 10, 10, 256)       0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 10, 10, 512)       1180160   
_________________________________________________________________
batch_normalization_3 (Batch (None, 10, 10, 512)       2048      
_________________________________________________________________
activation_3 (Activation)    (None, 10, 10, 512)       0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 5, 5, 512)         0         
_________________________________________________________________
zero_padding2d_2 (ZeroPaddin (None, 7, 7, 512)         0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 7, 7, 1024)        4719616   
_________________________________________________________________
batch_normalization_4 (Batch (None, 7, 7, 1024)        4096      
_________________________________________________________________
activation_4 (Activation)    (None, 7, 7, 1024)        0         
_________________________________________________________________
zero_padding2d_3 (ZeroPaddin (None, 9, 9, 1024)        0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 9, 9, 1024)        9438208   
_________________________________________________________________
batch_normalization_5 (Batch (None, 9, 9, 1024)        4096      
_________________________________________________________________
activation_5 (Activation)    (None, 9, 9, 1024)        0         
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 4, 4, 1024)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 16384)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 3072)              50334720  
_________________________________________________________________
batch_normalization_6 (Batch (None, 3072)              12288     
_________________________________________________________________
activation_6 (Activation)    (None, 3072)              0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 3072)              0         
_________________________________________________________________
dense_2 (Dense)              (None, 4096)              12587008  
_________________________________________________________________
batch_normalization_7 (Batch (None, 4096)              16384     
_________________________________________________________________
activation_7 (Activation)    (None, 4096)              0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 4096)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 10)                40970     
_________________________________________________________________
batch_normalization_8 (Batch (None, 10)                40        
_________________________________________________________________
activation_8 (Activation)    (None, 10)                0         
=================================================================
Total params: 78,990,642
Trainable params: 78,970,462
Non-trainable params: 20,180
_________________________________________________________________