Machine Learning Algorithms I worked on using keras' MNIST dataset. My testing data is of 10,000 images but for sanity's sake I also drew some numbers on Paint.NET by resizing it to 28x28 pixels as my own testing data. The best solution, using CNN, ranked 58th out of 2,468 teams (Top 2.35%) on Kaggle's Digit Recognizer Competition.
For some reason GitHub seems to be having a problem on their backend sometimes. I just find that restarting it at a later time works. If the issue doesn't fix itself you can always use Jupyter's nbviewer to render and view the files. You can also just click the links in the table below to view the *.ipynb files.
| Model Name | Description | Testing Accuracy Rate % | Kaggle Public Leaderboard % |
|---|---|---|---|
| FCModel | Fully Connected Neural Network | ~96 | ~97 |
| SimpleConvModel | Straightforward Convolutional Model | ~98.5 | N/A |
| ConvModel_v1 | Convolutional Model | ~99.3 | ~99.9 |
| ConvModel_v2 | Changed placement of BN and added a dynamic learning rate |
~99.6 | ~99.94 |
INPUT (1x784) -> [FC -> RELU]*2 -> [FC -> SOFTMAX]
INPUT (28x28x1) -> [CONV2D -> RELU]*2 -> MAXPOOL2D -> [FC -> SOFTMAX]
INPUT (28x28x1) -> [[CONV2D -> RELU]*2 -> MAXPOOL2D -> BATCHNORMALIZATION]*2 -> [FC -> RELU]
-> [FC -> SOFTMAX]
INPUT (28x28x1) -> [[CONV2D -> RELU] -> BATCHNORMALIZATION]*2 -> MAXPOOL2D -> DROPOUT]*2 -> [FC -> RELU]
-> [FC -> SOFTMAX]
Although the authors of the original research paper on BatchNormalization have indicated that it should be included between linear and non-linear layers it has been found in practice to yield better results when adding it after the activation layer.
| Package | Version |
|---|---|
| numpy | 1.15.4 |
| matplotlib | 3.0.2 |
| keras | 2.2.4 |
| Pillow | 5.4.1 |
| tensorflow | 1.12.0 |