diff --git a/module2-backpropagation/LS_DS_422_Backprop_Assignment.ipynb b/module2-backpropagation/LS_DS_422_Backprop_Assignment.ipynb index 4a32868c..4e620f7d 100644 --- a/module2-backpropagation/LS_DS_422_Backprop_Assignment.ipynb +++ b/module2-backpropagation/LS_DS_422_Backprop_Assignment.ipynb @@ -1,304 +1,304 @@ { - "cells": [ - { - "cell_type": "markdown", - "source": [ - "\n", - "

\n", - "\n", - "# Backpropagation Practice\n", - "\n", - "## *Data Science Unit 4 Sprint 2 Assignment 2*\n", - "\n", - "Using TensorFlow Keras, Implement a 3 input, 4 node hidden-layer, 1 output node Multilayer Perceptron on the following dataset:\n", - "\n", - "| x1 | x2 | x3 | y |\n", - "|----|----|----|---|\n", - "| 0 | 0 | 1 | 0 |\n", - "| 0 | 1 | 1 | 1 |\n", - "| 1 | 0 | 1 | 1 |\n", - "| 0 | 1 | 0 | 1 |\n", - "| 1 | 0 | 0 | 1 |\n", - "| 1 | 1 | 1 | 0 |\n", - "| 0 | 0 | 0 | 0 |\n", - "\n", - "If you look at the data you'll notice that the first two columns behave like an XOR gate while the last column is mostly just noise. Remember that creating an XOR gate was what the perceptron was criticized for not being able to learn.\n", - "\n", - "This is your \"Hello World!\" of TensorFlow.\n", - "\n", - "### Example TensorFlow Starter Code\n", - "\n", - "```python \n", - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense\n", - "\n", - "model = Sequential([\n", - " Dense(3, activation='sigmoid', input_dim=2),\n", - " Dense(1, activation='sigmoid')\n", - "])\n", - "\n", - "model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['acc'])\n", - "\n", - "results = model.fit(X,y, epochs=100)\n", - "\n", - "```\n", - "\n", - "### Additional Written Tasks:\n", - "1. Investigate the various [loss functions](https://www.tensorflow.org/api_docs/python/tf/keras/losses). Which is best suited for the task at hand (predicting 1 / 0) and why? \n", - "2. What is the difference between a loss function and a metric? Why might we need both in Keras? \n", - "3. Investigate the various [optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers). Stochastic Gradient Descent (`sgd`) is not the learning algorithm dejour anyone. Why is that? What do newer optimizers such as `adam` have to offer? " - ], - "metadata": { - "colab_type": "text", - "id": "NGGrt9EYlCqY" - } - }, - { - "cell_type": "code", - "source": [ - "##### Your Code Here #####\n", - "\n" - ], - "outputs": [], - "execution_count": null, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "nEREYT-3wI1f" - } - }, - { - "cell_type": "markdown", - "source": [ - "### Build a Tensor Keras Perceptron\n", - "\n", - "Try to match the architecture we used on Monday - inputs nodes and one output node. Apply this architecture to the XOR-ish dataset above. \n", - "\n", - "After fitting your model answer these questions: \n", - "\n", - "Are you able to achieve the same results as a bigger architecture from the first part of the assignment? Why is this disparity the case? What properties of the XOR dataset would cause this disparity? \n", - "\n", - "Now extrapolate this behavior on a much larger dataset in terms of features. What kind of architecture decisions could we make to avoid the problems the XOR dataset presents at scale? \n", - "\n", - "*Note:* The bias term is baked in by default in the Dense layer." - ], - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "### Compare " - ], - "outputs": [], - "execution_count": null, - "metadata": {} - }, - { - "cell_type": "markdown", - "source": [ - "## Try building/training a more complex MLP on a bigger dataset.\n", - "\n", - "Use TensorFlow Keras & the [MNIST dataset](http://yann.lecun.com/exdb/mnist/) to build the canonical handwriting digit recognizer and see what kind of accuracy you can achieve. \n", - "\n", - "If you need inspiration, the Internet is chalk-full of tutorials, but I want you to see how far you can get on your own first. I've linked to the original MNIST dataset above but it will probably be easier to download data through a neural network library. If you reference outside resources make sure you understand every line of code that you're using from other sources, and share with your fellow students helpful resources that you find.\n", - "\n", - "\n", - "### Parts\n", - "1. Gathering & Transforming the Data\n", - "2. Making MNIST a Binary Problem\n", - "3. Estimating your Neural Network (the part you focus on)" - ], - "metadata": { - "colab_type": "text", - "id": "8b-r70o8p2Dm" - } - }, - { - "cell_type": "markdown", - "source": [ - "### Gathering the Data \n", - "\n", - "`keras` has a handy method to pull the mnist dataset for you. You'll notice that each observation is a 28x28 arrary which represents an image. Although most Neural Network frameworks can handle higher dimensional data, that is more overhead than necessary for us. We need to flatten the image to one long row which will be 784 values (28X28). Basically, you will be appending each row to one another to make on really long row. " - ], - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "import numpy as np\n", - "from tensorflow.keras.datasets import mnist\n", - "from tensorflow.keras.utils import to_categorical" - ], - "outputs": [], - "execution_count": 1, - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "# input image dimensions\n", - "img_rows, img_cols = 28, 28" - ], - "outputs": [], - "execution_count": 2, - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "# the data, split between train and test sets\n", - "(x_train, y_train), (x_test, y_test) = mnist.load_data()" - ], - "outputs": [], - "execution_count": 3, - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "x_train = x_train.reshape(x_train.shape[0], img_rows * img_cols)\n", - "x_test = x_test.reshape(x_test.shape[0], img_rows * img_cols)\n", - "\n", - "# Normalize Our Data\n", - "x_train = x_train / 255\n", - "x_test = x_test / 255" - ], - "outputs": [], - "execution_count": 32, - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "# Now the data should be in a format you're more familiar with\n", - "x_train.shape" - ], - "outputs": [ - { - "output_type": "execute_result", - "execution_count": 4, - "data": { - "text/plain": [ - "(60000, 28, 28)" - ] - }, - "metadata": {} - } - ], - "execution_count": 4, - "metadata": {} - }, - { - "cell_type": "markdown", - "source": [ - "### Making MNIST a Binary Problem \n", - "MNIST is multiclass classification problem; however we haven't covered all the necessary techniques to handle this yet. You would need to one-hot encode the target, use a different loss metric, and use softmax activations for the last layer. This is all stuff we'll cover later this week, but let us simplify the problem for now: Zero or all else." - ], - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "import numpy as np\n", - "\n", - "y_temp = np.zeros(y_train.shape)\n", - "y_temp[np.where(y_train == 0.0)[0]] = 1\n", - "y_train = y_temp\n", - "\n", - "y_temp = np.zeros(y_test.shape)\n", - "y_temp[np.where(y_test == 0.0)[0]] = 1\n", - "y_test = y_temp" - ], - "outputs": [], - "execution_count": 5, - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "# A Nice Binary target for ya to work with\n", - "y_train" - ], - "outputs": [ - { - "output_type": "execute_result", - "execution_count": 6, - "data": { - "text/plain": [ - "array([0., 1., 0., ..., 0., 0., 0.])" - ] - }, - "metadata": {} - } - ], - "execution_count": 6, - "metadata": {} - }, - { - "cell_type": "markdown", - "source": [ - "### Estimating Your `net" - ], - "metadata": {} - }, - { - "cell_type": "code", - "source": [ - "##### Your Code Here #####\n", - "\n" - ], - "outputs": [], - "execution_count": null, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "5MOPtYdk1HgA" - } - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "NGGrt9EYlCqY" + }, + "source": [ + "\n", + "

\n", + "\n", + "# Backpropagation Practice\n", + "\n", + "## *Data Science Unit 4 Sprint 2 Assignment 2*\n", + "\n", + "Using TensorFlow Keras, Implement a 3 input, 4 node hidden-layer, 1 output node Multilayer Perceptron on the following dataset:\n", + "\n", + "| x1 | x2 | x3 | y |\n", + "|----|----|----|---|\n", + "| 0 | 0 | 1 | 0 |\n", + "| 0 | 1 | 1 | 1 |\n", + "| 1 | 0 | 1 | 1 |\n", + "| 0 | 1 | 0 | 1 |\n", + "| 1 | 0 | 0 | 1 |\n", + "| 1 | 1 | 1 | 0 |\n", + "| 0 | 0 | 0 | 0 |\n", + "\n", + "If you look at the data you'll notice that the first two columns behave like an XOR gate while the last column is mostly just noise. Remember that creating an XOR gate was what the perceptron was criticized for not being able to learn.\n", + "\n", + "This is your \"Hello World!\" of TensorFlow.\n", + "\n", + "### Example TensorFlow Starter Code\n", + "\n", + "```python \n", + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras.layers import Dense\n", + "\n", + "model = Sequential([\n", + " Dense(3, activation='sigmoid', input_dim=2),\n", + " Dense(1, activation='sigmoid')\n", + "])\n", + "\n", + "model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['acc'])\n", + "\n", + "results = model.fit(X,y, epochs=100)\n", + "\n", + "```\n", + "\n", + "### Additional Written Tasks:\n", + "1. Investigate the various [loss functions](https://www.tensorflow.org/api_docs/python/tf/keras/losses). Which is best suited for the task at hand (predicting 1 / 0) and why? \n", + "2. What is the difference between a loss function and a metric? Why might we need both in Keras? \n", + "3. Investigate the various [optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers). Stochastic Gradient Descent (`sgd`) is not the learning algorithm dejour anyone. Why is that? What do newer optimizers such as `adam` have to offer? " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "nEREYT-3wI1f" + }, + "outputs": [], + "source": [ + "##### Your Code Here #####\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Build a Tensor Keras Perceptron\n", + "\n", + "Try to match the architecture we used on Monday - inputs nodes and one output node. Apply this architecture to the XOR-ish dataset above. \n", + "\n", + "After fitting your model answer these questions: \n", + "\n", + "Are you able to achieve the same results as a bigger architecture from the first part of the assignment? Why is this disparity the case? What properties of the XOR dataset would cause this disparity? \n", + "\n", + "Now extrapolate this behavior on a much larger dataset in terms of features. What kind of architecture decisions could we make to avoid the problems the XOR dataset presents at scale? \n", + "\n", + "*Note:* The bias term is baked in by default in the Dense layer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "### Compare " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "8b-r70o8p2Dm" + }, + "source": [ + "## Try building/training a more complex MLP on a bigger dataset.\n", + "\n", + "Use TensorFlow Keras & the [MNIST dataset](http://yann.lecun.com/exdb/mnist/) to build the canonical handwriting digit recognizer and see what kind of accuracy you can achieve. \n", + "\n", + "If you need inspiration, the Internet is chalk-full of tutorials, but I want you to see how far you can get on your own first. I've linked to the original MNIST dataset above but it will probably be easier to download data through a neural network library. If you reference outside resources make sure you understand every line of code that you're using from other sources, and share with your fellow students helpful resources that you find.\n", + "\n", + "\n", + "### Parts\n", + "1. Gathering & Transforming the Data\n", + "2. Making MNIST a Binary Problem\n", + "3. Estimating your Neural Network (the part you focus on)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Gathering the Data \n", + "\n", + "`keras` has a handy method to pull the mnist dataset for you. You'll notice that each observation is a 28x28 arrary which represents an image. Although most Neural Network frameworks can handle higher dimensional data, that is more overhead than necessary for us. We need to flatten the image to one long row which will be 784 values (28X28). Basically, you will be appending each row to one another to make on really long row. " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "from tensorflow.keras.datasets import mnist\n", + "from tensorflow.keras.utils import to_categorical" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# input image dimensions\n", + "img_rows, img_cols = 28, 28" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# the data, split between train and test sets\n", + "(x_train, y_train), (x_test, y_test) = mnist.load_data()" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [], + "source": [ + "x_train = x_train.reshape(x_train.shape[0], img_rows * img_cols)\n", + "x_test = x_test.reshape(x_test.shape[0], img_rows * img_cols)\n", + "\n", + "# Normalize Our Data\n", + "x_train = x_train / 255\n", + "x_test = x_test / 255" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "source": [ - "## Stretch Goals: \n", - "\n", - "- Make MNIST a multiclass problem using cross entropy & soft-max\n", - "- Implement Cross Validation model evaluation on your MNIST implementation \n", - "- Research different [Gradient Descent Based Optimizers](https://keras.io/optimizers/)\n", - " - [Siraj Raval the evolution of gradient descent](https://www.youtube.com/watch?v=nhqo0u1a6fw)\n", - "- Build a housing price estimation model using a neural network. How does its accuracy compare with the regression models that we fit earlier on in class?" - ], - "metadata": { - "colab_type": "text", - "id": "FwlRJSfBlCvy" - } + "data": { + "text/plain": [ + "(60000, 28, 28)" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" } - ], - "metadata": { - "colab": { - "name": "LS_DS_432_Backprop_Assignment.ipynb", - "provenance": [], - "version": "0.3.2" - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, - "nteract": { - "version": "0.22.4" + ], + "source": [ + "# Now the data should be in a format you're more familiar with\n", + "x_train.shape" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Making MNIST a Binary Problem \n", + "MNIST is multiclass classification problem; however we haven't covered all the necessary techniques to handle this yet. You would need to one-hot encode the target, use a different loss metric, and use softmax activations for the last layer. This is all stuff we'll cover later this week, but let us simplify the problem for now: Zero or all else." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "y_temp = np.zeros(y_train.shape)\n", + "y_temp[np.where(y_train == 0.0)[0]] = 1\n", + "y_train = y_temp\n", + "\n", + "y_temp = np.zeros(y_test.shape)\n", + "y_temp[np.where(y_test == 0.0)[0]] = 1\n", + "y_test = y_temp" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([0., 1., 0., ..., 0., 0., 0.])" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" } + ], + "source": [ + "# A Nice Binary target for ya to work with\n", + "y_train" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Estimating Your `net" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "5MOPtYdk1HgA" + }, + "outputs": [], + "source": [ + "##### Your Code Here #####\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "FwlRJSfBlCvy" + }, + "source": [ + "## Stretch Goals: \n", + "\n", + "- Make MNIST a multiclass problem using cross entropy & soft-max\n", + "- Implement Cross Validation model evaluation on your MNIST implementation \n", + "- Research different [Gradient Descent Based Optimizers](https://keras.io/optimizers/)\n", + " - [Siraj Raval the evolution of gradient descent](https://www.youtube.com/watch?v=nhqo0u1a6fw)\n", + "- Build a housing price estimation model using a neural network. How does its accuracy compare with the regression models that we fit earlier on in class?" + ] + } + ], + "metadata": { + "colab": { + "name": "LS_DS_432_Backprop_Assignment.ipynb", + "provenance": [], + "version": "0.3.2" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.2" }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file + "nteract": { + "version": "0.22.4" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/module2-backpropagation/LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb b/module2-backpropagation/LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb index 57faa0ff..43f342b3 100644 --- a/module2-backpropagation/LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb +++ b/module2-backpropagation/LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb @@ -1,2406 +1,2404 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "kernelspec": { - "display_name": "U4-S2-NNF (Python 3.7)", - "language": "python", - "name": "u4-s2-nnf" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "aS4GZ37Wgcjr" + }, + "source": [ + "Lambda School Data Science\n", + "\n", + "*Unit 4, Sprint 2, Module 2*\n", + "\n", + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "etFf1WLWgcjt", + "toc-hr-collapsed": false + }, + "source": [ + "# Backpropagation & Gradient Descent (Prepare)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hXB80QOhgcju" + }, + "source": [ + "## Learning Objectives\n", + "* Part 1: Explain the intutition behind backproprogation\n", + "* Part 2: Implement gradient descent + backpropagation on a feedforward neural network. \n", + "* Part 3: Introduce the Keras Sequential Model API" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "8YuQu2lfgcju" + }, + "source": [ + "## Summary of Yesterday\n", + "\n", + "Yesterday, we learned about some of the principal components of Neural Networks: Neurons, Weights, Activation Functions, and layers (input, output, & hidden). Today, we will reinforce our understanding of those components and introduce the mechanics of training a neural network. Feed-forward neural networks, such as multi-layer perceptrons (MLPs), are almost always trained using some variation of gradient descent where the gradient has been calculated by backpropagation.\n", + "\n", + "
\n", + "\n", + "- There are three kinds of layers: input, hidden, and output layers.\n", + "- Each layer is made up of **n** individual neurons (aka activation units) which have a corresponding weight and bias.\n", + "- Signal is passed from layer to layer through a network by:\n", + " - Taking in inputs from the training data (or previous layer)\n", + " - Multiplying each input by its corresponding weight (think arrow/connecting line)\n", + " - Adding a bias to this weighted some of inputs and weights\n", + " - Activating this weighted sum + bias by squishifying it with sigmoid or some other activation function. With a single perceptron with three inputs, calculating the output from the node is done like so:\n", + "\\begin{align}\n", + " y = sigmoid(\\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)\n", + "\\end{align}\n", + " - this final activated value is the signal that gets passed onto the next layer of the network.\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bpi4R03rgcjv" + }, + "source": [ + "## Training a Neural Network: *Formal Summary*\n", + "\n", + "0. Pick a network architecture\n", + " - No. of input units = No. of features\n", + " - No. of output units = Number of Classes (or expected targets)\n", + " - Select the number of hidden layers and number of neurons within each hidden layer\n", + "1. Randomly initialize weights\n", + "2. Implement forward propagation to get $h_{\\theta}(x^{(i)})$ for any $x^{(i)}$\n", + "3. Implement code to compute a cost function $J(\\theta)$\n", + "4. Implement backpropagation to compute partial derivatives $\\frac{\\delta}{\\delta\\theta_{jk}^{l}}{J(\\theta)}$\n", + "5. Use gradient descent (or other advanced optimizer) with backpropagation to minimize $J(\\theta)$ as a function of parameters $\\theta\\$\n", + "6. Repeat steps 2 - 5 until cost function is 'minimized' or some other stopping criteria is met. One pass over steps 2 - 5 is called an iteration or epoch." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "z04QRNGagcjv", + "toc-hr-collapsed": false + }, + "source": [ + "## Calculating *\"cost\"*, *\"loss\"* or *\"error\"*\n", + "\n", + "We've talked about how in order to evaluate a network's performance, the data is \"fed forward\" until predictions are obtained and then the \"loss\" or \"error\" for a given observation is ascertained by looking at what the network predicted for that observation and comparing it to what it *should* have predicted. \n", + "\n", + "The error for a given observation is calculated by taking the square of the difference between the predicted value and the actual value. \n", + "\n", + "We can summarize the overall quality of a network's predictions by finding the average error across all observations. This gives us the \"Mean Squared Error.\" which hopefully is a fairly familiar model evaluation metric by now. Graphing the MSE over each epoch (training cycle) is a common practice with Neural Networks. This is what you're seeing in the top right corner of the Tensorflow Playground website as the number of \"epochs\" climbs higher and higher." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "MrmTsXM5gcjw" + }, + "source": [ + "## What is an \"Epoch\"?\n", + "\n", + "An \"Epoch\" is one cycle of passing our data forward through the network, measuring error given our specified cost function, and then -via gradient descent- updating weights within our network to hopefully improve the quality of our predictions on the next iteration." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "wmp5pBqIgcjx" + }, + "source": [ + "### A note about Hyperparameters\n", + "\n", + "Neural Networks have many more hyperparameters than other machine learning algorithms which is part of what makes them a beast to train.\n", + "\n", + "1. You need more data to train them on. \n", + "2. They're complex so they take longer to train. \n", + "3. They have lots and lots of hyperparameters which we need to find the most optimal combination of, so we might end up training our model dozens or hundreds of times with different combinations of hyperparameters in order to try and squeeze out a few more tenths of a percent of accuracy. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "aM4CK1IarId4", + "toc-hr-collapsed": false + }, + "source": [ + "# Backpropagation (Learn)\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Ktm8Fmoagcjy", + "toc-hr-collapsed": true + }, + "source": [ + "## Overview\n", + "\n", + "Backpropagation is short for [\"Backwards Propagation of errors\"](https://en.wikipedia.org/wiki/Backpropagation) and refers to a specific (rather calculus intensive) algorithm for how weights in a neural network are updated in reverse order at the end of each training epoch. Our purpose today is to demonstrate the backpropagation algorithm on a simple Feedforward Neural Network and in so doing help you get a grasp on the main process. If you want to understand all of the underlying calculus of how the gradients are calculated then you'll need to dive into it yourself, [3Blue1Brown's video is a great starting place](https://www.youtube.com/watch?v=tIeHLnjs5U8). I also highly recommend this Welch Labs series [Neural Networks Demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs) if you want a rapid yet orderly walk through of the main intuitions and math behind the backpropagation algorithm. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "NXI2tEO9gcjy" + }, + "source": [ + "### What is a Gradient?\n", + "\n", + "> In vector calculus, the gradient is a multi-variable generalization of the derivative. \n", + "\n", + "The gradients that we will deal with today will be vector representations of the derivative of the activation function. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "UZY66kiUgcjz", + "toc-hr-collapsed": true + }, + "source": [ + "## Follow Along\n", + "\n", + "In this section, we will again implement a multi-layer perceptron using numpy. We'll focus on using a __Feed Forward Neural Network__ to predict test scores. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Dm2HPETcrgy6", + "toc-hr-collapsed": true + }, + "source": [ + "![231 Neural Network](https://cdn-images-1.medium.com/max/1600/1*IjY3wFF24sK9UhiOlf36Bw.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "4d4tzpwO6B47" + }, + "source": [ + "### Generate some Fake Data" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "ERyVgeO_IWyV" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "np.random.seed(812)\n", + "\n", + "# Imagine that our data is drawn from a linear function\n", + "# y = 2*hours_studying + 4*hours_sleeping + 50\n", + "\n", + "# hours studying, hours sleep\n", + "X = np.array(([2,9],\n", + " [1,5],\n", + " [3,6]), dtype=float)\n", + "\n", + "# Exam Scores\n", + "y = np.array(([90],\n", + " [72],\n", + " [80]), dtype=float)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bCJesGEUgcj4" + }, + "source": [ + "### Feature Normalization" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { "colab": { - "name": "LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb", - "provenance": [] + "base_uri": "https://localhost:8080/", + "height": 153 + }, + "colab_type": "code", + "id": "cDeUBW6k4Ri4", + "outputId": "ae92e269-3279-4710-a2c9-70a5ee483406" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Studying, Sleeping \n", + " [[0.66666667 1. ]\n", + " [0.33333333 0.55555556]\n", + " [1. 0.66666667]]\n", + "Test Score \n", + " [[0.9 ]\n", + " [0.72]\n", + " [0.8 ]]\n" + ] } + ], + "source": [ + "# Normalizing Data on feature \n", + "# Neural Network would probably do this on its own, but it will help us converge on a solution faster\n", + "X = X / np.amax(X, axis=0)\n", + "y = y / 100\n", + "\n", + "print(\"Studying, Sleeping \\n\", X)\n", + "print(\"Test Score \\n\", y)" + ] }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "aS4GZ37Wgcjr", - "colab_type": "text" - }, - "source": [ - "Lambda School Data Science\n", - "\n", - "*Unit 4, Sprint 2, Module 2*\n", - "\n", - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": false, - "id": "etFf1WLWgcjt", - "colab_type": "text" - }, - "source": [ - "# Backpropagation & Gradient Descent (Prepare)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hXB80QOhgcju", - "colab_type": "text" - }, - "source": [ - "## Learning Objectives\n", - "* Part 1: Explain the intutition behind backproprogation\n", - "* Part 2: Implement gradient descent + backpropagation on a feedforward neural network. \n", - "* Part 3: Introduce the Keras Sequential Model API" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8YuQu2lfgcju", - "colab_type": "text" - }, - "source": [ - "## Summary of Yesterday\n", - "\n", - "Yesterday, we learned about some of the principal components of Neural Networks: Neurons, Weights, Activation Functions, and layers (input, output, & hidden). Today, we will reinforce our understanding of those components and introduce the mechanics of training a neural network. Feed-forward neural networks, such as multi-layer perceptrons (MLPs), are almost always trained using some variation of gradient descent where the gradient has been calculated by backpropagation.\n", - "\n", - "
\n", - "\n", - "- There are three kinds of layers: input, hidden, and output layers.\n", - "- Each layer is made up of **n** individual neurons (aka activation units) which have a corresponding weight and bias.\n", - "- Signal is passed from layer to layer through a network by:\n", - " - Taking in inputs from the training data (or previous layer)\n", - " - Multiplying each input by its corresponding weight (think arrow/connecting line)\n", - " - Adding a bias to this weighted some of inputs and weights\n", - " - Activating this weighted sum + bias by squishifying it with sigmoid or some other activation function. With a single perceptron with three inputs, calculating the output from the node is done like so:\n", - "\\begin{align}\n", - " y = sigmoid(\\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)\n", - "\\end{align}\n", - " - this final activated value is the signal that gets passed onto the next layer of the network.\n", - " " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bpi4R03rgcjv", - "colab_type": "text" - }, - "source": [ - "## Training a Neural Network: *Formal Summary*\n", - "\n", - "0. Pick a network architecture\n", - " - No. of input units = No. of features\n", - " - No. of output units = Number of Classes (or expected targets)\n", - " - Select the number of hidden layers and number of neurons within each hidden layer\n", - "1. Randomly initialize weights\n", - "2. Implement forward propagation to get $h_{\\theta}(x^{(i)})$ for any $x^{(i)}$\n", - "3. Implement code to compute a cost function $J(\\theta)$\n", - "4. Implement backpropagation to compute partial derivatives $\\frac{\\delta}{\\delta\\theta_{jk}^{l}}{J(\\theta)}$\n", - "5. Use gradient descent (or other advanced optimizer) with backpropagation to minimize $J(\\theta)$ as a function of parameters $\\theta\\$\n", - "6. Repeat steps 2 - 5 until cost function is 'minimized' or some other stopping criteria is met. One pass over steps 2 - 5 is called an iteration or epoch." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": false, - "id": "z04QRNGagcjv", - "colab_type": "text" - }, - "source": [ - "## Calculating *\"cost\"*, *\"loss\"* or *\"error\"*\n", - "\n", - "We've talked about how in order to evaluate a network's performance, the data is \"fed forward\" until predictions are obtained and then the \"loss\" or \"error\" for a given observation is ascertained by looking at what the network predicted for that observation and comparing it to what it *should* have predicted. \n", - "\n", - "The error for a given observation is calculated by taking the square of the difference between the predicted value and the actual value. \n", - "\n", - "We can summarize the overall quality of a network's predictions by finding the average error across all observations. This gives us the \"Mean Squared Error.\" which hopefully is a fairly familiar model evaluation metric by now. Graphing the MSE over each epoch (training cycle) is a common practice with Neural Networks. This is what you're seeing in the top right corner of the Tensorflow Playground website as the number of \"epochs\" climbs higher and higher." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MrmTsXM5gcjw", - "colab_type": "text" - }, - "source": [ - "## What is an \"Epoch\"?\n", - "\n", - "An \"Epoch\" is one cycle of passing our data forward through the network, measuring error given our specified cost function, and then -via gradient descent- updating weights within our network to hopefully improve the quality of our predictions on the next iteration." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wmp5pBqIgcjx", - "colab_type": "text" - }, - "source": [ - "### A note about Hyperparameters\n", - "\n", - "Neural Networks have many more hyperparameters than other machine learning algorithms which is part of what makes them a beast to train.\n", - "\n", - "1. You need more data to train them on. \n", - "2. They're complex so they take longer to train. \n", - "3. They have lots and lots of hyperparameters which we need to find the most optimal combination of, so we might end up training our model dozens or hundreds of times with different combinations of hyperparameters in order to try and squeeze out a few more tenths of a percent of accuracy. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "aM4CK1IarId4", - "toc-hr-collapsed": false - }, - "source": [ - "# Backpropagation (Learn)\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "toc-hr-collapsed": true, - "id": "Ktm8Fmoagcjy" - }, - "source": [ - "## Overview\n", - "\n", - "Backpropagation is short for [\"Backwards Propagation of errors\"](https://en.wikipedia.org/wiki/Backpropagation) and refers to a specific (rather calculus intensive) algorithm for how weights in a neural network are updated in reverse order at the end of each training epoch. Our purpose today is to demonstrate the backpropagation algorithm on a simple Feedforward Neural Network and in so doing help you get a grasp on the main process. If you want to understand all of the underlying calculus of how the gradients are calculated then you'll need to dive into it yourself, [3Blue1Brown's video is a great starting place](https://www.youtube.com/watch?v=tIeHLnjs5U8). I also highly recommend this Welch Labs series [Neural Networks Demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs) if you want a rapid yet orderly walk through of the main intuitions and math behind the backpropagation algorithm. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "NXI2tEO9gcjy", - "colab_type": "text" - }, - "source": [ - "### What is a Gradient?\n", - "\n", - "> In vector calculus, the gradient is a multi-variable generalization of the derivative. \n", - "\n", - "The gradients that we will deal with today will be vector representations of the derivative of the activation function. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": true, - "id": "UZY66kiUgcjz", - "colab_type": "text" - }, - "source": [ - "## Follow Along\n", - "\n", - "In this section, we will again implement a multi-layer perceptron using numpy. We'll focus on using a __Feed Forward Neural Network__ to predict test scores. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "Dm2HPETcrgy6", - "toc-hr-collapsed": true - }, - "source": [ - "![231 Neural Network](https://cdn-images-1.medium.com/max/1600/1*IjY3wFF24sK9UhiOlf36Bw.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "4d4tzpwO6B47" - }, - "source": [ - "### Generate some Fake Data" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "ERyVgeO_IWyV", - "colab": {} - }, - "source": [ - "import numpy as np\n", - "\n", - "np.random.seed(812)\n", - "\n", - "# Imagine that our data is drawn from a linear function\n", - "# y = 2*hours_studying + 4*hours_sleeping + 50\n", - "\n", - "# hours studying, hours sleep\n", - "X = np.array(([2,9],\n", - " [1,5],\n", - " [3,6]), dtype=float)\n", - "\n", - "# Exam Scores\n", - "y = np.array(([90],\n", - " [72],\n", - " [80]), dtype=float)" - ], - "execution_count": 1, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bCJesGEUgcj4", - "colab_type": "text" - }, - "source": [ - "### Feature Normalization" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "cDeUBW6k4Ri4", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 153 - }, - "outputId": "ae92e269-3279-4710-a2c9-70a5ee483406" - }, - "source": [ - "# Normalizing Data on feature \n", - "# Neural Network would probably do this on its own, but it will help us converge on a solution faster\n", - "X = X / np.amax(X, axis=0)\n", - "y = y / 100\n", - "\n", - "print(\"Studying, Sleeping \\n\", X)\n", - "print(\"Test Score \\n\", y)" - ], - "execution_count": 2, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Studying, Sleeping \n", - " [[0.66666667 1. ]\n", - " [0.33333333 0.55555556]\n", - " [1. 0.66666667]]\n", - "Test Score \n", - " [[0.9 ]\n", - " [0.72]\n", - " [0.8 ]]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "bgTf6vTS69Sw" - }, - "source": [ - "### Neural Network Architecture\n", - "Lets create a Neural_Network class to contain this functionality" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "RUI8VSR5zyBv", - "colab": {} - }, - "source": [ - "class NeuralNetwork:\n", - " \n", - " def __init__(self):\n", - " # Set up Arch\n", - " self.inputs = 2\n", - " self.hiddenNodes = 3\n", - " self.outputNodes = 1\n", - " \n", - " # Initialize Weights\n", - " # 2x3\n", - " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", - " \n", - " # 3x1\n", - " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)" - ], - "execution_count": 3, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "gbyT_FJ88IlK" - }, - "source": [ - "### Randomly Initialize Weights\n", - "How many random weights do we need to initialize? \"Fully-connected Layers\"" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "IreIDe6P8H0H", - "colab": {} - }, - "source": [ - "nn = NeuralNetwork()" - ], - "execution_count": 4, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "lHhewvuJgckG", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 136 - }, - "outputId": "617e7c1f-c04d-42ae-ce1e-efc91a5d2880" - }, - "source": [ - "print(\"Layer 1 weights: \\n\", nn.weights1)\n", - "print(\"Layer 2 weights: \\n\", nn.weights2)" - ], - "execution_count": 5, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Layer 1 weights: \n", - " [[ 2.48783189 0.11697987 -1.97118428]\n", - " [-0.48325593 -1.50361209 0.57515126]]\n", - "Layer 2 weights: \n", - " [[-0.20672583]\n", - " [ 0.41271104]\n", - " [-0.57757999]]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "hbxDhyjQ-RwS" - }, - "source": [ - "### Implement Feedforward Functionality\n", - "\n", - "After this step our neural network should be able to generate an output even though it has not been trained." - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "0gGivpEk-VdP", - "colab": {} - }, - "source": [ - "class NeuralNetwork:\n", - " \n", - " def __init__(self):\n", - " # Set up Arch\n", - " self.inputs = 2\n", - " self.hiddenNodes = 3\n", - " self.outputNodes = 1\n", - " \n", - " # Initialize Weights\n", - " # 2x3\n", - " # Input to Hidden (1st set of weights)\n", - " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", - " \n", - " # 3x1\n", - " # Hidden to Output (2nd set of weights)\n", - " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)\n", - " \n", - " def sigmoid(self, s):\n", - " return 1 / (1+np.exp(-s))\n", - " \n", - " def feed_forward(self, X):\n", - " \"\"\"\n", - " Calculate the NN inference using feed forward.\n", - " \"\"\"\n", - " \n", - " # Weighted Sum\n", - " self.hidden_sum = np.dot(X, self.weights1)\n", - " \n", - " # Activate\n", - " self.activated_hidden = self.sigmoid(self.hidden_sum)\n", - " \n", - " # Weighted sum of activated hidden (which output layer will use)\n", - " self.output_sum = np.dot(self.activated_hidden, self.weights2)\n", - " \n", - " # Final Activation of Output (My Predictions)\n", - " self.activated_output = self.sigmoid(self.output_sum)\n", - " \n", - " return self.activated_output" - ], - "execution_count": 6, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "4X7_Z_GsgckM", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "5046b4fd-58b8-4a30-b218-2517c0e514ef" - }, - "source": [ - "X" - ], - "execution_count": 7, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.66666667, 1. ],\n", - " [0.33333333, 0.55555556],\n", - " [1. , 0.66666667]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 7 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "a1pxdfmDAaJg" - }, - "source": [ - "### Make a Prediction" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "GO8VaEc6gckP", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "outputId": "e0692165-a961-4e61-851a-334a278c4bf2" - }, - "source": [ - "# Try to make a prediction with our updated 'net\n", - "nn = NeuralNetwork()\n", - "print(X[0])\n", - "output = nn.feed_forward(X[0])\n", - "print(\"output\", output)" - ], - "execution_count": 8, - "outputs": [ - { - "output_type": "stream", - "text": [ - "[0.66666667 1. ]\n", - "output [0.25814933]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "3V61yNmAB2T5" - }, - "source": [ - "### Calculate Error" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "KeLKL9GDgckS", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "outputId": "e49ac3b5-ecc2-44de-816d-d276ca80075f" - }, - "source": [ - "error = y[0] - output\n", - "error" - ], - "execution_count": 9, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([0.64185067])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 9 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "tmRRaKgJgckV", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "outputId": "a9554aed-7985-4582-983a-8f43ce5832b2" - }, - "source": [ - "output_all = nn.feed_forward(X)\n", - "error_all = y - output_all\n", - "print(output_all)\n", - "print(error_all)" - ], - "execution_count": 10, - "outputs": [ - { - "output_type": "stream", - "text": [ - "[[0.25814933]\n", - " [0.33067192]\n", - " [0.22642076]]\n", - "[[0.64185067]\n", - " [0.38932808]\n", - " [0.57357924]]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "26wgCLU0TLvy" - }, - "source": [ - "Why is my error so big?\n", - "\n", - "My error is so big because my prediction is low.\n", - "\n", - "Why are my prediction low?\n", - "\n", - "Because either:\n", - "\n", - " 1) Second layer **weights** are low\n", - " \n", - " (or)\n", - " \n", - " 2) Activations coming from the first layer are low\n", - " \n", - "How are activations from the first layer determined? \n", - "\n", - " 1) By inputs - fixed\n", - " \n", - " 2) by **weights** - variable\n", - " \n", - "The only thing that I have control over throughout this process in order to increase the value of my final predictions is to either increase weights in layer 2 or increase weights in layer 1. \n", - "\n", - "Imagine that you could only change your weights by a fixed amount. Say you have .3 and you have to split that up and disperse it over your weights so as to increase your predictions as much as possible. (This isn't actually what happens, but it will help us identify which weights we would benefit the most from moving.)\n", - "\n", - "I need to increase weights of my model somewhere, I'll get the biggest bang for my buck if I increase weights in places where I'm already seeing high activation values -because they end up getting multiplied together before being passed to the sigmoid function. \n", - "\n", - "> \"Neurons that fire together, wire together\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "j_eyzItYIxgm" - }, - "source": [ - "### Implement Backpropagation \n", - "\n", - "> *Assigning blame for bad predictions and delivering justice - repeatedly and a little bit at a time*\n", - "\n", - "What in our model could be causing our predictions to suck so bad? \n", - "\n", - "Well, we know that our inputs (X) and outputs (y) are correct, if they weren't then we would have bigger problems than understanding backpropagation.\n", - "\n", - "We also know that our activation function (sigmoid) is working correctly. It can't be blamed because it just does whatever we tell it to and transforms the data in a known way.\n", - "\n", - "So what are the potential culprits for these terrible predictions? The **weights** of our model. Here's the problem though. I have weights that exist in both layers of my model. How do I know if the weights in the first layer are to blame, or the second layer, or both? \n", - "\n", - "Lets investigate. And see if we can just eyeball what should be updated." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "HGaWbXnZgckY", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 442 - }, - "outputId": "357a13a7-bc19-4e34-c042-dc13444d0536" - }, - "source": [ - "attributes = ['weights1', 'hidden_sum', 'activated_hidden', 'weights2', 'activated_output']\n", - "[print(i+'\\n', getattr(nn,i), '\\n'+'---'*3) for i in attributes if i[:2]!= '__'] " - ], - "execution_count": 11, - "outputs": [ - { - "output_type": "stream", - "text": [ - "weights1\n", - " [[-1.75351135 1.23279898 0.24464757]\n", - " [-0.06568225 0.30190098 0.79723428]] \n", - "---------\n", - "hidden_sum\n", - " [[-1.23468981 1.12376697 0.96033266]\n", - " [-0.62099392 0.57865576 0.52445712]\n", - " [-1.79729952 1.4340663 0.77613709]] \n", - "---------\n", - "activated_hidden\n", - " [[0.22536165 0.75468678 0.7231884 ]\n", - " [0.34955543 0.64075804 0.6281894 ]\n", - " [0.14218011 0.8075341 0.68484697]] \n", - "---------\n", - "weights2\n", - " [[ 1.23073545]\n", - " [-1.52187331]\n", - " [-0.25502715]] \n", - "---------\n", - "activated_output\n", - " [[0.25814933]\n", - " [0.33067192]\n", - " [0.22642076]] \n", - "---------\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "[None, None, None, None, None]" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 11 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "16Ujj6vNYQyX", - "toc-hr-collapsed": true - }, - "source": [ - "### Update Weights Based on Gradient\n", - "\n", - "Repeat steps 2-4 for every observation in a given batch, and then given the network's cost function, calculate its gradient using calculus and update weights associated with the (negative) gradient of the cost function. \n", - "\n", - "Remember that we have 9 weights in our network therefore the gradient that comes from our gradient descent calculation will be the vector that takes us in the most downward direction along some function in 9-dimensional hyperspace.\n", - "\n", - "\\begin{align}\n", - "C(w1, w2, w3, w4, w5, w6, w7, w8, w9)\n", - "\\end{align}\n", - "\n", - "You should also know that with neural networks it is common to have gradients that are not convex (like what we saw when we applied gradient descent to linear regression). Due to the high complexity of these models and their nonlinearity, it is common for gradient descent to get stuck in a local minimum, but there are ways to combat this:\n", - "\n", - "1) Stochastic Gradient Descent\n", - "\n", - "2) More advanced Gradient-Descent-based \"Optimizers\" - See Stretch Goals on assignment." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "h8mcKTvlgckb", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# I want activations that correspond to negative weights to be lower\n", - "# and activations that correspond to positive weights to be higher\n", - "\n", - "class NeuralNetwork:\n", - " \n", - " def __init__(self):\n", - " # Set up Arch\n", - " self.inputs = 2\n", - " self.hiddenNodes = 3\n", - " self.outputNodes = 1\n", - " \n", - " # Initialize Weights\n", - " # 2x3\n", - " # Input to Hidden (1st set of weights)\n", - " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", - " \n", - " # 3x1\n", - " # Hidden to Output (2nd set of weights)\n", - " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)\n", - " \n", - " def sigmoid(self, s):\n", - " return 1 / (1+np.exp(-s))\n", - " \n", - " def sigmoidPrime(self, s):\n", - " sx = self.sigmoid(s)\n", - " return sx * (1-sx)\n", - " \n", - " def feed_forward(self, X):\n", - " \"\"\"\n", - " Calculate the NN inference using feed forward.\n", - " \"\"\"\n", - " \n", - " # Weighted Sum\n", - " self.hidden_sum = np.dot(X, self.weights1)\n", - " \n", - " # Activate\n", - " self.activated_hidden = self.sigmoid(self.hidden_sum)\n", - " \n", - " # Weighted sum of activated hidden (which output layer will use)\n", - " self.output_sum = np.dot(self.activated_hidden, self.weights2)\n", - " \n", - " # Final Activation of Output (My Predictions)\n", - " self.activated_output = self.sigmoid(self.output_sum)\n", - " \n", - " return self.activated_output\n", - " \n", - " def backward(self, X, y, o):\n", - " \"\"\"\n", - " Back prop thru the network\n", - " \"\"\"\n", - " \n", - " self.o_error = y - o # Error in the output\n", - " \n", - " # Apply derivative of sigmoid to error\n", - " self.o_delta = self.o_error * self.sigmoidPrime(o)\n", - " \n", - " # z2 error: how much were our output layer weights off\n", - " self.z2_error = self.o_delta.dot(self.weights2.T)\n", - " \n", - " # z2 delta: how much were the weights off?\n", - " self.z2_delta = self.z2_error*self.sigmoidPrime(self.output_sum)\n", - "\n", - " self.weights1 += X.T.dot(self.z2_delta) #Adjust first set (input => hidden) weights\n", - " self.weights2 += self.activated_hidden.T.dot(self.o_delta) #adjust second set (hidden => output) weights\n", - " \n", - " def train(self, X,y):\n", - " o = self.feed_forward(X)\n", - " self.backward(X,y,o)" - ], - "execution_count": 12, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": true, - "id": "Qw7m9KPLgckd", - "colab_type": "text" - }, - "source": [ - "#### Let's look at the shape of the Gradient Componets\n" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "fWhDTvAIgcke", - "colab_type": "code", - "colab": {} - }, - "source": [ - "nn = NeuralNetwork()\n", - "\n", - "nn.train(X,y)" - ], - "execution_count": 13, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "59kruYhegckh", - "colab_type": "text" - }, - "source": [ - "##### Our Error Associated with Each Observation \n", - "aka how wrong were we?" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "vNuJJsLWgckh", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "2af4033a-2949-4892-df1e-7c615b53345f" - }, - "source": [ - "nn.o_error" - ], - "execution_count": 14, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.60746115],\n", - " [0.46613403],\n", - " [0.53667427]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 14 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vO2flHK5gckk", - "colab_type": "text" - }, - "source": [ - "##### 1st Gradient \n", - "Simple interpretation - how much more sigmoid activation would have pushed us towards the right answer?\n", - "\n", - "`self.o_delta = self.o_error * self.sigmoidPrime(self.output_sum)`" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Xj8eCxYEgckk", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "1df98eb6-4566-4ac1-e68a-ac831baa5746" - }, - "source": [ - "nn.o_delta" - ], - "execution_count": 15, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.14866196],\n", - " [0.11467591],\n", - " [0.13186936]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 15 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ckGNArjjgckm", - "colab_type": "text" - }, - "source": [ - "Let's take a look at the derivate of the sigmoid function to understand what's happening. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "u5QlrPOngckm", - "colab_type": "code", - "colab": {} - }, - "source": [ - "import matplotlib.pyplot as plt\n", - "\n", - "line_x = np.arange(-5, 5, 0.01)\n", - "\n", - "def sigmoid(x):\n", - " return 1 / (1+ np.exp(-x))\n", - "\n", - "def sigmoid_derivative(x):\n", - " sx = sigmoid(x)\n", - " return sx * (1-sx)\n", - "\n", - "# sigmoid\n", - "line_y = sigmoid(line_x)\n", - "y_d = sigmoid_derivative(line_x)\n", - "\n", - "x = nn.output_sum\n", - "s = sigmoid(x)\n", - "sx = sigmoid_derivative(x)" - ], - "execution_count": 51, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "BtEe1wCQgcko", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 331 - }, - "outputId": "a69f18db-8495-4a16-c9f4-cb776bfb16c9" - }, - "source": [ - "# call regplot on each axes\n", - "import seaborn as sns\n", - "plt.style.use('seaborn-darkgrid')\n", - "fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True, figsize=(15,5))\n", - "sns.lineplot(x=line_x, y=line_y, ax=ax1)\n", - "ax1.plot(x[0], s[0], 'ro')\n", - "ax1.set_title(\"Sigmoid of Weighted Sum\")\n", - "sns.lineplot(x=line_x, y=y_d, ax=ax2) \n", - "ax2.plot(x[0],sx[0],'ro');\n", - "ax2.set_title(\"Sigmoid Derivative of Weighted Sum\");" - ], - "execution_count": 52, - "outputs": [ - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAE6CAYAAAB585FmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUZf7+8XvSe4OE3iGUQOig9JKQUGxrw4a79rULVnRFBBR1dRVXLGt3XcVVVlZlKYqAlAABpQmhBFJoSUghvc3z+4Mv8yMmEJAkJ5m8X9flJWdmcs49n0zmmc+c55xjM8YYAQAAAADqnIvVAQAAAACgsaIhAwAAAACL0JABAAAAgEVoyAAAAADAIjRkAAAAAGARGjIAAAAAsAgNGWrMjh07dPPNNys2NlYxMTG69tprFR8fL0lavny5nnjiiVrP8M9//lOvvvpqlfdFR0drw4YN57W+hx9+WCNHjtRPP/3kuC0rK0vdu3dXWlqa47b4+Hh17dpV6enpjtvi4uI0fPjws67/0Ucf1YoVK876mIULF+qPf/xjlfclJiZq06ZN5/BMKvrjH/+ohQsXVrrdbrfr1Vdf1fjx4xUbG6uoqCg999xzKisrO+9tAEB91VDHq8cff1wXXXSRYmNjNXr0aF166aX66KOPZLfba3T752Lx4sXKy8uTdG5j2YXKzc3VZZddpnHjxikrK8tx+/LlyxUTE1PhsW+88YZiY2Mr3DZv3jxNmzbtrNuIjY1VRkbGWR/z+OOPa/78+VXe99NPP+nw4cNn/fmq9OjRQ6mpqZVuz8zM1LRp0xQTE6OYmBhNnDhRX3zxxXmvH/Wfm9UB4ByMMbrrrrs0e/ZsjRo1SpK0bNky3XPPPVq5cqWio6MVHR1d6zluvPHGGl3fd999p6VLl6pt27aO24KDg9W9e3fFxcXp0ksvlXSy+QoODtaGDRs0adIkx21Dhw496/pffPHFC8r3/fffq6ysTAMHDryg9ZyyYMECbd68WV9++aV8fX2Vl5en22+/Xe+//77uuOOOGtkGAFipoY9XU6ZM0d133y1JSkpK0vTp07V//349++yzdbL9U+bNm6d+/frJz8/vgseyc5GQkKDs7GytWrWqwu2DBw9WSkqKjh07pmbNmkk6Of7m5OQoLS1NYWFhjtuuuuqqs25jyZIlF5Txww8/1J///Ge1bNnygtZzyqxZs9SyZUu99NJLcnFx0cGDB3XttdeqS5cu6tu3b41sA/UDe8hQI7KyspSenq7evXs7bhs3bpwWLVokb2/vCnt5UlNTdfnll2vMmDF6+umndeeddzr21nTt2lVffPGFLrnkEo0cOVLr16/X1KlTNXr0aN12222OPTUbNmzQFVdcodjYWF199dXavn27JOn111/Xk08+KenkN6ATJ05UTEyMnnvuuTNmP3z4sG699VbFxMRo0qRJ+vrrryVJN910k+x2u2699dZKA8DQoUO1fv16x3JcXJyuvvrqCt9oxsXFaciQIZJONjqxsbEaM2aMpk6dqqKiIsc2Fi1aJOnknrChQ4fq0ksv1cKFC9W1a9cK23z22Wc1btw4TZw4UXv27NGKFSv09ttv6+OPP9bcuXPPup2UlBRdffXVioqK0rRp01ReXl5lLfbs2aPw8HD5+vpKkvz8/DR//nxNmTJFkjRmzBjHt8inL6empmrYsGH6xz/+4fgm75dfftEdd9yh4cOH18m3zQBwLhryePVb7dq10/z58/Xdd98pMTFR0skv6i655BKNHTtWt9xyizIzMx3be+qpp3TVVVfpww8/dGz/008/1V133eVYZ3l5uQYPHqz9+/crMTFR1113ncaPH6/o6Gh9++23kqQnnnhCBw4c0E033aT4+HjHWPbAAw/o/fffd6xr165dGjZsmOx2uzZv3qwrr7xS0dHRuuaaa5SSklLlc6qqXocPH9bDDz+s48ePKzY21vGcJCkgIEC9evVyjMnFxcU6cOBAhb2MhYWF2rZtm4YOHaqSkhLNnj1bMTExGjNmjN566y3Hurp27aqjR4/Kbrdr1qxZGjp0qK677jq98847uummmxyPy8nJ0e23365Ro0bp1ltvVV5enl599VXFxcXpkUce0eLFi8+6nVWrVik6Olrjx4/Xu+++e8bf7549exQZGSkXl5Mf19u3b69vvvlGkZGRSk1NVY8ePRyPPX154cKFuv/++zVt2jSNGjVKf/rTnxQfH6/JkydryJAhWrBgwRm3CYsYoAbY7XZz5ZVXmkmTJpkvvvjCJCcnV7j/q6++MjfffLMxxpj77rvPvPjii8YYY5YvX2569uxpvvrqK2OMMeHh4eatt94yxhgzd+5cM2DAAJOYmGiKi4vN8OHDzbp160xeXp4ZPHiwiY+PN8YYs2TJEjNu3DhTXl5u5s2bZ6ZPn26MMebKK680n3/+uTHGmMWLF5tu3bqZuLi4StlvueUWxzZTU1NN//79TUpKiiPPkSNHKv3M+vXrzejRo40xxhQWFprBgweb1NRUEx0dbYwxJi8vz0RERJj09HSzadMmc/HFF5ujR48aY4z5y1/+YubOnWuMMebGG280X3/9tcnKyjKRkZEmISHBlJeXm4ceesiEh4c7atenTx+zfft2Y4wxM2fONE888YQxxpjHHnvMvPHGG8YYc9bt3H///ebll182xhizdetW06NHD0fNT7dixQoTERFhZs2aZdavX2+Kiooq3D969GizadOmSsspKSmmR48e5j//+Y/jdzxq1Chz/Phxk5mZaXr27GmSkpIqbQ8A6lpDHq9Of88/3W233WY+++wzk5ycbPr27WsSEhKMMca89dZb5r777jPGGDNv3jwzbNgwc/z4ccfy9OnTTVpamunTp48pKCgwxpwc3yZNmmSMMebOO+80b7/9tjHGmI0bN5rIyEhTUlLieP6nxsdTY9l3331nbrjhBkeu1157zcyaNcvk5uaagQMHmjVr1hhjjPnmm2/MFVdcUel5nK1ecXFxJioqqtLPnNrOY489ZowxZt26dea2224zixYtMk8++aQxxpjVq1eb8ePHG2OM+fvf/25uvvlmU1xcbPLz883ll19uVqxYUeE5rVixwkRFRZm8vDyTlZVlYmNjzY033uj4HUycONFkZWWZ0tJSc9lllznGvtPHyDNtp6yszAwdOtT89NNPxhhj3nvvPRMeHu743HG6uXPnmosuusi89dZbZufOnaa8vNxxX0pKiunevXuVy6c+N5z+erzzzjtNWVmZWbFihRkxYkSVdYR12EOGGmGz2fTBBx8oOjpaH3/8saKiojRx4kQtW7as0mPj4+Md0/qioqIc0wlOiYqKkiSFh4erTZs26tChgzw8PNSuXTsdO3ZM27ZtU/PmzdW/f39JUkxMjLKysnTo0CHHOoqLi7V9+3ZNmDBB0sl54d7e3pWylJaWat26dbr++uslSa1atdLgwYMVFxd31ufbr18/ZWZmKjk5WVu2bFGvXr3UqlUrSdLRo0cVHx+vjh07qmnTplqxYoUmTJjgmEpx3XXXVarL1q1b1b59e4WHh8vFxUXXXXddhfs7deqknj17SpK6d++uY8eOVcp0tu3Ex8c7ahEZGamOHTtW+bxGjx6td955R8eOHdM999yjQYMG6fHHH1dOTs5Z6yFJZWVljjn74eHh6tWrl0JCQhQcHKzQ0NAKx9wBgFUa6nh1Nn5+fsrNzdXq1as1aNAghYeHS5ImT56sFStWOGZF9O7dWyEhIRV+NjQ0VD169NDatWslndzDNn78eEnS/Pnzdeutt0qS+vfvr+Li4grHSv/WqFGj9Ouvvyo7O1vSyeO7YmNjtXnzZjVr1swxjX/SpElKTk6udLzVudSrKkOGDHHsIYuLi9OgQYM0aNAgxx6y02es/Pjjj7r++uvl4eEhHx8fXXbZZZV+9/Hx8Ro1apR8fX0VFBSkiRMnVrh/xIgRCgoKkpubm7p06VLlmHym7Rw8eFAlJSUaNmyYJOmKK6444/N65JFH9NBDD2nNmjW65pprNGzYML3xxhvndMxg586dK7wehw0bJldXV4WHhzMe10McQ4Ya4+/vr/vvv1/333+/MjIytHDhQk2dOtUxJe+UEydOKDAw0LF8qoE45dR0ORcXF8e/JcnV1VV2u12ZmZkKCAiotO3jx487lk8NBn5+fpJODsC//ZlTjzPGyN/f33FbQEBAhekQVfHw8NCAAQO0fv16HT58WIMHD5YkDRw4UHFxcdqzZ49j4MnNzdXy5cu1Zs0aSSePXygtLT2vmpx6HqfqUNWUw7NtJycnp8I6qqrFKUOGDNGQIUNUXl6uLVu26IUXXtDMmTP1yiuvnLUmrq6u8vLyknTyd+fj41NtZgCwQkMcr87m0KFDGj58uNLS0hQfH1/hhBZ+fn6ObZz+XE4XExOjFStWKCoqSj/88IM++OADSSdPUvHmm28qKytLNptNxpizNgM+Pj4aMmSIVq5cqf79++vEiRPq37+/vv32W6WkpFTI5eHhoczMzArHW51LvarSp08f5ebm6sCBA9qwYYOmT5+u5s2by2az6ciRI9qwYYPuueceSSfHyueff94xppWUlCgyMrLC+k6cOFHhd/17x+SqtvPb8fhMvxPp5Ovqmmuu0TXXXKOCggKtXLlSs2bNUpMmTRwN3Zn89vV4akw+9dpE/UJDhhpx9OhRpaamasCAAZKkpk2b6o477tCSJUu0d+/eCo/19fVVQUGBY/ls37ZVpUmTJo7BRTrZeOTk5KhJkyaO2069weXl5cnf3192u73KvTzBwcFycXFRTk6O42eys7MrrOtMhg0bps2bN+vQoUN67LHHJEmDBg3Spk2btHfvXt17772SpLCwMF1xxRWOx1TFz8+vQk1+z7dXZ9tOQECA42xYks7YcK5atUr9+vWTv7+/XF1dNXDgQN19992OAcXFxaXCG/m57DkDgPqkoY5XZ5KSkqKEhARddNFFjj1B8+bNO6+cMTExevvtt7V9+3YFBgaqffv2Ki0t1YMPPqhXX31VI0eOrLJxOdO6li9frqysLMXExMhmsyksLEwdO3as8uy+pztbvc529kI3NzcNHjxY69at08GDBxURESHp5Jekq1ev1t69ezVo0CBJJ8fKW265RaNHjz7j+n47Jp/v7/1s29m/f/85jcf5+fnauHGj4+d9fHw0YcIEbdu2TXv27NHIkSNlt9tljJHNZtOJEyfOOyPqD6YsokYcOXJE99xzj3bs2OG4bdu2bTp8+LB69epV4bGRkZH63//+J+nkLv3zbT4iIyOVkZGhn3/+WdLJMyE2b95crVu3djzGy8tL3bp10/Llyx2PKS4urrQuNzc3DRs2zHGAa3JysuLj4x1TG85myJAh2rJli5KSkhxv/oMHD1Z8fLz279/vOPPhmDFjtGzZMseb7vfff6933nmnwroiIiKUkJCgpKQk2e12ffnll+dUCzc3N+Xm5la7nT59+jhqsWXLFiUnJ1e5vk8++UQvvfSSo1bFxcVaunSp47mEhoZq9+7dkk6e8riqmgJAfdZQx6uqpKamaurUqbr++uvVsmVLDRs2TPHx8Y4TZmzbtk2zZ8+udj3NmjVTmzZt9NZbbzmmKxYWFqqgoMAxXf6jjz6Su7u7o1Fxc3OrsgkYPXq0fv755wpTH3v37q309HRt3bpV0skm8pFHHpEx5rzrdSZDhw7V559/rt69e8vV1VXSyS9J//Wvf6lnz56OPUZjx47Vv//9b5WXl8sYo/nz52v16tUV1tWrVy+tXLlSRUVFOnHihOM1UJ3Tx+Qzbadt27ZydXV1TKdcuHChbDZbpXXZbDY98cQTFZrYjIwMrV27VgMHDlRwcLBcXV2VkJAgSY4TkqFhYg8ZakTfvn01a9YsPfPMM8rNzZXdblfTpk31t7/9zXFs1SmPPPKIpk2bpu+++04jRoxQnz59qnwzOhMfHx+9+uqrmjVrlgoKChQSEqJXXnml0jqeeeYZTZ8+XW+//bZGjBihTp06Vbm+mTNn6qmnntLChQvl7u6u2bNnq0WLFtXmCA8PV1FRkSIiIhxv/s2bN1d5ebl69OjhmL4XERGhu+66y3HWxiZNmmjmzJkV1hUWFqapU6dqypQpatq0qSZPnqz//Oc/1WYYPXq0Hn74YR06dEjz5s0743ZO1XzRokXq3bv3GRvOl19+WS+99JIuueQS2Ww2lZeXa+zYsXrggQckSXfffbdmzJihL774QjExMercuXO1GQGgPmnI45Ukffzxx/rvf/+rkpISubu7a/LkyY6zQoaFhWnWrFm65557VFpaKl9fX02fPv2cssbExGju3LmOWRYBAQG67bbbdPnll6tJkyb685//rKioKN1111369ttvFRsbq8mTJ1dq+Pz8/BxfMvbp00fSyaZz3rx5mjVrlvLz8+Xu7q4HHnigUh3OtV5VGTJkiGbNmqXLL7/ccdvgwYP16KOP6r777nPcdv311ys1NVUTJ06UMUY9e/bUzTffXGFd0dHRWrlypWJjY9WuXTuNHz++wpmVz1bDqVOn6v7779cNN9xQ5Xbc3d01a9YsTZ8+XR4eHvrDH/5QYYr/6bX48MMP9fLLLzvO0Oju7q4bbrjB0ejed999uu222xQWFlbhLJBoeGzmt19PAHXg1C52Sbryyisdb/SN2ek12bt3r66//vrfddFnAEDNYbxqnE7/vX/66adat26d3njjDYtTwVkxZRF17tRJIiQ5rnNyakpEY1VWVqbhw4c7pnMsXrzY8c0iAMAajFeN065duzR27Fjl5OSorKxMy5YtY0xGrWIPGepcWlqaHn30UR06dEguLi666667znra18Zi+fLlevnll2WMUWhoqObMmaN27dpZHQsAGi3Gq8Zr3rx5WrRokVxdXdWnTx/NnDnzvC9HAJwrGjIAAAAAsAhTFgEAAADAIjRkAAAAAGCROjntfXp6bl1sps74+XkqL4/rL52OmlRGTSqjJlVzprqEhvpbHaFBYXx0ftSkatSlMmpSmbPV5ExjJHvIfgc3N1erI9Q71KQyalIZNakadYGz4LVcGTWpGnWpjJpU1lhqQkMGAAAAABahIQMAAAAAi9CQAQAAAIBFaMgAAAAAwCI0ZAAAAABgkXNqyPbs2aOoqCj985//rHTfunXrdNVVV+naa6/VG2+8UeMBAQAAAMBZVduQFRQUaNasWbr44ourvH/27Nl6/fXX9dlnn2nt2rXat29fjYcEAAAAAGdUbUPm4eGhf/zjHwoLC6t0X0pKigIDA9WiRQu5uLho5MiRWr9+fa0EBQAAAABn41btA9zc5OZW9cPS09MVEhLiWA4JCVFKSkqlx/n5eTrVhd1cXV0UFORjdYx6hZpURk0qoyZVoy6NF+Oj86MmVaMulVGTyhpLTaptyGpCXl5xXWymzgQF+Sg7u8DqGPUKNamMmlRGTapWk3UpK7frRHGZcgrLlFNYqpyiUuUUlSm/pFx5xWXKLy5XfsnJ5fySMuX933JhSbnuGd5B47pVng1xPkJD/WvkeTQWjI/Oj5pUjbpURk0qc7aanGmMvKCGLCwsTBkZGY7lY8eOVTm1EQDw+xhjlFNUpoz8Eh0/7b9TyzlFpxqvk//PLyk/6/q83Fzk6+kmXw9X+f3f/0N8vOXn6aa2wd519KwAAMApF9SQtW7dWnl5eUpNTVXz5s31448/6q9//WtNZQMAp5dbVKYjR05oz6EcHT1RpCMninU0t0hHTxQ7mq4yu6n0c55uLmri66Egb3cFeburXYiPAr3cFOjlrkDvk/8POPV/Lzf5/1/z5ebK1U4AAKhPqm3IduzYoRdeeEGHDh2Sm5ubli5dqjFjxqh169aKjo7WM888o2nTpkmSJkyYoA4dOtR6aABoSHIKS5WUVajkrAIlZxUqKbNQKdmFOpxTVGmPlqebi5r5e6pFgKfaNwlSEx8PNfF1V1NfDzXx9XD839fDVTabzaJnBAAAakq1DVnPnj31ySefnPH+gQMHasGCBTUaCgAaotyiMu3LyNfe9DztTc9X4vECJWUWKKeozPEYVxebWgV6qW2wt/q2ClTzAE91bhGoAFepeYCXQnzcabQAAGhE6uSkHgDgbLIKSrTjSK52Hs3VnrQ87cvI15ET//8EDYFeburU1FdjwpuqbbCP2gV7q22wt1oFelWaNuhsBy0DAIBzR0MGANUoK7drd1qeth/J1c4jJ7TjSK4O5RRJklxtUrsQH0W2DNAfIn3VJcxPXZr6KtTPgz1dAACgWjRkAPAbZXajhLQ8bU7O1qaUbG09lKPCUrskKczPQz1bBOjK3i3Us0WAujfzk5e781xHCgAA1C0aMgCQdPREkdYkZmrdgUxtSc1xnGyjQ4iPJvZopv5tghTZMkBh/p4WJwUAAM6EhgxAo1RuN9px5ITWJGZqTWKm9mXkS5JaBXppXLdQDWgTpH5tgtTU18PipAAAwJnRkAFoNOzGaOuhE1qekK4f9qQrs6BUrjapT+tAPTCyo4Z1CFG7EG+O/QIAAHWGhgyAUzPG6NdjeVq2O03fJ6QrLa9Enm4uGtYxRGO6NNXF7UPk78VbIQAAsAafQgA4peyCUi3edUz/3XFU+zMK5O5q05D2Ibp/RKiGd2oiHw9OxAEAAKxHQwbAaRhjtDE5W19vO6JV+4+rtNwoorm/nojuoujwUPaEAQCAeodPJwAavKLSci3ZlabPthxS4vECBXq56creLXVZz+bqHOprdTwAAIAzoiED0GBl5Jfo378c1sKtR5RdWKouob6aERuu6K5h8nRzsToeAABAtWjIADQ4abnF+nhTir7eflQlZXaN6NRE1/VvpX6tAzlDIgAAaFBoyAA0GEdPFOmjjSlatOOo7Eaa2CNMNw9qq7bB3lZHAwAA+F1oyADUezmFpXp/Q7L+/cthGSNd0rOZbh7URq0CacQAAEDDRkMGoN4qLrPri58P6YMNKcovKdOkiGa6/eJ2ah7gZXU0AACAGkFDBqDeMcbo+z0Zen11oo6cKNaQDsG6b3hHzpgIAACcDg0ZgHolKbNAL/6wTxuTsxUe6qunrgrXoHbBVscCAACoFTRkAOqFotJyfbAxRZ9sSpGnm4seGdNZV/ZuIVcXzpoIAACcFw0ZAMv9nJqjZ5cmKDW7SBN6hOn+ER3VxNfD6lgAAAC1joYMgGWKSss1f81Bfb7lkFoGeunNqyM1oG2Q1bEAAADqDA0ZAEtsPZSjZ5fuUXJWoa7p01L3juggb3dXq2MBAADUKRoyAHWq3G70wYZk/WN9kpr7e7JXDAAANGo0ZADqzLETRXrwy22KT8lRbPcwPTa2s/w8eRsCAACNF5+EANSJdQcyNXPpHhUUl+kvMeG6JKKZbDbOoAgAABo3GjIAtcpujN6LS9Y765LUtZmfZl0dqQ5NfKyOBQAAUC/QkAGoNfklZXrmfwlaue+4JvQI0wtX9VZRfrHVsQAAAOoNGjIAtSIlq1DTFu1UcmaBHhrVUdf1ayUvd1cVWR0MAACgHqEhA1Dj4pOz9eh/f5WLTZp3ZS8NahdsdSQAAIB6iYYMQI1asitNM5ckqE2wt/52RYRaBXpbHQkAAKDeoiEDUCOMMfpoY4reWHNQ/VoH6qXLeijAy93qWAAAAPUaDRmAC1ZuN3ppxT59tfWIYrqF6umYrvJwc7E6FgAAQL1HQwbggpSW2/X04t36fk+Gpgxso3uGt5cL1xcDAAA4JzRkAH634jK7Hv/mV61JzNSDIzvqhgGtrY4EAADQoNCQAfhdCkvLNe3rnYpPztYTUZ31h94trY4EAADQ4NCQAThvecVlenDhDm0/ckLPjO+qCT2aWR0JAACgQaIhA3Be8kvKdP9X2/XrsTw9N6m7xoaHWh0JAACgwaIhA3DOikrLNfU/O/Xr0Vw9f0kPje7S1OpIAAAADRoNGYBzUlJm1yOLftXPqTmaPbEbzRgAAEAN4EJBAKpVVm7XE9/uUlxSlp6KCde4bmFWRwIAAHAKNGQAzspujGb8L0Gr9x/Xo2M769Keza2OBAAA4DRoyACc1WurErUsIV33De+gq/twansAAICadE7HkD333HPaunWrbDabpk+frsjISMd9n376qf773//KxcVFPXv21JNPPllrYQHUrX/Gp+pfmw9pcr9WumkgF30GAACoadXuIdu4caOSkpK0YMECzZkzR3PmzHHcl5eXp/fee0+ffvqpPvvsM+3fv1+//PJLrQYGUDeW7krTa6sSFRUeqodGdZTNZrM6EgAAgNOptiFbv369oqKiJEmdOnVSTk6O8vLyJEnu7u5yd3dXQUGBysrKVFhYqMDAwNpNDKDWbUzK0jNLEtSvdaCeGd9VLjRjAAAAtaLaKYsZGRmKiIhwLIeEhCg9PV1+fn7y9PTUPffco6ioKHl6emrixInq0KFDpXX4+XnKzc21ZpNbyNXVRUFBPlbHqFeoSWUNtSb70vL02De71LGpr/4xZYACvN1rbN0NtSa1jbo0XoyPzo+aVI26VEZNKmssNTnv65AZYxz/zsvL09tvv60lS5bIz89PN998s3bv3q1u3bpV+Jm8vOILT1qPBAX5KDu7wOoY9Qo1qawh1iS7sFS3/+tnebja9PJlPWQvLlV2cWmNrb8h1qQuOFNdQkP9rY7QoDA+Oj9qUjXqUhk1qczZanKmMbLaKYthYWHKyMhwLKelpSk0NFSStH//frVp00YhISHy8PDQgAEDtGPHjhqKDKAulZXb9cQ3v+pYbrFeuixCzQO8rI4EAADg9KptyIYOHaqlS5dKknbu3KmwsDD5+flJklq1aqX9+/erqKhIkrRjxw61b9++9tICqBXGGP31x/2KT8nRU+PCFdkywOpIAAAAjUK1Uxb79euniIgITZ48WTabTTNmzNDChQvl7++v6Oho3XrrrZoyZYpcXV3Vt29fDRgwoC5yA6hB//7lsL7aekRTBrbRhB7NrI4DAADQaJzTMWQPP/xwheXTjxGbPHmyJk+eXLOpANSZLanZeuXH/RreMUT3DG9vdRwAAIBGpdopiwCcV3pesZ74ZpdaB3nr2QndOL09AABAHTvvsywCcA4nT+KxSwUl5Zp/daT8PHk7AAAAqGvsIQMaqXmrD2jr4RP6Sxz5W9YAACAASURBVEy4OjX1tToOAABAo0RDBjRCyxPS9dmWQ7q2b0uN6xZmdRwAAIBGi4YMaGQOZhZo1tIERbYM0AMjO1odBwAAoFGjIQMakeIyu6Z/u0seri56flJ3ubvyFgAAAGAljuIHGpHXVydqb3q+Xrk8QmH+nlbHAQAAaPT4ehxoJFbvP64FPx/WtX1baninJlbHAQAAgGjIgEYhPa9Yzy5JUHior+4fwXFjAAAA9QUNGeDkyu1GTy/ereIyu+ZM6i4PN/7sAQAA6gs+mQFO7uNNKYpPydEjYzurfYiP1XEAAABwGhoywIklHMvT2+uSFN01VJdENLM6DgAAAH6DhgxwUiVlds1YslvB3u56bGxn2Ww2qyMBAADgN2jIACf19rqD2p9RoKdiwhXo7W51HAAAAFSBhgxwQr+k5uiTTam6IrK5hnYIsToOAAAAzoCGDHAyBSXlemZJgloEeumBkZziHgAAoD5zszoAgJo1b3WiDucU6e1re8vXgz9xAACA+ow9ZIATWX8wU19tPaIbBrRW39aBVscBAABANWjIACeRX1KmOcv2qkOIj+4a2t7qOAAAADgHNGSAk5j/00Gl5RbrqZhwebrxpw0AANAQ8KkNcAJbD+Xo378c1jV9WyqyZYDVcQAAAHCOaMiABq64zK7Zy/aoeYCn7h7Wweo4AAAAOA80ZEAD935ckg5mFmp6dBf5eLhaHQcAAADngYYMaMD2pOXpo02pmtgjTBe15wLQAAAADQ0NGdBAldmNZi/bo0AvNz04qpPVcQAAAPA70JABDdSCLYe061ieHh7TWUHe7lbHAQAAwO9AQwY0QMdyi/XOuiQN7RCiqPCmVscBAADA70RDBjRAr/y4X+XG6JGxnWSz2ayOAwAAgN+JhgxoYNYeyNSKvRm69aK2ahXobXUcAAAAXAAaMqABKSot10s/7FP7EG/dOKC11XEAAABwgWjIgAbkg40pOpRTpMfGdpG7K3++AAAADR2f6IAG4uDxAn28MUXju4dpQNsgq+MAAACgBtCQAQ2AMUYv/LBX3u6uemBkR6vjAAAAoIbQkAENwJLdaYpPydHdw9qria+H1XEAAABQQ2jIgHour7hMr65MVI/m/roisoXVcQAAAFCDaMiAeu7d9cnKKijVY2M7y9WFa44BAAA4ExoyoB47eLxAn/98SJf2bK4ezf2tjgMAAIAaRkMG1FPGGL28cr+83Fx09/D2VscBAABALaAhA+qpnxIzFXcwS3cMaacQH07kAQAA4IxoyIB6qLjMrld+3K8OIT66pk9Lq+MAAACgltCQAfXQvzan6lBOkaaN7iQ3V/5MAQAAnJXbuTzoueee09atW2Wz2TR9+nRFRkY67jty5IimTp2q0tJS9ejRQ88++2ythQUag2O5xXo/LlmjOjfR4PbBVscBAABALar2q/eNGzcqKSlJCxYs0Jw5czRnzpwK98+dO1e33HKLvvzyS7m6uurw4cO1FhZoDF5fnSi7MXpwVEerowAAAKCWVduQrV+/XlFRUZKkTp06KScnR3l5eZIku92uzZs3a8yYMZKkGTNmqGVLjncBfq9fUnO0dHe6bhrYRq0Cva2OAwAAgFpW7ZTFjIwMRUREOJZDQkKUnp4uPz8/ZWZmytfXV88//7x27typAQMGaNq0aZXW4efnKTc315pNbiFXVxcFBflYHaNeoSaVnW9Nyu1Gr6z6WS0CvfRAdFd5ezjP38wpvE6qRl0aL8ZH50dNqkZdKqMmlTWWmpzTMWSnM8ZU+PexY8c0ZcoUtWrVSnfccYdWrlypUaNGVfiZvLziCw5anwQF+Sg7u8DqGPUKNansfGuyaPsR7TqaqzkTu6m4oFjFTlhOXidVc6a6hIZyAfPzwfjo/KhJ1ahLZdSkMmeryZnGyGqnLIaFhSkjI8OxnJaWptDQUElScHCwWrZsqbZt28rV1VUXX3yx9u7dW0ORgcajoKRcb65NUq8WAYruGmp1HAAAANSRahuyoUOHaunSpZKknTt3KiwsTH5+fpIkNzc3tWnTRgcPHnTc36FDh9pLCzipTzal6Hh+iR4c1VE2m83qOAAAAKgj1U5Z7NevnyIiIjR58mTZbDbNmDFDCxculL+/v6KjozV9+nQ9/vjjMsYoPDzccYIPAOcmLbdYn8SnKio8VJEtA6yOAwAAgDp0TseQPfzwwxWWu3Xr5vh3u3bt9Nlnn9VsKqAReXPtQdmN0b0j2lsdBQAAAHWs2imLAGpPQlqevtt5TNf2bcVp7gEAABohGjLAIsYYvboqUQFebrplcFur4wAAAMACNGSARdYkZio+OVu3X9xO/l7nfQUKAAAAOAEaMsACZeV2zVudqLbB3rqydwur4wAAAMAiNGSABf6z/agOZhbq/hEd5ObKnyEAAEBjxSdBoI7lFZfpnXVJ6tc6UCM6NbE6DgAAACxEQwbUsQ82pCi7sJSLQAMAAICGDKhLh3OK9PmWVE3oEabuzfytjgMAAACL0ZABdWj+mgOy2Wz689D2VkcBAABAPUBDBtSRnUdOaOnudN3Qv5WaB3hZHQcAAAD1AA0ZUAeMMfrbykSF+LhryqA2VscBAABAPUFDBtSBH/dmaOvhE7pzaHv5enARaAAAAJxEQwbUstJyu17/6YA6NvHRpT2bWx0HAAAA9QgNGVDL/v3LYaVmF+mBkR3l5sJp7gEAAPD/0ZABtSinsFTvxSXronbBGtIhxOo4AAAAqGdoyIBa9F5csvKKy/TAyI5WRwEAAEA9REMG1JKk4/n69y+HdUnP5uoc6mt1HAAAANRDNGRALXlx2R65u9p0FxeBBgAAwBnQkAG14OfUHC379ZimDGyjpr4eVscBAABAPUVDBtQwuzF6dVWimgV46sYBra2OAwAAgHqMhgyoYct2p+vXo7maGhUuL3dXq+MAAACgHqMhA2pQUWm53vjpgLqG+eny3i2tjgMAAIB6joYMqEGfbzmko7nFenBkR7lwEWgAAABUg4YMqCGZBSX6cGOKhncM0YC2QVbHAQAAQANAQwbUkHfWJamotFz3j+Ai0AAAADg3NGRADThwvEBfbzuiP/RuqfZNfKyOAwAAgAaChgyoAfNWJ8rL3VW3X9zW6igAAABoQGjIgAu0ISlLaxIzdetFbRXsw0WgAQAAcO5oyIALUG43em1VoloGeOqavq2sjgMAAIAGhoYMuADf7jyqven5undER3m68ecEAACA88MnSOB3Kigp15trk9SrRYCiwptaHQcAAAANEA0Z8Dt9vClFx/NL9NCojrLZuAg0AAAAzh8NGfA7HMst1j/jUzWua6h6tQywOg4AAAAaKBoy4Hd4c80BGWN0z/AOVkcBAABAA0ZDBpynXcdy9d2vaZrcr7VaBnpZHQcAAAANGA0ZcB6MMXp1ZaKCvd31p8FtrI4DAACABo6GDDgPq/Yd15bUHN0xpJ38PN2sjgMAAIAGjoYMOEel5Xa9/tMBdQjx0eWRLayOAwAAACdAQwacoy+3HlFyVqEeGNVRbi6c5h4AAAAXjoYMOAc5haV6d32SBrcL0pD2wVbHAQAAgJOgIQPOwfsbkpVXXKYHR3biItAAAACoMefUkD333HO69tprNXnyZG3btq3Kx7z88su66aabajQcUB8cPF6gBT8f1qU9m6tzqK/VcQAAAOBEqm3INm7cqKSkJC1YsEBz5szRnDlzKj1m37592rRpU60EBKz2t1X75e3uoj8Pa291FAAAADiZahuy9evXKyoqSpLUqVMn5eTkKC8vr8Jj5s6dq4ceeqh2EgIWWpN4XOsOZOn2i9spxMfD6jgAAABwMtVeSCkjI0MRERGO5ZCQEKWnp8vPz0+StHDhQg0aNEitWrU64zr8/Dzl5uZaA3HrB1dXFwUF+Vgdo15xxpqUlNn12uoD6tjUV7eN7CwPt/M75NIZa3KhqEnVqEvjxfjo/KhJ1ahLZdSkssZSk/O+sq0xxvHv7OxsLVy4UB988IGOHTt2xp/Jyyv+fenqqaAgH2VnF1gdo15xxpp8silFB48X6LU/9FRBXpHO99k5Y00uFDWpmjPVJTTU3+oIDQrjo/OjJlWjLpVRk8qcrSZnGiOr/co/LCxMGRkZjuW0tDSFhoZKkuLi4pSZmakbbrhB9957r3bu3KnnnnuuhiID1snIL9F7ccka1jFEQzqEWB0HAAAATqrahmzo0KFaunSpJGnnzp0KCwtzTFeMjY3V4sWL9cUXX+jvf/+7IiIiNH369NpNDNQiz6++UEi/CHXrEKplr03Rs7m/WB0JAAAATqzaKYv9+vVTRESEJk+eLJvNphkzZmjhwoXy9/dXdHR0XWQE6oTnV1/If+p9shUWSpJanUiX+cs05fp6qPjKayxOBwAAAGdkM6cfFFZL0tNza3sTdcrZ5rPWBGeoSUi/CLmmplS6vbx1G2Vu2Xne63OGmtQ0alI1Z6oLx5CdH8ZH50dNqkZdKqMmlTlbTX73MWRAY+FyKPW8bgcAAAAuFA0Z8H/KW1Z96QZ7q9Z1nAQAAACNBQ0Z8H8WXnm3Ctw8K9xmvL2V/+QMixIBAADA2dGQAZL2ZeTrCZ9IfXn7Uypv3UbGZlN56zbKfeV1TugBAACAWnPeF4YGnI0xRi/+sE9+nm66+LG7lTnzAasjAQAAoJFgDxkavf/tStPPqTm6Z3gHBfm4Wx0HAAAAjQgNGRq13KIyvbYqUT1b+OuyXs2tjgMAAIBGhoYMjdqbaw8qu7BUj4/tIhebzeo4AAAAaGRoyNBo7T6Wq6+2HtZVvVuqazM/q+MAAACgEaIhQ6NkN0Yv/LBPQd7uumtoe6vjAAAAoJGiIUOj9PX2o9pxJFcPjOwofy9ONgoAAABr0JCh0cnIK9brqxPVv02gxncPszoOAAAAGjEaMjQ6f/1xv0rK7JoeHS4bJ/IAAACAhWjI0Kis2ndcP+zJ0G0Xt1PbYG+r4wAAAKCRoyFDo5FXXKYXf9irzk19ddOA1lbHAQAAAGjI0HjMX3NQ6XklenJcF7m58tIHAACA9fhUikZh2+ET+vKXw7qmb0v1bBFgdRwAAABAEg0ZGoHScrvmLNujMH9P/XlYe6vjAAAAAA40ZHB6H2xIVuLxAj0e1Vm+HlxzDAAAAPUHDRmcWsKxPL2/IUXju4dpWMcmVscBAAAAKqAhg9MqLbfrmSUJCvZ217TRnayOAwAAAFRCQwan9W5csvZl5Gt6dBcFertbHQcAAACohIYMTunXo7n6aEOyJkU00/BOTFUEAABA/URDBqdTUnZyqmITXw9NHcVURQAAANRfNGRwOu+sT9KB4wV6cly4/L04qyIAAADqLxoyOJUtqdn6eGOKLuvVXEM6hFgdBwAAADgrGjI4jdyiMs1YnKDWQV5MVQQAAECDwHwuOAVjjOZ+v1fpecV677o+8vFwtToSAAAAUC32kMEp/G9XmpYlpOuOIe0V0SLA6jgAAADAOaEhQ4N3KKdQL/6wT31aBejmQW2sjgMAAACcMxoyNGhldqOnFydIkp6d0E2uLjaLEwEAAADnjmPI0KD9Y32Sth0+odkTuqlFgJfVcQAAaPSMMcoqLNXBzAIdzCzU0RNFyi4sVU5hmfJLyiRJbm6ucjFGAd7uCvRyU/MAL7UN9lbbIG+1DPTiC1Y0KjRkaLDWH8zUB3HJuiSimWK6h1kdBwCARskYo30Z+dqckqOth3L0y6ETysgvcdzvapMCvd0V6O0uPw9XSTa52aX8olLtTc9XdmGpisrsjsf7ergqorm/IlsGqF+bQPVtFSg3VyZ1wXnRkKFBOpZbrKcXJ6hjUx89Oraz1XEAAHBqnl99Id85M+VyKFX2Vq2VN/1prR8cox/3Zuin/cd1+ESxJKlFgKf6twlUj+b+6tDER+1DfNTM31Mutop7vIKCfJSdXSDpZEOXXViq5KxCJWUWatexXG07fELvb0jWu3EnG7SL24doVOcmGtm5ibzcOZMynAsNGRqcsnK7nvx2l0rK7Jp7SQ/emAEAqEWeX30h/6n3yVZYKElyTU2RxwP3anHMvVrae6wGtg3Snwa31UXtg9X8dxw+YLPZFOzjoWAfD/VuFahLezWXJOWXlCk+OVs/7c/UT4nH9f2edPl6uCqqa6guiWimyJYBstmY2oiGj4YMDc78NQe19f+OG2sf4mN1HAAAnJrvnJmOZuwU79JiPbfpMz389lPyrqUvRn093DSyc1ON7NxUdmP0c2qOvtl5TMt2p2nR9qPqEuqr6/q1Uky3MHm4MaURDRevXjQoq/cf1yfxqbqydwuOGwMAoBbZjdHyhHTZUlOrvN837UitNWO/5WKzqX+bID0T21VL7rpYT43rIrsxenbpHl3yjw36YEOy44QhQEPDHjI0GAePF+jpxbvVLcxPD43qZHUcAACckjFGq/cf11trk7QvI19jgkLVPDut0uPsrVpbkE7y8XDVZb1a6NKezbUxKVufbk7V/DUH9Wl8qqYMbKOr+7ass0YRqAk0ZGgQcovKNG3RTnm4uuily3rIk6kJAADUuA0HszR/7UH9ejRXbYO9NXtCN3m1nSPz8P0Vpi0ab2/lPznDwqQnjz0b3D5Yg9sHa+eRE3pnfZJe/+mA/hmfqlsuaqurerfg7IxoEGjIUO+V243+sni3DuUU6c2rI3/XAcMAAODMkjIL9OqqRK1JzFSLAE/9JSZcE3o0k5uLTaXdr1Wui63CWRbzn5yh4iuvsTq2Q0SLAL32h17adviE3lx7UC//uF9f/nJYD47qqKEdQjj5B+o1GjLUe2+vO6i1BzL12NjO6ts60Oo4AAA4jbziMr0Xl6zPtxySp5uLHhjZUdf0aVnpJBnFV15TrxqwM4lsGaD5V/XSmsRMvboqUQ/9Z6cuahesB0d1VKemvlbHA6p0Tg3Zc889p61bt8pms2n69OmKjIx03BcXF6dXXnlFLi4u6tChg+bMmSMXF3YPo2Z8n5CuDzak6LJezXVl7xZWxwEAwCkYY/S/XWl6bVWisgpKdUnPZrp7WAc18fWwOtoFs9lsGt6piS5qH6wvtx7RP9Yl6YZPtujGAa1120VtuVwO6p1qO6eNGzcqKSlJCxYs0Jw5czRnzpwK9z/99NOaN2+ePv/8c+Xn5+unn36qtbBoXLYfPqFnliQosmWAHh3TmekGAADUgNTsQt3/1Q7N+F+CWgZ66cMb+uovMV2dohk7nburi67r10oLbxmoCd3D9NHGFF370WatO5BpdTSggmr3kK1fv15RUVGSpE6dOiknJ0d5eXny8/OTJC1cuNDx75CQEGVlZdViXDQWqdmFmvb1TjX19dBfL+vB9UUAALhAZXajzzan6u11SXJzsemRMZ11VZ8WcnHyLzyDfNz1dGxXTYxopueX79UDC3coKjxU00Z3VFM/T6vjAdXvIcvIyFBwcLBjOSQkROnp6Y7lU81YWlqa1q5dq5EjR9ZCTDQmOYWlenDhDtmN0Wt/6KlgH+f6xg4AgLq2+1iu/vTpz5q3+oAuahesBX8coGv6tnT6Zux0/dsE6V9T+uvOIe20en+Grv1os/6365iMMVZHQyN33if1qOpFe/z4cd11112aMWNGhebtFD8/T7m5Oc98XVdXFwUF+Vgdo16pqZoUl9l191fbdfhEkT7640D1bh9SA+msweukMmpSNerSeDE+Oj+ra1JSZtf8Vfv11upEhfh46PXJfRTTo5nlhwFYWZeHx3fXHwa00eP/2aGnFydoVWKWnr20h8L8rT2Ls9WvlfqosdSk2oYsLCxMGRkZjuW0tDSFhoY6lvPy8nT77bfrwQcf1LBhw6pcR15ecQ1ErT+CgnyUnV1gdYx6pSZqYjdGTy/erU0HszRrQjd1CfJq0HXmdVIZNamaM9UlNNTf6ggNCuOj87OyJvsy8vXM/xKUkJaniRHNNG1UJ/l7uSknp7D6H65lVr9WQtxd9OZVvfTZlkN6a+1BjZ+3Rg+P6aTYbmGWNatW16Q+craanGmMrHbK4tChQ7V06VJJ0s6dOxUWFuaYpihJc+fO1c0336wRI0bUUFQ0RsYYvbxiv5buTtfdw9ortnuY1ZEAAGiQyu1Gn2xK0ZR/blF6XrFeurSHnontKn8vrnZ0OlcXm24c0Fr/vKmf2gV76+nFCXpk0a/KyC+xOhoamWr/Mvv166eIiAhNnjxZNptNM2bM0MKFC+Xv769hw4bp66+/VlJSkr788ktJ0qRJk3TttdfWenA4l3fWJemLXw7r+v6t9MdBbayOAwBAg5SaXaiZSxL0y6ETGt2lqZ6I6syx2NVoH+Kjf0zuo39tTtVbaw9q8ofxenRsZ43rxpfDqBvn9FXJww8/XGG5W7dujn/v2LGjZhOh0fnX5lS9G5esS3s204MjO1o+rx0AgIbGGKOF247otVWJcnWxaeb4rhrf3brpdw2Nq4tNNw1so+Edm+iZJQl68rvd+nHvcT02trOCfNytjgcnx75rWOqbHUf1t5WJGtOlqaZHhzNwAABwntJyizVr2R7FHczS4HZBempcuJoHWHuCioaqfRMfvXtdH32yKUXvrEvSltRsTY8O18jOTayOBidGQwbLLP71mGYv26OL2gVr1oRucnWhGQMA4FwZY7Rkd5pe+mG/SsvtenRsZ13VuwVfbl4gNxeb/jS4rYZ2CNEzSxL08KKdmhTRTNNGd5KfJx+dUfN4VcES3+08pplLEtS/bZBe4sLPAACcl+yCUs39Ya9+2JOhyJYBmhHbVW2Dva2O5VTCw/z00Q199e76JH24MUUbk7L0dGxXDW5X+RJPwIXgUzDq3Lc7j2rmkgQNbBukv10eIS9357kGDwAAtW1N4nFd+1G8Vu07rnuGtdc71/amGasl7q4u+vOwDnrvuj7ydnfVvV9u1wvf71VBSbnV0eBE2EOGOvXfHUc1e+keDWoXpL9eRjMGAMC5yi8p06srE/X19qPq3NRXr1/ZS+FhftX/IC5YzxYB+udN/fTm2oP6bPMhxSVlaUZMV/VpHWh1NDgB9pChzizYckizl+7R4PbBNGMAAJyHX1JzdP3HW7Ro+1FNGdhGH93Ql2asjnm5u+qhUZ305jWRshvpjgVb9erKRBWX2a2OhgaOPWSodcYYvbUuSe/HJWtU5yaaPbG7PDlmDACAapWU2fX2uoP6ZFOqWgR66Z1re7NXxmL92wTpX1P6ad6qA/p0c6rWHcjUM+O7qkdzf6ujoYHiUzFqVbnd6Pnv9+r9uGRd1qu5nr+kB80YAADnYE9anm7+9Gd9vClVl0c217+m9KMZqyd8Pdz0RHQXzbuyp/JLynTLv37WW2sPqrScvWU4f+whQ60pLrPrL4t368e9GbplcBvdNbQ9p+IFAKAaZXbjuA5WgJeb/nZFhIZ15DpY9dHF7UP02c399fKP+/VeXLLWJGbqmdiu6hzqa3U0NCA0ZKgVGfklemTRTu04kqupozvpun6trI4EAEC9tzc9T7OW7tGuY3kaG95Uj4/toiAfd6tj4SwCvNw1c3w3jercVM8v36spn27RnUPa68YBrbnGKs4JDRlq3J60PE39eqdyCkv1wqU9NKZLU6sjAQBQr5WW2/XhhhS9vyFZAV5umntJd40ND7U6Fs7D6C5N1adVgJ7/fp/+/tMBrdqXoRmxXdUuxMfqaKjnaMhQo1bty9BfFu+Wv6eb3p3cR12bcQYoAADOZtexXM1aukd70/MV2z1M00Z1Yq9YAxXs46EXLumupbvT9eIP+3TDJ1t07/AOuqZvS7lw2AbOgIYMNcJuN3ovLklvr01S9+b+evmyHmrq52l1LAAA6q3iMrveXZ+kTzalKNjHQ3+9LEIjO3OsWENns9kU2z1M/dsEavayPXr5x/0nv7CO6aqWgV5Wx0M9REOGC5ZdWKqH//urVu3NUEy3UD01LpxrjAEAcBYbDmbphR/2KiW7SJdENNNDozrJ34uPZc4k1M9Tr17RU4u2H9XfVibquo82686h7XRN31Zy49gynIa/fFyQHUdO6Ilvdul4QYkej+qsP0S24EyKAACcQUZ+iV5duV9Ld6erbbC3/n5VLw1uF2x1LNQSm82myyNbaFC7YL34wz79bWWivt15TI9HdVFkywCr46GeoCHD72I3Rp9vOaTXVx9QqJ+HFtx+kdr4Mt8dAICqlNuNvtp6RPPXHFBJuV13XNxOUwa14dqcjUTLQC/97YoI/bjvuF5esU+3fvaLrohsrnuGdVCgN5+fGjsaMpy3Y7nFmrkkQZuSszW8Y8jJMwi1CFR2doHV0QAAqHd2HDmhl1bs169HczWobZAei+qitsHeVsdCHbPZbBrTpakGtwvSO+uStGDLIa3ce1z3jeigiRHNrI4HC9GQ4bws252mud/vU2m5XU9Ed9EVvZozRREAgCocyy3W7O/3adHWw2ri66HZE7ppXLdQxs1GztfDTQ+N6qRJEc30/PJ9enbpHv37l8OacUmEOgVyQrTGiIYM5+R4fole/nG/liekq2cLfz07vpva8O0eAACVFJWW65P4VH28MUV2SX8a3EY3D2ojXw8+duH/6xLqp3ev662lu9P099UHNPndDYruGqr7RnRQiwDOxtiY8M6AszLG6Judx/TaqkQVlpbrziHt9MfBbTk7EAAAv2E3Rkt3p+mNnw7qWG6xosKbavqkHvJnyMQZuNhsGt+9mUZ1bqp/bz+qd/7vgtI3DmitKTTxjQa/ZZxRclahnl++R/EpOerTKkDTo8PVoQlXmwcA4HTGGP2UmKk31xzUvoz/1969R0dZ33kcf08mc78lk8zkDoQEEggXi4IVCqgFdL1Vai1x9Wy3Z6v/dNvVtbtVzp5D123tkdNaPVTr9oj2oqsp1JVuj1fOoS7ITaoVjIIQcoMQkpBMkkkmmUwy+8eEgZi4gAWfJPN5nTNnnrnlfH0i+cz3+f2e39NDWdDNQzeUsaAwg4wMp86xlnNyWMx899oZrCrN4okddTyzp5GXJ6tVAgAAEVZJREFU9jfz94uKuG1+ni4nNMmpIZNRwv0xnt3TwAvvHseWnsaDK2dw69xcXWFeRETkE/7cGOKJ7XUcONFFUYadH91YzoqygDJTPpNcr53/uKGcygUFPLWjjsfeOsp//fkY//DFKdwyJ5d0s1blnIzUkEnS4FCcP1Y38+SOOtp7B7ixIod//NI0st06wVRERORsB5q6+OXOenbXdxB0W3lw5QxuqcjRF2a5KCpyPWz42txkw//jrUf47b5j3H3VVFaVB3XqyCSjhkyIx+PsrQ+xYXsth1rCzM/38rPVc5id6zG6NBERkXEjHo/zTkOIZ/c0sK+xE589nX9aPp2vaUqZXCKXF2Ww8Y75vF3bzpM76lj36iF+ubOev1tUxE2zc7DqOnaTghqyFPfusRBPvV3Pe8c6yfXY+NGN5aws05K8IiIipw3F42yvaefZPQ1UN3cTcFu57+rp3Do3D6dVjZhcWiaTiS9Nz2JxsZ/tNad4Zk8jP37zME/vqufOywtZPU//H050ashS1IGmLv5zZx176kNku6z8y7Wl3Do3V0daREREhkUGBnnlw5O8+O5x6toj5PvsPLiilJsqlJfy+UszmVhems2ykqzESO3eRh576yjP7Gng1rl53H5ZHrlaLn9CUkOWQobicXbWtvObd47x3rFOMhwW7l0+Xav3iIiInKW5q4/fvdfEywea6e6PMSsnsWriyjKduyPGM5lMLJqayaKpmRxo6uK3+47x3L5Gnt/XyDUzslnzhQLmF3g122kCUUOWAqKxIV4/2MJz+45x9FQvOR6bplqIiIicZXAozu66Dl4+cILtNacAuGZGNpULCpiXry+3Mj7Nzfey/pbZnOjqY9PwQYStH7dRHnSzen4eq8oCuG36uj/e6Tc0iR0LRfjv/c38zwfNdEQGKM128e9/U8aqsoBWgRIREQGOd0b4wwcn+eMHzbSEo2Q6LNx5RSG3X5av6V8yYeR57Xx3+XTuXjyVVz88ye/+0sSP3zzMz7bVsKIswFfm5GrUbBxTQzbJxAaHeLu2g9+/38Tuug5MJlhWksVt8/O4cmqm/iGKiEjKC/fH+NORNl79sIW9DSFMwFXFmdx/bSlLp/ux6KClTFAOi5mvzs9n9bw8qpu72XKgmTcPtfLH6pNMzXRw85xcVpUHyNPBhnFFDdkkEI/H+bC5m1c/auGNg610RAYIuK1866opfGVuHjkeXUdMRERSW9/AIDuOtvP6wRZ21rYTHYyT77Vxz+Kp3FyRo9EwmVRMJhNz8rzMyfPyz9eUsPVQK3/4oJmfb6/l59trmZvnZVV5gBUzs3W92XFADdkEVnuql60ft/LaRy00dESwmk0sK8ni+llBlhT7NS1RRERSWndfjF117bx15BQ7jrbTOzBIlsvKV+fns6oswJw8j2aOyKTnsJi5eU4uN8/J5XhnhK2H2njjYAs/3VbDo9tqWFDk45rSbJaWZJHv04EJI6ghm0CGhkfCth0+xVtH2qjviGACLi/y8Y2FRVw7M1snboqISEo73hlhe007/1tzinePdTI4FCfTYWFleYDrygMsKMzArJUSJUUV+Bx8Y1ER31hURN2pXt481Mqbh1r5ybYafrKthtJsF0tL/CydnkVFnoc0HbD4XOjb+zjXGRlgX2OI3XUdvF3bTms4ijnNxOWFPtYsKGB5SRZBTUkUEZEUFe6P8efGEO80hNhbH6K2vReA4iwnd11RyLKSLCpyPWrCRD5hWpaTuxdP5e7FU2noiLC95hTbj57iN3sbeXZPI36nhUVTM1k4JYNFUzI0rfcSUkM2zvTHhqhu7mJPfYg9dR18dLKboTi4rGaunJrJ8tIsvjTdj9duMbpUERGRz11vdJDq5i72NSSasOrmRE7a09O4rNDHV+bmsqwki6JMh9GlikwYUzId3HlFIXdeUUhX3wA7azvYcfQUe+s7eO2jFgCKMuzJBm1+vlfnnl1EasgMFuod4P2mLvY3dfKX4118dLKbgcE4ZhNU5Hn51hencuW0TGbnenQxShERSSnxeJwTXf3sb+pK3g63hhmKg9kEs3O9fPPKKSycksHcPC/WdJ07LfLX8totXD8ryPWzgsTjcWraetnb0ME7DSFe/bCF379/AoB8n515+d7krSTbpe+qn5Eass9RTzTGxy09HGwJc+hkN9XN3dS1RwBITzMxO9dD5RcKmF/g4/Iin84HExGRlHG6+TrUEuZQS5iPW8J8dDJMW08UAKfFTEWeh29eOYW5+V7m53uVkyKXmMlkojTgojTg4m8vLyQ2OMRHJ8PJAyTvNISSI2hOi5myoIuZQTflOW7Kgm6K/U4tMnce9JfsEhgcinOiq4/aU73UnurlUEuYgy1hGjsixIffk+2yUp7j5sbZOVxW4GNWrgebjuyJiEgK6IwMUNfeS117L0eHc/Ljlh66+2MApJlgqt+ZGPnS0XeRcSPdnMbcfC9z873cyehR7IMnw2w50EzVe0MAWM0mSrITTdr0LCfFWU6K/U5yPDatcHoWNWR/he6+GE2dfRzrjCSbr9r2Xho6IvTHhpLvy/XYKM9xc8PsIOVBD2VBl+bdiojIpNYfG+JEVx9NnX3UtfdS3x6htr2X+vZe2nsHku+zpadRmu1iRVk2ZcHEUfXSbBd2i9nA6kXkfJhMJvJ9dvJ9dq6fFQQSAxMNHZHkaPfBljB/OtzGlgOx5OccljSm+RMN2jS/k6IMBwUZdgp89pRcJ0EN2aeIx+OE+wdpCffTGu6nqbOP48O3k+EoDe29dPXFRnwm32ujOMvFoimZFGc5KM5yUex34rFrN4uIyOTSE43RGo4mM7Kps4+2SIy6tp7E9vBUw9O89nSm+Z0snZ7FVL8j+UUsz2vXCogik4g5zZQYCctyJps0gI7eKLXtvclBjLr2XvY1hHjlw5YRn3fbzBT4HBT47EzP8ZBlMxP02Ai6rQQ9NjIclkm3HH/KdQoDg0OEIgPJW0fvAG09UVq6o7T19NMSjtIW7qc1HKXvrFEuAIvZRJ7XzrRsF2UBFwU++/DNwRS/A4eO5omIyAQWjQ3R2TdAZyRGKDJAe290uOlKZGRbz/B2OErvwOCIz6aZIM9nJ9dj46ppmcmj5vleO1P8DjIdFk1REklhmU4rmU4rCwozRjzfE03MODseOjP4cbwzQk1bDztq24l+4vt4epqJgNtKwJ1o0gJuG9kuK5lOS+LmsJDhtJDpsOKwpE2Ivzvn1ZA9/PDDvP/++5hMJtauXcu8efOSr+3cuZNHH30Us9nMsmXL+Pa3v33Jij1tYHCIcH+Mnugg4f4Y4f7BkY+jMbr7BglFooSGQ+X0rSc6OObPtKWnke2yEnRbmZXjYWmJlaDblvyF5/vsBNxW0kwmMjKchEK9l/y/U0RE5ELE43H6Y0PJPOyJDtITTeTk6ftwf4yuvpHZ2NkXo/M8MjLgtjIz4GZJsZVsl5Vsd+K5fJ+dHLeN7Cy38lFELojLms6MgJsZAfeo17xeB0eOh2gNJwZNRtx39/Nxaw9v17YTGRga4ycn/nZlOM40aT57Oh5bOh57Om5rOu7Tj21mPLZ03Ge99nmu2nrOhmzv3r3U19dTVVVFTU0Na9eupaqqKvn6D3/4QzZu3EhOTg533XUX1113HaWlpRe1yN+9d5yq95qS4dIfG3unn23EL8BhoTDDTsbwdqbTktz2OSwEXFa89vQJ0UGLiIicFu6P8a9/+JCmzr5kRsaG4uf8nMtqxudIfDnJcFiY5ncOZ2L6mXy0J/Iy4LbisSkjReTzl5ZmSkxX9Nio+JT3xONxIgNDdESidPQmZr91RAYIDd+fvX08FKG7f5Du/hiD5/hbaTWbcFjMuKxmCjMcPLp6ziVbgO+cDdmuXbtYsWIFACUlJXR2dhIOh3G73TQ2NuLz+cjLywNg+fLl7Nq166I3ZNkuK2VBN26bGZc1HbfNjNuajmv43m07ezvxHl2LREREJrv0tMQJ9VkuKy6rOZGH53Fv0TLUIjJJmEwmnFYzTquDAt/5XRA+Ho/TFxuiuy9Gd3+McP/p+0Sz1t0XoyeaOMjVGx3EY0u/pOe6nrMha2tro6LiTE/q9/tpbW3F7XbT2tqK3+8f8VpjY+NFL/LamQGunRm46D9XRERkIrNbzPzbqplGlyEiMqGYTInRL4clsWCI0S54UY94/NxTIT7J7baRnj55Frwwm9PIyHAaXca4on0ymvbJaNonY9N+SV3Kx8lP+2Rs2i+jaZ+Mlir75JwNWTAYpK2tLfm4paWFQCAw5msnT54kGAyO+hnhcP/FqHXc0KIeo2mfjKZ9Mpr2ydgm034JBDxGlzChKB8nP+2TsWm/jKZ9Mtpk2yeflpHnnES+ZMkSXn/9dQCqq6sJBoO43YlVUAoLCwmHwxw7doxYLMa2bdtYsmTJRSxbRERERERk8jrnCNmCBQuoqKigsrISk8nEunXreOmll/B4PKxcuZIf/OAH3H///QDccMMNFBcXX/KiRUREREREJoPzOofse9/73ojH5eXlye2FCxeOWAZfREREREREzo/WvRURERERETGIGjIRERERERGDqCETERERERExiBoyERERERERg6ghExERERERMYgpHo/HjS5CREREREQkFWmETERERERExCBqyERERERERAyihkxERERERMQgasg+o7a2NhYuXMiePXuMLmVciMVifP/73+eOO+7g61//Ovv27TO6JEM9/PDDrFmzhsrKSvbv3290OePC+vXrWbNmDbfddhtvvPGG0eWMG319faxYsYKXXnrJ6FJELhpl5BnKx5GUj2NTRo4tVTIy3egCJqr169dTVFRkdBnjxpYtW3A4HLzwwgscPnyYBx98kM2bNxtdliH27t1LfX09VVVV1NTUsHbtWqqqqowuy1C7d+/m8OHDVFVV0dHRwerVq1m1apXRZY0Lv/jFL/D5fEaXIXJRKSPPUD6eoXwcmzLy06VKRqoh+wx27dqFy+Vi5syZRpcybtxyyy3cdNNNAPj9fkKhkMEVGWfXrl2sWLECgJKSEjo7OwmHw7jdboMrM87ChQuZN28eAF6vl0gkwuDgIGaz2eDKjFVTU8ORI0e4+uqrjS5F5KJRRo6kfDxD+Tg2ZeTYUikjNWXxAkWjUZ544gnuu+8+o0sZVywWCzabDYBf//rXyfBJRW1tbWRmZiYf+/1+WltbDazIeGazGafTCcDmzZtZtmxZygcNwCOPPMIDDzxgdBkiF40ycjTl4xnKx7EpI8eWShmpEbL/x6ZNm9i0adOI55YtW8btt9+O1+s1qCrjjbVfvvOd77B06VKef/55qqureeqppwyqbvzRpf7O2Lp1K5s3b+aZZ54xuhTDvfzyy1x22WWa1iUTljJyNOXjhVE+jqSMPCPVMlIXhr5AlZWVDA0NAdDQ0IDf7+fxxx9nxowZBldmvE2bNvHaa6/x5JNPJo8GpqINGzYQCASorKwE4Mtf/jJbtmxJ+SkZ27dv5/HHH+fpp58mIyPD6HIMd++999LY2IjZbKa5uRmr1cpDDz3E4sWLjS5N5DNTRo5N+ZigfPx0ysiRUi0jNUJ2gV588cXk9gMPPMDq1atTPmgAGhsbefHFF3nuuedSOmwAlixZwoYNG6isrKS6uppgMJjyYdPd3c369ev51a9+paAZ9thjjyW3N2zYQEFBwaQNGkkdysjRlI9nKB/HpowcLdUyUg2ZXBSbNm0iFApxzz33JJ/buHEjVqvVwKqMsWDBAioqKqisrMRkMrFu3TqjSzLcK6+8QkdHB/fee2/yuUceeYT8/HwDqxIRufSUj2coH8emjBRNWRQRERERETGIVlkUERERERExiBoyERERERERg6ghExERERERMYgaMhEREREREYOoIRMRERERETGIGjIRERERERGDqCETERERERExiBoyERERERERg/wfl/DB2aiuIRQAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "tags": [] - } - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "S4_rHECFgckq", - "colab_type": "text" - }, - "source": [ - "Look at the derivate graph. The derivative multiplied by the error tells us where to assign blame and update the weights most effective. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "aY4djmVHgckr", - "colab_type": "text" - }, - "source": [ - "##### 2nd Error\n", - "Justice hasn't been served yet - tho. We still have neurons to blame. Let's go back another layer. \n", - "\n", - "`self.z2_error = self.o_delta.dot(self.weights2.T)`\n", - "\n", - "__Discussion:__ Why is this shape different?" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "unOJ8HKtgckr", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "42f2ed68-8c6c-466d-9df1-562b8f71d552" - }, - "source": [ - "nn.o_delta.dot(nn.weights2.T)" - ], - "execution_count": 18, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[-0.23952061, -0.19347426, 0.1111231 ],\n", - " [-0.1847631 , -0.14924354, 0.08571892],\n", - " [-0.21246478, -0.17161974, 0.09857082]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 18 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3EaJGaqhgckt", - "colab_type": "text" - }, - "source": [ - "##### 2nd Gradient\n", - "For each observation, how much more sigmoid activation from this layer would have pushed us towards the right answer?\n", - "\n", - "`self.z2_delta = self.z2_error * self.sigmoidPrime(self.activated_hidden)`" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "OPf5OrdYgcku", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "91d27661-c401-4678-c290-6de149465f82" - }, - "source": [ - "nn.z2_delta" - ], - "execution_count": 19, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[-0.05483346, -0.04408088, 0.01744341],\n", - " [-0.03871267, -0.0311213 , 0.01231513],\n", - " [-0.04559029, -0.03665025, 0.01450301]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 19 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "F1PcMJoygckw", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "outputId": "cb9374e9-3d59-4042-f8c8-ef00e2ce62a1" - }, - "source": [ - "X.T.shape == nn.weights1.shape" - ], - "execution_count": 20, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "True" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 20 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Oo8wACzsgckz", - "colab_type": "code", - "colab": {}, - "outputId": "8b4ec8aa-de30-4cec-fca0-3332e493190e" - }, - "source": [ - "" - ], - "execution_count": null, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.66666667, 1. ],\n", - " [0.33333333, 0.55555556],\n", - " [1. , 0.66666667]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 44 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "YvjEuIAvgck1", - "colab_type": "text" - }, - "source": [ - "##### Descent\n", - "\n", - "*Discussion:* Input to Hidden Weight Update\n", - "- We multiply the gradient by the inputs. Why?\n", - "- Why do we need to transpose the inputs? " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "3vtKK4CVgck1", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "outputId": "1ca86f11-d561-4fc3-9fde-61d1d9b4e1b9" - }, - "source": [ - "X.T" - ], - "execution_count": 21, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.66666667, 0.33333333, 1. ],\n", - " [1. , 0.55555556, 0.66666667]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 21 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "bOwpGy-mgck3", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "outputId": "9f8abcb2-bfca-4d53-e8f8-63b8f7105766" - }, - "source": [ - "X.T.dot(nn.z2_delta)" - ], - "execution_count": 22, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[-0.09505015, -0.07641127, 0.03023699],\n", - " [-0.10673402, -0.08580399, 0.03395382]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 22 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MCXdRzRrgck6", - "colab_type": "text" - }, - "source": [ - "*Discussion:* Hidden to Output Weight Update\n", - "- Why is output the shape 3x1? \n", - "- We multiply the gradient by the inputs. Why?\n", - "- Why do we need to transpose the inputs?" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "V3_1WJgIgck6", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - }, - "outputId": "6ec79203-2c90-4702-b50b-9c69dc1099e9" - }, - "source": [ - "nn.activated_hidden.T.dot(nn.o_delta)" - ], - "execution_count": 23, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0.17103686],\n", - " [0.13129211],\n", - " [0.18053762]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 23 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7iziWWURgck8", - "colab_type": "text" - }, - "source": [ - "### Train the Network (fo real this time)" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "6VPrkCXvgck9", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "outputId": "3fcc3281-5089-4a6f-bd1a-5318758f909e" - }, - "source": [ - "# Train my 'net\n", - "nn = NeuralNetwork()\n", - "\n", - "# Number of Epochs / Iterations\n", - "for i in range(5000):\n", - " if (i+1 in [1,2,3,4,5]) or ((i+1) % 500 ==0):\n", - " print('+' + '---' * 3 + f'EPOCH {i+1}' + '---'*3 + '+')\n", - " print('Input: \\n', X)\n", - " print('Actual Output: \\n', y)\n", - " print('Predicted Output: \\n', str(nn.feed_forward(X)))\n", - " print(\"Loss: \\n\", str(np.mean(np.square(y - nn.feed_forward(X)))))\n", - " nn.train(X,y)" - ], - "execution_count": 24, - "outputs": [ - { - "output_type": "stream", - "text": [ - "+---------EPOCH 1---------+\n", - "Input: \n", - " [[0.66666667 1. ]\n", - " [0.33333333 0.55555556]\n", - " [1. 0.66666667]]\n", - "Actual Output: \n", - " [0.00669285 0.00675966 0.00682713 0.00689527 0.00696409 0.00703359\n", - " 0.00710377 0.00717466 0.00724624 0.00731853 0.00739154 0.00746527\n", - " 0.00753973 0.00761493 0.00769088 0.00776757 0.00784502 0.00792324\n", - " 0.00800223 0.00808201 0.00816257 0.00824393 0.00832609 0.00840907\n", - " 0.00849286 0.00857749 0.00866294 0.00874925 0.0088364 0.00892442\n", - " 0.0090133 0.00910306 0.00919371 0.00928525 0.00937769 0.00947104\n", - " 0.00956532 0.00966052 0.00975667 0.00985376 0.0099518 0.01005081\n", - " 0.0101508 0.01025177 0.01035374 0.01045671 0.01056069 0.01066569\n", - " 0.01077173 0.01087881 0.01098694 0.01109614 0.01120641 0.01131776\n", - " 0.0114302 0.01154375 0.01165842 0.01177421 0.01189113 0.0120092\n", - " 0.01212843 0.01224883 0.01237041 0.01249319 0.01261716 0.01274235\n", - " 0.01286876 0.01299642 0.01312532 0.01325548 0.01338692 0.01351964\n", - " 0.01365366 0.01378899 0.01392564 0.01406363 0.01420296 0.01434366\n", - " 0.01448572 0.01462918 0.01477403 0.0149203 0.01506799 0.01521712\n", - " 0.01536771 0.01551976 0.01567329 0.01582831 0.01598485 0.01614291\n", - " 0.0163025 0.01646364 0.01662636 0.01679065 0.01695654 0.01712403\n", - " 0.01729316 0.01746392 0.01763634 0.01781043 0.01798621 0.01816369\n", - " 0.01834289 0.01852382 0.01870651 0.01889096 0.0190772 0.01926523\n", - " 0.01945508 0.01964677 0.01984031 0.02003571 0.020233 0.02043219\n", - " 0.0206333 0.02083634 0.02104135 0.02124832 0.02145729 0.02166827\n", - " 0.02188127 0.02209632 0.02231344 0.02253264 0.02275394 0.02297737\n", - " 0.02320294 0.02343067 0.02366058 0.02389269 0.02412702 0.02436359\n", - " 0.02460243 0.02484354 0.02508696 0.0253327 0.02558079 0.02583124\n", - " 0.02608408 0.02633932 0.02659699 0.02685712 0.02711972 0.02738481\n", - " 0.02765242 0.02792257 0.02819529 0.02847059 0.0287485 0.02902904\n", - " 0.02931223 0.0295981 0.02988668 0.03017798 0.03047203 0.03076886\n", - " 0.03106848 0.03137093 0.03167623 0.0319844 0.03229546 0.03260946\n", - " 0.03292639 0.03324631 0.03356922 0.03389516 0.03422416 0.03455623\n", - " 0.03489141 0.03522972 0.03557119 0.03591585 0.03626372 0.03661483\n", - " 0.03696921 0.03732689 0.03768789 0.03805225 0.03841999 0.03879113\n", - " 0.03916572 0.03954378 0.03992533 0.04031042 0.04069905 0.04109128\n", - " 0.04148712 0.04188661 0.04228977 0.04269664 0.04310725 0.04352163\n", - " 0.04393982 0.04436183 0.0447877 0.04521747 0.04565117 0.04608883\n", - " 0.04653047 0.04697615 0.04742587 0.04787969 0.04833763 0.04879972\n", - " 0.04926601 0.04973651 0.05021127 0.05069032 0.0511737 0.05166144\n", - " 0.05215356 0.05265012 0.05315114 0.05365665 0.0541667 0.05468132\n", - " 0.05520054 0.0557244 0.05625293 0.05678618 0.05732418 0.05786696\n", - " 0.05841456 0.05896701 0.05952437 0.06008665 0.0606539 0.06122616\n", - " 0.06180347 0.06238585 0.06297336 0.06356602 0.06416388 0.06476697\n", - " 0.06537533 0.06598901 0.06660804 0.06723245 0.06786229 0.0684976\n", - " 0.06913842 0.06978478 0.07043673 0.0710943 0.07175754 0.07242649\n", - " 0.07310117 0.07378165 0.07446795 0.07516011 0.07585818 0.0765622\n", - " 0.0772722 0.07798824 0.07871034 0.07943855 0.08017291 0.08091347\n", - " 0.08166026 0.08241332 0.0831727 0.08393843 0.08471057 0.08548914\n", - " 0.08627419 0.08706577 0.08786391 0.08866866 0.08948006 0.09029814\n", - " 0.09112296 0.09195455 0.09279295 0.09363821 0.09449037 0.09534946\n", - " 0.09621554 0.09708864 0.0979688 0.09885607 0.09975049 0.10065209\n", - " 0.10156093 0.10247703 0.10340045 0.10433122 0.10526939 0.10621499\n", - " 0.10716807 0.10812867 0.10909682 0.11007257 0.11105597 0.11204704\n", - " 0.11304583 0.11405238 0.11506673 0.11608892 0.11711899 0.11815698\n", - " 0.11920292 0.12025686 0.12131884 0.12238889 0.12346705 0.12455336\n", - " 0.12564786 0.12675058 0.12786157 0.12898085 0.13010847 0.13124447\n", - " 0.13238887 0.13354172 0.13470305 0.1358729 0.13705129 0.13823827\n", - " 0.13943387 0.14063813 0.14185106 0.14307272 0.14430313 0.14554233\n", - " 0.14679034 0.1480472 0.14931293 0.15058758 0.15187116 0.15316372\n", - " 0.15446527 0.15577584 0.15709547 0.15842418 0.159762 0.16110895\n", - " 0.16246506 0.16383036 0.16520487 0.16658861 0.16798161 0.1693839\n", - " 0.17079548 0.17221639 0.17364665 0.17508627 0.17653527 0.17799369\n", - " 0.17946152 0.18093879 0.18242552 0.18392173 0.18542742 0.18694261\n", - " 0.18846733 0.19000157 0.19154535 0.19309868 0.19466158 0.19623406\n", - " 0.19781611 0.19940776 0.201009 0.20261985 0.2042403 0.20587037\n", - " 0.20751006 0.20915937 0.21081829 0.21248684 0.21416502 0.21585281\n", - " 0.21755022 0.21925725 0.22097389 0.22270014 0.22443599 0.22618143\n", - " 0.22793645 0.22970105 0.23147522 0.23325894 0.2350522 0.23685498\n", - " 0.23866729 0.24048908 0.24232036 0.2441611 0.24601128 0.24787089\n", - " 0.24973989 0.25161828 0.25350602 0.25540308 0.25730945 0.2592251\n", - " 0.26114999 0.2630841 0.2650274 0.26697985 0.26894142 0.27091208\n", - " 0.27289178 0.2748805 0.27687819 0.27888482 0.28090034 0.28292471\n", - " 0.28495789 0.28699984 0.2890505 0.29110983 0.29317778 0.2952543\n", - " 0.29733935 0.29943286 0.30153478 0.30364507 0.30576366 0.3078905\n", - " 0.31002552 0.31216867 0.31431989 0.31647911 0.31864627 0.3208213\n", - " 0.32300414 0.32519473 0.32739298 0.32959884 0.33181223 0.33403307\n", - " 0.3362613 0.33849684 0.34073961 0.34298954 0.34524654 0.34751054\n", - " 0.34978145 0.3520592 0.35434369 0.35663485 0.35893259 0.36123682\n", - " 0.36354746 0.36586441 0.36818758 0.37051689 0.37285223 0.37519353\n", - " 0.37754067 0.37989357 0.38225213 0.38461624 0.38698582 0.38936077\n", - " 0.39174097 0.39412633 0.39651675 0.39891212 0.40131234 0.4037173\n", - " 0.4061269 0.40854102 0.41095957 0.41338242 0.41580948 0.41824062\n", - " 0.42067575 0.42311474 0.42555748 0.42800387 0.43045378 0.4329071\n", - " 0.43536371 0.4378235 0.44028635 0.44275215 0.44522076 0.44769209\n", - " 0.450166 0.45264238 0.45512111 0.45760206 0.46008512 0.46257015\n", - " 0.46505705 0.46754569 0.47003595 0.4725277 0.47502081 0.47751518\n", - " 0.48001066 0.48250714 0.4850045 0.4875026 0.49000133 0.49250056\n", - " 0.49500017 0.49750002 0.5 0.50249998 0.50499983 0.50749944\n", - " 0.50999867 0.5124974 0.5149955 0.51749286 0.51998934 0.52248482\n", - " 0.52497919 0.5274723 0.52996405 0.53245431 0.53494295 0.53742985\n", - " 0.53991488 0.54239794 0.54487889 0.54735762 0.549834 0.55230791\n", - " 0.55477924 0.55724785 0.55971365 0.5621765 0.56463629 0.5670929\n", - " 0.56954622 0.57199613 0.57444252 0.57688526 0.57932425 0.58175938\n", - " 0.58419052 0.58661758 0.58904043 0.59145898 0.5938731 0.5962827\n", - " 0.59868766 0.60108788 0.60348325 0.60587367 0.60825903 0.61063923\n", - " 0.61301418 0.61538376 0.61774787 0.62010643 0.62245933 0.62480647\n", - " 0.62714777 0.62948311 0.63181242 0.63413559 0.63645254 0.63876318\n", - " 0.64106741 0.64336515 0.64565631 0.6479408 0.65021855 0.65248946\n", - " 0.65475346 0.65701046 0.65926039 0.66150316 0.6637387 0.66596693\n", - " 0.66818777 0.67040116 0.67260702 0.67480527 0.67699586 0.6791787\n", - " 0.68135373 0.68352089 0.68568011 0.68783133 0.68997448 0.6921095\n", - " 0.69423634 0.69635493 0.69846522 0.70056714 0.70266065 0.7047457\n", - " 0.70682222 0.70889017 0.7109495 0.71300016 0.71504211 0.71707529\n", - " 0.71909966 0.72111518 0.72312181 0.7251195 0.72710822 0.72908792\n", - " 0.73105858 0.73302015 0.7349726 0.7369159 0.73885001 0.7407749\n", - " 0.74269055 0.74459692 0.74649398 0.74838172 0.75026011 0.75212911\n", - " 0.75398872 0.7558389 0.75767964 0.75951092 0.76133271 0.76314502\n", - " 0.7649478 0.76674106 0.76852478 0.77029895 0.77206355 0.77381857\n", - " 0.77556401 0.77729986 0.77902611 0.78074275 0.78244978 0.78414719\n", - " 0.78583498 0.78751316 0.78918171 0.79084063 0.79248994 0.79412963\n", - " 0.7957597 0.79738015 0.798991 0.80059224 0.80218389 0.80376594\n", - " 0.80533842 0.80690132 0.80845465 0.80999843 0.81153267 0.81305739\n", - " 0.81457258 0.81607827 0.81757448 0.81906121 0.82053848 0.82200631\n", - " 0.82346473 0.82491373 0.82635335 0.82778361 0.82920452 0.8306161\n", - " 0.83201839 0.83341139 0.83479513 0.83616964 0.83753494 0.83889105\n", - " 0.840238 0.84157582 0.84290453 0.84422416 0.84553473 0.84683628\n", - " 0.84812884 0.84941242 0.85068707 0.8519528 0.85320966 0.85445767\n", - " 0.85569687 0.85692728 0.85814894 0.85936187 0.86056613 0.86176173\n", - " 0.86294871 0.8641271 0.86529695 0.86645828 0.86761113 0.86875553\n", - " 0.86989153 0.87101915 0.87213843 0.87324942 0.87435214 0.87544664\n", - " 0.87653295 0.87761111 0.87868116 0.87974314 0.88079708 0.88184302\n", - " 0.88288101 0.88391108 0.88493327 0.88594762 0.88695417 0.88795296\n", - " 0.88894403 0.88992743 0.89090318 0.89187133 0.89283193 0.89378501\n", - " 0.89473061 0.89566878 0.89659955 0.89752297 0.89843907 0.89934791\n", - " 0.90024951 0.90114393 0.9020312 0.90291136 0.90378446 0.90465054\n", - " 0.90550963 0.90636179 0.90720705 0.90804545 0.90887704 0.90970186\n", - " 0.91051994 0.91133134 0.91213609 0.91293423 0.91372581 0.91451086\n", - " 0.91528943 0.91606157 0.9168273 0.91758668 0.91833974 0.91908653\n", - " 0.91982709 0.92056145 0.92128966 0.92201176 0.9227278 0.9234378\n", - " 0.92414182 0.92483989 0.92553205 0.92621835 0.92689883 0.92757351\n", - " 0.92824246 0.9289057 0.92956327 0.93021522 0.93086158 0.9315024\n", - " 0.93213771 0.93276755 0.93339196 0.93401099 0.93462467 0.93523303\n", - " 0.93583612 0.93643398 0.93702664 0.93761415 0.93819653 0.93877384\n", - " 0.9393461 0.93991335 0.94047563 0.94103299 0.94158544 0.94213304\n", - " 0.94267582 0.94321382 0.94374707 0.9442756 0.94479946 0.94531868\n", - " 0.9458333 0.94634335 0.94684886 0.94734988 0.94784644 0.94833856\n", - " 0.9488263 0.94930968 0.94978873 0.95026349 0.95073399 0.95120028\n", - " 0.95166237 0.95212031 0.95257413 0.95302385 0.95346953 0.95391117\n", - " 0.95434883 0.95478253 0.9552123 0.95563817 0.95606018 0.95647837\n", - " 0.95689275 0.95730336 0.95771023 0.95811339 0.95851288 0.95890872\n", - " 0.95930095 0.95968958 0.96007467 0.96045622 0.96083428 0.96120887\n", - " 0.96158001 0.96194775 0.96231211 0.96267311 0.96303079 0.96338517\n", - " 0.96373628 0.96408415 0.96442881 0.96477028 0.96510859 0.96544377\n", - " 0.96577584 0.96610484 0.96643078 0.96675369 0.96707361 0.96739054\n", - " 0.96770454 0.9680156 0.96832377 0.96862907 0.96893152 0.96923114\n", - " 0.96952797 0.96982202 0.97011332 0.9704019 0.97068777 0.97097096\n", - " 0.9712515 0.97152941 0.97180471 0.97207743 0.97234758 0.97261519\n", - " 0.97288028 0.97314288 0.97340301 0.97366068 0.97391592 0.97416876\n", - " 0.97441921 0.9746673 0.97491304 0.97515646 0.97539757 0.97563641\n", - " 0.97587298 0.97610731 0.97633942 0.97656933 0.97679706 0.97702263\n", - " 0.97724606 0.97746736 0.97768656 0.97790368 0.97811873 0.97833173\n", - " 0.97854271 0.97875168 0.97895865 0.97916366 0.9793667 0.97956781\n", - " 0.979767 0.97996429 0.98015969 0.98035323 0.98054492 0.98073477\n", - " 0.9809228 0.98110904 0.98129349 0.98147618 0.98165711 0.98183631\n", - " 0.98201379 0.98218957 0.98236366 0.98253608 0.98270684 0.98287597\n", - " 0.98304346 0.98320935 0.98337364 0.98353636 0.9836975 0.98385709\n", - " 0.98401515 0.98417169 0.98432671 0.98448024 0.98463229 0.98478288\n", - " 0.98493201 0.9850797 0.98522597 0.98537082 0.98551428 0.98565634\n", - " 0.98579704 0.98593637 0.98607436 0.98621101 0.98634634 0.98648036\n", - " 0.98661308 0.98674452 0.98687468 0.98700358 0.98713124 0.98725765\n", - " 0.98738284 0.98750681 0.98762959 0.98775117 0.98787157 0.9879908\n", - " 0.98810887 0.98822579 0.98834158 0.98845625 0.9885698 0.98868224\n", - " 0.98879359 0.98890386 0.98901306 0.98912119 0.98922827 0.98933431\n", - " 0.98943931 0.98954329 0.98964626 0.98974823 0.9898492 0.98994919\n", - " 0.9900482 0.99014624 0.99024333 0.99033948 0.99043468 0.99052896\n", - " 0.99062231 0.99071475 0.99080629 0.99089694 0.9909867 0.99107558\n", - " 0.9911636 0.99125075 0.99133706 0.99142251 0.99150714 0.99159093\n", - " 0.99167391 0.99175607 0.99183743 0.99191799 0.99199777 0.99207676\n", - " 0.99215498 0.99223243 0.99230912 0.99238507 0.99246027 0.99253473\n", - " 0.99260846 0.99268147 0.99275376 0.99282534 0.99289623 0.99296641\n", - " 0.99303591 0.99310473 0.99317287 0.99324034]\n", - "Predicted Output: \n", - " [[0.37771334]\n", - " [0.41477855]\n", - " [0.40348484]]\n", - "Loss: \n", - " 0.16174922917818682\n" - ], - "name": "stdout" - }, - { - "output_type": "error", - "ename": "ValueError", - "evalue": "ignored", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Predicted Output: \\n'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Loss: \\n\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msquare\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[0mo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 68\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, X, y, o)\u001b[0m\n\u001b[1;32m 56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0;31m# z2 error: how much were our output layer weights off\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mz2_error\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mo_delta\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweights2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 60\u001b[0m \u001b[0;31m# z2 delta: how much were the weights off?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mValueError\u001b[0m: shapes (3,1000) and (1,3) not aligned: 1000 (dim 1) != 1 (dim 0)" - ] - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "iKUVGoRxgck_", - "colab_type": "text" - }, - "source": [ - "## Challenge\n", - "\n", - "In the module project, you will implement backpropagation inside a multi-layer perceptron (aka a feedforward neural network). " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": true, - "id": "vTqZg-6igclA", - "colab_type": "text" - }, - "source": [ - "# Stochastic Gradient Descent (Learn)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0nrm-racgclA", - "colab_type": "text" - }, - "source": [ - "## Overview\n", - "\n", - "The What - Stochastic Gradient Descent calculates an approximation of the gradient over the entire dataset by reviewing the predictions of a random sample. \n", - "\n", - "The Why - *Speed*. Calculating the gradient over the entire dataset is extremely expensive computationally. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "ZF7UE-KluPsX" - }, - "source": [ - "## Follow Along\n", - "\n", - "A true Stochastic GD-based implementation from [Welch Labs](https://www.youtube.com/watch?v=bxe2T-V8XRs)" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "P-ChVGikgclD", - "colab_type": "code", - "colab": {} - }, - "source": [ - "class Neural_Network(object):\n", - " def __init__(self): \n", - " #Define Hyperparameters\n", - " self.inputLayerSize = 2\n", - " self.outputLayerSize = 1\n", - " self.hiddenLayerSize = 3\n", - " \n", - " #Weights (parameters)\n", - " self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)\n", - " self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)\n", - " \n", - " def forward(self, X):\n", - " #Propogate inputs though network\n", - " self.z2 = np.dot(X, self.W1)\n", - " self.a2 = self.sigmoid(self.z2)\n", - " self.z3 = np.dot(self.a2, self.W2)\n", - " yHat = self.sigmoid(self.z3) \n", - " return yHat\n", - " \n", - " def sigmoid(self, z):\n", - " #Apply sigmoid activation function to scalar, vector, or matrix\n", - " return 1/(1+np.exp(-z))\n", - " \n", - " def sigmoidPrime(self,z):\n", - " #Gradient of sigmoid\n", - " return np.exp(-z)/((1+np.exp(-z))**2)\n", - " \n", - " def costFunction(self, X, y):\n", - " #Compute cost for given X,y, use weights already stored in class.\n", - " self.yHat = self.forward(X)\n", - " J = 0.5*sum((y-self.yHat)**2)\n", - " return J\n", - " \n", - " def costFunctionPrime(self, X, y):\n", - " #Compute derivative with respect to W and W2 for a given X and y:\n", - " self.yHat = self.forward(X)\n", - " \n", - " delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))\n", - " dJdW2 = np.dot(self.a2.T, delta3)\n", - " \n", - " delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)\n", - " dJdW1 = np.dot(X.T, delta2) \n", - " \n", - " return dJdW1, dJdW2\n", - " \n", - " #Helper Functions for interacting with other classes:\n", - " def getParams(self):\n", - " #Get W1 and W2 unrolled into vector:\n", - " params = np.concatenate((self.W1.ravel(), self.W2.ravel()))\n", - " return params\n", - " \n", - " def setParams(self, params):\n", - " #Set W1 and W2 using single paramater vector.\n", - " W1_start = 0\n", - " W1_end = self.hiddenLayerSize * self.inputLayerSize\n", - " self.W1 = np.reshape(params[W1_start:W1_end], (self.inputLayerSize , self.hiddenLayerSize))\n", - " W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize\n", - " self.W2 = np.reshape(params[W1_end:W2_end], (self.hiddenLayerSize, self.outputLayerSize))\n", - " \n", - " def computeGradients(self, X, y):\n", - " dJdW1, dJdW2 = self.costFunctionPrime(X, y)\n", - " return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))" - ], - "execution_count": 26, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "uA9LaTgKr6rP", - "colab": {} - }, - "source": [ - "from scipy import optimize\n", - "class trainer(object):\n", - " def __init__(self, N):\n", - " #Make Local reference to network:\n", - " self.N = N\n", - " \n", - " def callbackF(self, params):\n", - " self.N.setParams(params)\n", - " self.J.append(self.N.costFunction(self.X, self.y)) \n", - " \n", - " def costFunctionWrapper(self, params, X, y):\n", - " self.N.setParams(params)\n", - " cost = self.N.costFunction(X, y)\n", - " grad = self.N.computeGradients(X,y)\n", - " \n", - " return cost, grad\n", - " \n", - " def train(self, X, y):\n", - " #Make an internal variable for the callback function:\n", - " self.X = X\n", - " self.y = y\n", - "\n", - " #Make empty list to store costs:\n", - " self.J = []\n", - " \n", - " params0 = self.N.getParams()\n", - "\n", - " options = {'maxiter': 200, 'disp' : True}\n", - " _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \\\n", - " args=(X, y), options=options, callback=self.callbackF)\n", - "\n", - " self.N.setParams(_res.x)\n", - " self.optimizationResults = _res" - ], - "execution_count": 29, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "g_kHb6Se1u9y", - "colab": {} - }, - "source": [ - "NN = Neural_Network()" - ], - "execution_count": 30, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "hYYVhFf4rn3q", - "colab": {} - }, - "source": [ - "T = trainer(NN)" - ], - "execution_count": 31, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "L-gYdVfgrysE", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 341 - }, - "outputId": "ca9e7fa2-920a-45c7-cc2c-c56e61e4638e" - }, - "source": [ - "T.train(X,y)" - ], - "execution_count": 32, - "outputs": [ - { - "output_type": "error", - "ename": "ValueError", - "evalue": "ignored", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mT\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 28\u001b[0m \u001b[0moptions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m'maxiter'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;36m200\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'disp'\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m \u001b[0m_res\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moptimize\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mminimize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunctionWrapper\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'BFGS'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbackF\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 31\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_res\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/_minimize.py\u001b[0m in \u001b[0;36mminimize\u001b[0;34m(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)\u001b[0m\n\u001b[1;32m 602\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_cg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 603\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'bfgs'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 604\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_bfgs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 605\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'newton-cg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 606\u001b[0m return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m_minimize_bfgs\u001b[0;34m(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)\u001b[0m\n\u001b[1;32m 1001\u001b[0m \u001b[0mfunc_calls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrap_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1002\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1003\u001b[0;31m \u001b[0mold_fval\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1004\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1005\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mfprime\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36mfunction_wrapper\u001b[0;34m(*wrapper_args)\u001b[0m\n\u001b[1;32m 325\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mwrapper_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 326\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 327\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 329\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, x, *args)\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 65\u001b[0;31m \u001b[0mfg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 66\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjac\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36mcostFunctionWrapper\u001b[0;34m(self, params, X, y)\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m \u001b[0mgrad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcomputeGradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36mcomputeGradients\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 60\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcomputeGradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m \u001b[0mdJdW1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdJdW2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunctionPrime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 62\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdJdW1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdJdW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36mcostFunctionPrime\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 39\u001b[0m \u001b[0mdJdW2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ma2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelta3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 41\u001b[0;31m \u001b[0mdelta2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdelta3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msigmoidPrime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mz2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 42\u001b[0m \u001b[0mdJdW1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelta2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 43\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mdot\u001b[0;34m(*args, **kwargs)\u001b[0m\n", - "\u001b[0;31mValueError\u001b[0m: shapes (3,1000) and (1,3) not aligned: 1000 (dim 1) != 1 (dim 0)" - ] - } - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "Jyv_L8Z2sKOA", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "outputId": "46e5f4d5-0aaf-4bfb-9d27-e5f2ee39fbd4" - }, - "source": [ - "print(\"Predicted Output: \\n\" + str(NN.forward(X))) \n", - "print(\"Loss: \\n\" + str(np.mean(np.square(y - NN.forward(X))))) # mean sum squared loss" - ], - "execution_count": 33, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Predicted Output: \n", - "[[0.19042135]\n", - " [0.21647809]\n", - " [0.19075219]]\n", - "Loss: \n", - "0.24166109849505304\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "Gtf9WI9FtGPk", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 361 - }, - "outputId": "d062b2a3-5a92-403e-8ce0-c070aa79907b" - }, - "source": [ - "import matplotlib.pyplot as plt\n", - "plt.plot(T.J)\n", - "plt.xlabel('Iterations')\n", - "plt.ylabel('Cost')\n", - "plt.show() " - ], - "execution_count": null, - "outputs": [ - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3df5zdVX3n8dd7fibMDIFk7iCSYEImiAFrcGOsFbsKpcRu12AXJbRa2tKiu7Drj91W6D7WWh5mV/pDtFvQxYJSioY8UNepZaHagFar+QFEJEBkSNBEKJlAIJkk8/uzf3zPJDeXOz8yc+/cmbnv58P7uN/v+Z5z7vl+H3E+nO/5fs9RRGBmZjZZNZVugJmZzQ4OKGZmVhIOKGZmVhIOKGZmVhIOKGZmVhJ1lW5AJbW2tsbixYsr3QwzsxnloYce2hcRucL0qg4oixcvZuvWrZVuhpnZjCLpp8XSfcvLzMxKwgHFzMxKwgHFzMxKwgHFzMxKwgHFzMxKwgHFzMxKoqwBRdJqSTskdUq6rsjxRkl3p+ObJC3OO3Z9St8h6ZKCcrWSHpH0zby0JamOp1KdDeU8NzMzO17ZAoqkWuBm4J3AcuAKScsLsl0F7I+IduAm4MZUdjmwFjgXWA3ckuob9iHgiYK6bgRuiohlwP5Ud1lsfPJ5bnmws1zVm5nNSOXsoawCOiNiZ0T0AeuBNQV51gB3pO17gIskKaWvj4jeiNgFdKb6kLQQ+HfA3wxXkspcmOog1XlpWc4K+O5P9nHLA0+Xq3ozsxmpnAHlDGB33v6elFY0T0QMAC8DC8Yo+xngj4ChvOMLgJdSHSP9FgCSrpa0VdLWrq6uEz0nAHItjXT3DnCkb3BC5c3MZqNyBhQVSStcHnKkPEXTJf06sDciHprAb2WJEbdGxMqIWJnLvWIqmnHJNTcCsK+7d0Llzcxmo3IGlD3Aorz9hcCzI+WRVAfMA14cpexbgXdJeobsFtqFkv4O2AeckuoY6bdKprUlG+/vckAxMzuqnAFlC7AsPX3VQDbI3lGQpwO4Mm1fBmyMbJH7DmBtegpsCbAM2BwR10fEwohYnOrbGBHvS2UeSHWQ6vxGuU6sdbiHctABxcxsWNkCShrPuBa4n+yJrA0RsV3SDZLelbLdBiyQ1Al8FLguld0ObAAeB+4DromIsQYsPgZ8NNW1INVdFrmWLKC4h2JmdkxZp6+PiHuBewvSPp633QO8Z4Sy64B1o9T9IPBg3v5O0pNg5bagabiH0jcVP2dmNiP4TfkJaKir4ZST6j0ob2aWxwFlglqbG+nyGIqZ2VEOKBPU2tzgHoqZWR4HlAnKtcxxQDEzy+OAMkGtzQ2+5WVmlscBZYJamxs51DfI4b6BsTObmVUBB5QJGn4XxY8Om5llHFAmaHg+L7/caGaWcUCZoKM9FAcUMzPAAWXChufz8sC8mVnGAWWCFjRnMw67h2JmlnFAmaD62hpO9fQrZmZHOaBMgqdfMTM7xgFlElqbG9nX7ceGzczAAWVSci2NvuVlZpaUNaBIWi1ph6ROSdcVOd4o6e50fJOkxXnHrk/pOyRdktLmSNos6UeStkv607z8X5K0S9K29FlRznMD3/IyM8tXtgW2JNUCNwMXk60Rv0VSR0Q8npftKmB/RLRLWgvcCFwuaTnZEr/nAq8Gvi3pbKAXuDAiuiXVA9+T9P8i4oepvj+MiHvKdU6Fci2NHO4b5FDvAE2NZV2rzMxs2itnD2UV0BkROyOiD1gPrCnIswa4I23fA1wkSSl9fUT0RsQuoBNYFZnulL8+faKM5zCqVj86bGZ2VDkDyhnA7rz9PSmtaJ60Bv3LZOvBj1hWUq2kbcBe4FsRsSkv3zpJj0q6SVJjKU+mmFa/LW9mdlQ5A4qKpBX2JkbKM2LZiBiMiBXAQmCVpPPS8euBc4A3AfOBjxVtlHS1pK2StnZ1dY19FqM4Op+XJ4g0MytrQNkDLMrbXwg8O1IeSXXAPODF8ZSNiJeAB4HVaf+5dEusF/gi2S23V4iIWyNiZUSszOVyEzuzZHg+L08QaWZW3oCyBVgmaYmkBrJB9o6CPB3AlWn7MmBjRERKX5ueAlsCLAM2S8pJOgVA0lzgV4An0/7p6VvApcBjZTw3AOY3pTEUP+llZla+p7wiYkDStcD9QC1we0Rsl3QDsDUiOoDbgDsldZL1TNamstslbQAeBwaAayJiMAWNO9ITZDXAhoj4ZvrJuyTlyG6XbQM+WK5zG1ZfW8P8Jq8tb2YGZQwoABFxL3BvQdrH87Z7gPeMUHYdsK4g7VHg/BHyXzjZ9k6ElwI2M8v4TflJ8tvyZmYZB5RJ8nxeZmYZB5RJ8vQrZmYZB5RJyrU0cqQ/m37FzKyaOaBMkpcCNjPLOKBMkufzMjPLOKBMUs7zeZmZAQ4ok5bzLS8zM8ABZdLmNzUgQZcfHTazKueAMkl1tTXMP8nTr5iZOaCUgN9FMTNzQCkJT79iZuaAUhKtzb7lZWbmgFICw7e8sqVczMyqkwNKCeRaGunpH+JQ32Clm2JmVjEOKCXg6VfMzMocUCStlrRDUqek64ocb5R0dzq+SdLivGPXp/Qdki5JaXMkbZb0I0nbJf1pXv4lqY6nUp0N5Ty3fH5b3sysjAElLdN7M/BOYDlwhaTlBdmuAvZHRDtwE3BjKrucbDngc4HVwC2pvl7gwoh4A7ACWC3pF1NdNwI3RcQyYH+qe0oM91C8tryZVbNy9lBWAZ0RsTMi+oD1wJqCPGuAO9L2PcBFkpTS10dEb0TsAjqBVZHpTvnr0ydSmQtTHaQ6Ly3XiRVqbck6Q13uoZhZFStnQDkD2J23vyelFc0TEQPAy8CC0cpKqpW0DdgLfCsiNqUyL6U6RvotUvmrJW2VtLWrq2sSp3fMgqZGauQeiplVt3IGFBVJK3yudqQ8I5aNiMGIWAEsBFZJOm+cv0Uqf2tErIyIlblcbsTGn4jaGjG/qcHzeZlZVStnQNkDLMrbXwg8O1IeSXXAPODF8ZSNiJeAB8nGWPYBp6Q6RvqtsvL0K2ZW7coZULYAy9LTVw1kg+wdBXk6gCvT9mXAxsjeDuwA1qanwJYAy4DNknKSTgGQNBf4FeDJVOaBVAepzm+U8dxewdOvmFm1qxs7y8RExICka4H7gVrg9ojYLukGYGtEdAC3AXdK6iTrmaxNZbdL2gA8DgwA10TEoKTTgTvSE181wIaI+Gb6yY8B6yV9Engk1T1lWpsb2bXv0FT+pJnZtFK2gAIQEfcC9xakfTxvuwd4zwhl1wHrCtIeBc4fIf9OsifLKiLXcmz6leyhMzOz6uI35UuktbmB3oEhunsHxs5sZjYLOaCUiKdfMbNq54BSIsemX/Gjw2ZWnRxQSuTo9Ct+0svMqpQDSon4lpeZVTsHlBKZ39SQTb/iHoqZVSkHlBLJpl/xy41mVr0cUEpo+F0UM7Nq5IBSQq3NniDSzKqXA0oJ5ZobPYW9mVUtB5QSyrU00tWdTb9iZlZtHFBKqLW5kb6BIQ56+hUzq0IOKCV0dClg3/YysyrkgFJCueY5gJcCNrPq5IBSQsM9FM/nZWbVyAGlhI5Nv9JT4ZaYmU29sgYUSasl7ZDUKem6IscbJd2djm+StDjv2PUpfYekS1LaIkkPSHpC0nZJH8rL/wlJP5e0LX1+rZznVsypJzVQWyP3UMysKpVtxca0TO/NwMXAHmCLpI6IeDwv21XA/ohol7QWuBG4XNJysuWAzwVeDXxb0tlkywH/14h4WFIL8JCkb+XVeVNE/EW5zmks2fQrDZ5+xcyqUjl7KKuAzojYGRF9wHpgTUGeNcAdafse4CJl6+euAdZHRG9E7AI6gVUR8VxEPAwQEQeBJ4AzyngOJyzX7OlXzKw6lTOgnAHsztvfwyv/+B/NExEDwMvAgvGUTbfHzgc25SVfK+lRSbdLOrVYoyRdLWmrpK1dXV0nek5jam3xBJFmVp3KGVBUJK3wFfKR8oxaVlIz8FXgwxFxICV/DlgKrACeA/6yWKMi4taIWBkRK3O53OhnMAGtzQ0eQzGzqlTOgLIHWJS3vxB4dqQ8kuqAecCLo5WVVE8WTO6KiK8NZ4iI5yNiMCKGgC+Q3XKbcsMzDnv6FTOrNuUMKFuAZZKWSGogG2TvKMjTAVyZti8DNkb2l7gDWJueAlsCLAM2p/GV24AnIuLT+RVJOj1v993AYyU/o3HINTfSNzjEgR5Pv2Jm1aVsT3lFxICka4H7gVrg9ojYLukGYGtEdJAFhzsldZL1TNamstslbQAeJ3uy65qIGJR0AfB+4MeStqWf+uOIuBf4M0kryG6NPQN8oFznNpr8pYDnza2vRBPMzCqibAEFIP2hv7cg7eN52z3Ae0Youw5YV5D2PYqPrxAR759se0sh15IFlH3dvbS3NVe4NWZmU8dvypfYcA/FT3qZWbVxQCmx4R6K30Uxs2rjgFJip8ytT9OvOKCYWXVxQCmxmhqxoKmBfQf9LoqZVRcHlDIYXgrYzKyaOKCUQWuzp18xs+rjgFIGrc2NXrXRzKqOA0oZ5Foa2dfd5+lXzKyqOKCUQWtzQzb9yhFPv2Jm1cMBpQyOvovS7aWAzax6jCugSLpzPGmWyR2dz8uPDptZ9RhvD+Xc/J20vO+/KX1zZofWFk+/YmbVZ9SAIul6SQeBX5B0IH0OAnuBb0xJC2egXLOnXzGz6jNqQImI/xURLcCfR8TJ6dMSEQsi4vopauOMM29uPXWefsXMqsx4b3l9U1ITgKT3Sfq0pNeUsV0zWk2NyLU08vwBBxQzqx7jDSifAw5LegPwR8BPgb8dq5Ck1ZJ2SOqUdF2R442S7k7HN0lanHfs+pS+Q9IlKW2RpAckPSFpu6QP5eWfL+lbkp5K36eO89zKYvGCJnbu665kE8zMptR4A8pAWpp3DfDZiPgs0DJagTRwfzPwTmA5cIWk5QXZrgL2R0Q7cBNwYyq7nGz1xnOB1cAtqb4B4L9GxOuAXwSuyavzOuCfImIZ8E9pv2KWtjXx9N5uv9xoZlVjvAHloKTryZbf/Yf0x32s9W1XAZ0RsTMi+oD1ZAEp3xrgjrR9D3BRWjd+DbA+InojYhfQCayKiOci4mGAiDgIPAGcUaSuO4BLx3luZdGea+ZAz4AniTSzqjHegHI50Av8XkT8K9kf8T8fo8wZwO68/T0c++P/ijwRMQC8DCwYT9l0e+x8YFNKOi0inkt1PQe0FWuUpKslbZW0taura4xTmLilafnfp/ceKttvmJlNJ+MKKCmI3AXMk/TrQE9EjDWGUmzt98L7PyPlGbWspGbgq8CHI+LAGO04vpKIWyNiZUSszOVyJ1L0hAyvJ9/Z5XEUM6sO431T/r3AZuA9wHuBTZIuG6PYHmBR3v5C4NmR8kiqA+YBL45WVlI9WTC5KyK+lpfneUmnpzynk70rUzGvOnkOTQ21PL3XAcXMqsN4b3n9d+BNEXFlRPw22fjI/xijzBZgmaQlkhrIBtk7CvJ0AFem7cuAjWnwvwNYm54CWwIsAzan8ZXbgCci4tOj1HUlFX7xUhJL25p52j0UM6sS4w0oNRGR/1/8L4xVNo2JXAvcTzZ4viEitku6QdK7UrbbgAWSOoGPkp7MiojtwAbgceA+4JqIGATeSvZgwIWStqXPr6W6PgVcLOkp4OK0X1FLc83uoZhZ1agbZ777JN0PfCXtXw7cO1ahiLi3MF9EfDxvu4fsNlqxsuuAdQVp36P4+AoR8QJw0Vhtmkrtbc18/ZGfc6h3gKbG8V5qM7OZadS/cpLayZ6e+kNJvwFcQPYH/Qdkg/Q2iqW5JgB2dh3i9QvnVbg1ZmblNdYtr88ABwEi4msR8dGI+AhZr+Mz5W7cTHfsSa+DFW6JmVn5jRVQFkfEo4WJEbEVWFyWFs0iZ85vorZGfhfFzKrCWAFlzijH5payIbNRQ10Nr1lwEp0emDezKjBWQNki6Q8KEyVdBTxUnibNLktzfnTYzKrDWI8efRj4uqTf4lgAWQk0AO8uZ8Nmi/a2Zh7csZeBwSHqasf7lLaZ2cwzakCJiOeBX5L0DuC8lPwPEbGx7C2bJZbmmukfDH724mHOyjVXujlmZmUzrpcjIuIB4IEyt2VWGn50+OmuQw4oZjar+R5MmQ3POuyBeTOb7RxQyuzkOfW0tTR6YN7MZj0HlCnQ3tbsHoqZzXoOKFNg+NFhLwdsZrOZA8oUaG9r5mDPAF0HvRywmc1eDihTYGnOqzea2ezngDIF2o+uL++AYmazlwPKFDjt5EaaG+t4usuTRJrZ7FXWgCJptaQdkjolXVfkeKOku9PxTZIW5x27PqXvkHRJXvrtkvZKeqygrk9I+nmRlRwrThJLc01+0svMZrWyBRRJtcDNwDuB5cAVkpYXZLsK2B8R7cBNwI2p7HKyNejPBVYDt6T6AL6U0oq5KSJWpM+YK0pOJU8SaWazXTl7KKuAzojYGRF9wHpgTUGeNcAdafse4CJJSunrI6I3InYBnak+IuK7wItlbHdZLG1r5rmXe+juHah0U8zMyqKcAeUMYHfe/p6UVjRPRAwALwMLxlm2mGslPZpui51aLIOkqyVtlbS1q6trfGdSAsNPeu10L8XMZqlyBhQVSSt8s2+kPOMpW+hzwFJgBfAc8JfFMkXErRGxMiJW5nK5MaosnXbP6WVms1w5A8oeYFHe/kLg2ZHySKoD5pHdzhpP2eNExPMRMRgRQ8AXSLfIpovXLDiJuhp5HMXMZq1yBpQtwDJJSyQ1kA2ydxTk6QCuTNuXARsjm5+kA1ibngJbAiwDNo/2Y5JOz9t9N/DYSHkrob62hjO9HLCZzWLjWg9lIiJiQNK1wP1ALXB7RGyXdAOwNSI6gNuAOyV1kvVM1qay2yVtAB4HBoBrImIQQNJXgLcDrZL2AH8SEbcBfyZpBdmtsWeAD5Tr3CaqPdfsd1HMbNYqW0ABSI/u3luQ9vG87R7gPSOUXQesK5J+xQj53z+pxk6BpW3NbHxyL/2DQ9R7OWAzm2X8V20KteeaGRjKlgM2M5ttHFCm0FLP6WVms5gDyhQaXl/esw6b2WzkgDKFWubUc9rJjTy91wPzZjb7OKBMsfa2ZvdQzGxWckCZYktzzezc6+WAzWz2cUCZYu1tzRzsHWCvlwM2s1nGAWWKDU8S6Se9zGy2cUCZYkcnifQ4ipnNMg4oU6ytJS0H7B6Kmc0yDihTTBJL/aSXmc1CDigVsDTX5HdRzGzWcUCpgPa2Zv71QA8He/or3RQzs5JxQKmAY8sBu5diZrOHA0oFDAcUL7ZlZrOJA0oFeDlgM5uNyhpQJK2WtENSp6TrihxvlHR3Or5J0uK8Y9en9B2SLslLv13SXkmPFdQ1X9K3JD2Vvk8t57lNRn1tDa/xcsBmNsuULaBIqgVuBt4JLAeukLS8INtVwP6IaAduAm5MZZeTLQd8LrAauCXVB/CllFboOuCfImIZ8E9pf9pqb2t2D8XMZpVy9lBWAZ0RsTMi+oD1wJqCPGuAO9L2PcBFkpTS10dEb0TsAjpTfUTEd8nWny+UX9cdwKWlPJlSW5pr5qcvHKZ/cKjSTTEzK4lyBpQzgN15+3tSWtE8ETEAvAwsGGfZQqdFxHOprueAtmKZJF0taaukrV1dXeM8ldJrb8uWA/7pC14O2Mxmh3IGFBVJK5yzfaQ84yk7IRFxa0SsjIiVuVyuFFVOyGtf1QLAPQ/tqVgbzMxKqZwBZQ+wKG9/IfDsSHkk1QHzyG5njadsoeclnZ7qOh3YO+GWT4Hlp5/M2jct4vPfeZqvP+KgYmYzXzkDyhZgmaQlkhrIBtk7CvJ0AFem7cuAjZGtPNUBrE1PgS0BlgGbx/i9/LquBL5RgnMoG0ncsOY83nLWAj52z4/Z+kyxYSEzs5mjbAEljYlcC9wPPAFsiIjtkm6Q9K6U7TZggaRO4KOkJ7MiYjuwAXgcuA+4JiIGASR9BfgB8FpJeyRdler6FHCxpKeAi9P+tNZQV8Pn3vdGzjh1Lh+48yF2v+jxFDObuVTNS9GuXLkytm7dWulmsLOrm3ff8i+0tTTy1f/0S5w8p77STTIzG5GkhyJiZWG635SfBs7KNfO533oju/Yd4j9/+REG/Cixmc1ADijTxC+1t/LJS8/jOz/p4pP/8ESlm2NmdsLqKt0AO2btqjN5uqubL/zzLpbmmnj/WxZXuklmZuPmgDLNXPfO17Gz6xCf+PvHec2CJn757Mq9K2NmdiJ8y2uaqa0Rn73ifJa1NXPNXQ/z1PMHK90kM7Nx8VNe0+Apr2J+/tIR1vz19zncN0BrcyONdTXMqa9lTn0NjXXHvhvrazirtYm3v7aNc199MtlUaGZm5TPSU14OKNM0oAA88dwB7vzhTzncO0DvwBA9/YP09A/RO3D8989fOgLAaSc38o7XtvGOc9q4oL2Vpkbf0TSz0nNAKWK6B5Tx6jrYy4M79vLAjr3880/2cbB3gIbaGt581nze8do2LjynjcWtTZVuppnNEg4oRcyWgJKvf3CILc+8yANP7mXjk3t5Oq1bf+E5bVz3znM4+7SWCrfQzGY6B5QiZmNAKfSzFw7z948+y+e/8zSHegd478pFfOTisznt5DmVbpqZzVAOKEVUQ0AZtv9QH3/9QCd3/uCn1NTA719wFh/4t2fR4mlezOwEOaAUUU0BZdjuFw/zF/+4g29se5b5TQ186KJlXLHqTBrq/AS5mY2P5/IyABbNP4nPrj2fv7/2Al57Wgt/0rGdX73pO9z32HOVbpqZzXAOKFXq9Qvn8eU/eDNf/N030VhXywf/7mH+8h93UM09VjObHL+oUMUk8Y7XtvG29lb++9cf439v7OTlI/184t+fS02NX5A0sxNT1h6KpNWSdkjqlHRdkeONku5OxzdJWpx37PqUvkPSJWPVKelLknZJ2pY+K8p5brNJXW0Nn/oPr+cDv3wWf/uDn/KRDdvo9xT6ZnaCytZDkVQL3Ey2euIeYIukjoh4PC/bVcD+iGiXtBa4Ebhc0nKyJYPPBV4NfFvS2anMaHX+YUTcU65zms0kcf2vvY55J9XzZ/ft4GDPADf/5huZ21Bb6aaZ2QxRzh7KKqAzInZGRB+wHlhTkGcNcEfavge4SNlkVGuA9RHRGxG7gM5U33jqtEn4T29vZ927z+OBHXu58vbNHOjpr3STzGyGKGdAOQPYnbe/J6UVzZPWoH8ZWDBK2bHqXCfpUUk3SWos1ihJV0vaKmlrV1fXiZ9VFfitN7+Gv1p7Po/s3s/a//ND9nX3VrpJZjYDlDOgFBvVLXyEaKQ8J5oOcD1wDvAmYD7wsWKNiohbI2JlRKzM5bzWyEj+/RtezRd+eyU793Xz3s//gD37D1e6SWY2zZUzoOwBFuXtLwSeHSmPpDpgHvDiKGVHrDMinotML/BFsttjNglvf20bf3fVm9nX3ct7Pv8DOvd6bRYzG1k5A8oWYJmkJZIayAbZOwrydABXpu3LgI2RvQjRAaxNT4EtAZYBm0erU9Lp6VvApcBjZTy3qrFy8Xzu/sBb6B8MrvjCJp5NU+WbmRUqW0BJYyLXAvcDTwAbImK7pBskvStluw1YIKkT+ChwXSq7HdgAPA7cB1wTEYMj1ZnqukvSj4EfA63AJ8t1btXmdaefzJf/4M309A3ye1/awkEP1JtZEZ7Lq8rm8pqMf36qi9/54hYuaG/ltitXUlfriRbMqpHn8rJJe9uyHJ+89Dy+85MuPvH32z1Ni5kdx1Ov2Am5YtWZPPPCIf7Pd3ayeEETv/+2syrdJDObJhxQ7IR97JJz+NkLh1l37xOcOf8kfvXcV1W6SWY2DfiWl52wmhrx6feu4BcWnsKH1m/jx3ternSTzGwacECxCZnbUMvf/PZK5jc18Ht3bOHnfpzYrOo5oNiE5Voa+eLvvomevkGu8uPEZlXPAcUm5ezTWrjlfW/kqb3dXPvlRxjwtPdmVcsBxSYt/3Hif/dX32PD1t309A9WullmNsUcUKwkrlh1Jn91xflI8Ef3PMoFN27kM9/+iWcqNqsiflPeb8qXVETwL0+/wG3f28XGJ/fSUFfDu1ecwVVvW8LZp7VUunlmVgIjvSnv91CspCTx1vZW3treSufebr74/V189eE93L11N29b1srvvnUxv3jWAk5q8D89s9nGPRT3UMpu/6E+vrz5Z9zxL8+w92AvtTXi7NNaWLFoHisWncKKRafS3tZMbU2x5W7MbLoZqYfigOKAMmX6Bob4fuc+HvnZfh7Z/RI/2v0SB3oGAGhqqOX1C+exYtGpnPOqFlqbG2ltaWBBUyOnnlTviSjNphHf8rKKa6ir4R3ntPGOc9qAbLxl175DbNv9EttSgLntezvpHzz+P3IkOPWkBlqbswCzoLmBpoY6GupqaKirob42+26sq6Ghtob6WtFQV0ttTXYLrkaiRlBbo7QPtRISFC4CqoJO0nj6TCosNJ4yRet5ZRvEcTvHlZWUt30sv5Ty6Pj94XMf/q5J10bD2zXH0mprRK1EbW32XVMDdTU1R9PqakRDbQ017lVaHgcUqxhJnJVr5qxcM7/xxoUA9PQPsmf/EV7o7uWFQ3280N3Lvu4+9nX38kJ3Hy8c6mX7swc43DdA38AQ/YNB38AQfX7/pSJqa44Fl/q6LJjX1WTBvbG+ljn1NcypS9/1telTQ2NdLSc11DK3vpa5DelTn7ef9z2nYNu3RqcvBxSbVubU19Le1kx7W/MJlYsI+gbzAszAEEMR2WeIY9vBcenH1UEU1DnZsxmprSP/bv6x/GzDt6bjuHxxXJlI2xFxbJsg/Y9I5x6k77xrE+k3BtP+4FDeJ4KBoWAo7Q8MZdd5YDDoHxyifzAL6MP7fYPZ9e/pH6J3YJCe/kH2dQ/Q0z9Iz8AgPf1D2Xb/4Ct6o+PRUFeTgksNTQ11tMyt5+Q5dcybW8/Jc+uz7zn1nDw3S5t/UgO5lkZyLY3Mm1s/oR6ljU9ZA4qk1cBngVrgbyLiUwXHG4G/Bf4N8AJweUQ8k45dD1wFDAL/JSLuH63OtFTwemA+8DDw/ojoK+f52fQhica6WhrrgMZKt8bGq38wCy5H+gY50j/I4fR9pC/b7unP9ocD0JG+oeP2u3sHONAzwMNBE4QAAAlJSURBVIEj/fx8/xEO9PTz8pH+EQNVfa3INTceDTC5lkZamxtpaqxjTl0Ncxtqj/ak5uZ9z22oSUGqnjn1tVN8lWaOsgUUSbXAzcDFwB5gi6SOiHg8L9tVwP6IaJe0FrgRuFzScrL14s8FXg18W9LZqcxIdd4I3BQR6yV9PtX9uXKdn5lNXn1tNgbWMqe+ZHVGBD39Q7x8JAsuLx7qo6u7l66D2Wdf2n72pR5+tOdlXujuZegEOkoNdTWpF3SsV3TynHpOOame+U0Nxz4nNXBqUwMLmho45aQGGupm/4Ml5eyhrAI6I2IngKT1wBqydeKHrQE+kbbvAf5aWX90DbA+InqBXWnN+VUp3yvqlPQEcCHwmynPHaleBxSzKiPp6LjMq+bNGTP/0FDQO5D1fI4c7QkN94KGUs/pWE/owJF+DvT0c+DIwNGAtWvfIV46nAWwkbQ01tFYXzPyQxE6/kGLYydU5BzHcQ3G8j/f/XpWLZk/Zr4TUc6AcgawO29/D/DmkfJExICkl4EFKf2HBWXPSNvF6lwAvBQRA0XyH0fS1cDVAGeeeeaJnZGZzTo1NccC0GQNDA6x/3A/+w/38UJ3H/sP9/HioWOf/sEhhtI4V/6YXhwd2zu+vmKvdYzZmRpnb6upsfS37soZUIqFyMJTHSnPSOnF+oyj5X9lYsStwK2QvYdSLI+Z2UTU1dYcHZvhtEq3ZuqV86beHmBR3v5C4NmR8kiqA+YBL45SdqT0fcApqY6RfsvMzMqonAFlC7BM0hJJDWSD7B0FeTqAK9P2ZcDGyPp4HcBaSY3p6a1lwOaR6kxlHkh1kOr8RhnPzczMCpTtllcaE7kWuJ/sEd/bI2K7pBuArRHRAdwG3JkG3V8kCxCkfBvIBvAHgGsiYhCgWJ3pJz8GrJf0SeCRVLeZmU0Rz+XlubzMzE7ISHN5zf4Ho83MbEo4oJiZWUk4oJiZWUk4oJiZWUlU9aC8pC7gpxMs3kr2/stMMZPaO5PaCjOrvTOprTCz2juT2gqTa+9rIiJXmFjVAWUyJG0t9pTDdDWT2juT2gozq70zqa0ws9o7k9oK5Wmvb3mZmVlJOKCYmVlJOKBM3K2VbsAJmkntnUlthZnV3pnUVphZ7Z1JbYUytNdjKGZmVhLuoZiZWUk4oJiZWUk4oEyApNWSdkjqlHRdpdszGknPSPqxpG2Spt1MmJJul7RX0mN5afMlfUvSU+n71Eq2cdgIbf2EpJ+n67tN0q9Vso35JC2S9ICkJyRtl/ShlD7tru8obZ2W11fSHEmbJf0otfdPU/oSSZvStb07LbMxXdv6JUm78q7tikn/lsdQToykWuAnwMVkC35tAa6IiMcr2rARSHoGWBkR0/KFK0m/DHQDfxsR56W0PwNejIhPpYB9akR8rJLtTO0q1tZPAN0R8ReVbFsxkk4HTo+IhyW1AA8BlwK/wzS7vqO09b1Mw+urbNH2pojollQPfA/4EPBR4GsRsV7S54EfRcTnpmlbPwh8MyLuKdVvuYdy4lYBnRGxMyL6gPXAmgq3acaKiO+SrYWTbw1wR9q+g+wPS8WN0NZpKyKei4iH0/ZB4AngDKbh9R2lrdNSZLrTbn36BHAhMPwHerpc25HaWnIOKCfuDGB33v4epvE/fLJ/OP8o6SFJV1e6MeN0WkQ8B9kfGqCtwu0Zy7WSHk23xCp++6gYSYuB84FNTPPrW9BWmKbXV1KtpG3AXuBbwNPASxExkLJMm78NhW2NiOFruy5d25skNU72dxxQTpyKpE3n+4ZvjYg3Au8Erkm3bax0PgcsBVYAzwF/WdnmvJKkZuCrwIcj4kCl2zOaIm2dttc3IgYjYgWwkOzOxeuKZZvaVhVX2FZJ5wHXA+cAbwLmk616OykOKCduD7Aob38h8GyF2jKmiHg2fe8Fvk72D3+6ez7dUx++t763wu0ZUUQ8n/7POgR8gWl2fdM9868Cd0XE11LytLy+xdo63a8vQES8BDwI/CJwiqThpdWn3d+GvLauTrcZIyJ6gS9SgmvrgHLitgDL0tMcDcBaoKPCbSpKUlMa4ERSE/CrwGOjl5oWOoAr0/aVwDcq2JZRDf9hTt7NNLq+aTD2NuCJiPh03qFpd31Haut0vb6ScpJOSdtzgV8hG/d5ALgsZZsu17ZYW5/M+48KkY31TPra+imvCUiPLn4GqAVuj4h1FW5SUZLOIuuVANQBX55ubZX0FeDtZFNpPw/8CfB/gQ3AmcDPgPdERMUHw0do69vJbscE8AzwgeHxiUqTdAHwz8CPgaGU/MdkYxPT6vqO0tYrmIbXV9IvkA2615L9h/mGiLgh/X9uPdktpEeA96UeQMWM0taNQI7sNv424IN5g/cT+y0HFDMzKwXf8jIzs5JwQDEzs5JwQDEzs5JwQDEzs5JwQDEzs5JwQDGbBEnd6XuxpN8scd1/XLD/L6Ws36zUHFDMSmMxcEIBJc1cPZrjAkpE/NIJtslsSjmgmJXGp4C3pXUlPpIm4/tzSVvS5HsfAJD09rTux5fJXuJD0v9Nk3duH57AU9KngLmpvrtS2nBvSKnux5StdXN5Xt0PSrpH0pOS7kpvQSPpU5IeT22ZVlPB2+xRN3YWMxuH64D/FhG/DpACw8sR8aY0i+v3Jf1jyrsKOC8idqX934uIF9O0GFskfTUirpN0bZrQr9BvkL09/gayt/a3SPpuOnY+cC7ZHFLfB94q6XGyaUvOiYgYnobDrNTcQzErj18FfjtNGb4JWAAsS8c25wUTgP8i6UfAD8kmHl3G6C4AvpImTXwe+A7ZjLHDde9JkyluI7sVdwDoAf5G0m8Ahyd9dmZFOKCYlYeA/xwRK9JnSUQM91AOHc0kvZ1ssr63RMQbyOZ/mjOOukeSP2/UIFCX1udYRTaT76XAfSd0Jmbj5IBiVhoHgZa8/fuB/5imZEfS2WnG50LzgP0RcVjSOWRToA/rHy5f4LvA5WmcJgf8MrB5pIalNUbmRcS9wIfJbpeZlZzHUMxK41FgIN26+hLwWbLbTQ+ngfEuii8Hex/wQUmPAjvIbnsNuxV4VNLDEfFbeelfB94C/IhsFt4/ioh/TQGpmBbgG5LmkPVuPjKxUzQbnWcbNjOzkvAtLzMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzK4n/D7ZL64LpHqfzAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "tags": [], - "needs_background": "light" - } - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4kZ2vUYYgclS", - "colab_type": "text" - }, - "source": [ - "## Challenge\n", - "\n", - "This is a reference implementation for you to explore. You will not be expected to apply it to today's module project. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "46cP9Pm_gclS", - "colab_type": "text" - }, - "source": [ - "# Keras Sequential API (Learn)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": true, - "id": "Bna67ADZgclT", - "colab_type": "text" - }, - "source": [ - "## Overview\n", - "\n", - "> \"Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research. Use Keras if you need a deep learning library that:\n", - "\n", - "> Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).\n", - "Supports both convolutional networks and recurrent networks, as well as combinations of the two.\n", - "Runs seamlessly on CPU and GPU.\" " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "AIJoRBxHy27n" - }, - "source": [ - "### Keras Perceptron Sample" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "ZGQHzp4rgclU", - "colab_type": "code", - "colab": {} - }, - "source": [ - "import pandas as pd\n", - "\n", - "data = { 'x1': [0,1,0,1],\n", - " 'x2': [0,0,1,1],\n", - " 'y': [1,1,1,0]\n", - " }\n", - "\n", - "df = pd.DataFrame.from_dict(data).astype('int')\n", - "X = df[['x1', 'x2']].values\n", - "y = df['y'].values" - ], - "execution_count": 34, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "TQxyONqKvFxB", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "outputId": "f5009153-ab32-4b49-f301-4cbd4051ee21" - }, - "source": [ - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense\n", - "\n", - "# This is our perceptron from Monday's by-hand: \n", - "model = Sequential()\n", - "model.add(Dense(1,input_dim=2, activation='sigmoid'))\n", - "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n", - "model.fit(X,y, epochs=5)" - ], - "execution_count": 35, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Epoch 1/5\n", - "1/1 [==============================] - 0s 2ms/step - loss: 0.9824 - accuracy: 0.5000\n", - "Epoch 2/5\n", - "1/1 [==============================] - 0s 1ms/step - loss: 0.9820 - accuracy: 0.5000\n", - "Epoch 3/5\n", - "1/1 [==============================] - 0s 1ms/step - loss: 0.9816 - accuracy: 0.5000\n", - "Epoch 4/5\n", - "1/1 [==============================] - 0s 1ms/step - loss: 0.9813 - accuracy: 0.5000\n", - "Epoch 5/5\n", - "1/1 [==============================] - 0s 2ms/step - loss: 0.9809 - accuracy: 0.5000\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 35 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "Z1wfKUxszPKa", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "outputId": "e458eb38-835a-42f7-f226-b81bd4824741" - }, - "source": [ - "# evaluate the model\n", - "scores = model.evaluate(X, y)\n", - "print(f\"{model.metrics_names[1]}: {scores[1]*100}\")" - ], - "execution_count": 36, - "outputs": [ - { - "output_type": "stream", - "text": [ - "1/1 [==============================] - 0s 1ms/step - loss: 0.9805 - accuracy: 0.5000\n", - "accuracy: 50.0\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "toc-hr-collapsed": true, - "id": "gsVYOn7bgcle", - "colab_type": "text" - }, - "source": [ - "## Follow Along\n", - "\n", - "In the `Sequential` api model, you specify a model architecture by 'sequentially specifying layers. This type of specification works well for feed forward neural networks in which the data flows in one direction (forward propagation) and the error flows in the opposite direction (backwards propagation). The Keras `Sequential` API follows a standardarized worklow to estimate a 'net: \n", - "\n", - "1. Load Data\n", - "2. Define Model\n", - "3. Compile Model\n", - "4. Fit Model\n", - "5. Evaluate Model\n", - "\n", - "You saw these steps in our Keras Perceptron Sample, but let's walk thru each step in detail." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "Md5D67XwqVAf", - "toc-hr-collapsed": false - }, - "source": [ - "### Load Data" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "bn09phMBpY1J", - "colab": {} - }, - "source": [ - "from tensorflow import keras\n", - "from tensorflow.keras.datasets import mnist\n", - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense, Dropout\n", - "\n", - "# Stretch - use dropout \n", - "import numpy as np" - ], - "execution_count": 37, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "P6kfiLLLgclh", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "outputId": "6b66d7ee-50e2-4865-d1e3-518ac5277cf7" - }, - "source": [ - "# Load the Data\n", - "(X_train, y_train), (X_test, y_test) = mnist.load_data()" - ], - "execution_count": 38, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n", - "11493376/11490434 [==============================] - 0s 0us/step\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Mt2QnmTVgclj", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "outputId": "de7f522a-29b4-478e-a09e-29d92376ec08" - }, - "source": [ - "X_train[0].shape" - ], - "execution_count": 39, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "(28, 28)" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 39 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "1Zm3Q8ezgclm", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "outputId": "a9cfed2b-d0aa-4a17-c3ee-2a9efee627b2" - }, - "source": [ - "X_train.shape" - ], - "execution_count": 40, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "(60000, 28, 28)" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 40 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "U1ekYH_3gcln", - "colab_type": "code", - "colab": {} - }, - "source": [ - "# X Variable Types\n", - "X_train = X_train.astype('float32') / 255.\n", - "X_test = X_test.astype('float32') /255.\n", - "\n", - "# Correct Encoding on Y\n", - "# What softmax expects = [0,0,0,0,0,1,0,0,0,0]\n", - "num_classes = 10\n", - "y_train = keras.utils.to_categorical(y_train, num_classes)\n", - "y_test = keras.utils.to_categorical(y_test, num_classes)" - ], - "execution_count": 41, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "o0xMqOyTs5xt" - }, - "source": [ - "### Define Model" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "Bp9USczrfu6M", - "colab": {} - }, - "source": [ - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense, Flatten\n", - "\n", - "import numpy as np\n", - "\n", - "np.random.seed(812)" - ], - "execution_count": 42, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "wAzHLg27thoN" - }, - "source": [ - "I'll instantiate my model as a \"sequential\" model. This just means that I'm going to tell Keras what my model's architecture should be one layer at a time." - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "DSNsL49Xp6KI", - "colab": {} - }, - "source": [ - "# https://keras.io/getting-started/sequential-model-guide/\n", - "model = Sequential()" - ], - "execution_count": 47, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "ZCYX6QzJtvpG" - }, - "source": [ - "Adding a \"Dense\" layer to our model is how we add \"vanilla\" perceptron-based layers to our neural network. These are also called \"fully-connected\" or \"densely-connected\" layers. They're used as a layer type in lots of other Neural Net Architectures but they're not referred to as perceptrons or multi-layer perceptrons very often in those situations even though that's what they are.\n", - "\n", - " > [\"Just your regular densely-connected NN layer.\"](https://keras.io/layers/core/)\n", - " \n", - " The first argument is how many neurons we want to have in that layer. To create a perceptron-esque model we will just set it to 10. Our architecture is just an input and output layer. We will tell it that there will be 784 inputs coming into this layer from our dataset and set it to use the sigmoid activation function." - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "GNzOLidxtvFa", - "colab": {} - }, - "source": [ - "model.add(Flatten(input_shape=(28,28)))\n", - "model.add(Dense(10,activation=\"softmax\")) #Relu is valid option. " - ], - "execution_count": 48, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "EnI3jwKMtBL2", - "toc-hr-collapsed": false - }, - "source": [ - "### Compile Model\n", - "Using binary_crossentropy as the loss function here is just telling keras that I'm doing binary classification so that it can use the appropriate loss function accordingly. If we were predicting non-binary categories we might assign something like `categorical_crossentropy`. We're also telling keras that we want it to report model accuracy as our main error metric for each epoch. We will also be able to see the overall accuracy once the model has finished training.\n", - "\n", - "#### Adam Optimizer\n", - "Check out this links for more background on the Adam optimizer and Stohastic Gradient Descent\n", - "* [Adam Optimization Algorithm](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)\n", - "* [Adam Optimizer - original paper](https://arxiv.org/abs/1412.6980)" - ] - }, - { - "cell_type": "code", - "metadata": { - "colab_type": "code", - "id": "qp6xwYaqurRO", - "colab": {} - }, - "source": [ - "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])" - ], - "execution_count": 49, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "5dW8SZ2Ls9SX", - "toc-hr-collapsed": false - }, - "source": [ - "### Fit Model\n", - "\n", - "Lets train it up! `model.fit()` has a `batch_size` parameter that we can use if we want to do mini-batch epochs, but since this tabular dataset is pretty small we're just going to delete that parameter. Keras' default `batch_size` is `None` so omiting it will tell Keras to do batch epochs." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "0g1Uh2STgcl1", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 341 - }, - "outputId": "07e6246a-0bbc-4581-92da-3570c3b8c0cc" - }, - "source": [ - "model.fit(X_train, y_train, epochs=10)" - ], - "execution_count": 50, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Epoch 1/10\n", - "1487/1875 [======================>.......] - ETA: 0s - loss: 0.4990 - accuracy: 0.8729" - ], - "name": "stdout" - }, - { - "output_type": "error", - "ename": "KeyboardInterrupt", - "evalue": "ignored", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 68\u001b[0m \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 846\u001b[0m batch_size=batch_size):\n\u001b[1;32m 847\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 848\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 849\u001b[0m \u001b[0;31m# Catch OutOfRangeError for Datasets of unknown size.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 850\u001b[0m \u001b[0;31m# This blocks until the batch has finished executing.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 578\u001b[0m \u001b[0mxla_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 579\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 580\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 581\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 582\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 609\u001b[0m \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 610\u001b[0m \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 611\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 612\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2418\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2419\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2420\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2421\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2422\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m 1663\u001b[0m if isinstance(t, (ops.Tensor,\n\u001b[1;32m 1664\u001b[0m resource_variable_ops.BaseResourceVariable))),\n\u001b[0;32m-> 1665\u001b[0;31m self.captured_inputs)\n\u001b[0m\u001b[1;32m 1666\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1667\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1744\u001b[0m \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1745\u001b[0m return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1746\u001b[0;31m ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m 1747\u001b[0m forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m 1748\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 596\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 597\u001b[0m \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 598\u001b[0;31m ctx=ctx)\n\u001b[0m\u001b[1;32m 599\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 600\u001b[0m outputs = execute.execute_with_cancellation(\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 58\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 59\u001b[0m tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 60\u001b[0;31m inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m 61\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mKeyboardInterrupt\u001b[0m: " - ] - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hJKAF0tEgcl4", - "colab_type": "text" - }, - "source": [ - "### Evaluate Model" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "StRwssm-gcl4", - "colab_type": "code", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 800 - }, - "outputId": "3c303f33-f677-4081-fc61-1395016b3c0b" - }, - "source": [ - "scores = model.evaluate(X_test,y_test)\n", - "print(\"\\n\")\n", - "print(\"Validation Data Metrics:\")\n", - "print(f\"{model.metrics_names[0]}: {scores[0]}\")\n", - "print(f\"{model.metrics_names[1]}: {scores[1]*100}\")" - ], - "execution_count": 46, - "outputs": [ - { - "output_type": "error", - "ename": "ValueError", - "evalue": "ignored", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Validation Data Metrics:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{model.metrics_names[0]}: {scores[0]}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{model.metrics_names[1]}: {scores[1]*100}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 68\u001b[0m \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict)\u001b[0m\n\u001b[1;32m 1079\u001b[0m step_num=step):\n\u001b[1;32m 1080\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_test_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1081\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtest_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1082\u001b[0m \u001b[0;31m# Catch OutOfRangeError for Datasets of unknown size.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1083\u001b[0m \u001b[0;31m# This blocks until the batch has finished executing.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 578\u001b[0m \u001b[0mxla_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 579\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 580\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 581\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 582\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 625\u001b[0m \u001b[0;31m# This is the first call of __call__, so we have to initialize.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 626\u001b[0m \u001b[0minitializers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 627\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_initialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madd_initializers_to\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitializers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 628\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 629\u001b[0m \u001b[0;31m# At this point we know that the initialization is complete (or less\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_initialize\u001b[0;34m(self, args, kwds, add_initializers_to)\u001b[0m\n\u001b[1;32m 504\u001b[0m self._concrete_stateful_fn = (\n\u001b[1;32m 505\u001b[0m self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access\n\u001b[0;32m--> 506\u001b[0;31m *args, **kwds))\n\u001b[0m\u001b[1;32m 507\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 508\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0minvalid_creator_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0munused_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0munused_kwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_get_concrete_function_internal_garbage_collected\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2444\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2445\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2446\u001b[0;31m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2447\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2448\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_maybe_define_function\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m 2775\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2776\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmissed\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcall_context_key\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2777\u001b[0;31m \u001b[0mgraph_function\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_graph_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2778\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprimary\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcache_key\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2779\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_create_graph_function\u001b[0;34m(self, args, kwargs, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m 2665\u001b[0m \u001b[0marg_names\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0marg_names\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2666\u001b[0m \u001b[0moverride_flat_arg_shapes\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moverride_flat_arg_shapes\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2667\u001b[0;31m capture_by_value=self._capture_by_value),\n\u001b[0m\u001b[1;32m 2668\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_attributes\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2669\u001b[0m \u001b[0;31m# Tell the ConcreteFunction to clean up its graph once it goes out of\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mfunc_graph_from_py_func\u001b[0;34m(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m 979\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal_func\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_decorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munwrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpython_func\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 980\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 981\u001b[0;31m \u001b[0mfunc_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpython_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mfunc_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfunc_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 982\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 983\u001b[0m \u001b[0;31m# invariant: `func_outputs` contains only Tensors, CompositeTensors,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36mwrapped_fn\u001b[0;34m(*args, **kwds)\u001b[0m\n\u001b[1;32m 439\u001b[0m \u001b[0;31m# __wrapped__ allows AutoGraph to swap in a converted function. We give\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 440\u001b[0m \u001b[0;31m# the function a weak reference to itself to avoid a reference cycle.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 441\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mweak_wrapped_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__wrapped__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 442\u001b[0m \u001b[0mweak_wrapped_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mweakref\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mref\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapped_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 443\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 966\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint:disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 967\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ag_error_metadata\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 968\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mag_error_metadata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 969\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 970\u001b[0m \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mValueError\u001b[0m: in user code:\n\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:941 test_function *\n outputs = self.distribute_strategy.run(\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:951 run **\n return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica\n return self._call_for_each_replica(fn, args, kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica\n return fn(*args, **kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:912 test_step **\n y, y_pred, sample_weight, regularization_losses=self.losses)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:205 __call__\n loss_value = loss_obj(y_t, y_p, sample_weight=sw)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:143 __call__\n losses = self.call(y_true, y_pred)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:246 call\n return self.fn(y_true, y_pred, **self._fn_kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:1527 categorical_crossentropy\n return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:4561 categorical_crossentropy\n target.shape.assert_is_compatible_with(output.shape)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1117 assert_is_compatible_with\n raise ValueError(\"Shapes %s and %s are incompatible\" % (self, other))\n\n ValueError: Shapes (None, 10) and (None, 28, 28) are incompatible\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "zHYB7k9q3O8T" - }, - "source": [ - "### Unstable Results\n", - "\n", - "You'll notice that if we rerun the results might differ from the origin run. This can be explain by a bunch of factors. Check out some of them in this article: \n", - "\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Kb2aiw_Sgcl7", - "colab_type": "text" - }, - "source": [ - "## Challenge\n", - "\n", - "You will be expected to leverage the Keras `Sequential` api to estimate a feed forward neural networks on a dataset.\n", - "\n", - "---" - ] + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bgTf6vTS69Sw" + }, + "source": [ + "### Neural Network Architecture\n", + "Lets create a Neural_Network class to contain this functionality" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "RUI8VSR5zyBv" + }, + "outputs": [], + "source": [ + "class NeuralNetwork:\n", + " \n", + " def __init__(self):\n", + " # Set up Arch\n", + " self.inputs = 2\n", + " self.hiddenNodes = 3\n", + " self.outputNodes = 1\n", + " \n", + " # Initialize Weights\n", + " # 2x3\n", + " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", + " \n", + " # 3x1\n", + " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gbyT_FJ88IlK" + }, + "source": [ + "### Randomly Initialize Weights\n", + "How many random weights do we need to initialize? \"Fully-connected Layers\"" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "IreIDe6P8H0H" + }, + "outputs": [], + "source": [ + "nn = NeuralNetwork()" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + }, + "colab_type": "code", + "id": "lHhewvuJgckG", + "outputId": "617e7c1f-c04d-42ae-ce1e-efc91a5d2880" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Layer 1 weights: \n", + " [[ 2.48783189 0.11697987 -1.97118428]\n", + " [-0.48325593 -1.50361209 0.57515126]]\n", + "Layer 2 weights: \n", + " [[-0.20672583]\n", + " [ 0.41271104]\n", + " [-0.57757999]]\n" + ] + } + ], + "source": [ + "print(\"Layer 1 weights: \\n\", nn.weights1)\n", + "print(\"Layer 2 weights: \\n\", nn.weights2)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hbxDhyjQ-RwS" + }, + "source": [ + "### Implement Feedforward Functionality\n", + "\n", + "After this step our neural network should be able to generate an output even though it has not been trained." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "0gGivpEk-VdP" + }, + "outputs": [], + "source": [ + "class NeuralNetwork:\n", + " \n", + " def __init__(self):\n", + " # Set up Arch\n", + " self.inputs = 2\n", + " self.hiddenNodes = 3\n", + " self.outputNodes = 1\n", + " \n", + " # Initialize Weights\n", + " # 2x3\n", + " # Input to Hidden (1st set of weights)\n", + " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", + " \n", + " # 3x1\n", + " # Hidden to Output (2nd set of weights)\n", + " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)\n", + " \n", + " def sigmoid(self, s):\n", + " return 1 / (1+np.exp(-s))\n", + " \n", + " def feed_forward(self, X):\n", + " \"\"\"\n", + " Calculate the NN inference using feed forward.\n", + " \"\"\"\n", + " \n", + " # Weighted Sum\n", + " self.hidden_sum = np.dot(X, self.weights1)\n", + " \n", + " # Activate\n", + " self.activated_hidden = self.sigmoid(self.hidden_sum)\n", + " \n", + " # Weighted sum of activated hidden (which output layer will use)\n", + " self.output_sum = np.dot(self.activated_hidden, self.weights2)\n", + " \n", + " # Final Activation of Output (My Predictions)\n", + " self.activated_output = self.sigmoid(self.output_sum)\n", + " \n", + " return self.activated_output" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "4X7_Z_GsgckM", + "outputId": "5046b4fd-58b8-4a30-b218-2517c0e514ef" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.66666667, 1. ],\n", + " [0.33333333, 0.55555556],\n", + " [1. , 0.66666667]])" + ] + }, + "execution_count": 7, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "a1pxdfmDAaJg" + }, + "source": [ + "### Make a Prediction" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "colab_type": "code", + "id": "GO8VaEc6gckP", + "outputId": "e0692165-a961-4e61-851a-334a278c4bf2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0.66666667 1. ]\n", + "output [0.25814933]\n" + ] + } + ], + "source": [ + "# Try to make a prediction with our updated 'net\n", + "nn = NeuralNetwork()\n", + "print(X[0])\n", + "output = nn.feed_forward(X[0])\n", + "print(\"output\", output)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "3V61yNmAB2T5" + }, + "source": [ + "### Calculate Error" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "colab_type": "code", + "id": "KeLKL9GDgckS", + "outputId": "e49ac3b5-ecc2-44de-816d-d276ca80075f" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([0.64185067])" + ] + }, + "execution_count": 9, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" } - ] + ], + "source": [ + "error = y[0] - output\n", + "error" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + }, + "colab_type": "code", + "id": "tmRRaKgJgckV", + "outputId": "a9554aed-7985-4582-983a-8f43ce5832b2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.25814933]\n", + " [0.33067192]\n", + " [0.22642076]]\n", + "[[0.64185067]\n", + " [0.38932808]\n", + " [0.57357924]]\n" + ] + } + ], + "source": [ + "output_all = nn.feed_forward(X)\n", + "error_all = y - output_all\n", + "print(output_all)\n", + "print(error_all)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "26wgCLU0TLvy" + }, + "source": [ + "Why is my error so big?\n", + "\n", + "My error is so big because my prediction is low.\n", + "\n", + "Why are my prediction low?\n", + "\n", + "Because either:\n", + "\n", + " 1) Second layer **weights** are low\n", + " \n", + " (or)\n", + " \n", + " 2) Activations coming from the first layer are low\n", + " \n", + "How are activations from the first layer determined? \n", + "\n", + " 1) By inputs - fixed\n", + " \n", + " 2) by **weights** - variable\n", + " \n", + "The only thing that I have control over throughout this process in order to increase the value of my final predictions is to either increase weights in layer 2 or increase weights in layer 1. \n", + "\n", + "Imagine that you could only change your weights by a fixed amount. Say you have .3 and you have to split that up and disperse it over your weights so as to increase your predictions as much as possible. (This isn't actually what happens, but it will help us identify which weights we would benefit the most from moving.)\n", + "\n", + "I need to increase weights of my model somewhere, I'll get the biggest bang for my buck if I increase weights in places where I'm already seeing high activation values -because they end up getting multiplied together before being passed to the sigmoid function. \n", + "\n", + "> \"Neurons that fire together, wire together\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "j_eyzItYIxgm" + }, + "source": [ + "### Implement Backpropagation \n", + "\n", + "> *Assigning blame for bad predictions and delivering justice - repeatedly and a little bit at a time*\n", + "\n", + "What in our model could be causing our predictions to suck so bad? \n", + "\n", + "Well, we know that our inputs (X) and outputs (y) are correct, if they weren't then we would have bigger problems than understanding backpropagation.\n", + "\n", + "We also know that our activation function (sigmoid) is working correctly. It can't be blamed because it just does whatever we tell it to and transforms the data in a known way.\n", + "\n", + "So what are the potential culprits for these terrible predictions? The **weights** of our model. Here's the problem though. I have weights that exist in both layers of my model. How do I know if the weights in the first layer are to blame, or the second layer, or both? \n", + "\n", + "Lets investigate. And see if we can just eyeball what should be updated." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 442 + }, + "colab_type": "code", + "id": "HGaWbXnZgckY", + "outputId": "357a13a7-bc19-4e34-c042-dc13444d0536" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "weights1\n", + " [[-1.75351135 1.23279898 0.24464757]\n", + " [-0.06568225 0.30190098 0.79723428]] \n", + "---------\n", + "hidden_sum\n", + " [[-1.23468981 1.12376697 0.96033266]\n", + " [-0.62099392 0.57865576 0.52445712]\n", + " [-1.79729952 1.4340663 0.77613709]] \n", + "---------\n", + "activated_hidden\n", + " [[0.22536165 0.75468678 0.7231884 ]\n", + " [0.34955543 0.64075804 0.6281894 ]\n", + " [0.14218011 0.8075341 0.68484697]] \n", + "---------\n", + "weights2\n", + " [[ 1.23073545]\n", + " [-1.52187331]\n", + " [-0.25502715]] \n", + "---------\n", + "activated_output\n", + " [[0.25814933]\n", + " [0.33067192]\n", + " [0.22642076]] \n", + "---------\n" + ] + }, + { + "data": { + "text/plain": [ + "[None, None, None, None, None]" + ] + }, + "execution_count": 11, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "attributes = ['weights1', 'hidden_sum', 'activated_hidden', 'weights2', 'activated_output']\n", + "[print(i+'\\n', getattr(nn,i), '\\n'+'---'*3) for i in attributes if i[:2]!= '__'] " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "16Ujj6vNYQyX", + "toc-hr-collapsed": true + }, + "source": [ + "### Update Weights Based on Gradient\n", + "\n", + "Repeat steps 2-4 for every observation in a given batch, and then given the network's cost function, calculate its gradient using calculus and update weights associated with the (negative) gradient of the cost function. \n", + "\n", + "Remember that we have 9 weights in our network therefore the gradient that comes from our gradient descent calculation will be the vector that takes us in the most downward direction along some function in 9-dimensional hyperspace.\n", + "\n", + "\\begin{align}\n", + "C(w1, w2, w3, w4, w5, w6, w7, w8, w9)\n", + "\\end{align}\n", + "\n", + "You should also know that with neural networks it is common to have gradients that are not convex (like what we saw when we applied gradient descent to linear regression). Due to the high complexity of these models and their nonlinearity, it is common for gradient descent to get stuck in a local minimum, but there are ways to combat this:\n", + "\n", + "1) Stochastic Gradient Descent\n", + "\n", + "2) More advanced Gradient-Descent-based \"Optimizers\" - See Stretch Goals on assignment." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "h8mcKTvlgckb" + }, + "outputs": [], + "source": [ + "# I want activations that correspond to negative weights to be lower\n", + "# and activations that correspond to positive weights to be higher\n", + "\n", + "class NeuralNetwork:\n", + " \n", + " def __init__(self):\n", + " # Set up Arch\n", + " self.inputs = 2\n", + " self.hiddenNodes = 3\n", + " self.outputNodes = 1\n", + " \n", + " # Initialize Weights\n", + " # 2x3\n", + " # Input to Hidden (1st set of weights)\n", + " self.weights1 = np.random.randn(self.inputs,self.hiddenNodes)\n", + " \n", + " # 3x1\n", + " # Hidden to Output (2nd set of weights)\n", + " self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes)\n", + " \n", + " def sigmoid(self, s):\n", + " return 1 / (1+np.exp(-s))\n", + " \n", + " def sigmoidPrime(self, s):\n", + " sx = self.sigmoid(s)\n", + " return sx * (1-sx)\n", + " \n", + " def feed_forward(self, X):\n", + " \"\"\"\n", + " Calculate the NN inference using feed forward.\n", + " \"\"\"\n", + " \n", + " # Weighted Sum\n", + " self.hidden_sum = np.dot(X, self.weights1)\n", + " \n", + " # Activate\n", + " self.activated_hidden = self.sigmoid(self.hidden_sum)\n", + " \n", + " # Weighted sum of activated hidden (which output layer will use)\n", + " self.output_sum = np.dot(self.activated_hidden, self.weights2)\n", + " \n", + " # Final Activation of Output (My Predictions)\n", + " self.activated_output = self.sigmoid(self.output_sum)\n", + " \n", + " return self.activated_output\n", + " \n", + " def backward(self, X, y, o):\n", + " \"\"\"\n", + " Back prop thru the network\n", + " \"\"\"\n", + " \n", + " self.o_error = y - o # Error in the output\n", + " \n", + " # Apply derivative of sigmoid to error\n", + " self.o_delta = self.o_error * self.sigmoidPrime(o)\n", + " \n", + " # z2 error: how much were our output layer weights off\n", + " self.z2_error = self.o_delta.dot(self.weights2.T)\n", + " \n", + " # z2 delta: how much were the weights off?\n", + " self.z2_delta = self.z2_error*self.sigmoidPrime(self.output_sum)\n", + "\n", + " self.weights1 += X.T.dot(self.z2_delta) #Adjust first set (input => hidden) weights\n", + " self.weights2 += self.activated_hidden.T.dot(self.o_delta) #adjust second set (hidden => output) weights\n", + " \n", + " def train(self, X,y):\n", + " o = self.feed_forward(X)\n", + " self.backward(X,y,o)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Qw7m9KPLgckd", + "toc-hr-collapsed": true + }, + "source": [ + "#### Let's look at the shape of the Gradient Componets\n" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "fWhDTvAIgcke" + }, + "outputs": [], + "source": [ + "nn = NeuralNetwork()\n", + "\n", + "nn.train(X,y)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "59kruYhegckh" + }, + "source": [ + "##### Our Error Associated with Each Observation \n", + "aka how wrong were we?" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "vNuJJsLWgckh", + "outputId": "2af4033a-2949-4892-df1e-7c615b53345f" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.60746115],\n", + " [0.46613403],\n", + " [0.53667427]])" + ] + }, + "execution_count": 14, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "nn.o_error" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vO2flHK5gckk" + }, + "source": [ + "##### 1st Gradient \n", + "Simple interpretation - how much more sigmoid activation would have pushed us towards the right answer?\n", + "\n", + "`self.o_delta = self.o_error * self.sigmoidPrime(self.output_sum)`" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "Xj8eCxYEgckk", + "outputId": "1df98eb6-4566-4ac1-e68a-ac831baa5746" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.14866196],\n", + " [0.11467591],\n", + " [0.13186936]])" + ] + }, + "execution_count": 15, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "nn.o_delta" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "ckGNArjjgckm" + }, + "source": [ + "Let's take a look at the derivate of the sigmoid function to understand what's happening. " + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "u5QlrPOngckm" + }, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "line_x = np.arange(-5, 5, 0.01)\n", + "\n", + "def sigmoid(x):\n", + " return 1 / (1+ np.exp(-x))\n", + "\n", + "def sigmoid_derivative(x):\n", + " sx = sigmoid(x)\n", + " return sx * (1-sx)\n", + "\n", + "# sigmoid\n", + "line_y = sigmoid(line_x)\n", + "y_d = sigmoid_derivative(line_x)\n", + "\n", + "x = nn.output_sum\n", + "s = sigmoid(x)\n", + "sx = sigmoid_derivative(x)" + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 331 + }, + "colab_type": "code", + "id": "BtEe1wCQgcko", + "outputId": "a69f18db-8495-4a16-c9f4-cb776bfb16c9" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAE6CAYAAAB585FmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUZf7+8XvSe4OE3iGUQOig9JKQUGxrw4a79rULVnRFBBR1dRVXLGt3XcVVVlZlKYqAlAABpQmhBFJoSUghvc3z+4Mv8yMmEJAkJ5m8X9flJWdmcs49n0zmmc+c55xjM8YYAQAAAADqnIvVAQAAAACgsaIhAwAAAACL0JABAAAAgEVoyAAAAADAIjRkAAAAAGARGjIAAAAAsAgNGWrMjh07dPPNNys2NlYxMTG69tprFR8fL0lavny5nnjiiVrP8M9//lOvvvpqlfdFR0drw4YN57W+hx9+WCNHjtRPP/3kuC0rK0vdu3dXWlqa47b4+Hh17dpV6enpjtvi4uI0fPjws67/0Ucf1YoVK876mIULF+qPf/xjlfclJiZq06ZN5/BMKvrjH/+ohQsXVrrdbrfr1Vdf1fjx4xUbG6uoqCg999xzKisrO+9tAEB91VDHq8cff1wXXXSRYmNjNXr0aF166aX66KOPZLfba3T752Lx4sXKy8uTdG5j2YXKzc3VZZddpnHjxikrK8tx+/LlyxUTE1PhsW+88YZiY2Mr3DZv3jxNmzbtrNuIjY1VRkbGWR/z+OOPa/78+VXe99NPP+nw4cNn/fmq9OjRQ6mpqZVuz8zM1LRp0xQTE6OYmBhNnDhRX3zxxXmvH/Wfm9UB4ByMMbrrrrs0e/ZsjRo1SpK0bNky3XPPPVq5cqWio6MVHR1d6zluvPHGGl3fd999p6VLl6pt27aO24KDg9W9e3fFxcXp0ksvlXSy+QoODtaGDRs0adIkx21Dhw496/pffPHFC8r3/fffq6ysTAMHDryg9ZyyYMECbd68WV9++aV8fX2Vl5en22+/Xe+//77uuOOOGtkGAFipoY9XU6ZM0d133y1JSkpK0vTp07V//349++yzdbL9U+bNm6d+/frJz8/vgseyc5GQkKDs7GytWrWqwu2DBw9WSkqKjh07pmbNmkk6Of7m5OQoLS1NYWFhjtuuuuqqs25jyZIlF5Txww8/1J///Ge1bNnygtZzyqxZs9SyZUu99NJLcnFx0cGDB3XttdeqS5cu6tu3b41sA/UDe8hQI7KyspSenq7evXs7bhs3bpwWLVokb2/vCnt5UlNTdfnll2vMmDF6+umndeeddzr21nTt2lVffPGFLrnkEo0cOVLr16/X1KlTNXr0aN12222OPTUbNmzQFVdcodjYWF199dXavn27JOn111/Xk08+KenkN6ATJ05UTEyMnnvuuTNmP3z4sG699VbFxMRo0qRJ+vrrryVJN910k+x2u2699dZKA8DQoUO1fv16x3JcXJyuvvrqCt9oxsXFaciQIZJONjqxsbEaM2aMpk6dqqKiIsc2Fi1aJOnknrChQ4fq0ksv1cKFC9W1a9cK23z22Wc1btw4TZw4UXv27NGKFSv09ttv6+OPP9bcuXPPup2UlBRdffXVioqK0rRp01ReXl5lLfbs2aPw8HD5+vpKkvz8/DR//nxNmTJFkjRmzBjHt8inL6empmrYsGH6xz/+4fgm75dfftEdd9yh4cOH18m3zQBwLhryePVb7dq10/z58/Xdd98pMTFR0skv6i655BKNHTtWt9xyizIzMx3be+qpp3TVVVfpww8/dGz/008/1V133eVYZ3l5uQYPHqz9+/crMTFR1113ncaPH6/o6Gh9++23kqQnnnhCBw4c0E033aT4+HjHWPbAAw/o/fffd6xr165dGjZsmOx2uzZv3qwrr7xS0dHRuuaaa5SSklLlc6qqXocPH9bDDz+s48ePKzY21vGcJCkgIEC9evVyjMnFxcU6cOBAhb2MhYWF2rZtm4YOHaqSkhLNnj1bMTExGjNmjN566y3Hurp27aqjR4/Kbrdr1qxZGjp0qK677jq98847uummmxyPy8nJ0e23365Ro0bp1ltvVV5enl599VXFxcXpkUce0eLFi8+6nVWrVik6Olrjx4/Xu+++e8bf7549exQZGSkXl5Mf19u3b69vvvlGkZGRSk1NVY8ePRyPPX154cKFuv/++zVt2jSNGjVKf/rTnxQfH6/JkydryJAhWrBgwRm3CYsYoAbY7XZz5ZVXmkmTJpkvvvjCJCcnV7j/q6++MjfffLMxxpj77rvPvPjii8YYY5YvX2569uxpvvrqK2OMMeHh4eatt94yxhgzd+5cM2DAAJOYmGiKi4vN8OHDzbp160xeXp4ZPHiwiY+PN8YYs2TJEjNu3DhTXl5u5s2bZ6ZPn26MMebKK680n3/+uTHGmMWLF5tu3bqZuLi4StlvueUWxzZTU1NN//79TUpKiiPPkSNHKv3M+vXrzejRo40xxhQWFprBgweb1NRUEx0dbYwxJi8vz0RERJj09HSzadMmc/HFF5ujR48aY4z5y1/+YubOnWuMMebGG280X3/9tcnKyjKRkZEmISHBlJeXm4ceesiEh4c7atenTx+zfft2Y4wxM2fONE888YQxxpjHHnvMvPHGG8YYc9bt3H///ebll182xhizdetW06NHD0fNT7dixQoTERFhZs2aZdavX2+Kiooq3D969GizadOmSsspKSmmR48e5j//+Y/jdzxq1Chz/Phxk5mZaXr27GmSkpIqbQ8A6lpDHq9Of88/3W233WY+++wzk5ycbPr27WsSEhKMMca89dZb5r777jPGGDNv3jwzbNgwc/z4ccfy9OnTTVpamunTp48pKCgwxpwc3yZNmmSMMebOO+80b7/9tjHGmI0bN5rIyEhTUlLieP6nxsdTY9l3331nbrjhBkeu1157zcyaNcvk5uaagQMHmjVr1hhjjPnmm2/MFVdcUel5nK1ecXFxJioqqtLPnNrOY489ZowxZt26dea2224zixYtMk8++aQxxpjVq1eb8ePHG2OM+fvf/25uvvlmU1xcbPLz883ll19uVqxYUeE5rVixwkRFRZm8vDyTlZVlYmNjzY033uj4HUycONFkZWWZ0tJSc9lllznGvtPHyDNtp6yszAwdOtT89NNPxhhj3nvvPRMeHu743HG6uXPnmosuusi89dZbZufOnaa8vNxxX0pKiunevXuVy6c+N5z+erzzzjtNWVmZWbFihRkxYkSVdYR12EOGGmGz2fTBBx8oOjpaH3/8saKiojRx4kQtW7as0mPj4+Md0/qioqIc0wlOiYqKkiSFh4erTZs26tChgzw8PNSuXTsdO3ZM27ZtU/PmzdW/f39JUkxMjLKysnTo0CHHOoqLi7V9+3ZNmDBB0sl54d7e3pWylJaWat26dbr++uslSa1atdLgwYMVFxd31ufbr18/ZWZmKjk5WVu2bFGvXr3UqlUrSdLRo0cVHx+vjh07qmnTplqxYoUmTJjgmEpx3XXXVarL1q1b1b59e4WHh8vFxUXXXXddhfs7deqknj17SpK6d++uY8eOVcp0tu3Ex8c7ahEZGamOHTtW+bxGjx6td955R8eOHdM999yjQYMG6fHHH1dOTs5Z6yFJZWVljjn74eHh6tWrl0JCQhQcHKzQ0NAKx9wBgFUa6nh1Nn5+fsrNzdXq1as1aNAghYeHS5ImT56sFStWOGZF9O7dWyEhIRV+NjQ0VD169NDatWslndzDNn78eEnS/Pnzdeutt0qS+vfvr+Li4grHSv/WqFGj9Ouvvyo7O1vSyeO7YmNjtXnzZjVr1swxjX/SpElKTk6udLzVudSrKkOGDHHsIYuLi9OgQYM0aNAgxx6y02es/Pjjj7r++uvl4eEhHx8fXXbZZZV+9/Hx8Ro1apR8fX0VFBSkiRMnVrh/xIgRCgoKkpubm7p06VLlmHym7Rw8eFAlJSUaNmyYJOmKK6444/N65JFH9NBDD2nNmjW65pprNGzYML3xxhvndMxg586dK7wehw0bJldXV4WHhzMe10McQ4Ya4+/vr/vvv1/333+/MjIytHDhQk2dOtUxJe+UEydOKDAw0LF8qoE45dR0ORcXF8e/JcnV1VV2u12ZmZkKCAiotO3jx487lk8NBn5+fpJODsC//ZlTjzPGyN/f33FbQEBAhekQVfHw8NCAAQO0fv16HT58WIMHD5YkDRw4UHFxcdqzZ49j4MnNzdXy5cu1Zs0aSSePXygtLT2vmpx6HqfqUNWUw7NtJycnp8I6qqrFKUOGDNGQIUNUXl6uLVu26IUXXtDMmTP1yiuvnLUmrq6u8vLyknTyd+fj41NtZgCwQkMcr87m0KFDGj58uNLS0hQfH1/hhBZ+fn6ObZz+XE4XExOjFStWKCoqSj/88IM++OADSSdPUvHmm28qKytLNptNxpizNgM+Pj4aMmSIVq5cqf79++vEiRPq37+/vv32W6WkpFTI5eHhoczMzArHW51LvarSp08f5ebm6sCBA9qwYYOmT5+u5s2by2az6ciRI9qwYYPuueceSSfHyueff94xppWUlCgyMrLC+k6cOFHhd/17x+SqtvPb8fhMvxPp5Ovqmmuu0TXXXKOCggKtXLlSs2bNUpMmTRwN3Zn89vV4akw+9dpE/UJDhhpx9OhRpaamasCAAZKkpk2b6o477tCSJUu0d+/eCo/19fVVQUGBY/ls37ZVpUmTJo7BRTrZeOTk5KhJkyaO2069weXl5cnf3192u73KvTzBwcFycXFRTk6O42eys7MrrOtMhg0bps2bN+vQoUN67LHHJEmDBg3Spk2btHfvXt17772SpLCwMF1xxRWOx1TFz8+vQk1+z7dXZ9tOQECA42xYks7YcK5atUr9+vWTv7+/XF1dNXDgQN19992OAcXFxaXCG/m57DkDgPqkoY5XZ5KSkqKEhARddNFFjj1B8+bNO6+cMTExevvtt7V9+3YFBgaqffv2Ki0t1YMPPqhXX31VI0eOrLJxOdO6li9frqysLMXExMhmsyksLEwdO3as8uy+pztbvc529kI3NzcNHjxY69at08GDBxURESHp5Jekq1ev1t69ezVo0CBJJ8fKW265RaNHjz7j+n47Jp/v7/1s29m/f/85jcf5+fnauHGj4+d9fHw0YcIEbdu2TXv27NHIkSNlt9tljJHNZtOJEyfOOyPqD6YsokYcOXJE99xzj3bs2OG4bdu2bTp8+LB69epV4bGRkZH63//+J+nkLv3zbT4iIyOVkZGhn3/+WdLJMyE2b95crVu3djzGy8tL3bp10/Llyx2PKS4urrQuNzc3DRs2zHGAa3JysuLj4x1TG85myJAh2rJli5KSkhxv/oMHD1Z8fLz279/vOPPhmDFjtGzZMseb7vfff6933nmnwroiIiKUkJCgpKQk2e12ffnll+dUCzc3N+Xm5la7nT59+jhqsWXLFiUnJ1e5vk8++UQvvfSSo1bFxcVaunSp47mEhoZq9+7dkk6e8riqmgJAfdZQx6uqpKamaurUqbr++uvVsmVLDRs2TPHx8Y4TZmzbtk2zZ8+udj3NmjVTmzZt9NZbbzmmKxYWFqqgoMAxXf6jjz6Su7u7o1Fxc3OrsgkYPXq0fv755wpTH3v37q309HRt3bpV0skm8pFHHpEx5rzrdSZDhw7V559/rt69e8vV1VXSyS9J//Wvf6lnz56OPUZjx47Vv//9b5WXl8sYo/nz52v16tUV1tWrVy+tXLlSRUVFOnHihOM1UJ3Tx+Qzbadt27ZydXV1TKdcuHChbDZbpXXZbDY98cQTFZrYjIwMrV27VgMHDlRwcLBcXV2VkJAgSY4TkqFhYg8ZakTfvn01a9YsPfPMM8rNzZXdblfTpk31t7/9zXFs1SmPPPKIpk2bpu+++04jRoxQnz59qnwzOhMfHx+9+uqrmjVrlgoKChQSEqJXXnml0jqeeeYZTZ8+XW+//bZGjBihTp06Vbm+mTNn6qmnntLChQvl7u6u2bNnq0WLFtXmCA8PV1FRkSIiIhxv/s2bN1d5ebl69OjhmL4XERGhu+66y3HWxiZNmmjmzJkV1hUWFqapU6dqypQpatq0qSZPnqz//Oc/1WYYPXq0Hn74YR06dEjz5s0743ZO1XzRokXq3bv3GRvOl19+WS+99JIuueQS2Ww2lZeXa+zYsXrggQckSXfffbdmzJihL774QjExMercuXO1GQGgPmnI45Ukffzxx/rvf/+rkpISubu7a/LkyY6zQoaFhWnWrFm65557VFpaKl9fX02fPv2cssbExGju3LmOWRYBAQG67bbbdPnll6tJkyb685//rKioKN1111369ttvFRsbq8mTJ1dq+Pz8/BxfMvbp00fSyaZz3rx5mjVrlvLz8+Xu7q4HHnigUh3OtV5VGTJkiGbNmqXLL7/ccdvgwYP16KOP6r777nPcdv311ys1NVUTJ06UMUY9e/bUzTffXGFd0dHRWrlypWJjY9WuXTuNHz++wpmVz1bDqVOn6v7779cNN9xQ5Xbc3d01a9YsTZ8+XR4eHvrDH/5QYYr/6bX48MMP9fLLLzvO0Oju7q4bbrjB0ejed999uu222xQWFlbhLJBoeGzmt19PAHXg1C52Sbryyisdb/SN2ek12bt3r66//vrfddFnAEDNYbxqnE7/vX/66adat26d3njjDYtTwVkxZRF17tRJIiQ5rnNyakpEY1VWVqbhw4c7pnMsXrzY8c0iAMAajFeN065duzR27Fjl5OSorKxMy5YtY0xGrWIPGepcWlqaHn30UR06dEguLi666667znra18Zi+fLlevnll2WMUWhoqObMmaN27dpZHQsAGi3Gq8Zr3rx5WrRokVxdXdWnTx/NnDnzvC9HAJwrGjIAAAAAsAhTFgEAAADAIjRkAAAAAGCROjntfXp6bl1sps74+XkqL4/rL52OmlRGTSqjJlVzprqEhvpbHaFBYXx0ftSkatSlMmpSmbPV5ExjJHvIfgc3N1erI9Q71KQyalIZNakadYGz4LVcGTWpGnWpjJpU1lhqQkMGAAAAABahIQMAAAAAi9CQAQAAAIBFaMgAAAAAwCI0ZAAAAABgkXNqyPbs2aOoqCj985//rHTfunXrdNVVV+naa6/VG2+8UeMBAQAAAMBZVduQFRQUaNasWbr44ourvH/27Nl6/fXX9dlnn2nt2rXat29fjYcEAAAAAGdUbUPm4eGhf/zjHwoLC6t0X0pKigIDA9WiRQu5uLho5MiRWr9+fa0EBQAAAABn41btA9zc5OZW9cPS09MVEhLiWA4JCVFKSkqlx/n5eTrVhd1cXV0UFORjdYx6hZpURk0qoyZVoy6NF+Oj86MmVaMulVGTyhpLTaptyGpCXl5xXWymzgQF+Sg7u8DqGPUKNamMmlRGTapWk3UpK7frRHGZcgrLlFNYqpyiUuUUlSm/pFx5xWXKLy5XfsnJ5fySMuX933JhSbnuGd5B47pVng1xPkJD/WvkeTQWjI/Oj5pUjbpURk0qc7aanGmMvKCGLCwsTBkZGY7lY8eOVTm1EQDw+xhjlFNUpoz8Eh0/7b9TyzlFpxqvk//PLyk/6/q83Fzk6+kmXw9X+f3f/0N8vOXn6aa2wd519KwAAMApF9SQtW7dWnl5eUpNTVXz5s31448/6q9//WtNZQMAp5dbVKYjR05oz6EcHT1RpCMninU0t0hHTxQ7mq4yu6n0c55uLmri66Egb3cFeburXYiPAr3cFOjlrkDvk/8POPV/Lzf5/1/z5ebK1U4AAKhPqm3IduzYoRdeeEGHDh2Sm5ubli5dqjFjxqh169aKjo7WM888o2nTpkmSJkyYoA4dOtR6aABoSHIKS5WUVajkrAIlZxUqKbNQKdmFOpxTVGmPlqebi5r5e6pFgKfaNwlSEx8PNfF1V1NfDzXx9XD839fDVTabzaJnBAAAakq1DVnPnj31ySefnPH+gQMHasGCBTUaCgAaotyiMu3LyNfe9DztTc9X4vECJWUWKKeozPEYVxebWgV6qW2wt/q2ClTzAE91bhGoAFepeYCXQnzcabQAAGhE6uSkHgDgbLIKSrTjSK52Hs3VnrQ87cvI15ET//8EDYFeburU1FdjwpuqbbCP2gV7q22wt1oFelWaNuhsBy0DAIBzR0MGANUoK7drd1qeth/J1c4jJ7TjSK4O5RRJklxtUrsQH0W2DNAfIn3VJcxPXZr6KtTPgz1dAACgWjRkAPAbZXajhLQ8bU7O1qaUbG09lKPCUrskKczPQz1bBOjK3i3Us0WAujfzk5e781xHCgAA1C0aMgCQdPREkdYkZmrdgUxtSc1xnGyjQ4iPJvZopv5tghTZMkBh/p4WJwUAAM6EhgxAo1RuN9px5ITWJGZqTWKm9mXkS5JaBXppXLdQDWgTpH5tgtTU18PipAAAwJnRkAFoNOzGaOuhE1qekK4f9qQrs6BUrjapT+tAPTCyo4Z1CFG7EG+O/QIAAHWGhgyAUzPG6NdjeVq2O03fJ6QrLa9Enm4uGtYxRGO6NNXF7UPk78VbIQAAsAafQgA4peyCUi3edUz/3XFU+zMK5O5q05D2Ibp/RKiGd2oiHw9OxAEAAKxHQwbAaRhjtDE5W19vO6JV+4+rtNwoorm/nojuoujwUPaEAQCAeodPJwAavKLSci3ZlabPthxS4vECBXq56creLXVZz+bqHOprdTwAAIAzoiED0GBl5Jfo378c1sKtR5RdWKouob6aERuu6K5h8nRzsToeAABAtWjIADQ4abnF+nhTir7eflQlZXaN6NRE1/VvpX6tAzlDIgAAaFBoyAA0GEdPFOmjjSlatOOo7Eaa2CNMNw9qq7bB3lZHAwAA+F1oyADUezmFpXp/Q7L+/cthGSNd0rOZbh7URq0CacQAAEDDRkMGoN4qLrPri58P6YMNKcovKdOkiGa6/eJ2ah7gZXU0AACAGkFDBqDeMcbo+z0Zen11oo6cKNaQDsG6b3hHzpgIAACcDg0ZgHolKbNAL/6wTxuTsxUe6qunrgrXoHbBVscCAACoFTRkAOqFotJyfbAxRZ9sSpGnm4seGdNZV/ZuIVcXzpoIAACcFw0ZAMv9nJqjZ5cmKDW7SBN6hOn+ER3VxNfD6lgAAAC1joYMgGWKSss1f81Bfb7lkFoGeunNqyM1oG2Q1bEAAADqDA0ZAEtsPZSjZ5fuUXJWoa7p01L3juggb3dXq2MBAADUKRoyAHWq3G70wYZk/WN9kpr7e7JXDAAANGo0ZADqzLETRXrwy22KT8lRbPcwPTa2s/w8eRsCAACNF5+EANSJdQcyNXPpHhUUl+kvMeG6JKKZbDbOoAgAABo3GjIAtcpujN6LS9Y765LUtZmfZl0dqQ5NfKyOBQAAUC/QkAGoNfklZXrmfwlaue+4JvQI0wtX9VZRfrHVsQAAAOoNGjIAtSIlq1DTFu1UcmaBHhrVUdf1ayUvd1cVWR0MAACgHqEhA1Dj4pOz9eh/f5WLTZp3ZS8NahdsdSQAAIB6iYYMQI1asitNM5ckqE2wt/52RYRaBXpbHQkAAKDeoiEDUCOMMfpoY4reWHNQ/VoH6qXLeijAy93qWAAAAPUaDRmAC1ZuN3ppxT59tfWIYrqF6umYrvJwc7E6FgAAQL1HQwbggpSW2/X04t36fk+Gpgxso3uGt5cL1xcDAAA4JzRkAH634jK7Hv/mV61JzNSDIzvqhgGtrY4EAADQoNCQAfhdCkvLNe3rnYpPztYTUZ31h94trY4EAADQ4NCQAThvecVlenDhDm0/ckLPjO+qCT2aWR0JAACgQaIhA3Be8kvKdP9X2/XrsTw9N6m7xoaHWh0JAACgwaIhA3DOikrLNfU/O/Xr0Vw9f0kPje7S1OpIAAAADRoNGYBzUlJm1yOLftXPqTmaPbEbzRgAAEAN4EJBAKpVVm7XE9/uUlxSlp6KCde4bmFWRwIAAHAKNGQAzspujGb8L0Gr9x/Xo2M769Keza2OBAAA4DRoyACc1WurErUsIV33De+gq/twansAAICadE7HkD333HPaunWrbDabpk+frsjISMd9n376qf773//KxcVFPXv21JNPPllrYQHUrX/Gp+pfmw9pcr9WumkgF30GAACoadXuIdu4caOSkpK0YMECzZkzR3PmzHHcl5eXp/fee0+ffvqpPvvsM+3fv1+//PJLrQYGUDeW7krTa6sSFRUeqodGdZTNZrM6EgAAgNOptiFbv369oqKiJEmdOnVSTk6O8vLyJEnu7u5yd3dXQUGBysrKVFhYqMDAwNpNDKDWbUzK0jNLEtSvdaCeGd9VLjRjAAAAtaLaKYsZGRmKiIhwLIeEhCg9PV1+fn7y9PTUPffco6ioKHl6emrixInq0KFDpXX4+XnKzc21ZpNbyNXVRUFBPlbHqFeoSWUNtSb70vL02De71LGpr/4xZYACvN1rbN0NtSa1jbo0XoyPzo+aVI26VEZNKmssNTnv65AZYxz/zsvL09tvv60lS5bIz89PN998s3bv3q1u3bpV+Jm8vOILT1qPBAX5KDu7wOoY9Qo1qawh1iS7sFS3/+tnebja9PJlPWQvLlV2cWmNrb8h1qQuOFNdQkP9rY7QoDA+Oj9qUjXqUhk1qczZanKmMbLaKYthYWHKyMhwLKelpSk0NFSStH//frVp00YhISHy8PDQgAEDtGPHjhqKDKAulZXb9cQ3v+pYbrFeuixCzQO8rI4EAADg9KptyIYOHaqlS5dKknbu3KmwsDD5+flJklq1aqX9+/erqKhIkrRjxw61b9++9tICqBXGGP31x/2KT8nRU+PCFdkywOpIAAAAjUK1Uxb79euniIgITZ48WTabTTNmzNDChQvl7++v6Oho3XrrrZoyZYpcXV3Vt29fDRgwoC5yA6hB//7lsL7aekRTBrbRhB7NrI4DAADQaJzTMWQPP/xwheXTjxGbPHmyJk+eXLOpANSZLanZeuXH/RreMUT3DG9vdRwAAIBGpdopiwCcV3pesZ74ZpdaB3nr2QndOL09AABAHTvvsywCcA4nT+KxSwUl5Zp/daT8PHk7AAAAqGvsIQMaqXmrD2jr4RP6Sxz5W9YAACAASURBVEy4OjX1tToOAABAo0RDBjRCyxPS9dmWQ7q2b0uN6xZmdRwAAIBGi4YMaGQOZhZo1tIERbYM0AMjO1odBwAAoFGjIQMakeIyu6Z/u0seri56flJ3ubvyFgAAAGAljuIHGpHXVydqb3q+Xrk8QmH+nlbHAQAAaPT4ehxoJFbvP64FPx/WtX1baninJlbHAQAAgGjIgEYhPa9Yzy5JUHior+4fwXFjAAAA9QUNGeDkyu1GTy/ereIyu+ZM6i4PN/7sAQAA6gs+mQFO7uNNKYpPydEjYzurfYiP1XEAAABwGhoywIklHMvT2+uSFN01VJdENLM6DgAAAH6DhgxwUiVlds1YslvB3u56bGxn2Ww2qyMBAADgN2jIACf19rqD2p9RoKdiwhXo7W51HAAAAFSBhgxwQr+k5uiTTam6IrK5hnYIsToOAAAAzoCGDHAyBSXlemZJgloEeumBkZziHgAAoD5zszoAgJo1b3WiDucU6e1re8vXgz9xAACA+ow9ZIATWX8wU19tPaIbBrRW39aBVscBAABANWjIACeRX1KmOcv2qkOIj+4a2t7qOAAAADgHNGSAk5j/00Gl5RbrqZhwebrxpw0AANAQ8KkNcAJbD+Xo378c1jV9WyqyZYDVcQAAAHCOaMiABq64zK7Zy/aoeYCn7h7Wweo4AAAAOA80ZEAD935ckg5mFmp6dBf5eLhaHQcAAADngYYMaMD2pOXpo02pmtgjTBe15wLQAAAADQ0NGdBAldmNZi/bo0AvNz04qpPVcQAAAPA70JABDdSCLYe061ieHh7TWUHe7lbHAQAAwO9AQwY0QMdyi/XOuiQN7RCiqPCmVscBAADA70RDBjRAr/y4X+XG6JGxnWSz2ayOAwAAgN+JhgxoYNYeyNSKvRm69aK2ahXobXUcAAAAXAAaMqABKSot10s/7FP7EG/dOKC11XEAAABwgWjIgAbkg40pOpRTpMfGdpG7K3++AAAADR2f6IAG4uDxAn28MUXju4dpQNsgq+MAAACgBtCQAQ2AMUYv/LBX3u6uemBkR6vjAAAAoIbQkAENwJLdaYpPydHdw9qria+H1XEAAABQQ2jIgHour7hMr65MVI/m/roisoXVcQAAAFCDaMiAeu7d9cnKKijVY2M7y9WFa44BAAA4ExoyoB47eLxAn/98SJf2bK4ezf2tjgMAAIAaRkMG1FPGGL28cr+83Fx09/D2VscBAABALaAhA+qpnxIzFXcwS3cMaacQH07kAQAA4IxoyIB6qLjMrld+3K8OIT66pk9Lq+MAAACgltCQAfXQvzan6lBOkaaN7iQ3V/5MAQAAnJXbuTzoueee09atW2Wz2TR9+nRFRkY67jty5IimTp2q0tJS9ejRQ88++2ythQUag2O5xXo/LlmjOjfR4PbBVscBAABALar2q/eNGzcqKSlJCxYs0Jw5czRnzpwK98+dO1e33HKLvvzyS7m6uurw4cO1FhZoDF5fnSi7MXpwVEerowAAAKCWVduQrV+/XlFRUZKkTp06KScnR3l5eZIku92uzZs3a8yYMZKkGTNmqGVLjncBfq9fUnO0dHe6bhrYRq0Cva2OAwAAgFpW7ZTFjIwMRUREOJZDQkKUnp4uPz8/ZWZmytfXV88//7x27typAQMGaNq0aZXW4efnKTc315pNbiFXVxcFBflYHaNeoSaVnW9Nyu1Gr6z6WS0CvfRAdFd5ezjP38wpvE6qRl0aL8ZH50dNqkZdKqMmlTWWmpzTMWSnM8ZU+PexY8c0ZcoUtWrVSnfccYdWrlypUaNGVfiZvLziCw5anwQF+Sg7u8DqGPUKNansfGuyaPsR7TqaqzkTu6m4oFjFTlhOXidVc6a6hIZyAfPzwfjo/KhJ1ahLZdSkMmeryZnGyGqnLIaFhSkjI8OxnJaWptDQUElScHCwWrZsqbZt28rV1VUXX3yx9u7dW0ORgcajoKRcb65NUq8WAYruGmp1HAAAANSRahuyoUOHaunSpZKknTt3KiwsTH5+fpIkNzc3tWnTRgcPHnTc36FDh9pLCzipTzal6Hh+iR4c1VE2m83qOAAAAKgj1U5Z7NevnyIiIjR58mTZbDbNmDFDCxculL+/v6KjozV9+nQ9/vjjMsYoPDzccYIPAOcmLbdYn8SnKio8VJEtA6yOAwAAgDp0TseQPfzwwxWWu3Xr5vh3u3bt9Nlnn9VsKqAReXPtQdmN0b0j2lsdBQAAAHWs2imLAGpPQlqevtt5TNf2bcVp7gEAABohGjLAIsYYvboqUQFebrplcFur4wAAAMACNGSARdYkZio+OVu3X9xO/l7nfQUKAAAAOAEaMsACZeV2zVudqLbB3rqydwur4wAAAMAiNGSABf6z/agOZhbq/hEd5ObKnyEAAEBjxSdBoI7lFZfpnXVJ6tc6UCM6NbE6DgAAACxEQwbUsQ82pCi7sJSLQAMAAICGDKhLh3OK9PmWVE3oEabuzfytjgMAAACL0ZABdWj+mgOy2Wz689D2VkcBAABAPUBDBtSRnUdOaOnudN3Qv5WaB3hZHQcAAAD1AA0ZUAeMMfrbykSF+LhryqA2VscBAABAPUFDBtSBH/dmaOvhE7pzaHv5enARaAAAAJxEQwbUstJyu17/6YA6NvHRpT2bWx0HAAAA9QgNGVDL/v3LYaVmF+mBkR3l5sJp7gEAAPD/0ZABtSinsFTvxSXronbBGtIhxOo4AAAAqGdoyIBa9F5csvKKy/TAyI5WRwEAAEA9REMG1JKk4/n69y+HdUnP5uoc6mt1HAAAANRDNGRALXlx2R65u9p0FxeBBgAAwBnQkAG14OfUHC379ZimDGyjpr4eVscBAABAPUVDBtQwuzF6dVWimgV46sYBra2OAwAAgHqMhgyoYct2p+vXo7maGhUuL3dXq+MAAACgHqMhA2pQUWm53vjpgLqG+eny3i2tjgMAAIB6joYMqEGfbzmko7nFenBkR7lwEWgAAABUg4YMqCGZBSX6cGOKhncM0YC2QVbHAQAAQANAQwbUkHfWJamotFz3j+Ai0AAAADg3NGRADThwvEBfbzuiP/RuqfZNfKyOAwAAgAaChgyoAfNWJ8rL3VW3X9zW6igAAABoQGjIgAu0ISlLaxIzdetFbRXsw0WgAQAAcO5oyIALUG43em1VoloGeOqavq2sjgMAAIAGhoYMuADf7jyqven5undER3m68ecEAACA88MnSOB3Kigp15trk9SrRYCiwptaHQcAAAANEA0Z8Dt9vClFx/NL9NCojrLZuAg0AAAAzh8NGfA7HMst1j/jUzWua6h6tQywOg4AAAAaKBoy4Hd4c80BGWN0z/AOVkcBAABAA0ZDBpynXcdy9d2vaZrcr7VaBnpZHQcAAAANGA0ZcB6MMXp1ZaKCvd31p8FtrI4DAACABo6GDDgPq/Yd15bUHN0xpJ38PN2sjgMAAIAGjoYMOEel5Xa9/tMBdQjx0eWRLayOAwAAACdAQwacoy+3HlFyVqEeGNVRbi6c5h4AAAAXjoYMOAc5haV6d32SBrcL0pD2wVbHAQAAgJOgIQPOwfsbkpVXXKYHR3biItAAAACoMefUkD333HO69tprNXnyZG3btq3Kx7z88su66aabajQcUB8cPF6gBT8f1qU9m6tzqK/VcQAAAOBEqm3INm7cqKSkJC1YsEBz5szRnDlzKj1m37592rRpU60EBKz2t1X75e3uoj8Pa291FAAAADiZahuy9evXKyoqSpLUqVMn5eTkKC8vr8Jj5s6dq4ceeqh2EgIWWpN4XOsOZOn2i9spxMfD6jgAAABwMtVeSCkjI0MRERGO5ZCQEKWnp8vPz0+StHDhQg0aNEitWrU64zr8/Dzl5uZaA3HrB1dXFwUF+Vgdo15xxpqUlNn12uoD6tjUV7eN7CwPt/M75NIZa3KhqEnVqEvjxfjo/KhJ1ahLZdSkssZSk/O+sq0xxvHv7OxsLVy4UB988IGOHTt2xp/Jyyv+fenqqaAgH2VnF1gdo15xxpp8silFB48X6LU/9FRBXpHO99k5Y00uFDWpmjPVJTTU3+oIDQrjo/OjJlWjLpVRk8qcrSZnGiOr/co/LCxMGRkZjuW0tDSFhoZKkuLi4pSZmakbbrhB9957r3bu3KnnnnuuhiID1snIL9F7ccka1jFEQzqEWB0HAAAATqrahmzo0KFaunSpJGnnzp0KCwtzTFeMjY3V4sWL9cUXX+jvf/+7IiIiNH369NpNDNQiz6++UEi/CHXrEKplr03Rs7m/WB0JAAAATqzaKYv9+vVTRESEJk+eLJvNphkzZmjhwoXy9/dXdHR0XWQE6oTnV1/If+p9shUWSpJanUiX+cs05fp6qPjKayxOBwAAAGdkM6cfFFZL0tNza3sTdcrZ5rPWBGeoSUi/CLmmplS6vbx1G2Vu2Xne63OGmtQ0alI1Z6oLx5CdH8ZH50dNqkZdKqMmlTlbTX73MWRAY+FyKPW8bgcAAAAuFA0Z8H/KW1Z96QZ7q9Z1nAQAAACNBQ0Z8H8WXnm3Ctw8K9xmvL2V/+QMixIBAADA2dGQAZL2ZeTrCZ9IfXn7Uypv3UbGZlN56zbKfeV1TugBAACAWnPeF4YGnI0xRi/+sE9+nm66+LG7lTnzAasjAQAAoJFgDxkavf/tStPPqTm6Z3gHBfm4Wx0HAAAAjQgNGRq13KIyvbYqUT1b+OuyXs2tjgMAAIBGhoYMjdqbaw8qu7BUj4/tIhebzeo4AAAAaGRoyNBo7T6Wq6+2HtZVvVuqazM/q+MAAACgEaIhQ6NkN0Yv/LBPQd7uumtoe6vjAAAAoJGiIUOj9PX2o9pxJFcPjOwofy9ONgoAAABr0JCh0cnIK9brqxPVv02gxncPszoOAAAAGjEaMjQ6f/1xv0rK7JoeHS4bJ/IAAACAhWjI0Kis2ndcP+zJ0G0Xt1PbYG+r4wAAAKCRoyFDo5FXXKYXf9irzk19ddOA1lbHAQAAAGjI0HjMX3NQ6XklenJcF7m58tIHAACA9fhUikZh2+ET+vKXw7qmb0v1bBFgdRwAAABAEg0ZGoHScrvmLNujMH9P/XlYe6vjAAAAAA40ZHB6H2xIVuLxAj0e1Vm+HlxzDAAAAPUHDRmcWsKxPL2/IUXju4dpWMcmVscBAAAAKqAhg9MqLbfrmSUJCvZ217TRnayOAwAAAFRCQwan9W5csvZl5Gt6dBcFertbHQcAAACohIYMTunXo7n6aEOyJkU00/BOTFUEAABA/URDBqdTUnZyqmITXw9NHcVURQAAANRfNGRwOu+sT9KB4wV6cly4/L04qyIAAADqLxoyOJUtqdn6eGOKLuvVXEM6hFgdBwAAADgrGjI4jdyiMs1YnKDWQV5MVQQAAECDwHwuOAVjjOZ+v1fpecV677o+8vFwtToSAAAAUC32kMEp/G9XmpYlpOuOIe0V0SLA6jgAAADAOaEhQ4N3KKdQL/6wT31aBejmQW2sjgMAAACcMxoyNGhldqOnFydIkp6d0E2uLjaLEwEAAADnjmPI0KD9Y32Sth0+odkTuqlFgJfVcQAAaPSMMcoqLNXBzAIdzCzU0RNFyi4sVU5hmfJLyiRJbm6ucjFGAd7uCvRyU/MAL7UN9lbbIG+1DPTiC1Y0KjRkaLDWH8zUB3HJuiSimWK6h1kdBwCARskYo30Z+dqckqOth3L0y6ETysgvcdzvapMCvd0V6O0uPw9XSTa52aX8olLtTc9XdmGpisrsjsf7ergqorm/IlsGqF+bQPVtFSg3VyZ1wXnRkKFBOpZbrKcXJ6hjUx89Oraz1XEAAHBqnl99Id85M+VyKFX2Vq2VN/1prR8cox/3Zuin/cd1+ESxJKlFgKf6twlUj+b+6tDER+1DfNTM31Mutop7vIKCfJSdXSDpZEOXXViq5KxCJWUWatexXG07fELvb0jWu3EnG7SL24doVOcmGtm5ibzcOZMynAsNGRqcsnK7nvx2l0rK7Jp7SQ/emAEAqEWeX30h/6n3yVZYKElyTU2RxwP3anHMvVrae6wGtg3Snwa31UXtg9X8dxw+YLPZFOzjoWAfD/VuFahLezWXJOWXlCk+OVs/7c/UT4nH9f2edPl6uCqqa6guiWimyJYBstmY2oiGj4YMDc78NQe19f+OG2sf4mN1HAAAnJrvnJmOZuwU79JiPbfpMz389lPyrqUvRn093DSyc1ON7NxUdmP0c2qOvtl5TMt2p2nR9qPqEuqr6/q1Uky3MHm4MaURDRevXjQoq/cf1yfxqbqydwuOGwMAoBbZjdHyhHTZUlOrvN837UitNWO/5WKzqX+bID0T21VL7rpYT43rIrsxenbpHl3yjw36YEOy44QhQEPDHjI0GAePF+jpxbvVLcxPD43qZHUcAACckjFGq/cf11trk7QvI19jgkLVPDut0uPsrVpbkE7y8XDVZb1a6NKezbUxKVufbk7V/DUH9Wl8qqYMbKOr+7ass0YRqAk0ZGgQcovKNG3RTnm4uuily3rIk6kJAADUuA0HszR/7UH9ejRXbYO9NXtCN3m1nSPz8P0Vpi0ab2/lPznDwqQnjz0b3D5Yg9sHa+eRE3pnfZJe/+mA/hmfqlsuaqurerfg7IxoEGjIUO+V243+sni3DuUU6c2rI3/XAcMAAODMkjIL9OqqRK1JzFSLAE/9JSZcE3o0k5uLTaXdr1Wui63CWRbzn5yh4iuvsTq2Q0SLAL32h17adviE3lx7UC//uF9f/nJYD47qqKEdQjj5B+o1GjLUe2+vO6i1BzL12NjO6ts60Oo4AAA4jbziMr0Xl6zPtxySp5uLHhjZUdf0aVnpJBnFV15TrxqwM4lsGaD5V/XSmsRMvboqUQ/9Z6cuahesB0d1VKemvlbHA6p0Tg3Zc889p61bt8pms2n69OmKjIx03BcXF6dXXnlFLi4u6tChg+bMmSMXF3YPo2Z8n5CuDzak6LJezXVl7xZWxwEAwCkYY/S/XWl6bVWisgpKdUnPZrp7WAc18fWwOtoFs9lsGt6piS5qH6wvtx7RP9Yl6YZPtujGAa1120VtuVwO6p1qO6eNGzcqKSlJCxYs0Jw5czRnzpwK9z/99NOaN2+ePv/8c+Xn5+unn36qtbBoXLYfPqFnliQosmWAHh3TmekGAADUgNTsQt3/1Q7N+F+CWgZ66cMb+uovMV2dohk7nburi67r10oLbxmoCd3D9NHGFF370WatO5BpdTSggmr3kK1fv15RUVGSpE6dOiknJ0d5eXny8/OTJC1cuNDx75CQEGVlZdViXDQWqdmFmvb1TjX19dBfL+vB9UUAALhAZXajzzan6u11SXJzsemRMZ11VZ8WcnHyLzyDfNz1dGxXTYxopueX79UDC3coKjxU00Z3VFM/T6vjAdXvIcvIyFBwcLBjOSQkROnp6Y7lU81YWlqa1q5dq5EjR9ZCTDQmOYWlenDhDtmN0Wt/6KlgH+f6xg4AgLq2+1iu/vTpz5q3+oAuahesBX8coGv6tnT6Zux0/dsE6V9T+uvOIe20en+Grv1os/6365iMMVZHQyN33if1qOpFe/z4cd11112aMWNGhebtFD8/T7m5Oc98XVdXFwUF+Vgdo16pqZoUl9l191fbdfhEkT7640D1bh9SA+msweukMmpSNerSeDE+Oj+ra1JSZtf8Vfv11upEhfh46PXJfRTTo5nlhwFYWZeHx3fXHwa00eP/2aGnFydoVWKWnr20h8L8rT2Ls9WvlfqosdSk2oYsLCxMGRkZjuW0tDSFhoY6lvPy8nT77bfrwQcf1LBhw6pcR15ecQ1ErT+CgnyUnV1gdYx6pSZqYjdGTy/erU0HszRrQjd1CfJq0HXmdVIZNamaM9UlNNTf6ggNCuOj87OyJvsy8vXM/xKUkJaniRHNNG1UJ/l7uSknp7D6H65lVr9WQtxd9OZVvfTZlkN6a+1BjZ+3Rg+P6aTYbmGWNatW16Q+craanGmMrHbK4tChQ7V06VJJ0s6dOxUWFuaYpihJc+fO1c0336wRI0bUUFQ0RsYYvbxiv5buTtfdw9ortnuY1ZEAAGiQyu1Gn2xK0ZR/blF6XrFeurSHnontKn8vrnZ0OlcXm24c0Fr/vKmf2gV76+nFCXpk0a/KyC+xOhoamWr/Mvv166eIiAhNnjxZNptNM2bM0MKFC+Xv769hw4bp66+/VlJSkr788ktJ0qRJk3TttdfWenA4l3fWJemLXw7r+v6t9MdBbayOAwBAg5SaXaiZSxL0y6ETGt2lqZ6I6syx2NVoH+Kjf0zuo39tTtVbaw9q8ofxenRsZ43rxpfDqBvn9FXJww8/XGG5W7dujn/v2LGjZhOh0fnX5lS9G5esS3s204MjO1o+rx0AgIbGGKOF247otVWJcnWxaeb4rhrf3brpdw2Nq4tNNw1so+Edm+iZJQl68rvd+nHvcT02trOCfNytjgcnx75rWOqbHUf1t5WJGtOlqaZHhzNwAABwntJyizVr2R7FHczS4HZBempcuJoHWHuCioaqfRMfvXtdH32yKUXvrEvSltRsTY8O18jOTayOBidGQwbLLP71mGYv26OL2gVr1oRucnWhGQMA4FwZY7Rkd5pe+mG/SsvtenRsZ13VuwVfbl4gNxeb/jS4rYZ2CNEzSxL08KKdmhTRTNNGd5KfJx+dUfN4VcES3+08pplLEtS/bZBe4sLPAACcl+yCUs39Ya9+2JOhyJYBmhHbVW2Dva2O5VTCw/z00Q199e76JH24MUUbk7L0dGxXDW5X+RJPwIXgUzDq3Lc7j2rmkgQNbBukv10eIS9357kGDwAAtW1N4nFd+1G8Vu07rnuGtdc71/amGasl7q4u+vOwDnrvuj7ydnfVvV9u1wvf71VBSbnV0eBE2EOGOvXfHUc1e+keDWoXpL9eRjMGAMC5yi8p06srE/X19qPq3NRXr1/ZS+FhftX/IC5YzxYB+udN/fTm2oP6bPMhxSVlaUZMV/VpHWh1NDgB9pChzizYckizl+7R4PbBNGMAAJyHX1JzdP3HW7Ro+1FNGdhGH93Ql2asjnm5u+qhUZ305jWRshvpjgVb9erKRBWX2a2OhgaOPWSodcYYvbUuSe/HJWtU5yaaPbG7PDlmDACAapWU2fX2uoP6ZFOqWgR66Z1re7NXxmL92wTpX1P6ad6qA/p0c6rWHcjUM+O7qkdzf6ujoYHiUzFqVbnd6Pnv9+r9uGRd1qu5nr+kB80YAADnYE9anm7+9Gd9vClVl0c217+m9KMZqyd8Pdz0RHQXzbuyp/JLynTLv37WW2sPqrScvWU4f+whQ60pLrPrL4t368e9GbplcBvdNbQ9p+IFAKAaZXbjuA5WgJeb/nZFhIZ15DpY9dHF7UP02c399fKP+/VeXLLWJGbqmdiu6hzqa3U0NCA0ZKgVGfklemTRTu04kqupozvpun6trI4EAEC9tzc9T7OW7tGuY3kaG95Uj4/toiAfd6tj4SwCvNw1c3w3jercVM8v36spn27RnUPa68YBrbnGKs4JDRlq3J60PE39eqdyCkv1wqU9NKZLU6sjAQBQr5WW2/XhhhS9vyFZAV5umntJd40ND7U6Fs7D6C5N1adVgJ7/fp/+/tMBrdqXoRmxXdUuxMfqaKjnaMhQo1bty9BfFu+Wv6eb3p3cR12bcQYoAADOZtexXM1aukd70/MV2z1M00Z1Yq9YAxXs46EXLumupbvT9eIP+3TDJ1t07/AOuqZvS7lw2AbOgIYMNcJuN3ovLklvr01S9+b+evmyHmrq52l1LAAA6q3iMrveXZ+kTzalKNjHQ3+9LEIjO3OsWENns9kU2z1M/dsEavayPXr5x/0nv7CO6aqWgV5Wx0M9REOGC5ZdWKqH//urVu3NUEy3UD01LpxrjAEAcBYbDmbphR/2KiW7SJdENNNDozrJ34uPZc4k1M9Tr17RU4u2H9XfVibquo82686h7XRN31Zy49gynIa/fFyQHUdO6Ilvdul4QYkej+qsP0S24EyKAACcQUZ+iV5duV9Ld6erbbC3/n5VLw1uF2x1LNQSm82myyNbaFC7YL34wz79bWWivt15TI9HdVFkywCr46GeoCHD72I3Rp9vOaTXVx9QqJ+HFtx+kdr4Mt8dAICqlNuNvtp6RPPXHFBJuV13XNxOUwa14dqcjUTLQC/97YoI/bjvuF5esU+3fvaLrohsrnuGdVCgN5+fGjsaMpy3Y7nFmrkkQZuSszW8Y8jJMwi1CFR2doHV0QAAqHd2HDmhl1bs169HczWobZAei+qitsHeVsdCHbPZbBrTpakGtwvSO+uStGDLIa3ce1z3jeigiRHNrI4HC9GQ4bws252mud/vU2m5XU9Ed9EVvZozRREAgCocyy3W7O/3adHWw2ri66HZE7ppXLdQxs1GztfDTQ+N6qRJEc30/PJ9enbpHv37l8OacUmEOgVyQrTGiIYM5+R4fole/nG/liekq2cLfz07vpva8O0eAACVFJWW65P4VH28MUV2SX8a3EY3D2ojXw8+duH/6xLqp3ev662lu9P099UHNPndDYruGqr7RnRQiwDOxtiY8M6AszLG6Judx/TaqkQVlpbrziHt9MfBbTk7EAAAv2E3Rkt3p+mNnw7qWG6xosKbavqkHvJnyMQZuNhsGt+9mUZ1bqp/bz+qd/7vgtI3DmitKTTxjQa/ZZxRclahnl++R/EpOerTKkDTo8PVoQlXmwcA4HTGGP2UmKk31xzUvoz/1969R0dZ33kcf08mc78lk8zkDoQEEggXi4IVCqgFdL1Vai1x9Wy3Z6v/dNvVtbtVzp5D123tkdNaPVTr9oj2oqsp1JVuj1fOoS7ITaoVjIIQcoMQkpBMkkkmmUwy+8eEgZi4gAWfJPN5nTNnnrnlfH0i+cz3+f2e39NDWdDNQzeUsaAwg4wMp86xlnNyWMx899oZrCrN4okddTyzp5GXJ6tVAgAAEVZJREFU9jfz94uKuG1+ni4nNMmpIZNRwv0xnt3TwAvvHseWnsaDK2dw69xcXWFeRETkE/7cGOKJ7XUcONFFUYadH91YzoqygDJTPpNcr53/uKGcygUFPLWjjsfeOsp//fkY//DFKdwyJ5d0s1blnIzUkEnS4FCcP1Y38+SOOtp7B7ixIod//NI0st06wVRERORsB5q6+OXOenbXdxB0W3lw5QxuqcjRF2a5KCpyPWz42txkw//jrUf47b5j3H3VVFaVB3XqyCSjhkyIx+PsrQ+xYXsth1rCzM/38rPVc5id6zG6NBERkXEjHo/zTkOIZ/c0sK+xE589nX9aPp2vaUqZXCKXF2Ww8Y75vF3bzpM76lj36iF+ubOev1tUxE2zc7DqOnaTghqyFPfusRBPvV3Pe8c6yfXY+NGN5aws05K8IiIipw3F42yvaefZPQ1UN3cTcFu57+rp3Do3D6dVjZhcWiaTiS9Nz2JxsZ/tNad4Zk8jP37zME/vqufOywtZPU//H050ashS1IGmLv5zZx176kNku6z8y7Wl3Do3V0daREREhkUGBnnlw5O8+O5x6toj5PvsPLiilJsqlJfy+UszmVhems2ykqzESO3eRh576yjP7Gng1rl53H5ZHrlaLn9CUkOWQobicXbWtvObd47x3rFOMhwW7l0+Xav3iIiInKW5q4/fvdfEywea6e6PMSsnsWriyjKduyPGM5lMLJqayaKpmRxo6uK3+47x3L5Gnt/XyDUzslnzhQLmF3g122kCUUOWAqKxIV4/2MJz+45x9FQvOR6bplqIiIicZXAozu66Dl4+cILtNacAuGZGNpULCpiXry+3Mj7Nzfey/pbZnOjqY9PwQYStH7dRHnSzen4eq8oCuG36uj/e6Tc0iR0LRfjv/c38zwfNdEQGKM128e9/U8aqsoBWgRIREQGOd0b4wwcn+eMHzbSEo2Q6LNx5RSG3X5av6V8yYeR57Xx3+XTuXjyVVz88ye/+0sSP3zzMz7bVsKIswFfm5GrUbBxTQzbJxAaHeLu2g9+/38Tuug5MJlhWksVt8/O4cmqm/iGKiEjKC/fH+NORNl79sIW9DSFMwFXFmdx/bSlLp/ux6KClTFAOi5mvzs9n9bw8qpu72XKgmTcPtfLH6pNMzXRw85xcVpUHyNPBhnFFDdkkEI/H+bC5m1c/auGNg610RAYIuK1866opfGVuHjkeXUdMRERSW9/AIDuOtvP6wRZ21rYTHYyT77Vxz+Kp3FyRo9EwmVRMJhNz8rzMyfPyz9eUsPVQK3/4oJmfb6/l59trmZvnZVV5gBUzs3W92XFADdkEVnuql60ft/LaRy00dESwmk0sK8ni+llBlhT7NS1RRERSWndfjF117bx15BQ7jrbTOzBIlsvKV+fns6oswJw8j2aOyKTnsJi5eU4uN8/J5XhnhK2H2njjYAs/3VbDo9tqWFDk45rSbJaWZJHv04EJI6ghm0CGhkfCth0+xVtH2qjviGACLi/y8Y2FRVw7M1snboqISEo73hlhe007/1tzinePdTI4FCfTYWFleYDrygMsKMzArJUSJUUV+Bx8Y1ER31hURN2pXt481Mqbh1r5ybYafrKthtJsF0tL/CydnkVFnoc0HbD4XOjb+zjXGRlgX2OI3XUdvF3bTms4ijnNxOWFPtYsKGB5SRZBTUkUEZEUFe6P8efGEO80hNhbH6K2vReA4iwnd11RyLKSLCpyPWrCRD5hWpaTuxdP5e7FU2noiLC95hTbj57iN3sbeXZPI36nhUVTM1k4JYNFUzI0rfcSUkM2zvTHhqhu7mJPfYg9dR18dLKboTi4rGaunJrJ8tIsvjTdj9duMbpUERGRz11vdJDq5i72NSSasOrmRE7a09O4rNDHV+bmsqwki6JMh9GlikwYUzId3HlFIXdeUUhX3wA7azvYcfQUe+s7eO2jFgCKMuzJBm1+vlfnnl1EasgMFuod4P2mLvY3dfKX4118dLKbgcE4ZhNU5Hn51hencuW0TGbnenQxShERSSnxeJwTXf3sb+pK3g63hhmKg9kEs3O9fPPKKSycksHcPC/WdJ07LfLX8totXD8ryPWzgsTjcWraetnb0ME7DSFe/bCF379/AoB8n515+d7krSTbpe+qn5Eass9RTzTGxy09HGwJc+hkN9XN3dS1RwBITzMxO9dD5RcKmF/g4/Iin84HExGRlHG6+TrUEuZQS5iPW8J8dDJMW08UAKfFTEWeh29eOYW5+V7m53uVkyKXmMlkojTgojTg4m8vLyQ2OMRHJ8PJAyTvNISSI2hOi5myoIuZQTflOW7Kgm6K/U4tMnce9JfsEhgcinOiq4/aU73UnurlUEuYgy1hGjsixIffk+2yUp7j5sbZOVxW4GNWrgebjuyJiEgK6IwMUNfeS117L0eHc/Ljlh66+2MApJlgqt+ZGPnS0XeRcSPdnMbcfC9z873cyehR7IMnw2w50EzVe0MAWM0mSrITTdr0LCfFWU6K/U5yPDatcHoWNWR/he6+GE2dfRzrjCSbr9r2Xho6IvTHhpLvy/XYKM9xc8PsIOVBD2VBl+bdiojIpNYfG+JEVx9NnX3UtfdS3x6htr2X+vZe2nsHku+zpadRmu1iRVk2ZcHEUfXSbBd2i9nA6kXkfJhMJvJ9dvJ9dq6fFQQSAxMNHZHkaPfBljB/OtzGlgOx5OccljSm+RMN2jS/k6IMBwUZdgp89pRcJ0EN2aeIx+OE+wdpCffTGu6nqbOP48O3k+EoDe29dPXFRnwm32ujOMvFoimZFGc5KM5yUex34rFrN4uIyOTSE43RGo4mM7Kps4+2SIy6tp7E9vBUw9O89nSm+Z0snZ7FVL8j+UUsz2vXCogik4g5zZQYCctyJps0gI7eKLXtvclBjLr2XvY1hHjlw5YRn3fbzBT4HBT47EzP8ZBlMxP02Ai6rQQ9NjIclkm3HH/KdQoDg0OEIgPJW0fvAG09UVq6o7T19NMSjtIW7qc1HKXvrFEuAIvZRJ7XzrRsF2UBFwU++/DNwRS/A4eO5omIyAQWjQ3R2TdAZyRGKDJAe290uOlKZGRbz/B2OErvwOCIz6aZIM9nJ9dj46ppmcmj5vleO1P8DjIdFk1REklhmU4rmU4rCwozRjzfE03MODseOjP4cbwzQk1bDztq24l+4vt4epqJgNtKwJ1o0gJuG9kuK5lOS+LmsJDhtJDpsOKwpE2Ivzvn1ZA9/PDDvP/++5hMJtauXcu8efOSr+3cuZNHH30Us9nMsmXL+Pa3v33Jij1tYHCIcH+Mnugg4f4Y4f7BkY+jMbr7BglFooSGQ+X0rSc6OObPtKWnke2yEnRbmZXjYWmJlaDblvyF5/vsBNxW0kwmMjKchEK9l/y/U0RE5ELE43H6Y0PJPOyJDtITTeTk6ftwf4yuvpHZ2NkXo/M8MjLgtjIz4GZJsZVsl5Vsd+K5fJ+dHLeN7Cy38lFELojLms6MgJsZAfeo17xeB0eOh2gNJwZNRtx39/Nxaw9v17YTGRga4ycn/nZlOM40aT57Oh5bOh57Om5rOu7Tj21mPLZ03Ge99nmu2nrOhmzv3r3U19dTVVVFTU0Na9eupaqqKvn6D3/4QzZu3EhOTg533XUX1113HaWlpRe1yN+9d5yq95qS4dIfG3unn23EL8BhoTDDTsbwdqbTktz2OSwEXFa89vQJ0UGLiIicFu6P8a9/+JCmzr5kRsaG4uf8nMtqxudIfDnJcFiY5ncOZ2L6mXy0J/Iy4LbisSkjReTzl5ZmSkxX9Nio+JT3xONxIgNDdESidPQmZr91RAYIDd+fvX08FKG7f5Du/hiD5/hbaTWbcFjMuKxmCjMcPLp6ziVbgO+cDdmuXbtYsWIFACUlJXR2dhIOh3G73TQ2NuLz+cjLywNg+fLl7Nq166I3ZNkuK2VBN26bGZc1HbfNjNuajmv43m07ezvxHl2LREREJrv0tMQJ9VkuKy6rOZGH53Fv0TLUIjJJmEwmnFYzTquDAt/5XRA+Ho/TFxuiuy9Gd3+McP/p+0Sz1t0XoyeaOMjVGx3EY0u/pOe6nrMha2tro6LiTE/q9/tpbW3F7XbT2tqK3+8f8VpjY+NFL/LamQGunRm46D9XRERkIrNbzPzbqplGlyEiMqGYTInRL4clsWCI0S54UY94/NxTIT7J7baRnj55Frwwm9PIyHAaXca4on0ymvbJaNonY9N+SV3Kx8lP+2Rs2i+jaZ+Mlir75JwNWTAYpK2tLfm4paWFQCAw5msnT54kGAyO+hnhcP/FqHXc0KIeo2mfjKZ9Mpr2ydgm034JBDxGlzChKB8nP+2TsWm/jKZ9Mtpk2yeflpHnnES+ZMkSXn/9dQCqq6sJBoO43YlVUAoLCwmHwxw7doxYLMa2bdtYsmTJRSxbRERERERk8jrnCNmCBQuoqKigsrISk8nEunXreOmll/B4PKxcuZIf/OAH3H///QDccMMNFBcXX/KiRUREREREJoPzOofse9/73ojH5eXlye2FCxeOWAZfREREREREzo/WvRURERERETGIGjIRERERERGDqCETERERERExiBoyERERERERg6ghExERERERMYgpHo/HjS5CREREREQkFWmETERERERExCBqyERERERERAyihkxERERERMQgasg+o7a2NhYuXMiePXuMLmVciMVifP/73+eOO+7g61//Ovv27TO6JEM9/PDDrFmzhsrKSvbv3290OePC+vXrWbNmDbfddhtvvPGG0eWMG319faxYsYKXXnrJ6FJELhpl5BnKx5GUj2NTRo4tVTIy3egCJqr169dTVFRkdBnjxpYtW3A4HLzwwgscPnyYBx98kM2bNxtdliH27t1LfX09VVVV1NTUsHbtWqqqqowuy1C7d+/m8OHDVFVV0dHRwerVq1m1apXRZY0Lv/jFL/D5fEaXIXJRKSPPUD6eoXwcmzLy06VKRqoh+wx27dqFy+Vi5syZRpcybtxyyy3cdNNNAPj9fkKhkMEVGWfXrl2sWLECgJKSEjo7OwmHw7jdboMrM87ChQuZN28eAF6vl0gkwuDgIGaz2eDKjFVTU8ORI0e4+uqrjS5F5KJRRo6kfDxD+Tg2ZeTYUikjNWXxAkWjUZ544gnuu+8+o0sZVywWCzabDYBf//rXyfBJRW1tbWRmZiYf+/1+WltbDazIeGazGafTCcDmzZtZtmxZygcNwCOPPMIDDzxgdBkiF40ycjTl4xnKx7EpI8eWShmpEbL/x6ZNm9i0adOI55YtW8btt9+O1+s1qCrjjbVfvvOd77B06VKef/55qqureeqppwyqbvzRpf7O2Lp1K5s3b+aZZ54xuhTDvfzyy1x22WWa1iUTljJyNOXjhVE+jqSMPCPVMlIXhr5AlZWVDA0NAdDQ0IDf7+fxxx9nxowZBldmvE2bNvHaa6/x5JNPJo8GpqINGzYQCASorKwE4Mtf/jJbtmxJ+SkZ27dv5/HHH+fpp58mIyPD6HIMd++999LY2IjZbKa5uRmr1cpDDz3E4sWLjS5N5DNTRo5N+ZigfPx0ysiRUi0jNUJ2gV588cXk9gMPPMDq1atTPmgAGhsbefHFF3nuuedSOmwAlixZwoYNG6isrKS6uppgMJjyYdPd3c369ev51a9+paAZ9thjjyW3N2zYQEFBwaQNGkkdysjRlI9nKB/HpowcLdUyUg2ZXBSbNm0iFApxzz33JJ/buHEjVqvVwKqMsWDBAioqKqisrMRkMrFu3TqjSzLcK6+8QkdHB/fee2/yuUceeYT8/HwDqxIRufSUj2coH8emjBRNWRQRERERETGIVlkUERERERExiBoyERERERERg6ghExERERERMYgaMhEREREREYOoIRMRERERETGIGjIRERERERGDqCETERERERExiBoyERERERERg/wfl/DB2aiuIRQAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "# call regplot on each axes\n", + "import seaborn as sns\n", + "plt.style.use('seaborn-darkgrid')\n", + "fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True, figsize=(15,5))\n", + "sns.lineplot(x=line_x, y=line_y, ax=ax1)\n", + "ax1.plot(x[0], s[0], 'ro')\n", + "ax1.set_title(\"Sigmoid of Weighted Sum\")\n", + "sns.lineplot(x=line_x, y=y_d, ax=ax2) \n", + "ax2.plot(x[0],sx[0],'ro');\n", + "ax2.set_title(\"Sigmoid Derivative of Weighted Sum\");" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "S4_rHECFgckq" + }, + "source": [ + "Look at the derivate graph. The derivative multiplied by the error tells us where to assign blame and update the weights most effective. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "aY4djmVHgckr" + }, + "source": [ + "##### 2nd Error\n", + "Justice hasn't been served yet - tho. We still have neurons to blame. Let's go back another layer. \n", + "\n", + "`self.z2_error = self.o_delta.dot(self.weights2.T)`\n", + "\n", + "__Discussion:__ Why is this shape different?" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "unOJ8HKtgckr", + "outputId": "42f2ed68-8c6c-466d-9df1-562b8f71d552" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[-0.23952061, -0.19347426, 0.1111231 ],\n", + " [-0.1847631 , -0.14924354, 0.08571892],\n", + " [-0.21246478, -0.17161974, 0.09857082]])" + ] + }, + "execution_count": 18, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "nn.o_delta.dot(nn.weights2.T)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "3EaJGaqhgckt" + }, + "source": [ + "##### 2nd Gradient\n", + "For each observation, how much more sigmoid activation from this layer would have pushed us towards the right answer?\n", + "\n", + "`self.z2_delta = self.z2_error * self.sigmoidPrime(self.activated_hidden)`" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "OPf5OrdYgcku", + "outputId": "91d27661-c401-4678-c290-6de149465f82" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[-0.05483346, -0.04408088, 0.01744341],\n", + " [-0.03871267, -0.0311213 , 0.01231513],\n", + " [-0.04559029, -0.03665025, 0.01450301]])" + ] + }, + "execution_count": 19, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "nn.z2_delta" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "colab_type": "code", + "id": "F1PcMJoygckw", + "outputId": "cb9374e9-3d59-4042-f8c8-ef00e2ce62a1" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 20, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X.T.shape == nn.weights1.shape" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "Oo8wACzsgckz", + "outputId": "8b4ec8aa-de30-4cec-fca0-3332e493190e" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.66666667, 1. ],\n", + " [0.33333333, 0.55555556],\n", + " [1. , 0.66666667]])" + ] + }, + "execution_count": 44, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "YvjEuIAvgck1" + }, + "source": [ + "##### Descent\n", + "\n", + "*Discussion:* Input to Hidden Weight Update\n", + "- We multiply the gradient by the inputs. Why?\n", + "- Why do we need to transpose the inputs? " + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "colab_type": "code", + "id": "3vtKK4CVgck1", + "outputId": "1ca86f11-d561-4fc3-9fde-61d1d9b4e1b9" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.66666667, 0.33333333, 1. ],\n", + " [1. , 0.55555556, 0.66666667]])" + ] + }, + "execution_count": 21, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X.T" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "colab_type": "code", + "id": "bOwpGy-mgck3", + "outputId": "9f8abcb2-bfca-4d53-e8f8-63b8f7105766" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[-0.09505015, -0.07641127, 0.03023699],\n", + " [-0.10673402, -0.08580399, 0.03395382]])" + ] + }, + "execution_count": 22, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X.T.dot(nn.z2_delta)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "MCXdRzRrgck6" + }, + "source": [ + "*Discussion:* Hidden to Output Weight Update\n", + "- Why is output the shape 3x1? \n", + "- We multiply the gradient by the inputs. Why?\n", + "- Why do we need to transpose the inputs?" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "V3_1WJgIgck6", + "outputId": "6ec79203-2c90-4702-b50b-9c69dc1099e9" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[0.17103686],\n", + " [0.13129211],\n", + " [0.18053762]])" + ] + }, + "execution_count": 23, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "nn.activated_hidden.T.dot(nn.o_delta)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7iziWWURgck8" + }, + "source": [ + "### Train the Network (fo real this time)" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "colab_type": "code", + "id": "6VPrkCXvgck9", + "outputId": "3fcc3281-5089-4a6f-bd1a-5318758f909e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+---------EPOCH 1---------+\n", + "Input: \n", + " [[0.66666667 1. ]\n", + " [0.33333333 0.55555556]\n", + " [1. 0.66666667]]\n", + "Actual Output: \n", + " [0.00669285 0.00675966 0.00682713 0.00689527 0.00696409 0.00703359\n", + " 0.00710377 0.00717466 0.00724624 0.00731853 0.00739154 0.00746527\n", + " 0.00753973 0.00761493 0.00769088 0.00776757 0.00784502 0.00792324\n", + " 0.00800223 0.00808201 0.00816257 0.00824393 0.00832609 0.00840907\n", + " 0.00849286 0.00857749 0.00866294 0.00874925 0.0088364 0.00892442\n", + " 0.0090133 0.00910306 0.00919371 0.00928525 0.00937769 0.00947104\n", + " 0.00956532 0.00966052 0.00975667 0.00985376 0.0099518 0.01005081\n", + " 0.0101508 0.01025177 0.01035374 0.01045671 0.01056069 0.01066569\n", + " 0.01077173 0.01087881 0.01098694 0.01109614 0.01120641 0.01131776\n", + " 0.0114302 0.01154375 0.01165842 0.01177421 0.01189113 0.0120092\n", + " 0.01212843 0.01224883 0.01237041 0.01249319 0.01261716 0.01274235\n", + " 0.01286876 0.01299642 0.01312532 0.01325548 0.01338692 0.01351964\n", + " 0.01365366 0.01378899 0.01392564 0.01406363 0.01420296 0.01434366\n", + " 0.01448572 0.01462918 0.01477403 0.0149203 0.01506799 0.01521712\n", + " 0.01536771 0.01551976 0.01567329 0.01582831 0.01598485 0.01614291\n", + " 0.0163025 0.01646364 0.01662636 0.01679065 0.01695654 0.01712403\n", + " 0.01729316 0.01746392 0.01763634 0.01781043 0.01798621 0.01816369\n", + " 0.01834289 0.01852382 0.01870651 0.01889096 0.0190772 0.01926523\n", + " 0.01945508 0.01964677 0.01984031 0.02003571 0.020233 0.02043219\n", + " 0.0206333 0.02083634 0.02104135 0.02124832 0.02145729 0.02166827\n", + " 0.02188127 0.02209632 0.02231344 0.02253264 0.02275394 0.02297737\n", + " 0.02320294 0.02343067 0.02366058 0.02389269 0.02412702 0.02436359\n", + " 0.02460243 0.02484354 0.02508696 0.0253327 0.02558079 0.02583124\n", + " 0.02608408 0.02633932 0.02659699 0.02685712 0.02711972 0.02738481\n", + " 0.02765242 0.02792257 0.02819529 0.02847059 0.0287485 0.02902904\n", + " 0.02931223 0.0295981 0.02988668 0.03017798 0.03047203 0.03076886\n", + " 0.03106848 0.03137093 0.03167623 0.0319844 0.03229546 0.03260946\n", + " 0.03292639 0.03324631 0.03356922 0.03389516 0.03422416 0.03455623\n", + " 0.03489141 0.03522972 0.03557119 0.03591585 0.03626372 0.03661483\n", + " 0.03696921 0.03732689 0.03768789 0.03805225 0.03841999 0.03879113\n", + " 0.03916572 0.03954378 0.03992533 0.04031042 0.04069905 0.04109128\n", + " 0.04148712 0.04188661 0.04228977 0.04269664 0.04310725 0.04352163\n", + " 0.04393982 0.04436183 0.0447877 0.04521747 0.04565117 0.04608883\n", + " 0.04653047 0.04697615 0.04742587 0.04787969 0.04833763 0.04879972\n", + " 0.04926601 0.04973651 0.05021127 0.05069032 0.0511737 0.05166144\n", + " 0.05215356 0.05265012 0.05315114 0.05365665 0.0541667 0.05468132\n", + " 0.05520054 0.0557244 0.05625293 0.05678618 0.05732418 0.05786696\n", + " 0.05841456 0.05896701 0.05952437 0.06008665 0.0606539 0.06122616\n", + " 0.06180347 0.06238585 0.06297336 0.06356602 0.06416388 0.06476697\n", + " 0.06537533 0.06598901 0.06660804 0.06723245 0.06786229 0.0684976\n", + " 0.06913842 0.06978478 0.07043673 0.0710943 0.07175754 0.07242649\n", + " 0.07310117 0.07378165 0.07446795 0.07516011 0.07585818 0.0765622\n", + " 0.0772722 0.07798824 0.07871034 0.07943855 0.08017291 0.08091347\n", + " 0.08166026 0.08241332 0.0831727 0.08393843 0.08471057 0.08548914\n", + " 0.08627419 0.08706577 0.08786391 0.08866866 0.08948006 0.09029814\n", + " 0.09112296 0.09195455 0.09279295 0.09363821 0.09449037 0.09534946\n", + " 0.09621554 0.09708864 0.0979688 0.09885607 0.09975049 0.10065209\n", + " 0.10156093 0.10247703 0.10340045 0.10433122 0.10526939 0.10621499\n", + " 0.10716807 0.10812867 0.10909682 0.11007257 0.11105597 0.11204704\n", + " 0.11304583 0.11405238 0.11506673 0.11608892 0.11711899 0.11815698\n", + " 0.11920292 0.12025686 0.12131884 0.12238889 0.12346705 0.12455336\n", + " 0.12564786 0.12675058 0.12786157 0.12898085 0.13010847 0.13124447\n", + " 0.13238887 0.13354172 0.13470305 0.1358729 0.13705129 0.13823827\n", + " 0.13943387 0.14063813 0.14185106 0.14307272 0.14430313 0.14554233\n", + " 0.14679034 0.1480472 0.14931293 0.15058758 0.15187116 0.15316372\n", + " 0.15446527 0.15577584 0.15709547 0.15842418 0.159762 0.16110895\n", + " 0.16246506 0.16383036 0.16520487 0.16658861 0.16798161 0.1693839\n", + " 0.17079548 0.17221639 0.17364665 0.17508627 0.17653527 0.17799369\n", + " 0.17946152 0.18093879 0.18242552 0.18392173 0.18542742 0.18694261\n", + " 0.18846733 0.19000157 0.19154535 0.19309868 0.19466158 0.19623406\n", + " 0.19781611 0.19940776 0.201009 0.20261985 0.2042403 0.20587037\n", + " 0.20751006 0.20915937 0.21081829 0.21248684 0.21416502 0.21585281\n", + " 0.21755022 0.21925725 0.22097389 0.22270014 0.22443599 0.22618143\n", + " 0.22793645 0.22970105 0.23147522 0.23325894 0.2350522 0.23685498\n", + " 0.23866729 0.24048908 0.24232036 0.2441611 0.24601128 0.24787089\n", + " 0.24973989 0.25161828 0.25350602 0.25540308 0.25730945 0.2592251\n", + " 0.26114999 0.2630841 0.2650274 0.26697985 0.26894142 0.27091208\n", + " 0.27289178 0.2748805 0.27687819 0.27888482 0.28090034 0.28292471\n", + " 0.28495789 0.28699984 0.2890505 0.29110983 0.29317778 0.2952543\n", + " 0.29733935 0.29943286 0.30153478 0.30364507 0.30576366 0.3078905\n", + " 0.31002552 0.31216867 0.31431989 0.31647911 0.31864627 0.3208213\n", + " 0.32300414 0.32519473 0.32739298 0.32959884 0.33181223 0.33403307\n", + " 0.3362613 0.33849684 0.34073961 0.34298954 0.34524654 0.34751054\n", + " 0.34978145 0.3520592 0.35434369 0.35663485 0.35893259 0.36123682\n", + " 0.36354746 0.36586441 0.36818758 0.37051689 0.37285223 0.37519353\n", + " 0.37754067 0.37989357 0.38225213 0.38461624 0.38698582 0.38936077\n", + " 0.39174097 0.39412633 0.39651675 0.39891212 0.40131234 0.4037173\n", + " 0.4061269 0.40854102 0.41095957 0.41338242 0.41580948 0.41824062\n", + " 0.42067575 0.42311474 0.42555748 0.42800387 0.43045378 0.4329071\n", + " 0.43536371 0.4378235 0.44028635 0.44275215 0.44522076 0.44769209\n", + " 0.450166 0.45264238 0.45512111 0.45760206 0.46008512 0.46257015\n", + " 0.46505705 0.46754569 0.47003595 0.4725277 0.47502081 0.47751518\n", + " 0.48001066 0.48250714 0.4850045 0.4875026 0.49000133 0.49250056\n", + " 0.49500017 0.49750002 0.5 0.50249998 0.50499983 0.50749944\n", + " 0.50999867 0.5124974 0.5149955 0.51749286 0.51998934 0.52248482\n", + " 0.52497919 0.5274723 0.52996405 0.53245431 0.53494295 0.53742985\n", + " 0.53991488 0.54239794 0.54487889 0.54735762 0.549834 0.55230791\n", + " 0.55477924 0.55724785 0.55971365 0.5621765 0.56463629 0.5670929\n", + " 0.56954622 0.57199613 0.57444252 0.57688526 0.57932425 0.58175938\n", + " 0.58419052 0.58661758 0.58904043 0.59145898 0.5938731 0.5962827\n", + " 0.59868766 0.60108788 0.60348325 0.60587367 0.60825903 0.61063923\n", + " 0.61301418 0.61538376 0.61774787 0.62010643 0.62245933 0.62480647\n", + " 0.62714777 0.62948311 0.63181242 0.63413559 0.63645254 0.63876318\n", + " 0.64106741 0.64336515 0.64565631 0.6479408 0.65021855 0.65248946\n", + " 0.65475346 0.65701046 0.65926039 0.66150316 0.6637387 0.66596693\n", + " 0.66818777 0.67040116 0.67260702 0.67480527 0.67699586 0.6791787\n", + " 0.68135373 0.68352089 0.68568011 0.68783133 0.68997448 0.6921095\n", + " 0.69423634 0.69635493 0.69846522 0.70056714 0.70266065 0.7047457\n", + " 0.70682222 0.70889017 0.7109495 0.71300016 0.71504211 0.71707529\n", + " 0.71909966 0.72111518 0.72312181 0.7251195 0.72710822 0.72908792\n", + " 0.73105858 0.73302015 0.7349726 0.7369159 0.73885001 0.7407749\n", + " 0.74269055 0.74459692 0.74649398 0.74838172 0.75026011 0.75212911\n", + " 0.75398872 0.7558389 0.75767964 0.75951092 0.76133271 0.76314502\n", + " 0.7649478 0.76674106 0.76852478 0.77029895 0.77206355 0.77381857\n", + " 0.77556401 0.77729986 0.77902611 0.78074275 0.78244978 0.78414719\n", + " 0.78583498 0.78751316 0.78918171 0.79084063 0.79248994 0.79412963\n", + " 0.7957597 0.79738015 0.798991 0.80059224 0.80218389 0.80376594\n", + " 0.80533842 0.80690132 0.80845465 0.80999843 0.81153267 0.81305739\n", + " 0.81457258 0.81607827 0.81757448 0.81906121 0.82053848 0.82200631\n", + " 0.82346473 0.82491373 0.82635335 0.82778361 0.82920452 0.8306161\n", + " 0.83201839 0.83341139 0.83479513 0.83616964 0.83753494 0.83889105\n", + " 0.840238 0.84157582 0.84290453 0.84422416 0.84553473 0.84683628\n", + " 0.84812884 0.84941242 0.85068707 0.8519528 0.85320966 0.85445767\n", + " 0.85569687 0.85692728 0.85814894 0.85936187 0.86056613 0.86176173\n", + " 0.86294871 0.8641271 0.86529695 0.86645828 0.86761113 0.86875553\n", + " 0.86989153 0.87101915 0.87213843 0.87324942 0.87435214 0.87544664\n", + " 0.87653295 0.87761111 0.87868116 0.87974314 0.88079708 0.88184302\n", + " 0.88288101 0.88391108 0.88493327 0.88594762 0.88695417 0.88795296\n", + " 0.88894403 0.88992743 0.89090318 0.89187133 0.89283193 0.89378501\n", + " 0.89473061 0.89566878 0.89659955 0.89752297 0.89843907 0.89934791\n", + " 0.90024951 0.90114393 0.9020312 0.90291136 0.90378446 0.90465054\n", + " 0.90550963 0.90636179 0.90720705 0.90804545 0.90887704 0.90970186\n", + " 0.91051994 0.91133134 0.91213609 0.91293423 0.91372581 0.91451086\n", + " 0.91528943 0.91606157 0.9168273 0.91758668 0.91833974 0.91908653\n", + " 0.91982709 0.92056145 0.92128966 0.92201176 0.9227278 0.9234378\n", + " 0.92414182 0.92483989 0.92553205 0.92621835 0.92689883 0.92757351\n", + " 0.92824246 0.9289057 0.92956327 0.93021522 0.93086158 0.9315024\n", + " 0.93213771 0.93276755 0.93339196 0.93401099 0.93462467 0.93523303\n", + " 0.93583612 0.93643398 0.93702664 0.93761415 0.93819653 0.93877384\n", + " 0.9393461 0.93991335 0.94047563 0.94103299 0.94158544 0.94213304\n", + " 0.94267582 0.94321382 0.94374707 0.9442756 0.94479946 0.94531868\n", + " 0.9458333 0.94634335 0.94684886 0.94734988 0.94784644 0.94833856\n", + " 0.9488263 0.94930968 0.94978873 0.95026349 0.95073399 0.95120028\n", + " 0.95166237 0.95212031 0.95257413 0.95302385 0.95346953 0.95391117\n", + " 0.95434883 0.95478253 0.9552123 0.95563817 0.95606018 0.95647837\n", + " 0.95689275 0.95730336 0.95771023 0.95811339 0.95851288 0.95890872\n", + " 0.95930095 0.95968958 0.96007467 0.96045622 0.96083428 0.96120887\n", + " 0.96158001 0.96194775 0.96231211 0.96267311 0.96303079 0.96338517\n", + " 0.96373628 0.96408415 0.96442881 0.96477028 0.96510859 0.96544377\n", + " 0.96577584 0.96610484 0.96643078 0.96675369 0.96707361 0.96739054\n", + " 0.96770454 0.9680156 0.96832377 0.96862907 0.96893152 0.96923114\n", + " 0.96952797 0.96982202 0.97011332 0.9704019 0.97068777 0.97097096\n", + " 0.9712515 0.97152941 0.97180471 0.97207743 0.97234758 0.97261519\n", + " 0.97288028 0.97314288 0.97340301 0.97366068 0.97391592 0.97416876\n", + " 0.97441921 0.9746673 0.97491304 0.97515646 0.97539757 0.97563641\n", + " 0.97587298 0.97610731 0.97633942 0.97656933 0.97679706 0.97702263\n", + " 0.97724606 0.97746736 0.97768656 0.97790368 0.97811873 0.97833173\n", + " 0.97854271 0.97875168 0.97895865 0.97916366 0.9793667 0.97956781\n", + " 0.979767 0.97996429 0.98015969 0.98035323 0.98054492 0.98073477\n", + " 0.9809228 0.98110904 0.98129349 0.98147618 0.98165711 0.98183631\n", + " 0.98201379 0.98218957 0.98236366 0.98253608 0.98270684 0.98287597\n", + " 0.98304346 0.98320935 0.98337364 0.98353636 0.9836975 0.98385709\n", + " 0.98401515 0.98417169 0.98432671 0.98448024 0.98463229 0.98478288\n", + " 0.98493201 0.9850797 0.98522597 0.98537082 0.98551428 0.98565634\n", + " 0.98579704 0.98593637 0.98607436 0.98621101 0.98634634 0.98648036\n", + " 0.98661308 0.98674452 0.98687468 0.98700358 0.98713124 0.98725765\n", + " 0.98738284 0.98750681 0.98762959 0.98775117 0.98787157 0.9879908\n", + " 0.98810887 0.98822579 0.98834158 0.98845625 0.9885698 0.98868224\n", + " 0.98879359 0.98890386 0.98901306 0.98912119 0.98922827 0.98933431\n", + " 0.98943931 0.98954329 0.98964626 0.98974823 0.9898492 0.98994919\n", + " 0.9900482 0.99014624 0.99024333 0.99033948 0.99043468 0.99052896\n", + " 0.99062231 0.99071475 0.99080629 0.99089694 0.9909867 0.99107558\n", + " 0.9911636 0.99125075 0.99133706 0.99142251 0.99150714 0.99159093\n", + " 0.99167391 0.99175607 0.99183743 0.99191799 0.99199777 0.99207676\n", + " 0.99215498 0.99223243 0.99230912 0.99238507 0.99246027 0.99253473\n", + " 0.99260846 0.99268147 0.99275376 0.99282534 0.99289623 0.99296641\n", + " 0.99303591 0.99310473 0.99317287 0.99324034]\n", + "Predicted Output: \n", + " [[0.37771334]\n", + " [0.41477855]\n", + " [0.40348484]]\n", + "Loss: \n", + " 0.16174922917818682\n" + ] + }, + { + "ename": "ValueError", + "evalue": "ignored", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Predicted Output: \\n'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Loss: \\n\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msquare\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[0mo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 68\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, X, y, o)\u001b[0m\n\u001b[1;32m 56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0;31m# z2 error: how much were our output layer weights off\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mz2_error\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mo_delta\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweights2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 60\u001b[0m \u001b[0;31m# z2 delta: how much were the weights off?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mValueError\u001b[0m: shapes (3,1000) and (1,3) not aligned: 1000 (dim 1) != 1 (dim 0)" + ] + } + ], + "source": [ + "# Train my 'net\n", + "nn = NeuralNetwork()\n", + "\n", + "# Number of Epochs / Iterations\n", + "for i in range(5000):\n", + " if (i+1 in [1,2,3,4,5]) or ((i+1) % 500 ==0):\n", + " print('+' + '---' * 3 + f'EPOCH {i+1}' + '---'*3 + '+')\n", + " print('Input: \\n', X)\n", + " print('Actual Output: \\n', y)\n", + " print('Predicted Output: \\n', str(nn.feed_forward(X)))\n", + " print(\"Loss: \\n\", str(np.mean(np.square(y - nn.feed_forward(X)))))\n", + " nn.train(X,y)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "iKUVGoRxgck_" + }, + "source": [ + "## Challenge\n", + "\n", + "In the module project, you will implement backpropagation inside a multi-layer perceptron (aka a feedforward neural network). " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vTqZg-6igclA", + "toc-hr-collapsed": true + }, + "source": [ + "# Stochastic Gradient Descent (Learn)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "0nrm-racgclA" + }, + "source": [ + "## Overview\n", + "\n", + "The What - Stochastic Gradient Descent calculates an approximation of the gradient over the entire dataset by reviewing the predictions of a random sample. \n", + "\n", + "The Why - *Speed*. Calculating the gradient over the entire dataset is extremely expensive computationally. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "ZF7UE-KluPsX" + }, + "source": [ + "## Follow Along\n", + "\n", + "A true Stochastic GD-based implementation from [Welch Labs](https://www.youtube.com/watch?v=bxe2T-V8XRs)" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "P-ChVGikgclD" + }, + "outputs": [], + "source": [ + "class Neural_Network(object):\n", + " def __init__(self): \n", + " #Define Hyperparameters\n", + " self.inputLayerSize = 2\n", + " self.outputLayerSize = 1\n", + " self.hiddenLayerSize = 3\n", + " \n", + " #Weights (parameters)\n", + " self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)\n", + " self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)\n", + " \n", + " def forward(self, X):\n", + " #Propogate inputs though network\n", + " self.z2 = np.dot(X, self.W1)\n", + " self.a2 = self.sigmoid(self.z2)\n", + " self.z3 = np.dot(self.a2, self.W2)\n", + " yHat = self.sigmoid(self.z3) \n", + " return yHat\n", + " \n", + " def sigmoid(self, z):\n", + " #Apply sigmoid activation function to scalar, vector, or matrix\n", + " return 1/(1+np.exp(-z))\n", + " \n", + " def sigmoidPrime(self,z):\n", + " #Gradient of sigmoid\n", + " return np.exp(-z)/((1+np.exp(-z))**2)\n", + " \n", + " def costFunction(self, X, y):\n", + " #Compute cost for given X,y, use weights already stored in class.\n", + " self.yHat = self.forward(X)\n", + " J = 0.5*sum((y-self.yHat)**2)\n", + " return J\n", + " \n", + " def costFunctionPrime(self, X, y):\n", + " #Compute derivative with respect to W and W2 for a given X and y:\n", + " self.yHat = self.forward(X)\n", + " \n", + " delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))\n", + " dJdW2 = np.dot(self.a2.T, delta3)\n", + " \n", + " delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)\n", + " dJdW1 = np.dot(X.T, delta2) \n", + " \n", + " return dJdW1, dJdW2\n", + " \n", + " #Helper Functions for interacting with other classes:\n", + " def getParams(self):\n", + " #Get W1 and W2 unrolled into vector:\n", + " params = np.concatenate((self.W1.ravel(), self.W2.ravel()))\n", + " return params\n", + " \n", + " def setParams(self, params):\n", + " #Set W1 and W2 using single paramater vector.\n", + " W1_start = 0\n", + " W1_end = self.hiddenLayerSize * self.inputLayerSize\n", + " self.W1 = np.reshape(params[W1_start:W1_end], (self.inputLayerSize , self.hiddenLayerSize))\n", + " W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize\n", + " self.W2 = np.reshape(params[W1_end:W2_end], (self.hiddenLayerSize, self.outputLayerSize))\n", + " \n", + " def computeGradients(self, X, y):\n", + " dJdW1, dJdW2 = self.costFunctionPrime(X, y)\n", + " return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "uA9LaTgKr6rP" + }, + "outputs": [], + "source": [ + "from scipy import optimize\n", + "class trainer(object):\n", + " def __init__(self, N):\n", + " #Make Local reference to network:\n", + " self.N = N\n", + " \n", + " def callbackF(self, params):\n", + " self.N.setParams(params)\n", + " self.J.append(self.N.costFunction(self.X, self.y)) \n", + " \n", + " def costFunctionWrapper(self, params, X, y):\n", + " self.N.setParams(params)\n", + " cost = self.N.costFunction(X, y)\n", + " grad = self.N.computeGradients(X,y)\n", + " \n", + " return cost, grad\n", + " \n", + " def train(self, X, y):\n", + " #Make an internal variable for the callback function:\n", + " self.X = X\n", + " self.y = y\n", + "\n", + " #Make empty list to store costs:\n", + " self.J = []\n", + " \n", + " params0 = self.N.getParams()\n", + "\n", + " options = {'maxiter': 200, 'disp' : True}\n", + " _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \\\n", + " args=(X, y), options=options, callback=self.callbackF)\n", + "\n", + " self.N.setParams(_res.x)\n", + " self.optimizationResults = _res" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "g_kHb6Se1u9y" + }, + "outputs": [], + "source": [ + "NN = Neural_Network()" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "hYYVhFf4rn3q" + }, + "outputs": [], + "source": [ + "T = trainer(NN)" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 341 + }, + "colab_type": "code", + "id": "L-gYdVfgrysE", + "outputId": "ca9e7fa2-920a-45c7-cc2c-c56e61e4638e" + }, + "outputs": [ + { + "ename": "ValueError", + "evalue": "ignored", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mT\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 28\u001b[0m \u001b[0moptions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m'maxiter'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;36m200\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'disp'\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m \u001b[0m_res\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moptimize\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mminimize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunctionWrapper\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'BFGS'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbackF\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 31\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_res\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/_minimize.py\u001b[0m in \u001b[0;36mminimize\u001b[0;34m(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)\u001b[0m\n\u001b[1;32m 602\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_cg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 603\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'bfgs'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 604\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_bfgs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 605\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'newton-cg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 606\u001b[0m return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m_minimize_bfgs\u001b[0;34m(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)\u001b[0m\n\u001b[1;32m 1001\u001b[0m \u001b[0mfunc_calls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrap_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1002\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1003\u001b[0;31m \u001b[0mold_fval\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1004\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1005\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mfprime\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36mfunction_wrapper\u001b[0;34m(*wrapper_args)\u001b[0m\n\u001b[1;32m 325\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mwrapper_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 326\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 327\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 329\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, x, *args)\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 65\u001b[0;31m \u001b[0mfg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 66\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjac\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36mcostFunctionWrapper\u001b[0;34m(self, params, X, y)\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetParams\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m \u001b[0mgrad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcomputeGradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36mcomputeGradients\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 60\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcomputeGradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m \u001b[0mdJdW1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdJdW2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcostFunctionPrime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 62\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdJdW1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdJdW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36mcostFunctionPrime\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m 39\u001b[0m \u001b[0mdJdW2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ma2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelta3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 41\u001b[0;31m \u001b[0mdelta2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdelta3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msigmoidPrime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mz2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 42\u001b[0m \u001b[0mdJdW1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelta2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 43\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mdot\u001b[0;34m(*args, **kwargs)\u001b[0m\n", + "\u001b[0;31mValueError\u001b[0m: shapes (3,1000) and (1,3) not aligned: 1000 (dim 1) != 1 (dim 0)" + ] + } + ], + "source": [ + "T.train(X,y)" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + }, + "colab_type": "code", + "id": "Jyv_L8Z2sKOA", + "outputId": "46e5f4d5-0aaf-4bfb-9d27-e5f2ee39fbd4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted Output: \n", + "[[0.19042135]\n", + " [0.21647809]\n", + " [0.19075219]]\n", + "Loss: \n", + "0.24166109849505304\n" + ] + } + ], + "source": [ + "print(\"Predicted Output: \\n\" + str(NN.forward(X))) \n", + "print(\"Loss: \\n\" + str(np.mean(np.square(y - NN.forward(X))))) # mean sum squared loss" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 361 + }, + "colab_type": "code", + "id": "Gtf9WI9FtGPk", + "outputId": "d062b2a3-5a92-403e-8ce0-c070aa79907b" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3df5zdVX3n8dd7fibMDIFk7iCSYEImiAFrcGOsFbsKpcRu12AXJbRa2tKiu7Drj91W6D7WWh5mV/pDtFvQxYJSioY8UNepZaHagFar+QFEJEBkSNBEKJlAIJkk8/uzf3zPJDeXOz8yc+/cmbnv58P7uN/v+Z5z7vl+H3E+nO/5fs9RRGBmZjZZNZVugJmZzQ4OKGZmVhIOKGZmVhIOKGZmVhIOKGZmVhJ1lW5AJbW2tsbixYsr3QwzsxnloYce2hcRucL0qg4oixcvZuvWrZVuhpnZjCLpp8XSfcvLzMxKwgHFzMxKwgHFzMxKwgHFzMxKwgHFzMxKwgHFzMxKoqwBRdJqSTskdUq6rsjxRkl3p+ObJC3OO3Z9St8h6ZKCcrWSHpH0zby0JamOp1KdDeU8NzMzO17ZAoqkWuBm4J3AcuAKScsLsl0F7I+IduAm4MZUdjmwFjgXWA3ckuob9iHgiYK6bgRuiohlwP5Ud1lsfPJ5bnmws1zVm5nNSOXsoawCOiNiZ0T0AeuBNQV51gB3pO17gIskKaWvj4jeiNgFdKb6kLQQ+HfA3wxXkspcmOog1XlpWc4K+O5P9nHLA0+Xq3ozsxmpnAHlDGB33v6elFY0T0QMAC8DC8Yo+xngj4ChvOMLgJdSHSP9FgCSrpa0VdLWrq6uEz0nAHItjXT3DnCkb3BC5c3MZqNyBhQVSStcHnKkPEXTJf06sDciHprAb2WJEbdGxMqIWJnLvWIqmnHJNTcCsK+7d0Llzcxmo3IGlD3Aorz9hcCzI+WRVAfMA14cpexbgXdJeobsFtqFkv4O2AeckuoY6bdKprUlG+/vckAxMzuqnAFlC7AsPX3VQDbI3lGQpwO4Mm1fBmyMbJH7DmBtegpsCbAM2BwR10fEwohYnOrbGBHvS2UeSHWQ6vxGuU6sdbiHctABxcxsWNkCShrPuBa4n+yJrA0RsV3SDZLelbLdBiyQ1Al8FLguld0ObAAeB+4DromIsQYsPgZ8NNW1INVdFrmWLKC4h2JmdkxZp6+PiHuBewvSPp633QO8Z4Sy64B1o9T9IPBg3v5O0pNg5bagabiH0jcVP2dmNiP4TfkJaKir4ZST6j0ob2aWxwFlglqbG+nyGIqZ2VEOKBPU2tzgHoqZWR4HlAnKtcxxQDEzy+OAMkGtzQ2+5WVmlscBZYJamxs51DfI4b6BsTObmVUBB5QJGn4XxY8Om5llHFAmaHg+L7/caGaWcUCZoKM9FAcUMzPAAWXChufz8sC8mVnGAWWCFjRnMw67h2JmlnFAmaD62hpO9fQrZmZHOaBMgqdfMTM7xgFlElqbG9nX7ceGzczAAWVSci2NvuVlZpaUNaBIWi1ph6ROSdcVOd4o6e50fJOkxXnHrk/pOyRdktLmSNos6UeStkv607z8X5K0S9K29FlRznMD3/IyM8tXtgW2JNUCNwMXk60Rv0VSR0Q8npftKmB/RLRLWgvcCFwuaTnZEr/nAq8Gvi3pbKAXuDAiuiXVA9+T9P8i4oepvj+MiHvKdU6Fci2NHO4b5FDvAE2NZV2rzMxs2itnD2UV0BkROyOiD1gPrCnIswa4I23fA1wkSSl9fUT0RsQuoBNYFZnulL8+faKM5zCqVj86bGZ2VDkDyhnA7rz9PSmtaJ60Bv3LZOvBj1hWUq2kbcBe4FsRsSkv3zpJj0q6SVJjKU+mmFa/LW9mdlQ5A4qKpBX2JkbKM2LZiBiMiBXAQmCVpPPS8euBc4A3AfOBjxVtlHS1pK2StnZ1dY19FqM4Op+XJ4g0MytrQNkDLMrbXwg8O1IeSXXAPODF8ZSNiJeAB4HVaf+5dEusF/gi2S23V4iIWyNiZUSszOVyEzuzZHg+L08QaWZW3oCyBVgmaYmkBrJB9o6CPB3AlWn7MmBjRERKX5ueAlsCLAM2S8pJOgVA0lzgV4An0/7p6VvApcBjZTw3AOY3pTEUP+llZla+p7wiYkDStcD9QC1we0Rsl3QDsDUiOoDbgDsldZL1TNamstslbQAeBwaAayJiMAWNO9ITZDXAhoj4ZvrJuyTlyG6XbQM+WK5zG1ZfW8P8Jq8tb2YGZQwoABFxL3BvQdrH87Z7gPeMUHYdsK4g7VHg/BHyXzjZ9k6ElwI2M8v4TflJ8tvyZmYZB5RJ8nxeZmYZB5RJ8vQrZmYZB5RJyrU0cqQ/m37FzKyaOaBMkpcCNjPLOKBMkufzMjPLOKBMUs7zeZmZAQ4ok5bzLS8zM8ABZdLmNzUgQZcfHTazKueAMkl1tTXMP8nTr5iZOaCUgN9FMTNzQCkJT79iZuaAUhKtzb7lZWbmgFICw7e8sqVczMyqkwNKCeRaGunpH+JQ32Clm2JmVjEOKCXg6VfMzMocUCStlrRDUqek64ocb5R0dzq+SdLivGPXp/Qdki5JaXMkbZb0I0nbJf1pXv4lqY6nUp0N5Ty3fH5b3sysjAElLdN7M/BOYDlwhaTlBdmuAvZHRDtwE3BjKrucbDngc4HVwC2pvl7gwoh4A7ACWC3pF1NdNwI3RcQyYH+qe0oM91C8tryZVbNy9lBWAZ0RsTMi+oD1wJqCPGuAO9L2PcBFkpTS10dEb0TsAjqBVZHpTvnr0ydSmQtTHaQ6Ly3XiRVqbck6Q13uoZhZFStnQDkD2J23vyelFc0TEQPAy8CC0cpKqpW0DdgLfCsiNqUyL6U6RvotUvmrJW2VtLWrq2sSp3fMgqZGauQeiplVt3IGFBVJK3yudqQ8I5aNiMGIWAEsBFZJOm+cv0Uqf2tErIyIlblcbsTGn4jaGjG/qcHzeZlZVStnQNkDLMrbXwg8O1IeSXXAPODF8ZSNiJeAB8nGWPYBp6Q6RvqtsvL0K2ZW7coZULYAy9LTVw1kg+wdBXk6gCvT9mXAxsjeDuwA1qanwJYAy4DNknKSTgGQNBf4FeDJVOaBVAepzm+U8dxewdOvmFm1qxs7y8RExICka4H7gVrg9ojYLukGYGtEdAC3AXdK6iTrmaxNZbdL2gA8DgwA10TEoKTTgTvSE181wIaI+Gb6yY8B6yV9Engk1T1lWpsb2bXv0FT+pJnZtFK2gAIQEfcC9xakfTxvuwd4zwhl1wHrCtIeBc4fIf9OsifLKiLXcmz6leyhMzOz6uI35UuktbmB3oEhunsHxs5sZjYLOaCUiKdfMbNq54BSIsemX/Gjw2ZWnRxQSuTo9Ct+0svMqpQDSon4lpeZVTsHlBKZ39SQTb/iHoqZVSkHlBLJpl/xy41mVr0cUEpo+F0UM7Nq5IBSQq3NniDSzKqXA0oJ5ZobPYW9mVUtB5QSyrU00tWdTb9iZlZtHFBKqLW5kb6BIQ56+hUzq0IOKCV0dClg3/YysyrkgFJCueY5gJcCNrPq5IBSQsM9FM/nZWbVyAGlhI5Nv9JT4ZaYmU29sgYUSasl7ZDUKem6IscbJd2djm+StDjv2PUpfYekS1LaIkkPSHpC0nZJH8rL/wlJP5e0LX1+rZznVsypJzVQWyP3UMysKpVtxca0TO/NwMXAHmCLpI6IeDwv21XA/ohol7QWuBG4XNJysuWAzwVeDXxb0tlkywH/14h4WFIL8JCkb+XVeVNE/EW5zmks2fQrDZ5+xcyqUjl7KKuAzojYGRF9wHpgTUGeNcAdafse4CJl6+euAdZHRG9E7AI6gVUR8VxEPAwQEQeBJ4AzyngOJyzX7OlXzKw6lTOgnAHsztvfwyv/+B/NExEDwMvAgvGUTbfHzgc25SVfK+lRSbdLOrVYoyRdLWmrpK1dXV0nek5jam3xBJFmVp3KGVBUJK3wFfKR8oxaVlIz8FXgwxFxICV/DlgKrACeA/6yWKMi4taIWBkRK3O53OhnMAGtzQ0eQzGzqlTOgLIHWJS3vxB4dqQ8kuqAecCLo5WVVE8WTO6KiK8NZ4iI5yNiMCKGgC+Q3XKbcsMzDnv6FTOrNuUMKFuAZZKWSGogG2TvKMjTAVyZti8DNkb2l7gDWJueAlsCLAM2p/GV24AnIuLT+RVJOj1v993AYyU/o3HINTfSNzjEgR5Pv2Jm1aVsT3lFxICka4H7gVrg9ojYLukGYGtEdJAFhzsldZL1TNamstslbQAeJ3uy65qIGJR0AfB+4MeStqWf+uOIuBf4M0kryG6NPQN8oFznNpr8pYDnza2vRBPMzCqibAEFIP2hv7cg7eN52z3Ae0Youw5YV5D2PYqPrxAR759se0sh15IFlH3dvbS3NVe4NWZmU8dvypfYcA/FT3qZWbVxQCmx4R6K30Uxs2rjgFJip8ytT9OvOKCYWXVxQCmxmhqxoKmBfQf9LoqZVRcHlDIYXgrYzKyaOKCUQWuzp18xs+rjgFIGrc2NXrXRzKqOA0oZ5Foa2dfd5+lXzKyqOKCUQWtzQzb9yhFPv2Jm1cMBpQyOvovS7aWAzax6jCugSLpzPGmWyR2dz8uPDptZ9RhvD+Xc/J20vO+/KX1zZofWFk+/YmbVZ9SAIul6SQeBX5B0IH0OAnuBb0xJC2egXLOnXzGz6jNqQImI/xURLcCfR8TJ6dMSEQsi4vopauOMM29uPXWefsXMqsx4b3l9U1ITgKT3Sfq0pNeUsV0zWk2NyLU08vwBBxQzqx7jDSifAw5LegPwR8BPgb8dq5Ck1ZJ2SOqUdF2R442S7k7HN0lanHfs+pS+Q9IlKW2RpAckPSFpu6QP5eWfL+lbkp5K36eO89zKYvGCJnbu665kE8zMptR4A8pAWpp3DfDZiPgs0DJagTRwfzPwTmA5cIWk5QXZrgL2R0Q7cBNwYyq7nGz1xnOB1cAtqb4B4L9GxOuAXwSuyavzOuCfImIZ8E9pv2KWtjXx9N5uv9xoZlVjvAHloKTryZbf/Yf0x32s9W1XAZ0RsTMi+oD1ZAEp3xrgjrR9D3BRWjd+DbA+InojYhfQCayKiOci4mGAiDgIPAGcUaSuO4BLx3luZdGea+ZAz4AniTSzqjHegHI50Av8XkT8K9kf8T8fo8wZwO68/T0c++P/ijwRMQC8DCwYT9l0e+x8YFNKOi0inkt1PQe0FWuUpKslbZW0taura4xTmLilafnfp/ceKttvmJlNJ+MKKCmI3AXMk/TrQE9EjDWGUmzt98L7PyPlGbWspGbgq8CHI+LAGO04vpKIWyNiZUSszOVyJ1L0hAyvJ9/Z5XEUM6sO431T/r3AZuA9wHuBTZIuG6PYHmBR3v5C4NmR8kiqA+YBL45WVlI9WTC5KyK+lpfneUmnpzynk70rUzGvOnkOTQ21PL3XAcXMqsN4b3n9d+BNEXFlRPw22fjI/xijzBZgmaQlkhrIBtk7CvJ0AFem7cuAjWnwvwNYm54CWwIsAzan8ZXbgCci4tOj1HUlFX7xUhJL25p52j0UM6sS4w0oNRGR/1/8L4xVNo2JXAvcTzZ4viEitku6QdK7UrbbgAWSOoGPkp7MiojtwAbgceA+4JqIGATeSvZgwIWStqXPr6W6PgVcLOkp4OK0X1FLc83uoZhZ1agbZ777JN0PfCXtXw7cO1ahiLi3MF9EfDxvu4fsNlqxsuuAdQVp36P4+AoR8QJw0Vhtmkrtbc18/ZGfc6h3gKbG8V5qM7OZadS/cpLayZ6e+kNJvwFcQPYH/Qdkg/Q2iqW5JgB2dh3i9QvnVbg1ZmblNdYtr88ABwEi4msR8dGI+AhZr+Mz5W7cTHfsSa+DFW6JmVn5jRVQFkfEo4WJEbEVWFyWFs0iZ85vorZGfhfFzKrCWAFlzijH5payIbNRQ10Nr1lwEp0emDezKjBWQNki6Q8KEyVdBTxUnibNLktzfnTYzKrDWI8efRj4uqTf4lgAWQk0AO8uZ8Nmi/a2Zh7csZeBwSHqasf7lLaZ2cwzakCJiOeBX5L0DuC8lPwPEbGx7C2bJZbmmukfDH724mHOyjVXujlmZmUzrpcjIuIB4IEyt2VWGn50+OmuQw4oZjar+R5MmQ3POuyBeTOb7RxQyuzkOfW0tTR6YN7MZj0HlCnQ3tbsHoqZzXoOKFNg+NFhLwdsZrOZA8oUaG9r5mDPAF0HvRywmc1eDihTYGnOqzea2ezngDIF2o+uL++AYmazlwPKFDjt5EaaG+t4usuTRJrZ7FXWgCJptaQdkjolXVfkeKOku9PxTZIW5x27PqXvkHRJXvrtkvZKeqygrk9I+nmRlRwrThJLc01+0svMZrWyBRRJtcDNwDuB5cAVkpYXZLsK2B8R7cBNwI2p7HKyNejPBVYDt6T6AL6U0oq5KSJWpM+YK0pOJU8SaWazXTl7KKuAzojYGRF9wHpgTUGeNcAdafse4CJJSunrI6I3InYBnak+IuK7wItlbHdZLG1r5rmXe+juHah0U8zMyqKcAeUMYHfe/p6UVjRPRAwALwMLxlm2mGslPZpui51aLIOkqyVtlbS1q6trfGdSAsNPeu10L8XMZqlyBhQVSSt8s2+kPOMpW+hzwFJgBfAc8JfFMkXErRGxMiJW5nK5MaosnXbP6WVms1w5A8oeYFHe/kLg2ZHySKoD5pHdzhpP2eNExPMRMRgRQ8AXSLfIpovXLDiJuhp5HMXMZq1yBpQtwDJJSyQ1kA2ydxTk6QCuTNuXARsjm5+kA1ibngJbAiwDNo/2Y5JOz9t9N/DYSHkrob62hjO9HLCZzWLjWg9lIiJiQNK1wP1ALXB7RGyXdAOwNSI6gNuAOyV1kvVM1qay2yVtAB4HBoBrImIQQNJXgLcDrZL2AH8SEbcBfyZpBdmtsWeAD5Tr3CaqPdfsd1HMbNYqW0ABSI/u3luQ9vG87R7gPSOUXQesK5J+xQj53z+pxk6BpW3NbHxyL/2DQ9R7OWAzm2X8V20KteeaGRjKlgM2M5ttHFCm0FLP6WVms5gDyhQaXl/esw6b2WzkgDKFWubUc9rJjTy91wPzZjb7OKBMsfa2ZvdQzGxWckCZYktzzezc6+WAzWz2cUCZYu1tzRzsHWCvlwM2s1nGAWWKDU8S6Se9zGy2cUCZYkcnifQ4ipnNMg4oU6ytJS0H7B6Kmc0yDihTTBJL/aSXmc1CDigVsDTX5HdRzGzWcUCpgPa2Zv71QA8He/or3RQzs5JxQKmAY8sBu5diZrOHA0oFDAcUL7ZlZrOJA0oFeDlgM5uNyhpQJK2WtENSp6TrihxvlHR3Or5J0uK8Y9en9B2SLslLv13SXkmPFdQ1X9K3JD2Vvk8t57lNRn1tDa/xcsBmNsuULaBIqgVuBt4JLAeukLS8INtVwP6IaAduAm5MZZeTLQd8LrAauCXVB/CllFboOuCfImIZ8E9pf9pqb2t2D8XMZpVy9lBWAZ0RsTMi+oD1wJqCPGuAO9L2PcBFkpTS10dEb0TsAjpTfUTEd8nWny+UX9cdwKWlPJlSW5pr5qcvHKZ/cKjSTTEzK4lyBpQzgN15+3tSWtE8ETEAvAwsGGfZQqdFxHOprueAtmKZJF0taaukrV1dXeM8ldJrb8uWA/7pC14O2Mxmh3IGFBVJK5yzfaQ84yk7IRFxa0SsjIiVuVyuFFVOyGtf1QLAPQ/tqVgbzMxKqZwBZQ+wKG9/IfDsSHkk1QHzyG5njadsoeclnZ7qOh3YO+GWT4Hlp5/M2jct4vPfeZqvP+KgYmYzXzkDyhZgmaQlkhrIBtk7CvJ0AFem7cuAjZGtPNUBrE1PgS0BlgGbx/i9/LquBL5RgnMoG0ncsOY83nLWAj52z4/Z+kyxYSEzs5mjbAEljYlcC9wPPAFsiIjtkm6Q9K6U7TZggaRO4KOkJ7MiYjuwAXgcuA+4JiIGASR9BfgB8FpJeyRdler6FHCxpKeAi9P+tNZQV8Pn3vdGzjh1Lh+48yF2v+jxFDObuVTNS9GuXLkytm7dWulmsLOrm3ff8i+0tTTy1f/0S5w8p77STTIzG5GkhyJiZWG635SfBs7KNfO533oju/Yd4j9/+REG/Cixmc1ADijTxC+1t/LJS8/jOz/p4pP/8ESlm2NmdsLqKt0AO2btqjN5uqubL/zzLpbmmnj/WxZXuklmZuPmgDLNXPfO17Gz6xCf+PvHec2CJn757Mq9K2NmdiJ8y2uaqa0Rn73ifJa1NXPNXQ/z1PMHK90kM7Nx8VNe0+Apr2J+/tIR1vz19zncN0BrcyONdTXMqa9lTn0NjXXHvhvrazirtYm3v7aNc199MtlUaGZm5TPSU14OKNM0oAA88dwB7vzhTzncO0DvwBA9/YP09A/RO3D8989fOgLAaSc38o7XtvGOc9q4oL2Vpkbf0TSz0nNAKWK6B5Tx6jrYy4M79vLAjr3880/2cbB3gIbaGt581nze8do2LjynjcWtTZVuppnNEg4oRcyWgJKvf3CILc+8yANP7mXjk3t5Oq1bf+E5bVz3znM4+7SWCrfQzGY6B5QiZmNAKfSzFw7z948+y+e/8zSHegd478pFfOTisznt5DmVbpqZzVAOKEVUQ0AZtv9QH3/9QCd3/uCn1NTA719wFh/4t2fR4mlezOwEOaAUUU0BZdjuFw/zF/+4g29se5b5TQ186KJlXLHqTBrq/AS5mY2P5/IyABbNP4nPrj2fv7/2Al57Wgt/0rGdX73pO9z32HOVbpqZzXAOKFXq9Qvn8eU/eDNf/N030VhXywf/7mH+8h93UM09VjObHL+oUMUk8Y7XtvG29lb++9cf439v7OTlI/184t+fS02NX5A0sxNT1h6KpNWSdkjqlHRdkeONku5OxzdJWpx37PqUvkPSJWPVKelLknZJ2pY+K8p5brNJXW0Nn/oPr+cDv3wWf/uDn/KRDdvo9xT6ZnaCytZDkVQL3Ey2euIeYIukjoh4PC/bVcD+iGiXtBa4Ebhc0nKyJYPPBV4NfFvS2anMaHX+YUTcU65zms0kcf2vvY55J9XzZ/ft4GDPADf/5huZ21Bb6aaZ2QxRzh7KKqAzInZGRB+wHlhTkGcNcEfavge4SNlkVGuA9RHRGxG7gM5U33jqtEn4T29vZ927z+OBHXu58vbNHOjpr3STzGyGKGdAOQPYnbe/J6UVzZPWoH8ZWDBK2bHqXCfpUUk3SWos1ihJV0vaKmlrV1fXiZ9VFfitN7+Gv1p7Po/s3s/a//ND9nX3VrpJZjYDlDOgFBvVLXyEaKQ8J5oOcD1wDvAmYD7wsWKNiohbI2JlRKzM5bzWyEj+/RtezRd+eyU793Xz3s//gD37D1e6SWY2zZUzoOwBFuXtLwSeHSmPpDpgHvDiKGVHrDMinotML/BFsttjNglvf20bf3fVm9nX3ct7Pv8DOvd6bRYzG1k5A8oWYJmkJZIayAbZOwrydABXpu3LgI2RvQjRAaxNT4EtAZYBm0erU9Lp6VvApcBjZTy3qrFy8Xzu/sBb6B8MrvjCJp5NU+WbmRUqW0BJYyLXAvcDTwAbImK7pBskvStluw1YIKkT+ChwXSq7HdgAPA7cB1wTEYMj1ZnqukvSj4EfA63AJ8t1btXmdaefzJf/4M309A3ye1/awkEP1JtZEZ7Lq8rm8pqMf36qi9/54hYuaG/ltitXUlfriRbMqpHn8rJJe9uyHJ+89Dy+85MuPvH32z1Ni5kdx1Ov2Am5YtWZPPPCIf7Pd3ayeEETv/+2syrdJDObJhxQ7IR97JJz+NkLh1l37xOcOf8kfvXcV1W6SWY2DfiWl52wmhrx6feu4BcWnsKH1m/jx3ternSTzGwacECxCZnbUMvf/PZK5jc18Ht3bOHnfpzYrOo5oNiE5Voa+eLvvomevkGu8uPEZlXPAcUm5ezTWrjlfW/kqb3dXPvlRxjwtPdmVcsBxSYt/3Hif/dX32PD1t309A9WullmNsUcUKwkrlh1Jn91xflI8Ef3PMoFN27kM9/+iWcqNqsiflPeb8qXVETwL0+/wG3f28XGJ/fSUFfDu1ecwVVvW8LZp7VUunlmVgIjvSnv91CspCTx1vZW3treSufebr74/V189eE93L11N29b1srvvnUxv3jWAk5q8D89s9nGPRT3UMpu/6E+vrz5Z9zxL8+w92AvtTXi7NNaWLFoHisWncKKRafS3tZMbU2x5W7MbLoZqYfigOKAMmX6Bob4fuc+HvnZfh7Z/RI/2v0SB3oGAGhqqOX1C+exYtGpnPOqFlqbG2ltaWBBUyOnnlTviSjNphHf8rKKa6ir4R3ntPGOc9qAbLxl175DbNv9EttSgLntezvpHzz+P3IkOPWkBlqbswCzoLmBpoY6GupqaKirob42+26sq6Ghtob6WtFQV0ttTXYLrkaiRlBbo7QPtRISFC4CqoJO0nj6TCosNJ4yRet5ZRvEcTvHlZWUt30sv5Ty6Pj94XMf/q5J10bD2zXH0mprRK1EbW32XVMDdTU1R9PqakRDbQ017lVaHgcUqxhJnJVr5qxcM7/xxoUA9PQPsmf/EV7o7uWFQ3280N3Lvu4+9nX38kJ3Hy8c6mX7swc43DdA38AQ/YNB38AQfX7/pSJqa44Fl/q6LJjX1WTBvbG+ljn1NcypS9/1telTQ2NdLSc11DK3vpa5DelTn7ef9z2nYNu3RqcvBxSbVubU19Le1kx7W/MJlYsI+gbzAszAEEMR2WeIY9vBcenH1UEU1DnZsxmprSP/bv6x/GzDt6bjuHxxXJlI2xFxbJsg/Y9I5x6k77xrE+k3BtP+4FDeJ4KBoWAo7Q8MZdd5YDDoHxyifzAL6MP7fYPZ9e/pH6J3YJCe/kH2dQ/Q0z9Iz8AgPf1D2Xb/4Ct6o+PRUFeTgksNTQ11tMyt5+Q5dcybW8/Jc+uz7zn1nDw3S5t/UgO5lkZyLY3Mm1s/oR6ljU9ZA4qk1cBngVrgbyLiUwXHG4G/Bf4N8AJweUQ8k45dD1wFDAL/JSLuH63OtFTwemA+8DDw/ojoK+f52fQhica6WhrrgMZKt8bGq38wCy5H+gY50j/I4fR9pC/b7unP9ocD0JG+oeP2u3sHONAzwMNBE4QAAAlJSURBVIEj/fx8/xEO9PTz8pH+EQNVfa3INTceDTC5lkZamxtpaqxjTl0Ncxtqj/ak5uZ9z22oSUGqnjn1tVN8lWaOsgUUSbXAzcDFwB5gi6SOiHg8L9tVwP6IaJe0FrgRuFzScrL14s8FXg18W9LZqcxIdd4I3BQR6yV9PtX9uXKdn5lNXn1tNgbWMqe+ZHVGBD39Q7x8JAsuLx7qo6u7l66D2Wdf2n72pR5+tOdlXujuZegEOkoNdTWpF3SsV3TynHpOOame+U0Nxz4nNXBqUwMLmho45aQGGupm/4Ml5eyhrAI6I2IngKT1wBqydeKHrQE+kbbvAf5aWX90DbA+InqBXWnN+VUp3yvqlPQEcCHwmynPHaleBxSzKiPp6LjMq+bNGTP/0FDQO5D1fI4c7QkN94KGUs/pWE/owJF+DvT0c+DIwNGAtWvfIV46nAWwkbQ01tFYXzPyQxE6/kGLYydU5BzHcQ3G8j/f/XpWLZk/Zr4TUc6AcgawO29/D/DmkfJExICkl4EFKf2HBWXPSNvF6lwAvBQRA0XyH0fS1cDVAGeeeeaJnZGZzTo1NccC0GQNDA6x/3A/+w/38UJ3H/sP9/HioWOf/sEhhtI4V/6YXhwd2zu+vmKvdYzZmRpnb6upsfS37soZUIqFyMJTHSnPSOnF+oyj5X9lYsStwK2QvYdSLI+Z2UTU1dYcHZvhtEq3ZuqV86beHmBR3v5C4NmR8kiqA+YBL45SdqT0fcApqY6RfsvMzMqonAFlC7BM0hJJDWSD7B0FeTqAK9P2ZcDGyPp4HcBaSY3p6a1lwOaR6kxlHkh1kOr8RhnPzczMCpTtllcaE7kWuJ/sEd/bI2K7pBuArRHRAdwG3JkG3V8kCxCkfBvIBvAHgGsiYhCgWJ3pJz8GrJf0SeCRVLeZmU0Rz+XlubzMzE7ISHN5zf4Ho83MbEo4oJiZWUk4oJiZWUk4oJiZWUlU9aC8pC7gpxMs3kr2/stMMZPaO5PaCjOrvTOprTCz2juT2gqTa+9rIiJXmFjVAWUyJG0t9pTDdDWT2juT2gozq70zqa0ws9o7k9oK5Wmvb3mZmVlJOKCYmVlJOKBM3K2VbsAJmkntnUlthZnV3pnUVphZ7Z1JbYUytNdjKGZmVhLuoZiZWUk4oJiZWUk4oEyApNWSdkjqlHRdpdszGknPSPqxpG2Spt1MmJJul7RX0mN5afMlfUvSU+n71Eq2cdgIbf2EpJ+n67tN0q9Vso35JC2S9ICkJyRtl/ShlD7tru8obZ2W11fSHEmbJf0otfdPU/oSSZvStb07LbMxXdv6JUm78q7tikn/lsdQToykWuAnwMVkC35tAa6IiMcr2rARSHoGWBkR0/KFK0m/DHQDfxsR56W0PwNejIhPpYB9akR8rJLtTO0q1tZPAN0R8ReVbFsxkk4HTo+IhyW1AA8BlwK/wzS7vqO09b1Mw+urbNH2pojollQPfA/4EPBR4GsRsV7S54EfRcTnpmlbPwh8MyLuKdVvuYdy4lYBnRGxMyL6gPXAmgq3acaKiO+SrYWTbw1wR9q+g+wPS8WN0NZpKyKei4iH0/ZB4AngDKbh9R2lrdNSZLrTbn36BHAhMPwHerpc25HaWnIOKCfuDGB33v4epvE/fLJ/OP8o6SFJV1e6MeN0WkQ8B9kfGqCtwu0Zy7WSHk23xCp++6gYSYuB84FNTPPrW9BWmKbXV1KtpG3AXuBbwNPASxExkLJMm78NhW2NiOFruy5d25skNU72dxxQTpyKpE3n+4ZvjYg3Au8Erkm3bax0PgcsBVYAzwF/WdnmvJKkZuCrwIcj4kCl2zOaIm2dttc3IgYjYgWwkOzOxeuKZZvaVhVX2FZJ5wHXA+cAbwLmk616OykOKCduD7Aob38h8GyF2jKmiHg2fe8Fvk72D3+6ez7dUx++t763wu0ZUUQ8n/7POgR8gWl2fdM9868Cd0XE11LytLy+xdo63a8vQES8BDwI/CJwiqThpdWn3d+GvLauTrcZIyJ6gS9SgmvrgHLitgDL0tMcDcBaoKPCbSpKUlMa4ERSE/CrwGOjl5oWOoAr0/aVwDcq2JZRDf9hTt7NNLq+aTD2NuCJiPh03qFpd31Haut0vb6ScpJOSdtzgV8hG/d5ALgsZZsu17ZYW5/M+48KkY31TPra+imvCUiPLn4GqAVuj4h1FW5SUZLOIuuVANQBX55ubZX0FeDtZFNpPw/8CfB/gQ3AmcDPgPdERMUHw0do69vJbscE8AzwgeHxiUqTdAHwz8CPgaGU/MdkYxPT6vqO0tYrmIbXV9IvkA2615L9h/mGiLgh/X9uPdktpEeA96UeQMWM0taNQI7sNv424IN5g/cT+y0HFDMzKwXf8jIzs5JwQDEzs5JwQDEzs5JwQDEzs5JwQDEzs5JwQDGbBEnd6XuxpN8scd1/XLD/L6Ws36zUHFDMSmMxcEIBJc1cPZrjAkpE/NIJtslsSjmgmJXGp4C3pXUlPpIm4/tzSVvS5HsfAJD09rTux5fJXuJD0v9Nk3duH57AU9KngLmpvrtS2nBvSKnux5StdXN5Xt0PSrpH0pOS7kpvQSPpU5IeT22ZVlPB2+xRN3YWMxuH64D/FhG/DpACw8sR8aY0i+v3Jf1jyrsKOC8idqX934uIF9O0GFskfTUirpN0bZrQr9BvkL09/gayt/a3SPpuOnY+cC7ZHFLfB94q6XGyaUvOiYgYnobDrNTcQzErj18FfjtNGb4JWAAsS8c25wUTgP8i6UfAD8kmHl3G6C4AvpImTXwe+A7ZjLHDde9JkyluI7sVdwDoAf5G0m8Ahyd9dmZFOKCYlYeA/xwRK9JnSUQM91AOHc0kvZ1ssr63RMQbyOZ/mjOOukeSP2/UIFCX1udYRTaT76XAfSd0Jmbj5IBiVhoHgZa8/fuB/5imZEfS2WnG50LzgP0RcVjSOWRToA/rHy5f4LvA5WmcJgf8MrB5pIalNUbmRcS9wIfJbpeZlZzHUMxK41FgIN26+hLwWbLbTQ+ngfEuii8Hex/wQUmPAjvIbnsNuxV4VNLDEfFbeelfB94C/IhsFt4/ioh/TQGpmBbgG5LmkPVuPjKxUzQbnWcbNjOzkvAtLzMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzKwkHFDMzK4n/D7ZL64LpHqfzAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "plt.plot(T.J)\n", + "plt.xlabel('Iterations')\n", + "plt.ylabel('Cost')\n", + "plt.show() " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "4kZ2vUYYgclS" + }, + "source": [ + "## Challenge\n", + "\n", + "This is a reference implementation for you to explore. You will not be expected to apply it to today's module project. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "46cP9Pm_gclS" + }, + "source": [ + "# Keras Sequential API (Learn)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Bna67ADZgclT", + "toc-hr-collapsed": true + }, + "source": [ + "## Overview\n", + "\n", + "> \"Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research. Use Keras if you need a deep learning library that:\n", + "\n", + "> Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).\n", + "Supports both convolutional networks and recurrent networks, as well as combinations of the two.\n", + "Runs seamlessly on CPU and GPU.\" " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "AIJoRBxHy27n" + }, + "source": [ + "### Keras Perceptron Sample" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "ZGQHzp4rgclU" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "data = { 'x1': [0,1,0,1],\n", + " 'x2': [0,0,1,1],\n", + " 'y': [1,1,1,0]\n", + " }\n", + "\n", + "df = pd.DataFrame.from_dict(data).astype('int')\n", + "X = df[['x1', 'x2']].values\n", + "y = df['y'].values" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + }, + "colab_type": "code", + "id": "TQxyONqKvFxB", + "outputId": "f5009153-ab32-4b49-f301-4cbd4051ee21" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/5\n", + "1/1 [==============================] - 0s 2ms/step - loss: 0.9824 - accuracy: 0.5000\n", + "Epoch 2/5\n", + "1/1 [==============================] - 0s 1ms/step - loss: 0.9820 - accuracy: 0.5000\n", + "Epoch 3/5\n", + "1/1 [==============================] - 0s 1ms/step - loss: 0.9816 - accuracy: 0.5000\n", + "Epoch 4/5\n", + "1/1 [==============================] - 0s 1ms/step - loss: 0.9813 - accuracy: 0.5000\n", + "Epoch 5/5\n", + "1/1 [==============================] - 0s 2ms/step - loss: 0.9809 - accuracy: 0.5000\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 35, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras.layers import Dense\n", + "\n", + "# This is our perceptron from Monday's by-hand: \n", + "model = Sequential()\n", + "model.add(Dense(1,input_dim=2, activation='sigmoid'))\n", + "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n", + "model.fit(X,y, epochs=5)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "colab_type": "code", + "id": "Z1wfKUxszPKa", + "outputId": "e458eb38-835a-42f7-f226-b81bd4824741" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1/1 [==============================] - 0s 1ms/step - loss: 0.9805 - accuracy: 0.5000\n", + "accuracy: 50.0\n" + ] + } + ], + "source": [ + "# evaluate the model\n", + "scores = model.evaluate(X, y)\n", + "print(f\"{model.metrics_names[1]}: {scores[1]*100}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gsVYOn7bgcle", + "toc-hr-collapsed": true + }, + "source": [ + "## Follow Along\n", + "\n", + "In the `Sequential` api model, you specify a model architecture by 'sequentially specifying layers. This type of specification works well for feed forward neural networks in which the data flows in one direction (forward propagation) and the error flows in the opposite direction (backwards propagation). The Keras `Sequential` API follows a standardarized worklow to estimate a 'net: \n", + "\n", + "1. Load Data\n", + "2. Define Model\n", + "3. Compile Model\n", + "4. Fit Model\n", + "5. Evaluate Model\n", + "\n", + "You saw these steps in our Keras Perceptron Sample, but let's walk thru each step in detail." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Md5D67XwqVAf", + "toc-hr-collapsed": false + }, + "source": [ + "### Load Data" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "bn09phMBpY1J" + }, + "outputs": [], + "source": [ + "from tensorflow import keras\n", + "from tensorflow.keras.datasets import mnist\n", + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras.layers import Dense, Dropout\n", + "\n", + "# Stretch - use dropout \n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "colab_type": "code", + "id": "P6kfiLLLgclh", + "outputId": "6b66d7ee-50e2-4865-d1e3-518ac5277cf7" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n", + "11493376/11490434 [==============================] - 0s 0us/step\n" + ] + } + ], + "source": [ + "# Load the Data\n", + "(X_train, y_train), (X_test, y_test) = mnist.load_data()" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "colab_type": "code", + "id": "Mt2QnmTVgclj", + "outputId": "de7f522a-29b4-478e-a09e-29d92376ec08" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "(28, 28)" + ] + }, + "execution_count": 39, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X_train[0].shape" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "colab_type": "code", + "id": "1Zm3Q8ezgclm", + "outputId": "a9cfed2b-d0aa-4a17-c3ee-2a9efee627b2" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "(60000, 28, 28)" + ] + }, + "execution_count": 40, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "X_train.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "U1ekYH_3gcln" + }, + "outputs": [], + "source": [ + "# X Variable Types\n", + "X_train = X_train.astype('float32') / 255.\n", + "X_test = X_test.astype('float32') /255.\n", + "\n", + "# Correct Encoding on Y\n", + "# What softmax expects = [0,0,0,0,0,1,0,0,0,0]\n", + "num_classes = 10\n", + "y_train = keras.utils.to_categorical(y_train, num_classes)\n", + "y_test = keras.utils.to_categorical(y_test, num_classes)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "o0xMqOyTs5xt" + }, + "source": [ + "### Define Model" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "Bp9USczrfu6M" + }, + "outputs": [], + "source": [ + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras.layers import Dense, Flatten\n", + "\n", + "import numpy as np\n", + "\n", + "np.random.seed(812)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "wAzHLg27thoN" + }, + "source": [ + "I'll instantiate my model as a \"sequential\" model. This just means that I'm going to tell Keras what my model's architecture should be one layer at a time." + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "DSNsL49Xp6KI" + }, + "outputs": [], + "source": [ + "# https://keras.io/getting-started/sequential-model-guide/\n", + "model = Sequential()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "ZCYX6QzJtvpG" + }, + "source": [ + "Adding a \"Dense\" layer to our model is how we add \"vanilla\" perceptron-based layers to our neural network. These are also called \"fully-connected\" or \"densely-connected\" layers. They're used as a layer type in lots of other Neural Net Architectures but they're not referred to as perceptrons or multi-layer perceptrons very often in those situations even though that's what they are.\n", + "\n", + " > [\"Just your regular densely-connected NN layer.\"](https://keras.io/layers/core/)\n", + " \n", + " The first argument is how many neurons we want to have in that layer. To create a perceptron-esque model we will just set it to 10. Our architecture is just an input and output layer. We will tell it that there will be 784 inputs coming into this layer from our dataset and set it to use the sigmoid activation function." + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "GNzOLidxtvFa" + }, + "outputs": [], + "source": [ + "model.add(Flatten(input_shape=(28,28)))\n", + "model.add(Dense(10,activation=\"softmax\")) #Relu is valid option. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "EnI3jwKMtBL2", + "toc-hr-collapsed": false + }, + "source": [ + "### Compile Model\n", + "Using binary_crossentropy as the loss function here is just telling keras that I'm doing binary classification so that it can use the appropriate loss function accordingly. If we were predicting non-binary categories we might assign something like `categorical_crossentropy`. We're also telling keras that we want it to report model accuracy as our main error metric for each epoch. We will also be able to see the overall accuracy once the model has finished training.\n", + "\n", + "#### Adam Optimizer\n", + "Check out this links for more background on the Adam optimizer and Stohastic Gradient Descent\n", + "* [Adam Optimization Algorithm](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)\n", + "* [Adam Optimizer - original paper](https://arxiv.org/abs/1412.6980)" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "qp6xwYaqurRO" + }, + "outputs": [], + "source": [ + "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "5dW8SZ2Ls9SX", + "toc-hr-collapsed": false + }, + "source": [ + "### Fit Model\n", + "\n", + "Lets train it up! `model.fit()` has a `batch_size` parameter that we can use if we want to do mini-batch epochs, but since this tabular dataset is pretty small we're just going to delete that parameter. Keras' default `batch_size` is `None` so omiting it will tell Keras to do batch epochs." + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 341 + }, + "colab_type": "code", + "id": "0g1Uh2STgcl1", + "outputId": "07e6246a-0bbc-4581-92da-3570c3b8c0cc" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/10\n", + "1487/1875 [======================>.......] - ETA: 0s - loss: 0.4990 - accuracy: 0.8729" + ] + }, + { + "ename": "KeyboardInterrupt", + "evalue": "ignored", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 68\u001b[0m \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 846\u001b[0m batch_size=batch_size):\n\u001b[1;32m 847\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 848\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 849\u001b[0m \u001b[0;31m# Catch OutOfRangeError for Datasets of unknown size.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 850\u001b[0m \u001b[0;31m# This blocks until the batch has finished executing.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 578\u001b[0m \u001b[0mxla_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 579\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 580\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 581\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 582\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 609\u001b[0m \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 610\u001b[0m \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 611\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 612\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2418\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2419\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2420\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2421\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2422\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m 1663\u001b[0m if isinstance(t, (ops.Tensor,\n\u001b[1;32m 1664\u001b[0m resource_variable_ops.BaseResourceVariable))),\n\u001b[0;32m-> 1665\u001b[0;31m self.captured_inputs)\n\u001b[0m\u001b[1;32m 1666\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1667\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1744\u001b[0m \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1745\u001b[0m return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1746\u001b[0;31m ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m 1747\u001b[0m forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m 1748\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 596\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 597\u001b[0m \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 598\u001b[0;31m ctx=ctx)\n\u001b[0m\u001b[1;32m 599\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 600\u001b[0m outputs = execute.execute_with_cancellation(\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 58\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 59\u001b[0m tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 60\u001b[0;31m inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m 61\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mKeyboardInterrupt\u001b[0m: " + ] + } + ], + "source": [ + "model.fit(X_train, y_train, epochs=10)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hJKAF0tEgcl4" + }, + "source": [ + "### Evaluate Model" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 800 + }, + "colab_type": "code", + "id": "StRwssm-gcl4", + "outputId": "3c303f33-f677-4081-fc61-1395016b3c0b" + }, + "outputs": [ + { + "ename": "ValueError", + "evalue": "ignored", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Validation Data Metrics:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{model.metrics_names[0]}: {scores[0]}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{model.metrics_names[1]}: {scores[1]*100}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 68\u001b[0m \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict)\u001b[0m\n\u001b[1;32m 1079\u001b[0m step_num=step):\n\u001b[1;32m 1080\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_test_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1081\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtest_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1082\u001b[0m \u001b[0;31m# Catch OutOfRangeError for Datasets of unknown size.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1083\u001b[0m \u001b[0;31m# This blocks until the batch has finished executing.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 578\u001b[0m \u001b[0mxla_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 579\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 580\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 581\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 582\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 625\u001b[0m \u001b[0;31m# This is the first call of __call__, so we have to initialize.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 626\u001b[0m \u001b[0minitializers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 627\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_initialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madd_initializers_to\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitializers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 628\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 629\u001b[0m \u001b[0;31m# At this point we know that the initialization is complete (or less\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_initialize\u001b[0;34m(self, args, kwds, add_initializers_to)\u001b[0m\n\u001b[1;32m 504\u001b[0m self._concrete_stateful_fn = (\n\u001b[1;32m 505\u001b[0m self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access\n\u001b[0;32m--> 506\u001b[0;31m *args, **kwds))\n\u001b[0m\u001b[1;32m 507\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 508\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0minvalid_creator_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0munused_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0munused_kwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_get_concrete_function_internal_garbage_collected\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2444\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2445\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2446\u001b[0;31m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2447\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2448\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_maybe_define_function\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m 2775\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2776\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmissed\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcall_context_key\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2777\u001b[0;31m \u001b[0mgraph_function\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_graph_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2778\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprimary\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcache_key\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2779\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_create_graph_function\u001b[0;34m(self, args, kwargs, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m 2665\u001b[0m \u001b[0marg_names\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0marg_names\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2666\u001b[0m \u001b[0moverride_flat_arg_shapes\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moverride_flat_arg_shapes\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2667\u001b[0;31m capture_by_value=self._capture_by_value),\n\u001b[0m\u001b[1;32m 2668\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_attributes\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2669\u001b[0m \u001b[0;31m# Tell the ConcreteFunction to clean up its graph once it goes out of\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mfunc_graph_from_py_func\u001b[0;34m(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m 979\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal_func\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_decorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munwrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpython_func\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 980\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 981\u001b[0;31m \u001b[0mfunc_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpython_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mfunc_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfunc_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 982\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 983\u001b[0m \u001b[0;31m# invariant: `func_outputs` contains only Tensors, CompositeTensors,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36mwrapped_fn\u001b[0;34m(*args, **kwds)\u001b[0m\n\u001b[1;32m 439\u001b[0m \u001b[0;31m# __wrapped__ allows AutoGraph to swap in a converted function. We give\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 440\u001b[0m \u001b[0;31m# the function a weak reference to itself to avoid a reference cycle.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 441\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mweak_wrapped_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__wrapped__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 442\u001b[0m \u001b[0mweak_wrapped_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mweakref\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mref\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapped_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 443\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 966\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint:disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 967\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ag_error_metadata\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 968\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mag_error_metadata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 969\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 970\u001b[0m \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mValueError\u001b[0m: in user code:\n\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:941 test_function *\n outputs = self.distribute_strategy.run(\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:951 run **\n return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica\n return self._call_for_each_replica(fn, args, kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica\n return fn(*args, **kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:912 test_step **\n y, y_pred, sample_weight, regularization_losses=self.losses)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:205 __call__\n loss_value = loss_obj(y_t, y_p, sample_weight=sw)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:143 __call__\n losses = self.call(y_true, y_pred)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:246 call\n return self.fn(y_true, y_pred, **self._fn_kwargs)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:1527 categorical_crossentropy\n return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:4561 categorical_crossentropy\n target.shape.assert_is_compatible_with(output.shape)\n /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1117 assert_is_compatible_with\n raise ValueError(\"Shapes %s and %s are incompatible\" % (self, other))\n\n ValueError: Shapes (None, 10) and (None, 28, 28) are incompatible\n" + ] + } + ], + "source": [ + "scores = model.evaluate(X_test,y_test)\n", + "print(\"\\n\")\n", + "print(\"Validation Data Metrics:\")\n", + "print(f\"{model.metrics_names[0]}: {scores[0]}\")\n", + "print(f\"{model.metrics_names[1]}: {scores[1]*100}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "zHYB7k9q3O8T" + }, + "source": [ + "### Unstable Results\n", + "\n", + "You'll notice that if we rerun the results might differ from the origin run. This can be explain by a bunch of factors. Check out some of them in this article: \n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Kb2aiw_Sgcl7" + }, + "source": [ + "## Challenge\n", + "\n", + "You will be expected to leverage the Keras `Sequential` api to estimate a feed forward neural networks on a dataset.\n", + "\n", + "---" + ] + } + ], + "metadata": { + "colab": { + "name": "LS_DS_422_Gradient_Descent_Backprop_Lecture.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.2" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } diff --git a/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Assignment.ipynb b/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Assignment.ipynb deleted file mode 100644 index 9c59d8c3..00000000 --- a/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Assignment.ipynb +++ /dev/null @@ -1,99 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "9Ryp-TVm4njD" - }, - "source": [ - "\n", - "

\n", - "\n", - "# Hyperparameter Tuning\n", - "\n", - "## *Data Science Unit 4 Sprint 2 Assignment 4*\n", - "\n", - "## Your Mission, should you choose to accept it...\n", - "\n", - "To hyperparameter tune and extract every ounce of accuracy out of this telecom customer churn dataset: [Available Here](https://lambdaschool-data-science.s3.amazonaws.com/telco-churn/WA_Fn-UseC_-Telco-Customer-Churn+(1).csv)\n", - "\n", - "## Requirements\n", - "\n", - "- Load the data\n", - "- Clean the data if necessary (it will be)\n", - "- Create and fit a baseline Keras MLP model to the data.\n", - "- Hyperparameter tune (at least) the following parameters:\n", - " - batch_size\n", - " - training epochs\n", - " - optimizer\n", - " - learning rate (if applicable to optimizer)\n", - " - momentum (if applicable to optimizer)\n", - " - activation functions\n", - " - network weight initialization\n", - " - dropout regularization\n", - " - number of neurons in the hidden layer\n", - " \n", - " You must use Grid Search and Cross Validation for your initial pass of the above hyperparameters\n", - " \n", - " Try and get the maximum accuracy possible out of this data! You'll save big telecoms millions! Doesn't that sound great?\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "NNJ-tOBs4jM1" - }, - "outputs": [], - "source": [ - "##### Your Code Here #####" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "FfZRtJ7MCN3x" - }, - "source": [ - "## Stretch Goals:\n", - "\n", - "- Try to implement Random Search Hyperparameter Tuning on this dataset\n", - "- Try to implement Bayesian Optimiation tuning on this dataset using hyperas or hyperopt (if you're brave)\n", - "- Practice hyperparameter tuning other datasets that we have looked at. How high can you get MNIST? Above 99%?\n", - "- Study for the Sprint Challenge\n", - " - Can you implement both perceptron and MLP models from scratch with forward and backpropagation?\n", - " - Can you implement both perceptron and MLP models in keras and tune their hyperparameters with cross validation?" - ] - } - ], - "metadata": { - "colab": { - "name": "LS_DS_434_Hyperparameter_Tuning_Assignment.ipynb", - "provenance": [], - "version": "0.3.2" - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Lecture.ipynb b/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Lecture.ipynb deleted file mode 100644 index 51e98d61..00000000 --- a/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Lecture.ipynb +++ /dev/null @@ -1,746 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "41TS0Sa0rDNx" - }, - "source": [ - "Lambda School Data Science\n", - "\n", - "*Unit 4, Sprint 2, Module 4*\n", - "\n", - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "41TS0Sa0rDNx" - }, - "source": [ - "# Neural Networks & GPUs (Prepare)\n", - "*aka Hyperparameter Tuning*\n", - "\n", - "*aka Big Servers for Big Problems*" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "41TS0Sa0rDNx" - }, - "source": [ - "## Learning Objectives\n", - "* Part 1: Describe the major hyperparemeters to tune\n", - "* Part 2: Implement an experiment tracking framework\n", - "* Part 3: Search the hyperparameter space using RandomSearch (Optional)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "wandb_group = \"...\"\n", - "wandb_project = \"...\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Hyperparameter Options (Learn)\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Overview\n", - "\n", - "Hyperparameter tuning is much more important with neural networks than it has been with any other models that we have considered up to this point. Other supervised learning models might have a couple of parameters, but neural networks can have dozens. These can substantially affect the accuracy of our models and although it can be a time consuming process is a necessary step when working with neural networks.\n", - "​\n", - "Hyperparameter tuning comes with a challenge. How can we compare models specified with different hyperparameters if our model's final error metric can vary somewhat erratically? How do we avoid just getting unlucky and selecting the wrong hyperparameter? This is a problem that to a certain degree we just have to live with as we test and test again. However, we can minimize it somewhat by pairing our experiments with Cross Validation to reduce the variance of our final accuracy values." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "0s0o2pqBs88q" - }, - "source": [ - "### Load Boston Housing Data" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from tensorflow.keras.datasets import boston_housing\n", - "\n", - "(x_train, y_train), (x_test, y_test) = boston_housing.load_data()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "Pm7zow5IvaTt" - }, - "source": [ - "### Normalizing Input Data\n", - "\n", - "It's not 100% necessary to normalize/scale your input data before feeding it to a neural network, the network can learn the appropriate weights to deal with data of as long as it is numerically represented, but it is recommended as it can help **make training faster** and **reduces the chances that gradient descent might get stuck in a local optimum**.\n", - "\n", - "" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.preprocessing import StandardScaler\n", - "\n", - "scaler = StandardScaler()\n", - "\n", - "x_train = scaler.fit_transform(x_train)\n", - "x_test = scaler.transform(x_test)\n", - "print(x_train[:10])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "l6hgCWbir90R" - }, - "source": [ - "### Model Validation using an automatic verification Dataset\n", - "\n", - "Instead of doing seperate train test split class, Keras has a really nice feature that you can set the validation.data argument when fitting your model and Keras will take that portion of your test data and use it as a validation dataset. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1750 - }, - "colab_type": "code", - "id": "GMXVfmzXp1Oo", - "outputId": "b05e251e-508f-46e6-865b-f869ae2a5dc4" - }, - "outputs": [], - "source": [ - "from tensorflow import keras\n", - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense\n", - "\n", - "# Important Hyperparameters\n", - "inputs = x_train.shape[1]\n", - "epochs = 75\n", - "batch_size = 10\n", - "\n", - "\n", - "# Create Model\n", - "model = Sequential()\n", - "model.add(Dense(64, activation='relu', input_shape=(inputs,)))\n", - "model.add(Dense(64, activation='relu'))\n", - "model.add(Dense(1))\n", - "\n", - "# Compile Model\n", - "model.compile(optimizer='adam', loss='mse', metrics=['mse', 'mae'])\n", - "\n", - "# Fit Model\n", - "model.fit(x_train, y_train, \n", - " validation_data=(x_test,y_test), \n", - " epochs=epochs, \n", - " batch_size=batch_size\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "sYJ8t_ezHP4W" - }, - "source": [ - "### Hyperparameter Tuning Approaches:\n", - "\n", - "#### 1) Babysitting AKA \"Grad Student Descent\".\n", - "\n", - "If you fiddled with any hyperparameters yesterday, this is basically what you did. This approach is 100% manual and is pretty common among researchers where finding that 1 exact specification that jumps your model to a level of accuracy never seen before is the difference between publishing and not publishing a paper. Of course the professors don't do this themselves, that's grunt work. This is also known as the fiddle with hyperparameters until you run out of time method.\n", - "\n", - "#### 2) Grid Search\n", - "\n", - "Grid Search is the Grad Student galaxy brain realization of: why don't I just specify all the experiments I want to run and let the computer try every possible combination of them while I go and grab lunch. This has a specific downside in that if I specify 5 hyperparameters with 5 options each then I've just created 5^5 combinations of hyperparameters to check. Which means that I have to train 3125 different versions of my model Then if I use 5-fold Cross Validation on that then my model has to run 15,525 times. This is the brute-force method of hyperparameter tuning, but it can be very profitable if done wisely. \n", - "\n", - "When using Grid Search here's what I suggest: don't use it to test combinations of different hyperparameters, only use it to test different specifications of **a single** hyperparameter. It's rare that combinations between different hyperparameters lead to big performance gains. You'll get 90-95% of the way there if you just Grid Search one parameter and take the best result, then retain that best result while you test another, and then retain the best specification from that while you train another. This at least makes the situation much more manageable and leads to pretty good results. \n", - "\n", - "#### 3) Random Search\n", - "\n", - "Do Grid Search for a couple of hours and you'll say to yourself - \"There's got to be a better way.\" Enter Random Search. For Random search you specify a hyperparameter space and it picks specifications from that randomly, tries them out, gives you the best results and says - That's going to have to be good enough, go home and spend time with your family. \n", - "\n", - "Grid Search treats every parameter as if it was equally important, but this just isn't the case, some are known to move the needle a lot more than others (we'll talk about that in a minute). Random Search allows searching to be specified along the most important parameter and experiments less along the dimensions of less important hyperparameters. The downside of Random search is that it won't find the absolute best hyperparameters, but it is much less costly to perform than Grid Search. \n", - "\n", - "#### 4) Bayesian Methods\n", - "\n", - "One thing that can make more manual methods like babysitting and gridsearch effective is that as the experimenter sees results he can then make updates to his future searches taking into account the results of past specifications. If only we could hyperparameter tune our hyperparameter tuning. Well, we kind of can. Enter Bayesian Optimization. Neural Networks are like an optimization problem within an optimization problem, and Bayesian Optimization is a search strategy that tries to take into account the results of past searches in order to improve future ones. Check out the new library `keras-tuner` for easy implementations of Bayesian methods. \n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "HfQ7D043OMMn" - }, - "source": [ - "## What Hyperparameters are there to test?\n", - "\n", - "- batch_size\n", - "- training epochs\n", - "- optimization algorithms\n", - "- learning rate\n", - "- momentum\n", - "- activation functions\n", - "- dropout regularization\n", - "- number of neurons in the hidden layer\n", - "\n", - "There are more, but these are the most important." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Follow Along" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "Mri5-kXzVKAa" - }, - "source": [ - "## Batch Size\n", - "\n", - "Batch size determines how many observations the model is shown before it calculates loss/error and updates the model weights via gradient descent. You're looking for a sweet spot here where you're showing it enough observations that you have enough information to updates the weights, but not such a large batch size that you don't get a lot of weight update iterations performed in a given epoch. Feed-forward Neural Networks aren't as sensitive to bach_size as other networks, but it is still an important hyperparameter to tune. Smaller batch sizes will also take longer to train. \n", - "\n", - "Traditionally, batch size is set in powers of 2 starting at 32 up to 512. Keras defaults to a batch size of 32 if you do not specify it. Yann LeCun famously Twitted: \n", - "\n", - "> Training with large minibatches is bad for your health.\n", - "More importantly, it's bad for your test error.\n", - "Friends dont let friends use minibatches larger than 32.\n", - "\n", - "Check out this paper for more reference on his tweet. https://arxiv.org/abs/1804.07612. \n", - "\n", - "Check out this SO question on why batch size is typically set in powers of two: https://datascience.stackexchange.com/questions/20179/what-is-the-advantage-of-keeping-batch-size-a-power-of-2\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 13561 - }, - "colab_type": "code", - "id": "2smXfriNAGn7", - "outputId": "ae996575-78e2-43fb-9dbe-5d44aaf0b430" - }, - "outputs": [], - "source": [ - "import numpy\n", - "import pandas as pd\n", - "from sklearn.model_selection import GridSearchCV\n", - "from tensorflow.keras.models import Sequential\n", - "from tensorflow.keras.layers import Dense\n", - "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n", - "\n", - "# fix random seed for reproducibility\n", - "seed = 7\n", - "numpy.random.seed(seed)\n", - "\n", - "# load dataset\n", - "url =\"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv\"\n", - "\n", - "dataset = pd.read_csv(url, header=None).values\n", - "\n", - "# split into input (X) and output (Y) variables\n", - "X = dataset[:,0:8]\n", - "Y = dataset[:,8]\n", - "\n", - "# Function to create model, required for KerasClassifier\n", - "def create_model():\n", - " # create model\n", - " model = Sequential()\n", - " model.add(Dense(12, input_dim=8, activation='relu'))\n", - " model.add(Dense(1, activation='sigmoid'))\n", - " # Compile model\n", - " model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n", - " return model\n", - "\n", - "# create model\n", - "model = KerasClassifier(build_fn=create_model, verbose=0)\n", - "\n", - "# define the grid search parameters\n", - "# batch_size = [10, 20, 40, 60, 80, 100]\n", - "# param_grid = dict(batch_size=batch_size, epochs=epochs)\n", - "\n", - "# define the grid search parameters\n", - "param_grid = {'batch_size': [10, 20, 40, 60, 80, 100],\n", - " 'epochs': [20]}\n", - "\n", - "# Create Grid Search\n", - "grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)\n", - "grid_result = grid.fit(X, Y)\n", - "\n", - "# Report Results\n", - "print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n", - "means = grid_result.cv_results_['mean_test_score']\n", - "stds = grid_result.cv_results_['std_test_score']\n", - "params = grid_result.cv_results_['params']\n", - "for mean, stdev, param in zip(means, stds, params):\n", - " print(f\"Means: {mean}, Stdev: {stdev} with: {param}\") " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "EKcuY6OiaLfz" - }, - "source": [ - "## Optimizer\n", - "\n", - "Remember that there's a different optimizers [optimizers](https://keras.io/optimizers/). At some point, take some time to read up on them a little bit. \"adam\" usually gives the best results. The thing to know about choosing an optimizer is that different optimizers have different hyperparameters like learning rate, momentum, etc. So based on the optimizer you choose you might also have to tune the learning rate and momentum of those optimizers after that. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "DG3wq5iOaLig" - }, - "source": [ - "## Learning Rate\n", - "\n", - "Remember that the Learning Rate is a hyperparameter that is specific to your gradient-descent based optimizer selection. A learning rate that is too high will cause divergent behavior, but a Learning Rate that is too low will fail to converge, again, you're looking for the sweet spot. I would start out tuning learning rates by orders of magnitude: [.001, .01, .1, .2, .3, .5] etc. I wouldn't go above .5, but you can try it and see what the behavior is like. \n", - "\n", - "Once you have narrowed it down, make the window even smaller and try it again. If after running the above specification your model reports that .1 is the best optimizer, then you should probably try things like [.05, .08, .1, .12, .15] to try and narrow it down. \n", - "\n", - "It can also be good to tune the number of epochs in combination with the learning rate since the number of iterations that you allow the learning rate to reach the minimum can determine if you have let it run long enough to converge to the minimum. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "gNTBUWd1aLlA" - }, - "source": [ - "## Momentum\n", - "\n", - "Momentum is a hyperparameter that is more commonly associated with Stochastic Gradient Descent. SGD is a common optimizer because it's what people understand and know, but I doubt it will get you the best results, you can try hyperparameter tuning its attributes and see if you can beat the performance from adam. Momentum is a property that decides the willingness of an optimizer to overshoot the minimum. Imagine a ball rolling down one side of a bowl and then up the opposite side a little bit before settling back to the bottom. The purpose of momentum is to try and escale local minima." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "xnEG-bCJaLnZ" - }, - "source": [ - "## Activation Functions\n", - "\n", - "We've talked about this a little bit, typically you'l want to use ReLU for hidden layers and either Sigmoid, or Softmax for output layers of binary and multi-class classification implementations respectively, but try other activation functions and see if you can get any better results with sigmoid or tanh or something. There are a lot of activation functions that we haven't really talked about. Maybe you'll get good results with them. Maybe you won't. :) " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "oul9sPq-dU-h" - }, - "source": [ - "## Network Weight Initialization\n", - "\n", - "You saw how big of an effect the way that we initialize our network's weights can have on our results. There are **a lot** of what are called initialization modes. I don't understand all of them, but they can have a big affect on your model's initial accuracy. Your model will get further with less epochs if you initialize it with weights that are well suited to the problem you're trying to solve.\n", - "\n", - "`init_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "bqtEuxeQaLqE" - }, - "source": [ - "## Dropout Regularization and the Weight Constraint\n", - "\n", - "the Dropout Regularization value is a percentage of neurons that you want to be randomly deactivated during training. The weight constraint is a second regularization parameter that works in tandem with dropout regularization. You should tune these two values at the same time. \n", - "\n", - "Using dropout on visible vs hidden layers might have a different effect. Using dropout on hidden layers might not have any effect while using dropout on hidden layers might have a substantial effect. You don't necessarily need to turn use dropout unless you see that your model has overfitting and generalizability problems." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "P2c5Cv6oaLtO" - }, - "source": [ - "## Neurons in Hidden Layer \n", - "\n", - "Remember that when we only had a single perceptron our model was only able to fit to linearly separable data, but as we have added layers and nodes to those layers our network has become a powerhouse of fitting nonlinearity in data. The larger the network and the more nodes generally the stronger the network's capacity to fit nonlinear patterns in data. The more nodes and layers the longer it will take to train a network, and higher the probability of overfitting. The larger your network gets the more you'll need dropout regularization or other regularization techniques to keep it in check. \n", - "\n", - "Typically depth (more layers) is more important than width (more nodes) for neural networks. This is part of why Deep Learning is so highly touted. Certain deep learning architectures have truly been huge breakthroughs for certain machine learning tasks. \n", - "\n", - "You might borrow ideas from other network architectures. For example if I was doing image recognition and I wasn't taking cues from state of the art architectures like resnet, alexnet, googlenet, etc. Then I'm probably going to have to do a lot more experimentation on my own before I find something that works.\n", - "\n", - "There are some heuristics, but I am highly skeptical of them. I think you're better off experimenting on your own and forming your own intuition for these kinds of problems. \n", - "\n", - "- https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Challenge\n", - "You will be expected to tune several hyperparameters in today's module project. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Experiment Tracking Framework (Learn)\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Overview\n", - "\n", - "You will notice quickly that managing the results of all the experiments you are running becomes challenging. Which set of parameters did the best? Are my results today different than my results yesterday? Although we use Ipython Notebooks to work, the format is not well suited to logging experimental results. Enter experiment tracking frameworks like [Comet.ml](https://comet.ml) and [Weights and Biases](https://wandb.ai/).\n", - "\n", - "Those tools will help you track your experiments, store the results, and the code associated with those experiments. Experimental results can also be readily visualized to see changes in performance across any metric you care about. Data is sent to the tool as each epoch is completed, so you can also see if your model is converging. Let's check out Weights & Biases today. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Follow Along\n", - "\n", - "Make sure you login into `wandb` in the terminal before running the next cell. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import wandb\n", - "from wandb.keras import WandbCallback" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1750 - }, - "colab_type": "code", - "id": "GMXVfmzXp1Oo", - "outputId": "b05e251e-508f-46e6-865b-f869ae2a5dc4" - }, - "outputs": [], - "source": [ - "wandb.init(project=wandb_project, entity=wand_group) #Initializes and Experiment\n", - "\n", - "# Important Hyperparameters\n", - "X = x_train\n", - "y = y_train\n", - "\n", - "inputs = X.shape[1]\n", - "wandb.config.epochs = 50\n", - "wandb.config.batch_size = 10\n", - "\n", - "# Create Model\n", - "model = Sequential()\n", - "model.add(Dense(64, activation='relu', input_shape=(inputs,)))\n", - "model.add(Dense(64, activation='relu'))\n", - "model.add(Dense(64, activation='relu'))\n", - "model.add(Dense(1))\n", - "# Compile Model\n", - "model.compile(optimizer='adam', loss='mse', metrics=['mse', 'mae'])\n", - "\n", - "# Fit Model\n", - "model.fit(X, y, \n", - " validation_split=0.33, \n", - " epochs=wandb.config.epochs, \n", - " batch_size=wandb.config.batch_size, \n", - " callbacks=[WandbCallback()]\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Your Turn\n", - "\n", - "Pick a few hyparameters that we *have not* tuned. Using the same code above, try changing a few parameters you're interested in and submitting the results to weights & biases. :) " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "wandb.init(project=wandb_project, entity=wand_group) #Initializes and Experiment\n", - "\n", - "wandb.config.epochs = 50\n", - "wandb.config.batch_size = 10\n", - "\n", - "\n", - "# Fit Model\n", - "model.fit(\n", - " callbacks=[WandbCallback()]\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Challenge\n", - "\n", - "You will be expected to use Weights & Biases to try to tune your model during your module assignment today. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Hyperparameters with RandomSearchCV (Learn)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Overview\n", - "\n", - "Basically `GridSearchCV` takes forever. You'll want to adopt a slightly more sophiscated strategy.\n", - "\n", - "Let's also take a look at an alternative with Keras-Tuner." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!pip install keras-tuner" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Follow Along" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from tensorflow import keras\n", - "from tensorflow.keras import layers\n", - "from kerastuner.tuners import RandomSearch\n", - "\n", - "\"\"\"\n", - "This model Tunes:\n", - "- Number of Neurons in the Hidden Layer\n", - "- Learning Rate in Adam\n", - "\n", - "\"\"\"\n", - "\n", - "def build_model(hp):\n", - " model = keras.Sequential()\n", - " model.add(layers.Dense(units=hp.Int('units',\n", - " min_value=32,\n", - " max_value=512,\n", - " step=32),\n", - " activation='relu'))\n", - " model.add(layers.Dense(10, activation='softmax'))\n", - " model.compile(\n", - " optimizer=keras.optimizers.Adam(\n", - " hp.Choice('learning_rate',\n", - " values=[1e-2, 1e-3, 1e-4])),\n", - " loss='sparse_categorical_crossentropy',\n", - " metrics=['accuracy'])\n", - " \n", - " return model" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tuner = RandomSearch(\n", - " build_model,\n", - " objective='val_accuracy',\n", - " max_trials=5,\n", - " executions_per_trial=3,\n", - " directory='./keras-tuner-trial',\n", - " project_name='helloworld')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tuner.search_space_summary()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tuner.search(x, y,\n", - " epochs=5,\n", - " validation_data=(val_x, val_y))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tuner.results_summary()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Challenge\n", - "\n", - "Try to apply RandomSearchCV to your module project today. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Review\n", - "* Part 1: Describe the major hyperparemeters to tune\n", - " - Activation Functions\n", - " - Optimizer\n", - " - Number of Layers\n", - " - Number of Neurons\n", - " - Batch Size\n", - " - Dropout Regulaization\n", - " - Learning Rate\n", - " - Number of Epochs\n", - " - and many more\n", - "* Part 2: Implement an experiment tracking framework\n", - " - Weights & Biases\n", - " - Comet.ml\n", - " - By Hand / GridSearch\n", - "* Part 3: Search the hyperparameter space using RandomSearch\n", - " - Sklearn still useful (haha)\n", - " - Integration with Wieghts & Biases\n", - "* Part 4: Discuss emerging hyperparameter tuning strategies\n", - " - Bayesian Optimization\n", - " - Hyperopt\n", - " - Genetic Evolution" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Sources" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Additional Reading\n", - "- https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/\n", - "- https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/\n", - "- https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/\n", - "- https://machinelearningmastery.com/introduction-to-weight-constraints-to-reduce-generalization-error-in-deep-learning/\n", - "- https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "U4-S2-NNF-DS12", - "language": "python", - "name": "u4-s2-nnf-ds12" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.7" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Assignment.ipynb b/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Assignment.ipynb new file mode 100644 index 00000000..4f8abbe8 --- /dev/null +++ b/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Assignment.ipynb @@ -0,0 +1 @@ +{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"LS_DS_434_Hyperparameter_Tuning_Assignment.ipynb","provenance":[{"file_id":"https://github.com/johnjdailey/DS-Unit-4-Sprint-2-Neural-Networks/blob/main/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Assignment.ipynb","timestamp":1596098576662}],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"9Ryp-TVm4njD"},"source":["\n","

\n","\n","# Hyperparameter Tuning\n","\n","## *Data Science Unit 4 Sprint 2 Assignment 4*\n","\n","## Your Mission, should you choose to accept it...\n","\n","To hyperparameter tune and extract every ounce of accuracy out of this telecom customer churn dataset: [Available Here](https://lambdaschool-data-science.s3.amazonaws.com/telco-churn/WA_Fn-UseC_-Telco-Customer-Churn+(1).csv)\n","\n","## Requirements\n","\n","- Load the data\n","- Clean the data if necessary (it will be)\n","- Create and fit a baseline Keras MLP model to the data.\n","- Hyperparameter tune (at least) the following parameters:\n"," - batch_size\n"," - training epochs\n"," - optimizer\n"," - learning rate (if applicable to optimizer)\n"," - momentum (if applicable to optimizer)\n"," - activation functions\n"," - network weight initialization\n"," - dropout regularization\n"," - number of neurons in the hidden layer\n"," \n"," You must use Grid Search and Cross Validation for your initial pass of the above hyperparameters\n"," \n"," Try and get the maximum accuracy possible out of this data! You'll save big telecoms millions! Doesn't that sound great?\n"]},{"cell_type":"code","metadata":{"id":"7qr00PC2JIn9","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":292},"executionInfo":{"status":"ok","timestamp":1596244207021,"user_tz":240,"elapsed":5408,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"45d6fea5-1585-4694-c48b-47a60bbd9f14"},"source":["# Install category_encoders\n","\n","!pip install category_encoders"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Collecting category_encoders\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/44/57/fcef41c248701ee62e8325026b90c432adea35555cbc870aff9cfba23727/category_encoders-2.2.2-py2.py3-none-any.whl (80kB)\n","\r\u001b[K |████ | 10kB 23.3MB/s eta 0:00:01\r\u001b[K |████████▏ | 20kB 5.7MB/s eta 0:00:01\r\u001b[K |████████████▏ | 30kB 5.5MB/s eta 0:00:01\r\u001b[K |████████████████▎ | 40kB 6.3MB/s eta 0:00:01\r\u001b[K |████████████████████▎ | 51kB 6.3MB/s eta 0:00:01\r\u001b[K |████████████████████████▍ | 61kB 6.6MB/s eta 0:00:01\r\u001b[K |████████████████████████████▍ | 71kB 7.5MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 81kB 3.6MB/s \n","\u001b[?25hRequirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (0.10.2)\n","Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (1.18.5)\n","Requirement already satisfied: pandas>=0.21.1 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (1.0.5)\n","Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (1.4.1)\n","Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (0.22.2.post1)\n","Requirement already satisfied: patsy>=0.5.1 in /usr/local/lib/python3.6/dist-packages (from category_encoders) (0.5.1)\n","Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.21.1->category_encoders) (2018.9)\n","Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.21.1->category_encoders) (2.8.1)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->category_encoders) (0.16.0)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from patsy>=0.5.1->category_encoders) (1.15.0)\n","Installing collected packages: category-encoders\n","Successfully installed category-encoders-2.2.2\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"mbamdA1bJd2b","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":530},"executionInfo":{"status":"ok","timestamp":1596244215310,"user_tz":240,"elapsed":7659,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"1216632a-1556-457a-c7f5-a5c0178aed63"},"source":["# Install keras-tuner\n","\n","!pip install keras-tuner"],"execution_count":4,"outputs":[{"output_type":"stream","text":["Collecting keras-tuner\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/a7/f7/4b41b6832abf4c9bef71a664dc563adb25afc5812831667c6db572b1a261/keras-tuner-1.0.1.tar.gz (54kB)\n","\r\u001b[K |██████ | 10kB 22.7MB/s eta 0:00:01\r\u001b[K |████████████ | 20kB 6.4MB/s eta 0:00:01\r\u001b[K |██████████████████ | 30kB 7.5MB/s eta 0:00:01\r\u001b[K |████████████████████████ | 40kB 8.1MB/s eta 0:00:01\r\u001b[K |██████████████████████████████ | 51kB 6.9MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 61kB 3.8MB/s \n","\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.16.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.18.5)\n","Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.8.7)\n","Collecting terminaltables\n"," Downloading https://files.pythonhosted.org/packages/9b/c4/4a21174f32f8a7e1104798c445dacdc1d4df86f2f26722767034e4de4bff/terminaltables-3.1.0.tar.gz\n","Collecting colorama\n"," Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl\n","Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (4.41.1)\n","Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (2.23.0)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.4.1)\n","Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.22.2.post1)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2.10)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (3.0.4)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2020.6.20)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (1.24.3)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->keras-tuner) (0.16.0)\n","Building wheels for collected packages: keras-tuner, terminaltables\n"," Building wheel for keras-tuner (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for keras-tuner: filename=keras_tuner-1.0.1-cp36-none-any.whl size=73200 sha256=7f50b624092d302d8cd2af6573d1bfc8bd938ab47252f5e8b1855d6d9342a304\n"," Stored in directory: /root/.cache/pip/wheels/b9/cc/62/52716b70dd90f3db12519233c3a93a5360bc672da1a10ded43\n"," Building wheel for terminaltables (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for terminaltables: filename=terminaltables-3.1.0-cp36-none-any.whl size=15356 sha256=752164d7f3e2d500a4685a0c9bf5d561a7dc7ac9086b06ec8eb4a72f04e7ea74\n"," Stored in directory: /root/.cache/pip/wheels/30/6b/50/6c75775b681fb36cdfac7f19799888ef9d8813aff9e379663e\n","Successfully built keras-tuner terminaltables\n","Installing collected packages: terminaltables, colorama, keras-tuner\n","Successfully installed colorama-0.4.3 keras-tuner-1.0.1 terminaltables-3.1.0\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab_type":"code","id":"NNJ-tOBs4jM1","colab":{"base_uri":"https://localhost:8080/","height":71},"executionInfo":{"status":"ok","timestamp":1596244219318,"user_tz":240,"elapsed":3062,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"3fe2eecf-f246-43ea-cbdd-47fbea09553d"},"source":["# Imports\n","\n","import pandas as pd\n","import numpy as np\n","import category_encoders as ce\n","from sklearn.preprocessing import StandardScaler\n","from sklearn.preprocessing import MinMaxScaler\n","from sklearn.model_selection import train_test_split\n","import os\n","import datetime\n","import tensorflow as tf\n","from tensorflow.keras.optimizers import Adam\n","from tensorflow.keras import layers\n","from sklearn.model_selection import GridSearchCV\n","from sklearn.model_selection import RandomizedSearchCV\n","from kerastuner.tuners import RandomSearch\n","from tensorflow.keras.models import Sequential\n","from tensorflow.keras.layers import Dense, Dropout\n","from tensorflow.keras import layers\n","from tensorflow.keras import regularizers\n","from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n","from tensorflow.keras.utils import to_categorical\n","import matplotlib.pyplot as plt"],"execution_count":5,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n"," import pandas.util.testing as tm\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"4NxoST0Xuozq","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244224643,"user_tz":240,"elapsed":912,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["%load_ext tensorboard\n","\n","logdir = os.path.join(\"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n","tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"id":"I19Ps3yO6JpJ","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244228631,"user_tz":240,"elapsed":1786,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["# Import the data\n","\n","df = pd.read_csv('https://lambdaschool-data-science.s3.amazonaws.com/telco-churn/WA_Fn-UseC_-Telco-Customer-Churn+(1).csv')"],"execution_count":7,"outputs":[]},{"cell_type":"code","metadata":{"id":"QNctM54e7iii","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":309},"executionInfo":{"status":"ok","timestamp":1596244230074,"user_tz":240,"elapsed":805,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"d675c4db-ddca-419f-80cf-dac27f0bd95a"},"source":["# See df\n","\n","df.head()"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
customerIDgenderSeniorCitizenPartnerDependentstenurePhoneServiceMultipleLinesInternetServiceOnlineSecurityOnlineBackupDeviceProtectionTechSupportStreamingTVStreamingMoviesContractPaperlessBillingPaymentMethodMonthlyChargesTotalChargesChurn
07590-VHVEGFemale0YesNo1NoNo phone serviceDSLNoYesNoNoNoNoMonth-to-monthYesElectronic check29.8529.85No
15575-GNVDEMale0NoNo34YesNoDSLYesNoYesNoNoNoOne yearNoMailed check56.951889.5No
23668-QPYBKMale0NoNo2YesNoDSLYesYesNoNoNoNoMonth-to-monthYesMailed check53.85108.15Yes
37795-CFOCWMale0NoNo45NoNo phone serviceDSLYesNoYesYesNoNoOne yearNoBank transfer (automatic)42.301840.75No
49237-HQITUFemale0NoNo2YesNoFiber opticNoNoNoNoNoNoMonth-to-monthYesElectronic check70.70151.65Yes
\n","
"],"text/plain":[" customerID gender SeniorCitizen ... MonthlyCharges TotalCharges Churn\n","0 7590-VHVEG Female 0 ... 29.85 29.85 No\n","1 5575-GNVDE Male 0 ... 56.95 1889.5 No\n","2 3668-QPYBK Male 0 ... 53.85 108.15 Yes\n","3 7795-CFOCW Male 0 ... 42.30 1840.75 No\n","4 9237-HQITU Female 0 ... 70.70 151.65 Yes\n","\n","[5 rows x 21 columns]"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"code","metadata":{"id":"M1dl-mTh7aAT","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596244232923,"user_tz":240,"elapsed":741,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"3ed4dc95-bfc5-4dec-e666-0ff8e85aa59b"},"source":["# df shape\n","\n","df.shape"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(7043, 21)"]},"metadata":{"tags":[]},"execution_count":9}]},{"cell_type":"code","metadata":{"id":"DKD6x39-7w3-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596244235307,"user_tz":240,"elapsed":732,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"a285d332-518d-4fce-f4b8-0e7e8d9a7534"},"source":["# Check for nans\n","\n","df.isnull().values.any()"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["False"]},"metadata":{"tags":[]},"execution_count":10}]},{"cell_type":"code","metadata":{"id":"kAIgsYnZGd5X","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":68},"executionInfo":{"status":"ok","timestamp":1596244236754,"user_tz":240,"elapsed":521,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"7ba08c19-de0c-4a0f-d702-4e99bd656f4e"},"source":["# Baseline\n","\n","df['Churn'].value_counts(normalize=True)"],"execution_count":11,"outputs":[{"output_type":"execute_result","data":{"text/plain":["No 0.73463\n","Yes 0.26537\n","Name: Churn, dtype: float64"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"id":"pRnmNZ8vGiJ1","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244239414,"user_tz":240,"elapsed":734,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["# Convert categries to binary\n","\n","df['Churn'] = df['Churn'].replace({'Yes':1, 'No':0})"],"execution_count":12,"outputs":[]},{"cell_type":"code","metadata":{"id":"1gdyF4v7G7OZ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":374},"executionInfo":{"status":"ok","timestamp":1596244242093,"user_tz":240,"elapsed":860,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"59aeca22-07f6-4651-f928-2db388b26a92"},"source":["# Check unique values in df\n","\n","for col in df.columns: print(col, df[col].nunique())"],"execution_count":13,"outputs":[{"output_type":"stream","text":["customerID 7043\n","gender 2\n","SeniorCitizen 2\n","Partner 2\n","Dependents 2\n","tenure 73\n","PhoneService 2\n","MultipleLines 3\n","InternetService 3\n","OnlineSecurity 3\n","OnlineBackup 3\n","DeviceProtection 3\n","TechSupport 3\n","StreamingTV 3\n","StreamingMovies 3\n","Contract 3\n","PaperlessBilling 2\n","PaymentMethod 4\n","MonthlyCharges 1585\n","TotalCharges 6531\n","Churn 2\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"M3GM7BD5HB4B","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244248243,"user_tz":240,"elapsed":744,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["# Drop customerID, too much noise and drop target\n","\n","X = df.drop(columns=['Churn', 'customerID']).values\n","y = df['Churn'].values"],"execution_count":14,"outputs":[]},{"cell_type":"code","metadata":{"id":"Dr6uVonEPksV","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":391},"executionInfo":{"status":"ok","timestamp":1596244249810,"user_tz":240,"elapsed":736,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"b89b95d8-6473-4756-8464-e31ab5b92c99"},"source":["# Check data types\n","\n","df.dtypes"],"execution_count":15,"outputs":[{"output_type":"execute_result","data":{"text/plain":["customerID object\n","gender object\n","SeniorCitizen int64\n","Partner object\n","Dependents object\n","tenure int64\n","PhoneService object\n","MultipleLines object\n","InternetService object\n","OnlineSecurity object\n","OnlineBackup object\n","DeviceProtection object\n","TechSupport object\n","StreamingTV object\n","StreamingMovies object\n","Contract object\n","PaperlessBilling object\n","PaymentMethod object\n","MonthlyCharges float64\n","TotalCharges object\n","Churn int64\n","dtype: object"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"id":"3LA4FjuAIQ4W","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244252938,"user_tz":240,"elapsed":747,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["# Ordinal encode so many low cardinality categories\n","\n","encode = ce.OrdinalEncoder()\n","\n","X = encode.fit_transform(X)"],"execution_count":16,"outputs":[]},{"cell_type":"code","metadata":{"id":"6HVKUNQMIRDB","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1596244254974,"user_tz":240,"elapsed":848,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}}},"source":["# Scale the data for the neural network\n","\n","#scale = MinMaxScaler()\n","\n","scale = StandardScaler()\n","\n","X = scale.fit_transform(X)"],"execution_count":17,"outputs":[]},{"cell_type":"code","metadata":{"id":"wQnTuKpLHCEU","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596244258545,"user_tz":240,"elapsed":755,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"2f421933-068b-48fd-b8e0-7c657a8e27a7"},"source":["# Train Test split\n","\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,random_state=7)\n","\n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":18,"outputs":[{"output_type":"stream","text":["(4930, 19) (2113, 19) (4930,) (2113,)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"zcrBfhTu-MqZ","colab_type":"code","colab":{}},"source":["# GridSearch and Cross Validation for initial pass\n","\n","# Fix random seed for reproducibility\n","\n","seed = 7\n","np.random.seed(seed)\n","\n","# Function to create model, required for KerasClassifier\n","\n","def create_model(lr):\n","\n"," # Create optimizer\n","\n"," adam = Adam(learning_rate=lr)\n","\n"," # Create model\n","\n"," model = Sequential()\n"," model.add(Dense(19, input_dim=19, activation='relu'))\n"," model.add(Dense(1, activation='sigmoid'))\n","\n"," # Compile model\n","\n"," model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n","\n"," return model\n","\n","# Create model\n","\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# Define the grid search parameters\n","\n","param_grid = {'lr': [.001, .01, .1, .2, .3, .5],\n"," 'batch_size': [10, 20, 40, 60, 80, 100],\n"," 'epochs': [100]}\n","\n","# Create Grid Search\n","\n","grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)\n","grid_result = grid.fit(X_train, y_train)\n","\n","# Report Results\n","\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\")\n","\n","# 1st Best: 0.787221086025238 using {'batch_size': 40, 'epochs': 100, 'lr': 0.001}\n","# This was run without GPU, just CPU, and it took about 30 minutes"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"nIOhOMrDu6Se","colab_type":"code","colab":{}},"source":["#%tensorboard --logdir logs"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"KFpxuTpDgC43","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":204},"executionInfo":{"status":"ok","timestamp":1596165426283,"user_tz":240,"elapsed":1949059,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"05fb14d2-9559-4875-e925-7999a69da9eb"},"source":["# RandomSearch and Cross Validation for initial pass\n","\n","# Fix random seed for reproducibility\n","\n","seed = 7\n","np.random.seed(seed)\n","\n","# Function to create model, required for KerasClassifier\n","\n","def create_model(lr):\n","\n"," # Create optimizer\n","\n"," adam = Adam(learning_rate=lr)\n","\n"," # Create model\n","\n"," model = Sequential()\n"," model.add(Dense(19, input_dim=19, activation='relu'))\n"," model.add(Dense(1, activation='sigmoid'))\n","\n"," # Compile model\n","\n"," model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n","\n"," return model\n","\n","# Create model\n","\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# Define the grid search parameters\n","\n","param_grid = {'lr': [.001, .01, .1, .2, .3, .5],\n"," 'batch_size': [10, 20, 40, 60, 80, 100],\n"," 'epochs': [100]}\n","\n","# Create Grid Search\n","\n","grid = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_jobs=1)\n","grid_result = grid.fit(X_train, y_train)\n","\n","# Report Results\n","\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\")\n","\n","# 1st Best: 0.786612582206726 using {'lr': 0.1, 'epochs': 100, 'batch_size': 100}\n","# This was run with GPU and it also took about 30 minutes, 0.0006 worse than Gridsearch"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Best: 0.786612582206726 using {'lr': 0.1, 'epochs': 100, 'batch_size': 100}\n","Means: 0.786612582206726, Stdev: 0.007274001630058567 with: {'lr': 0.1, 'epochs': 100, 'batch_size': 100}\n","Means: 0.7746450424194335, Stdev: 0.007761119619711015 with: {'lr': 0.1, 'epochs': 100, 'batch_size': 10}\n","Means: 0.7829614639282226, Stdev: 0.00650985323103509 with: {'lr': 0.001, 'epochs': 100, 'batch_size': 80}\n","Means: 0.7762677550315857, Stdev: 0.01015416027683366 with: {'lr': 0.01, 'epochs': 100, 'batch_size': 10}\n","Means: 0.776673436164856, Stdev: 0.0060648663294483734 with: {'lr': 0.001, 'epochs': 100, 'batch_size': 60}\n","Means: 0.7807302355766297, Stdev: 0.006459097335944383 with: {'lr': 0.01, 'epochs': 100, 'batch_size': 40}\n","Means: 0.7807302236557007, Stdev: 0.010606021239098977 with: {'lr': 0.2, 'epochs': 100, 'batch_size': 20}\n","Means: 0.778904664516449, Stdev: 0.00580845114156917 with: {'lr': 0.2, 'epochs': 100, 'batch_size': 40}\n","Means: 0.7791074991226197, Stdev: 0.006132353571338926 with: {'lr': 0.3, 'epochs': 100, 'batch_size': 60}\n","Means: 0.7750507116317749, Stdev: 0.008662951056421068 with: {'lr': 0.5, 'epochs': 100, 'batch_size': 10}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"jOUmo0yBu-ux","colab_type":"code","colab":{}},"source":["#%tensorboard --logdir logs"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"d9cIJ8Mf2BkB","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596254107434,"user_tz":240,"elapsed":3163794,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"05cce235-4ada-4bdd-8b6b-5a9801291dcb"},"source":["# GridSearch and Cross Validation for second pass of the following hyperparameters:\n","\n","# batch_size\n","# training epochs\n","# optimizer\n","# learning rate (if applicable to optimizer)\n","# momentum (if applicable to optimizer)\n","# activation functions\n","# network weight initialization\n","# dropout regularization\n","# number of neurons in the hidden layer\n","\n","# All of the hyperparameters\n","\n","#param_grid = {'batch_size' : [10, 20, 40, 60, 80, 100],\n","# 'epochs' : [50, 75, 100, 125, 150],\n","# 'optimizer' : ['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'],\n","# 'lr' : [.001, .01, .1],\n","# 'momentum' : [0.0, 0.2, 0.4, 0.6, 0.8, 0.9],\n","# 'activation' : ['softmax', 'softplus', 'softsign', 'relu', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'],\n","# 'net_weight_init' : ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'],\n","# 'dropout_rate' : [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],\n","# 'neurons' : [1, 5, 10, 15, 20, 25, 30],\n","# 'l1_reg': [1e-2, le-3, le-4], # weight decay, add this term to loss function, small coeffecient\n","# 'l2_reg': [1e-2, le-3, le-4]}\n","\n","# However, there are too many options, and colab will time-out in 12 hours before\n","# the model finishes running, so trim this down.\n","\n","\n","# Fix random seed for reproducibility\n","\n","seed = 7\n","np.random.seed(seed)\n","\n","# Function to create model, required for KerasClassifier\n","\n","def create_model(batch_size, epochs, optimizer, lr, momentum, activation, \n"," net_weight_init, dropout_rate, neurons, l1_reg, l2_reg):\n","\n"," # Create optimizer\n","\n"," adam = Adam(learning_rate=lr)\n","\n"," # Create model\n","\n"," model = Sequential()\n"," model.add(Dense(19, input_dim=19, activation=activation))\n"," model.add(Dense(1, activation='sigmoid'))\n","\n"," # Compile model\n","\n"," model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n","\n"," return model\n","\n","# Create model\n","\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# Define the grid search parameters\n","\n","param_grid = {'batch_size' : [60, 80],\n"," 'epochs' : [100],\n"," 'optimizer' : ['Adam', 'Nadam'],\n"," 'lr' : [.001, .01],\n"," 'momentum' : [0.1, 0.2],\n"," 'activation' : ['relu'],\n"," 'net_weight_init' : ['uniform', 'lecun_uniform'],\n"," 'dropout_rate' : [0.0, 0.1],\n"," 'neurons' : [1, 10],\n"," 'l1_reg': [1e-5],\n"," 'l2_reg': [1e-5]}\n","\n","# Create Grid Search\n","\n","grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)\n","grid_result = grid.fit(X_train, y_train)\n","\n","# Report Results\n","\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\")\n","\n","# 1st Best: 0.7912778854370117 using {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 10, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","# This was run with CPU, and it took about 10 minutes\n","# 2nd Best: Best: 0.7876267790794372 using {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}"],"execution_count":26,"outputs":[{"output_type":"stream","text":["Best: 0.7876267790794372 using {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7815415740013123, Stdev: 0.006653643495390422 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7791074991226197, Stdev: 0.009522676594026164 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7874239206314086, Stdev: 0.010234870703621352 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7837728261947632, Stdev: 0.004321959186483095 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7821501016616821, Stdev: 0.005643168953936855 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7815415740013123, Stdev: 0.002981135324358499 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7845841765403747, Stdev: 0.011907219795563225 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7866125702857971, Stdev: 0.010194594177474595 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7870182633399964, Stdev: 0.007643594498998371 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7862068891525269, Stdev: 0.006232170096007918 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.78052738904953, Stdev: 0.008353441807864303 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7795132040977478, Stdev: 0.006172469573814403 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7821501135826111, Stdev: 0.008249369019405468 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7805273771286011, Stdev: 0.006715178382244117 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7835699796676636, Stdev: 0.007330340629076998 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7831642866134644, Stdev: 0.0045987010999803705 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.779513168334961, Stdev: 0.009866437095236139 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7807302236557007, Stdev: 0.006490878015769221 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7809330582618713, Stdev: 0.00933944610408313 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7787018299102784, Stdev: 0.00567953033107649 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7827586174011231, Stdev: 0.004774269528134426 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7789046764373779, Stdev: 0.008533739025469762 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7770791172981262, Stdev: 0.003168464834387867 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7837728261947632, Stdev: 0.005187310507787045 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7858012199401856, Stdev: 0.00412716098951887 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7776876330375672, Stdev: 0.0028252378911536873 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7801217079162598, Stdev: 0.005606603675410994 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.781135904788971, Stdev: 0.006165785685048549 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7855983734130859, Stdev: 0.008808955462590962 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7868154048919678, Stdev: 0.007125430567766293 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7821501135826111, Stdev: 0.006132336616045893 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7858012199401856, Stdev: 0.00402625347017766 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7821501016616821, Stdev: 0.006490872800366439 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7845841765403747, Stdev: 0.0026909743228633863 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7793103456497192, Stdev: 0.008643925366417223 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7837728261947632, Stdev: 0.00417671858823589 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7819472551345825, Stdev: 0.01208557255936319 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7813387393951416, Stdev: 0.012348255919897213 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7809330701828003, Stdev: 0.005701206154813713 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7809330582618713, Stdev: 0.008214369775120394 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7784989953041077, Stdev: 0.006497217060495398 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7823529481887818, Stdev: 0.007871629419466277 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7784989953041077, Stdev: 0.009396546054498436 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7843813300132751, Stdev: 0.005115424124076285 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7813387393951416, Stdev: 0.008179239984970447 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7845841765403747, Stdev: 0.007659717808105952 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7839756488800049, Stdev: 0.003571366552926297 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7807302117347718, Stdev: 0.008662942124442732 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7825557708740234, Stdev: 0.003985169044661304 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7803245425224304, Stdev: 0.0026909644385760163 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7801217079162598, Stdev: 0.0058928428429681256 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7841785073280334, Stdev: 0.004859680724914396 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7776876091957092, Stdev: 0.008323852742320847 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7831642985343933, Stdev: 0.0036511103312606946 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7748478889465332, Stdev: 0.005171421334025931 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7778904676437378, Stdev: 0.010182468137671275 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7839756727218627, Stdev: 0.010402324769691062 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7829614520072937, Stdev: 0.005628565075408968 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7831643104553223, Stdev: 0.005381947747149567 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7837728142738343, Stdev: 0.00661643054588285 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.782758629322052, Stdev: 0.012024133131554829 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.78052738904953, Stdev: 0.00825434834371681 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7803245425224304, Stdev: 0.0030493528401823577 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7839756488800049, Stdev: 0.00756241917778455 with: {'activation': 'relu', 'batch_size': 60, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.787221097946167, Stdev: 0.006950027032487927 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7778904676437378, Stdev: 0.008289179080662306 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7817444205284119, Stdev: 0.008299095420559373 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7853955388069153, Stdev: 0.006776182678838023 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7795131802558899, Stdev: 0.005722803681954424 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7827586174011231, Stdev: 0.005892838329277931 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7843813419342041, Stdev: 0.0023654816174767633 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7795131802558899, Stdev: 0.006528800333703466 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7809330582618713, Stdev: 0.009448931105776623 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7839756608009338, Stdev: 0.006997223112275698 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7801216959953308, Stdev: 0.008274278342625946 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7809330582618713, Stdev: 0.007616626365315911 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7839756727218627, Stdev: 0.012187272086655548 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7853955507278443, Stdev: 0.007819201394497221 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7839756608009338, Stdev: 0.005517833676803774 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7805273771286011, Stdev: 0.004993320984005395 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7801216959953308, Stdev: 0.0037070316235804906 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7819472551345825, Stdev: 0.006152437391023557 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7864097356796265, Stdev: 0.0035482553073505002 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.776673412322998, Stdev: 0.006490878015769221 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7860040545463562, Stdev: 0.006727437155000261 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7799188613891601, Stdev: 0.000641438192612333 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7787018299102784, Stdev: 0.004730989812447577 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7821500897407532, Stdev: 0.003922733222226103 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7835699796676636, Stdev: 0.004782884744463644 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7815415859222412, Stdev: 0.007605817106617906 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7862069010734558, Stdev: 0.007627409209908486 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7860040545463562, Stdev: 0.005091229319434382 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.781135904788971, Stdev: 0.005962227659555538 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.786612582206726, Stdev: 0.004331463525883936 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7791075110435486, Stdev: 0.004462466998556633 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7770790934562684, Stdev: 0.004943623823669162 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.0, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.786612582206726, Stdev: 0.005969134623790323 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7821501016616821, Stdev: 0.010129817645411024 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7829614639282226, Stdev: 0.005517835867936976 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.784989869594574, Stdev: 0.012585875257911692 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7855983734130859, Stdev: 0.004378700080476693 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7823529362678527, Stdev: 0.009655681000708622 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.782758629322052, Stdev: 0.010190561993232821 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7839756608009338, Stdev: 0.007507801295159616 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7809330701828003, Stdev: 0.0033941576290812204 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7803245306015014, Stdev: 0.01005232936901078 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7843813419342041, Stdev: 0.0043787122294470845 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7772819519042968, Stdev: 0.00469607546253738 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.787221097946167, Stdev: 0.0050261651656186455 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7855983853340149, Stdev: 0.0018810531757666147 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7821501016616821, Stdev: 0.007038294988047891 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7843813538551331, Stdev: 0.011084041957432861 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.001, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7809330582618713, Stdev: 0.00972785703717467 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7797160267829895, Stdev: 0.005420030624147124 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7807302236557007, Stdev: 0.012075361205085048 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7801217198371887, Stdev: 0.005457859298137298 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7835699677467346, Stdev: 0.006497205151193081 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7829614520072937, Stdev: 0.007697233177843552 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7825557827949524, Stdev: 0.008128800947205616 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7876267790794372, Stdev: 0.006401505555056547 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.1, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7813387393951416, Stdev: 0.003933202384723378 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7833671331405639, Stdev: 0.005969145561201623 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7801216959953308, Stdev: 0.004176735377227978 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7839756608009338, Stdev: 0.004397467697514189 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n","Means: 0.7807302117347718, Stdev: 0.006030859560042987 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Adam'}\n","Means: 0.7825557827949524, Stdev: 0.005540154396164488 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 1, 'optimizer': 'Nadam'}\n","Means: 0.7809330701828003, Stdev: 0.004397464948133049 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Adam'}\n","Means: 0.7793103456497192, Stdev: 0.008524094239621218 with: {'activation': 'relu', 'batch_size': 80, 'dropout_rate': 0.1, 'epochs': 100, 'l1_reg': 1e-05, 'l2_reg': 1e-05, 'lr': 0.01, 'momentum': 0.2, 'net_weight_init': 'lecun_uniform', 'neurons': 10, 'optimizer': 'Nadam'}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"vjYkuImavBVG","colab_type":"code","colab":{}},"source":["%tensorboard --logdir logs"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"tDhlINEIMaO-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":224},"executionInfo":{"status":"ok","timestamp":1596250792182,"user_tz":240,"elapsed":195596,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"489c0397-0a74-4f41-fb47-f548f9b5e5d8"},"source":["# GridSearch and Cross Validation for second pass of the following hyperparameters:\n","\n","# batch_size\n","# training epochs\n","# optimizer\n","# learning rate (if applicable to optimizer)\n","# momentum (if applicable to optimizer)\n","# activation functions\n","# network weight initialization\n","# dropout regularization\n","# number of neurons in the hidden layer\n","\n","# All of the hyperparameters\n","\n","#param_grid = {'batch_size' : [10, 20, 40, 60, 80, 100],\n","# 'epochs' : [50, 75, 100, 125, 150],\n","# 'optimizer' : ['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'],\n","# 'lr' : [.001, .01, .1],\n","# 'momentum' : [0.0, 0.2, 0.4, 0.6, 0.8, 0.9],\n","# 'activation' : ['softmax', 'softplus', 'softsign', 'relu', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'],\n","# 'net_weight_init' : ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'],\n","# 'dropout_rate' : [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],\n","# 'neurons' : [1, 5, 10, 15, 20, 25, 30],\n","# 'l1_reg': [1e-2, le-3, le-4], # weight decay, add this term to loss function, small coeffecient\n","# 'l2_reg': [1e-2, le-3, le-4]}\n","\n","# However, there are too many options, and colab will time-out in 12 hours before\n","# the model finishes running, so trim this down.\n","\n","\n","# Fix random seed for reproducibility\n","\n","seed = 7\n","np.random.seed(seed)\n","\n","# Function to create model, required for KerasClassifier\n","\n","def create_model(batch_size, epochs, optimizer, lr, momentum, activation, \n"," net_weight_init, dropout_rate, neurons, l1_reg, l2_reg):\n","\n"," # Create optimizer\n","\n"," adam = Adam(learning_rate=lr)\n","\n"," # Create model\n","\n"," model = Sequential()\n"," model.add(Dense(19, input_dim=19, activation=activation))\n"," model.add(Dense(1, activation='sigmoid'))\n","\n"," # Compile model\n","\n"," model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n","\n"," return model\n","\n","# Create model\n","\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# Define the grid search parameters\n","\n","param_grid = {'batch_size' : [80, 100],\n"," 'epochs' : [100],\n"," 'optimizer' : ['Adam', 'Nadam'],\n"," 'lr' : [.01, .1],\n"," 'momentum' : [0.0, 0.2],\n"," 'activation' : ['relu'],\n"," 'net_weight_init' : ['uniform', 'lecun_uniform'],\n"," 'dropout_rate' : [0.0, 0.1],\n"," 'neurons' : [1, 5],\n"," 'l1_reg': [1e-5],\n"," 'l2_reg': [1e-5]}\n","\n","# Create Grid Search\n","\n","grid = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_jobs=1)\n","grid_result = grid.fit(X_train, y_train)\n","\n","# Report Results\n","\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\")\n","\n","# 1st Best: 0.7896551728248596 using {'optimizer': 'Nadam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.01, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 10, 'dropout_rate': 0.1, 'batch_size': 80, 'activation': 'relu'}\n","# This was run with CPU, and it took about 10 minutes\n","# 2nd Best: Best: 0.7851926922798157 using {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 100, 'activation': 'relu'}"],"execution_count":25,"outputs":[{"output_type":"stream","text":["Best: 0.7851926922798157 using {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7821501016616821, Stdev: 0.008519264629910673 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.0, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7851926922798157, Stdev: 0.007125412921382508 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7803245425224304, Stdev: 0.00473967400478512 with: {'optimizer': 'Nadam', 'neurons': 5, 'net_weight_init': 'lecun_uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7809330582618713, Stdev: 0.009317396799397845 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'lecun_uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7829614639282226, Stdev: 0.007723905789610606 with: {'optimizer': 'Nadam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.2, 'lr': 0.01, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 80, 'activation': 'relu'}\n","Means: 0.7827586174011231, Stdev: 0.009012109289772814 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'lecun_uniform', 'momentum': 0.0, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.0, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7839756608009338, Stdev: 0.005091243567673705 with: {'optimizer': 'Adam', 'neurons': 5, 'net_weight_init': 'lecun_uniform', 'momentum': 0.0, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.0, 'batch_size': 100, 'activation': 'relu'}\n","Means: 0.7821501016616821, Stdev: 0.006132321237977442 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'uniform', 'momentum': 0.0, 'lr': 0.01, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.1, 'batch_size': 80, 'activation': 'relu'}\n","Means: 0.7827586174011231, Stdev: 0.004321974292389376 with: {'optimizer': 'Adam', 'neurons': 1, 'net_weight_init': 'lecun_uniform', 'momentum': 0.2, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.0, 'batch_size': 80, 'activation': 'relu'}\n","Means: 0.7807302236557007, Stdev: 0.006165800195334738 with: {'optimizer': 'Nadam', 'neurons': 1, 'net_weight_init': 'lecun_uniform', 'momentum': 0.0, 'lr': 0.1, 'l2_reg': 1e-05, 'l1_reg': 1e-05, 'epochs': 100, 'dropout_rate': 0.0, 'batch_size': 100, 'activation': 'relu'}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"f-SOBwYFvDcg","colab_type":"code","colab":{}},"source":["%tensorboard --logdir logs"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"FfZRtJ7MCN3x"},"source":["## Stretch Goals:\n","\n","- Try to implement Random Search Hyperparameter Tuning on this dataset\n","- Try to implement Bayesian Optimiation tuning on this dataset using hyperas or hyperopt (if you're brave)\n","- Practice hyperparameter tuning other datasets that we have looked at. How high can you get MNIST? Above 99%?\n","- Study for the Sprint Challenge\n"," - Can you implement both perceptron and MLP models from scratch with forward and backpropagation?\n"," - Can you implement both perceptron and MLP models in keras and tune their hyperparameters with cross validation?"]}]} \ No newline at end of file diff --git a/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Lecture.ipynb b/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Lecture.ipynb new file mode 100644 index 00000000..691c0570 --- /dev/null +++ b/module4-Hyperparameter-Tuning/LS_DS_434_Hyperparameter_Tuning_Lecture.ipynb @@ -0,0 +1 @@ +{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"U4-S2-NNF-DS12","language":"python","name":"u4-s2-nnf-ds12"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.7"},"colab":{"name":"DSPT5_LS_DS_434_Hyperparameter_Tuning_Lecture.ipynb","provenance":[{"file_id":"1ovQzqSv5iV2jlRp7qXm8GnxprcETlgHw","timestamp":1596074412876},{"file_id":"https://github.com/LambdaSchool/DS-Unit-4-Sprint-2-Neural-Networks/blob/main/module4-Hyperparameter-Tuning/LS_DS_424_Hyperparameter_Tuning_Lecture.ipynb","timestamp":1596068345353}],"collapsed_sections":[]}},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"41TS0Sa0rDNx"},"source":["Lambda School Data Science\n","\n","*Unit 4, Sprint 2, Module 4*\n","\n","---"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"u5EbNocPE-rG"},"source":["# Neural Networks & GPUs (Prepare)\n","*aka Hyperparameter Tuning*\n","\n","*aka Big Servers for Big Problems*"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"RUtS3IB4E-rH"},"source":["## Learning Objectives\n","* Part 1: Describe the major hyperparemeters to tune\n","* Part 2: Implement an experiment tracking framework\n","* Part 3: Search the hyperparameter space using RandomSearch (Optional)"]},{"cell_type":"code","metadata":{"id":"6Z4Lo0FIE-rH","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596188361595,"user_tz":240,"elapsed":16371,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"8434a2bb-2c9a-422b-e079-8cd685f946cb"},"source":["#wandb_group = alexkim\n","#wandb_project = \"...\"\n","\n","!pip install --upgrade wandb\n","!wandb login 2c87e05ce2646c11a6c77b8dcea6b9d0e0f2eff4"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Collecting wandb\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/94/19/f8db9eff4b0173adf6dd2e8b0c3d8de0bfe10ec9ed63d247665980d82258/wandb-0.9.4-py2.py3-none-any.whl (1.4MB)\n","\u001b[K |████████████████████████████████| 1.4MB 2.6MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: Click>=7.0 in /usr/local/lib/python3.6/dist-packages (from wandb) (7.1.2)\n","Collecting configparser>=3.8.1\n"," Downloading https://files.pythonhosted.org/packages/4b/6b/01baa293090240cf0562cc5eccb69c6f5006282127f2b846fad011305c79/configparser-5.0.0-py3-none-any.whl\n","Requirement already satisfied, skipping upgrade: PyYAML>=3.10 in /usr/local/lib/python3.6/dist-packages (from wandb) (3.13)\n","Requirement already satisfied, skipping upgrade: psutil>=5.0.0 in /usr/local/lib/python3.6/dist-packages (from wandb) (5.4.8)\n","Requirement already satisfied, skipping upgrade: nvidia-ml-py3>=7.352.0 in /usr/local/lib/python3.6/dist-packages (from wandb) (7.352.0)\n","Collecting subprocess32>=3.5.3\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/32/c8/564be4d12629b912ea431f1a50eb8b3b9d00f1a0b1ceff17f266be190007/subprocess32-3.5.4.tar.gz (97kB)\n","\u001b[K |████████████████████████████████| 102kB 5.5MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: requests>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from wandb) (2.23.0)\n","Collecting shortuuid>=0.5.0\n"," Downloading https://files.pythonhosted.org/packages/25/a6/2ecc1daa6a304e7f1b216f0896b26156b78e7c38e1211e9b798b4716c53d/shortuuid-1.0.1-py3-none-any.whl\n","Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from wandb) (2.8.1)\n","Collecting watchdog>=0.8.3\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/0e/06/121302598a4fc01aca942d937f4a2c33430b7181137b35758913a8db10ad/watchdog-0.10.3.tar.gz (94kB)\n","\u001b[K |████████████████████████████████| 102kB 7.1MB/s \n","\u001b[?25hCollecting docker-pycreds>=0.4.0\n"," Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl\n","Collecting GitPython>=1.0.0\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/f9/1e/a45320cab182bf1c8656107b3d4c042e659742822fc6bff150d769a984dd/GitPython-3.1.7-py3-none-any.whl (158kB)\n","\u001b[K |████████████████████████████████| 163kB 11.5MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from wandb) (1.15.0)\n","Collecting gql==0.2.0\n"," Downloading https://files.pythonhosted.org/packages/c4/6f/cf9a3056045518f06184e804bae89390eb706168349daa9dff8ac609962a/gql-0.2.0.tar.gz\n","Collecting sentry-sdk>=0.4.0\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/19/2a/5461e1fe0026d6eab10571f81d052894328e97f15614abb6a576c65bc82d/sentry_sdk-0.16.2-py2.py3-none-any.whl (109kB)\n","\u001b[K |████████████████████████████████| 112kB 13.6MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->wandb) (2020.6.20)\n","Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->wandb) (1.24.3)\n","Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->wandb) (3.0.4)\n","Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->wandb) (2.10)\n","Collecting pathtools>=0.1.1\n"," Downloading https://files.pythonhosted.org/packages/e7/7f/470d6fcdf23f9f3518f6b0b76be9df16dcc8630ad409947f8be2eb0ed13a/pathtools-0.1.2.tar.gz\n","Collecting gitdb<5,>=4.0.1\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/48/11/d1800bca0a3bae820b84b7d813ad1eff15a48a64caea9c823fc8c1b119e8/gitdb-4.0.5-py3-none-any.whl (63kB)\n","\u001b[K |████████████████████████████████| 71kB 6.2MB/s \n","\u001b[?25hCollecting graphql-core<2,>=0.5.0\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/b0/89/00ad5e07524d8c523b14d70c685e0299a8b0de6d0727e368c41b89b7ed0b/graphql-core-1.1.tar.gz (70kB)\n","\u001b[K |████████████████████████████████| 71kB 6.2MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: promise<3,>=2.0 in /usr/local/lib/python3.6/dist-packages (from gql==0.2.0->wandb) (2.3)\n","Collecting smmap<4,>=3.0.1\n"," Downloading https://files.pythonhosted.org/packages/b0/9a/4d409a6234eb940e6a78dfdfc66156e7522262f5f2fecca07dc55915952d/smmap-3.0.4-py2.py3-none-any.whl\n","Building wheels for collected packages: subprocess32, watchdog, gql, pathtools, graphql-core\n"," Building wheel for subprocess32 (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for subprocess32: filename=subprocess32-3.5.4-cp36-none-any.whl size=6489 sha256=cedb31aa6b9c96b26a84d37f179eeed4025dfd2033919612f79b720bd01ec6e1\n"," Stored in directory: /root/.cache/pip/wheels/68/39/1a/5e402bdfdf004af1786c8b853fd92f8c4a04f22aad179654d1\n"," Building wheel for watchdog (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for watchdog: filename=watchdog-0.10.3-cp36-none-any.whl size=73870 sha256=c5f00aec98e448313d3f7c7ba54c5847e3d72287de80b39de3345c3bb0a8ff63\n"," Stored in directory: /root/.cache/pip/wheels/a8/1d/38/2c19bb311f67cc7b4d07a2ec5ea36ab1a0a0ea50db994a5bc7\n"," Building wheel for gql (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for gql: filename=gql-0.2.0-cp36-none-any.whl size=7630 sha256=daba685e7b844f5dfdceec7e4ce85f7af149fe42fffdc6851feb3de00d259ec0\n"," Stored in directory: /root/.cache/pip/wheels/ce/0e/7b/58a8a5268655b3ad74feef5aa97946f0addafb3cbb6bd2da23\n"," Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for pathtools: filename=pathtools-0.1.2-cp36-none-any.whl size=8784 sha256=142a77fee1020f2774c98edb0042541784f36e141585264d70fc680d107c6368\n"," Stored in directory: /root/.cache/pip/wheels/0b/04/79/c3b0c3a0266a3cb4376da31e5bfe8bba0c489246968a68e843\n"," Building wheel for graphql-core (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for graphql-core: filename=graphql_core-1.1-cp36-none-any.whl size=104650 sha256=ce55d38ff3b29f0ae13f7e069e00c6069ef575483e4f89f1b90c54b0a4da5afe\n"," Stored in directory: /root/.cache/pip/wheels/45/99/d7/c424029bb0fe910c63b68dbf2aa20d3283d023042521bcd7d5\n","Successfully built subprocess32 watchdog gql pathtools graphql-core\n","Installing collected packages: configparser, subprocess32, shortuuid, pathtools, watchdog, docker-pycreds, smmap, gitdb, GitPython, graphql-core, gql, sentry-sdk, wandb\n","Successfully installed GitPython-3.1.7 configparser-5.0.0 docker-pycreds-0.4.0 gitdb-4.0.5 gql-0.2.0 graphql-core-1.1 pathtools-0.1.2 sentry-sdk-0.16.2 shortuuid-1.0.1 smmap-3.0.4 subprocess32-3.5.4 wandb-0.9.4 watchdog-0.10.3\n","\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n","\u001b[32mSuccessfully logged in to Weights & Biases!\u001b[0m\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"95DDiPc8E-rK","colab_type":"text"},"source":["# Hyperparameter Options (Learn)\n",""]},{"cell_type":"markdown","metadata":{"id":"LNpcLpvLE-rK","colab_type":"text"},"source":["## Overview\n","\n","Hyperparameter tuning is much more important with neural networks than it has been with any other models that we have considered up to this point. Other supervised learning models might have a couple of parameters, but neural networks can have dozens. These can substantially affect the accuracy of our models and although it can be a time consuming process is a necessary step when working with neural networks.\n","​\n","Hyperparameter tuning comes with a challenge. How can we compare models specified with different hyperparameters if our model's final error metric can vary somewhat erratically? How do we avoid just getting unlucky and selecting the wrong hyperparameter? This is a problem that to a certain degree we just have to live with as we test and test again. However, we can minimize it somewhat by pairing our experiments with Cross Validation to reduce the variance of our final accuracy values."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"0s0o2pqBs88q"},"source":["### Load Boston Housing Data"]},{"cell_type":"code","metadata":{"id":"bwWjtILAE-rL","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":51},"executionInfo":{"status":"ok","timestamp":1596188378100,"user_tz":240,"elapsed":2178,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"56ed301d-6e3d-4084-90af-10cbe71aede6"},"source":["from tensorflow.keras.datasets import boston_housing\n","\n","(x_train, y_train), (x_test, y_test) = boston_housing.load_data()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/boston_housing.npz\n","57344/57026 [==============================] - 0s 0us/step\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Pm7zow5IvaTt"},"source":["### Normalizing Input Data\n","\n","It's not 100% necessary to normalize/scale your input data before feeding it to a neural network, the network can learn the appropriate weights to deal with data of as long as it is numerically represented, but it is recommended as it can help **make training faster** and **reduces the chances that gradient descent might get stuck in a local optimum**.\n","\n",""]},{"cell_type":"code","metadata":{"id":"AWVKdm-3E-rO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":527},"executionInfo":{"status":"ok","timestamp":1596188385713,"user_tz":240,"elapsed":879,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"a9a3d1f6-acbe-4a27-ae29-722d98bca5d5"},"source":["from sklearn.preprocessing import StandardScaler\n","\n","scaler = StandardScaler()\n","\n","x_train = scaler.fit_transform(x_train)\n","x_test = scaler.transform(x_test)\n","print(x_train[:10])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[[-0.27224633 -0.48361547 -0.43576161 -0.25683275 -0.1652266 -0.1764426\n"," 0.81306188 0.1166983 -0.62624905 -0.59517003 1.14850044 0.44807713\n"," 0.8252202 ]\n"," [-0.40342651 2.99178419 -1.33391162 -0.25683275 -1.21518188 1.89434613\n"," -1.91036058 1.24758524 -0.85646254 -0.34843254 -1.71818909 0.43190599\n"," -1.32920239]\n"," [ 0.1249402 -0.48361547 1.0283258 -0.25683275 0.62864202 -1.82968811\n"," 1.11048828 -1.18743907 1.67588577 1.5652875 0.78447637 0.22061726\n"," -1.30850006]\n"," [-0.40149354 -0.48361547 -0.86940196 -0.25683275 -0.3615597 -0.3245576\n"," -1.23667187 1.10717989 -0.51114231 -1.094663 0.78447637 0.44807713\n"," -0.65292624]\n"," [-0.0056343 -0.48361547 1.0283258 -0.25683275 1.32861221 0.15364225\n"," 0.69480801 -0.57857203 1.67588577 1.5652875 0.78447637 0.3898823\n"," 0.26349695]\n"," [-0.37502238 -0.48361547 -0.54747912 -0.25683275 -0.54935658 -0.78865126\n"," 0.18954148 0.48371503 -0.51114231 -0.71552978 0.51145832 0.38669063\n"," -0.13812828]\n"," [ 0.58963463 -0.48361547 1.0283258 -0.25683275 1.21764133 -1.03127774\n"," 1.11048828 -1.06518235 1.67588577 1.5652875 0.78447637 0.44807713\n"," 1.49873604]\n"," [ 0.0381708 -0.48361547 1.24588095 -0.25683275 2.67733525 -1.12719983\n"," 1.11048828 -1.14833073 -0.51114231 -0.01744323 -1.71818909 0.44807713\n"," 1.88793986]\n"," [-0.17228416 -0.48361547 1.24588095 -0.25683275 2.67733525 -0.90150078\n"," 1.11048828 -1.09664657 -0.51114231 -0.01744323 -1.71818909 -1.97365769\n"," 0.53952803]\n"," [-0.22932104 -0.48361547 1.58544339 -0.25683275 0.56888847 -1.76056777\n"," 1.11048828 -1.13471925 -0.62624905 0.18716835 1.23950646 0.44807713\n"," 2.99068404]]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"l6hgCWbir90R"},"source":["### Model Validation using an automatic verification Dataset\n","\n","Instead of doing seperate train test split class, Keras has a really nice feature that you can set the validation.data argument when fitting your model and Keras will take that portion of your test data and use it as a validation dataset. "]},{"cell_type":"code","metadata":{"colab_type":"code","id":"GMXVfmzXp1Oo","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596188402053,"user_tz":240,"elapsed":10859,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"b29149d2-9aa6-4d45-aac2-44bfc9fb4737"},"source":["from tensorflow import keras\n","from tensorflow.keras.models import Sequential\n","from tensorflow.keras.layers import Dense\n","\n","# Important Hyperparameters\n","inputs = x_train.shape[1]\n","epochs = 75\n","batch_size = 10\n","\n","\n","# Create Model\n","model = Sequential()\n","model.add(Dense(64, activation='relu', input_shape=(inputs,)))\n","model.add(Dense(64, activation='relu'))\n","model.add(Dense(1)) # no activation function, essentially linear for regression\n","\n","# Compile Model\n","model.compile(optimizer='adam', loss='mse', metrics=['mse', 'mae'])\n","\n","# Fit Model\n","model.fit(x_train, y_train, \n"," validation_data=(x_test,y_test), \n"," epochs=epochs, \n"," batch_size=batch_size\n"," )"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch 1/75\n","41/41 [==============================] - 0s 6ms/step - loss: 514.0092 - mse: 514.0092 - mae: 20.7252 - val_loss: 434.9840 - val_mse: 434.9841 - val_mae: 18.9418\n","Epoch 2/75\n","41/41 [==============================] - 0s 3ms/step - loss: 274.6827 - mse: 274.6827 - mae: 14.0831 - val_loss: 131.5931 - val_mse: 131.5931 - val_mae: 9.5006\n","Epoch 3/75\n","41/41 [==============================] - 0s 2ms/step - loss: 75.6898 - mse: 75.6898 - mae: 6.3636 - val_loss: 53.2264 - val_mse: 53.2264 - val_mae: 5.7947\n","Epoch 4/75\n","41/41 [==============================] - 0s 2ms/step - loss: 39.5782 - mse: 39.5782 - mae: 4.5561 - val_loss: 36.0183 - val_mse: 36.0183 - val_mae: 4.7729\n","Epoch 5/75\n","41/41 [==============================] - 0s 3ms/step - loss: 28.7986 - mse: 28.7986 - mae: 3.8497 - val_loss: 29.9567 - val_mse: 29.9567 - val_mae: 4.3366\n","Epoch 6/75\n","41/41 [==============================] - 0s 3ms/step - loss: 24.0347 - mse: 24.0347 - mae: 3.4889 - val_loss: 27.5862 - val_mse: 27.5862 - val_mae: 4.0782\n","Epoch 7/75\n","41/41 [==============================] - 0s 3ms/step - loss: 21.3435 - mse: 21.3435 - mae: 3.2991 - val_loss: 27.0629 - val_mse: 27.0629 - val_mae: 4.0411\n","Epoch 8/75\n","41/41 [==============================] - 0s 3ms/step - loss: 19.2274 - mse: 19.2274 - mae: 3.1361 - val_loss: 25.4956 - val_mse: 25.4956 - val_mae: 3.8080\n","Epoch 9/75\n","41/41 [==============================] - 0s 2ms/step - loss: 17.7311 - mse: 17.7311 - mae: 2.9682 - val_loss: 25.2189 - val_mse: 25.2189 - val_mae: 3.8059\n","Epoch 10/75\n","41/41 [==============================] - 0s 3ms/step - loss: 17.1372 - mse: 17.1372 - mae: 2.9428 - val_loss: 25.0563 - val_mse: 25.0563 - val_mae: 3.7397\n","Epoch 11/75\n","41/41 [==============================] - 0s 2ms/step - loss: 15.6228 - mse: 15.6228 - mae: 2.7895 - val_loss: 23.7237 - val_mse: 23.7237 - val_mae: 3.5330\n","Epoch 12/75\n","41/41 [==============================] - 0s 3ms/step - loss: 14.7233 - mse: 14.7233 - mae: 2.7174 - val_loss: 23.3178 - val_mse: 23.3178 - val_mae: 3.4798\n","Epoch 13/75\n","41/41 [==============================] - 0s 2ms/step - loss: 13.9190 - mse: 13.9190 - mae: 2.6471 - val_loss: 23.9869 - val_mse: 23.9869 - val_mae: 3.5014\n","Epoch 14/75\n","41/41 [==============================] - 0s 2ms/step - loss: 13.2388 - mse: 13.2388 - mae: 2.5739 - val_loss: 23.2438 - val_mse: 23.2438 - val_mae: 3.4058\n","Epoch 15/75\n","41/41 [==============================] - 0s 2ms/step - loss: 12.6390 - mse: 12.6390 - mae: 2.5190 - val_loss: 23.3307 - val_mse: 23.3307 - val_mae: 3.3796\n","Epoch 16/75\n","41/41 [==============================] - 0s 3ms/step - loss: 12.1060 - mse: 12.1060 - mae: 2.4517 - val_loss: 24.2746 - val_mse: 24.2746 - val_mae: 3.4283\n","Epoch 17/75\n","41/41 [==============================] - 0s 3ms/step - loss: 11.9750 - mse: 11.9750 - mae: 2.4555 - val_loss: 22.7051 - val_mse: 22.7051 - val_mae: 3.2850\n","Epoch 18/75\n","41/41 [==============================] - 0s 3ms/step - loss: 11.5243 - mse: 11.5243 - mae: 2.4357 - val_loss: 22.4600 - val_mse: 22.4600 - val_mae: 3.2506\n","Epoch 19/75\n","41/41 [==============================] - 0s 3ms/step - loss: 11.1250 - mse: 11.1250 - mae: 2.3653 - val_loss: 22.1321 - val_mse: 22.1321 - val_mae: 3.2142\n","Epoch 20/75\n","41/41 [==============================] - 0s 3ms/step - loss: 10.6568 - mse: 10.6568 - mae: 2.3552 - val_loss: 23.2313 - val_mse: 23.2313 - val_mae: 3.3010\n","Epoch 21/75\n","41/41 [==============================] - 0s 2ms/step - loss: 10.5037 - mse: 10.5037 - mae: 2.2871 - val_loss: 22.1407 - val_mse: 22.1407 - val_mae: 3.1812\n","Epoch 22/75\n","41/41 [==============================] - 0s 3ms/step - loss: 10.2905 - mse: 10.2905 - mae: 2.3019 - val_loss: 22.7514 - val_mse: 22.7514 - val_mae: 3.2306\n","Epoch 23/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.9972 - mse: 9.9972 - mae: 2.2813 - val_loss: 22.2856 - val_mse: 22.2856 - val_mae: 3.1909\n","Epoch 24/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.9144 - mse: 9.9144 - mae: 2.2598 - val_loss: 22.3396 - val_mse: 22.3396 - val_mae: 3.1586\n","Epoch 25/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.7556 - mse: 9.7556 - mae: 2.2391 - val_loss: 22.1466 - val_mse: 22.1466 - val_mae: 3.1379\n","Epoch 26/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.7293 - mse: 9.7293 - mae: 2.2511 - val_loss: 22.4811 - val_mse: 22.4811 - val_mae: 3.1405\n","Epoch 27/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.4003 - mse: 9.4003 - mae: 2.2010 - val_loss: 23.0302 - val_mse: 23.0302 - val_mae: 3.1815\n","Epoch 28/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.2906 - mse: 9.2906 - mae: 2.1883 - val_loss: 21.9236 - val_mse: 21.9236 - val_mae: 3.0802\n","Epoch 29/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.4611 - mse: 9.4611 - mae: 2.2199 - val_loss: 23.9851 - val_mse: 23.9851 - val_mae: 3.2426\n","Epoch 30/75\n","41/41 [==============================] - 0s 3ms/step - loss: 9.1702 - mse: 9.1702 - mae: 2.2024 - val_loss: 22.2426 - val_mse: 22.2426 - val_mae: 3.1385\n","Epoch 31/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.7888 - mse: 8.7888 - mae: 2.1481 - val_loss: 21.8907 - val_mse: 21.8907 - val_mae: 3.0539\n","Epoch 32/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.7062 - mse: 8.7062 - mae: 2.1216 - val_loss: 22.2148 - val_mse: 22.2148 - val_mae: 3.1105\n","Epoch 33/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.8687 - mse: 8.8687 - mae: 2.1537 - val_loss: 21.3159 - val_mse: 21.3159 - val_mae: 3.0741\n","Epoch 34/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.5699 - mse: 8.5699 - mae: 2.1485 - val_loss: 19.9102 - val_mse: 19.9102 - val_mae: 2.9768\n","Epoch 35/75\n","41/41 [==============================] - 0s 4ms/step - loss: 8.5797 - mse: 8.5797 - mae: 2.1115 - val_loss: 21.7396 - val_mse: 21.7396 - val_mae: 3.0293\n","Epoch 36/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.3488 - mse: 8.3488 - mae: 2.0929 - val_loss: 23.3645 - val_mse: 23.3645 - val_mae: 3.2148\n","Epoch 37/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.3953 - mse: 8.3953 - mae: 2.0882 - val_loss: 22.7452 - val_mse: 22.7452 - val_mae: 3.1476\n","Epoch 38/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.0026 - mse: 8.0026 - mae: 2.0462 - val_loss: 22.1009 - val_mse: 22.1009 - val_mae: 3.0715\n","Epoch 39/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.9713 - mse: 7.9713 - mae: 2.0357 - val_loss: 21.6011 - val_mse: 21.6011 - val_mae: 3.0637\n","Epoch 40/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.9509 - mse: 7.9509 - mae: 2.0274 - val_loss: 21.1624 - val_mse: 21.1624 - val_mae: 2.9840\n","Epoch 41/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.9093 - mse: 7.9093 - mae: 2.0216 - val_loss: 22.8402 - val_mse: 22.8402 - val_mae: 3.1792\n","Epoch 42/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.8368 - mse: 7.8368 - mae: 2.0164 - val_loss: 20.3073 - val_mse: 20.3073 - val_mae: 2.9283\n","Epoch 43/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.6577 - mse: 7.6577 - mae: 1.9821 - val_loss: 21.2846 - val_mse: 21.2846 - val_mae: 3.0244\n","Epoch 44/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.4597 - mse: 7.4597 - mae: 1.9815 - val_loss: 22.1813 - val_mse: 22.1813 - val_mae: 3.0481\n","Epoch 45/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.5839 - mse: 7.5839 - mae: 1.9770 - val_loss: 21.2119 - val_mse: 21.2119 - val_mae: 3.0103\n","Epoch 46/75\n","41/41 [==============================] - 0s 3ms/step - loss: 8.1784 - mse: 8.1784 - mae: 2.0882 - val_loss: 22.3565 - val_mse: 22.3565 - val_mae: 3.0048\n","Epoch 47/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.5082 - mse: 7.5082 - mae: 1.9784 - val_loss: 20.9093 - val_mse: 20.9093 - val_mae: 2.9362\n","Epoch 48/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.2457 - mse: 7.2457 - mae: 1.9256 - val_loss: 20.6428 - val_mse: 20.6428 - val_mae: 2.9141\n","Epoch 49/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.2925 - mse: 7.2925 - mae: 1.9627 - val_loss: 21.1908 - val_mse: 21.1908 - val_mae: 2.9858\n","Epoch 50/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.1714 - mse: 7.1714 - mae: 1.9291 - val_loss: 22.0807 - val_mse: 22.0807 - val_mae: 3.0624\n","Epoch 51/75\n","41/41 [==============================] - 0s 3ms/step - loss: 7.0935 - mse: 7.0935 - mae: 1.9333 - val_loss: 19.7454 - val_mse: 19.7454 - val_mae: 2.8568\n","Epoch 52/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.9767 - mse: 6.9767 - mae: 1.9226 - val_loss: 18.7870 - val_mse: 18.7870 - val_mae: 2.7679\n","Epoch 53/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.8090 - mse: 6.8090 - mae: 1.8592 - val_loss: 20.1801 - val_mse: 20.1801 - val_mae: 2.8538\n","Epoch 54/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.7521 - mse: 6.7521 - mae: 1.8715 - val_loss: 20.1352 - val_mse: 20.1352 - val_mae: 2.8272\n","Epoch 55/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.5944 - mse: 6.5944 - mae: 1.8518 - val_loss: 19.5575 - val_mse: 19.5575 - val_mae: 2.8644\n","Epoch 56/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.5074 - mse: 6.5074 - mae: 1.8393 - val_loss: 19.6137 - val_mse: 19.6137 - val_mae: 2.8024\n","Epoch 57/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.3821 - mse: 6.3821 - mae: 1.8463 - val_loss: 19.0578 - val_mse: 19.0578 - val_mae: 2.7660\n","Epoch 58/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.7939 - mse: 6.7939 - mae: 1.8796 - val_loss: 18.8629 - val_mse: 18.8629 - val_mae: 2.7922\n","Epoch 59/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.4643 - mse: 6.4643 - mae: 1.8152 - val_loss: 19.2437 - val_mse: 19.2437 - val_mae: 2.7562\n","Epoch 60/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.3447 - mse: 6.3447 - mae: 1.8147 - val_loss: 19.4260 - val_mse: 19.4260 - val_mae: 2.8777\n","Epoch 61/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.8662 - mse: 6.8662 - mae: 1.9098 - val_loss: 18.4795 - val_mse: 18.4795 - val_mae: 2.7544\n","Epoch 62/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.4291 - mse: 6.4291 - mae: 1.8315 - val_loss: 18.4120 - val_mse: 18.4120 - val_mae: 2.7480\n","Epoch 63/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.2472 - mse: 6.2472 - mae: 1.8005 - val_loss: 19.0121 - val_mse: 19.0121 - val_mae: 2.7526\n","Epoch 64/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.0105 - mse: 6.0105 - mae: 1.7618 - val_loss: 19.4557 - val_mse: 19.4557 - val_mae: 2.8173\n","Epoch 65/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.1950 - mse: 6.1950 - mae: 1.7818 - val_loss: 20.6050 - val_mse: 20.6050 - val_mae: 2.9213\n","Epoch 66/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.1843 - mse: 6.1843 - mae: 1.8095 - val_loss: 18.8913 - val_mse: 18.8913 - val_mae: 2.7239\n","Epoch 67/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.0272 - mse: 6.0272 - mae: 1.7711 - val_loss: 18.4684 - val_mse: 18.4684 - val_mae: 2.7306\n","Epoch 68/75\n","41/41 [==============================] - 0s 3ms/step - loss: 6.0355 - mse: 6.0355 - mae: 1.7559 - val_loss: 18.5036 - val_mse: 18.5036 - val_mae: 2.7438\n","Epoch 69/75\n","41/41 [==============================] - 0s 3ms/step - loss: 5.8888 - mse: 5.8888 - mae: 1.7477 - val_loss: 18.4388 - val_mse: 18.4388 - val_mae: 2.6962\n","Epoch 70/75\n","41/41 [==============================] - 0s 3ms/step - loss: 5.7796 - mse: 5.7796 - mae: 1.7381 - val_loss: 19.0011 - val_mse: 19.0011 - val_mae: 2.8476\n","Epoch 71/75\n","41/41 [==============================] - 0s 3ms/step - loss: 5.7611 - mse: 5.7611 - mae: 1.7503 - val_loss: 19.2333 - val_mse: 19.2333 - val_mae: 2.8507\n","Epoch 72/75\n","41/41 [==============================] - 0s 3ms/step - loss: 5.6884 - mse: 5.6884 - mae: 1.7017 - val_loss: 18.3141 - val_mse: 18.3141 - val_mae: 2.7772\n","Epoch 73/75\n","41/41 [==============================] - 0s 4ms/step - loss: 6.2922 - mse: 6.2922 - mae: 1.8430 - val_loss: 18.2414 - val_mse: 18.2414 - val_mae: 2.6995\n","Epoch 74/75\n","41/41 [==============================] - 0s 3ms/step - loss: 5.4324 - mse: 5.4324 - mae: 1.6724 - val_loss: 18.0608 - val_mse: 18.0608 - val_mae: 2.6306\n","Epoch 75/75\n","41/41 [==============================] - 0s 4ms/step - loss: 5.5657 - mse: 5.5657 - mae: 1.7053 - val_loss: 18.2394 - val_mse: 18.2394 - val_mae: 2.7435\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"sYJ8t_ezHP4W"},"source":["### Hyperparameter Tuning Approaches:\n","\n","#### 1) Babysitting AKA \"Grad Student Descent\".\n","\n","If you fiddled with any hyperparameters yesterday, this is basically what you did. This approach is 100% manual and is pretty common among researchers where finding that 1 exact specification that jumps your model to a level of accuracy never seen before is the difference between publishing and not publishing a paper. Of course the professors don't do this themselves, that's grunt work. This is also known as the fiddle with hyperparameters until you run out of time method.\n","\n","#### 2) Grid Search\n","\n","Grid Search is the Grad Student galaxy brain realization of: why don't I just specify all the experiments I want to run and let the computer try every possible combination of them while I go and grab lunch. This has a specific downside in that if I specify 5 hyperparameters with 5 options each then I've just created 5^5 combinations of hyperparameters to check. Which means that I have to train 3125 different versions of my model Then if I use 5-fold Cross Validation on that then my model has to run 15,525 times. This is the brute-force method of hyperparameter tuning, but it can be very profitable if done wisely. \n","\n","When using Grid Search here's what I suggest: don't use it to test combinations of different hyperparameters, only use it to test different specifications of **a single** hyperparameter. It's rare that combinations between different hyperparameters lead to big performance gains. You'll get 90-95% of the way there if you just Grid Search one parameter and take the best result, then retain that best result while you test another, and then retain the best specification from that while you train another. This at least makes the situation much more manageable and leads to pretty good results. \n","\n","#### 3) Random Search\n","\n","Do Grid Search for a couple of hours and you'll say to yourself - \"There's got to be a better way.\" Enter Random Search. For Random search you specify a hyperparameter space and it picks specifications from that randomly, tries them out, gives you the best results and says - That's going to have to be good enough, go home and spend time with your family. \n","\n","Grid Search treats every parameter as if it was equally important, but this just isn't the case, some are known to move the needle a lot more than others (we'll talk about that in a minute). Random Search allows searching to be specified along the most important parameter and experiments less along the dimensions of less important hyperparameters. The downside of Random search is that it won't find the absolute best hyperparameters, but it is much less costly to perform than Grid Search. \n","\n","#### 4) Bayesian Methods\n","\n","One thing that can make more manual methods like babysitting and gridsearch effective is that as the experimenter sees results he can then make updates to his future searches taking into account the results of past specifications. If only we could hyperparameter tune our hyperparameter tuning. Well, we kind of can. Enter Bayesian Optimization. Neural Networks are like an optimization problem within an optimization problem, and Bayesian Optimization is a search strategy that tries to take into account the results of past searches in order to improve future ones. Check out the new library `keras-tuner` for easy implementations of Bayesian methods. \n"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"HfQ7D043OMMn"},"source":["## What Hyperparameters are there to test?\n","\n","- batch_size - to the power of 2, so data is not idle\n","- training epochs - ie: set early stopping\n","- optimization algorithms - there are several, generally adam is fine to start with\n","- learning rate - size of the step\n","- momentum - tendency of the \"ball\" to continue its trajectory as it travels towards the minimum, ie: travel down up and through a local minimum to reach global minimum\n","- activation functions\n","- dropout regularization - dropout some neurons, some have different probabilities\n","- number of neurons in the hidden layer\n","\n","There are more, but these are the most important."]},{"cell_type":"markdown","metadata":{"id":"uSKW-Bx9E-rT","colab_type":"text"},"source":["## Follow Along"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Mri5-kXzVKAa"},"source":["## Batch Size\n","\n","Batch size determines how many observations the model is shown before it calculates loss/error and updates the model weights via gradient descent. You're looking for a sweet spot here where you're showing it enough observations that you have enough information to updates the weights, but not such a large batch size that you don't get a lot of weight update iterations performed in a given epoch. Feed-forward Neural Networks aren't as sensitive to bach_size as other networks, but it is still an important hyperparameter to tune. Smaller batch sizes will also take longer to train. \n","\n","Traditionally, batch size is set in powers of 2 starting at 32 up to 512. Keras defaults to a batch size of 32 if you do not specify it. Yann LeCun famously Twitted: \n","\n","> Training with large minibatches is bad for your health.\n","More importantly, it's bad for your test error.\n","Friends dont let friends use minibatches larger than 32.\n","\n","Check out this paper for more reference on his tweet. https://arxiv.org/abs/1804.07612. \n","\n","Check out this SO question on why batch size is typically set in powers of two: https://datascience.stackexchange.com/questions/20179/what-is-the-advantage-of-keeping-batch-size-a-power-of-2\n","\n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"2smXfriNAGn7","colab":{"base_uri":"https://localhost:8080/","height":136},"executionInfo":{"status":"ok","timestamp":1596188438600,"user_tz":240,"elapsed":28630,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"9a1435ec-939b-4270-c55f-34d3dce8c36e"},"source":["import numpy\n","import pandas as pd\n","from sklearn.model_selection import GridSearchCV\n","from tensorflow.keras.models import Sequential\n","from tensorflow.keras.layers import Dense\n","from tensorflow.keras.wrappers.scikit_learn import KerasClassifier # helps sklearn and keras play well together\n","\n","# fix random seed for reproducibility\n","seed = 7\n","numpy.random.seed(seed)\n","\n","# load dataset\n","url =\"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv\"\n","\n","dataset = pd.read_csv(url, header=None).values\n","\n","# split into input (X) and output (Y) variables\n","X = dataset[:,0:8]\n","Y = dataset[:,8]\n","\n","# Function to create model, required for KerasClassifier\n","def create_model():\n"," # create model\n"," model = Sequential()\n"," model.add(Dense(12, input_dim=8, activation='relu'))\n"," model.add(Dense(1, activation='sigmoid'))\n"," # Compile model\n"," model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n"," return model\n","\n","# create model\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# define the grid search parameters\n","# batch_size = [10, 20, 40, 60, 80, 100]\n","# param_grid = dict(batch_size=batch_size, epochs=epochs)\n","\n","# define the grid search parameters\n","param_grid = {'batch_size': [10, 20, 40, 60, 80, 100],\n"," 'epochs': [20]}\n","\n","# Create Grid Search\n","grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)\n","grid_result = grid.fit(X, Y)\n","\n","# Report Results\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\") "],"execution_count":null,"outputs":[{"output_type":"stream","text":["Best: 0.6288854956626893 using {'batch_size': 10, 'epochs': 20}\n","Means: 0.6288854956626893, Stdev: 0.025613360036190976 with: {'batch_size': 10, 'epochs': 20}\n","Means: 0.5950428724288941, Stdev: 0.05920059978356421 with: {'batch_size': 20, 'epochs': 20}\n","Means: 0.5754519999027252, Stdev: 0.06369198916468542 with: {'batch_size': 40, 'epochs': 20}\n","Means: 0.6173160195350647, Stdev: 0.05204448217039456 with: {'batch_size': 60, 'epochs': 20}\n","Means: 0.5585689008235931, Stdev: 0.05896946221610665 with: {'batch_size': 80, 'epochs': 20}\n","Means: 0.5637382328510284, Stdev: 0.06841783362212868 with: {'batch_size': 100, 'epochs': 20}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"EKcuY6OiaLfz"},"source":["## Optimizer\n","\n","Remember that there's a different optimizers [optimizers](https://keras.io/optimizers/). At some point, take some time to read up on them a little bit. \"adam\" usually gives the best results. The thing to know about choosing an optimizer is that different optimizers have different hyperparameters like learning rate, momentum, etc. So based on the optimizer you choose you might also have to tune the learning rate and momentum of those optimizers after that. "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"DG3wq5iOaLig"},"source":["## Learning Rate\n","\n","Remember that the Learning Rate is a hyperparameter that is specific to your gradient-descent based optimizer selection. A learning rate that is too high will cause divergent behavior, but a Learning Rate that is too low will fail to converge, again, you're looking for the sweet spot. I would start out tuning learning rates by orders of magnitude: [.001, .01, .1, .2, .3, .5] etc. I wouldn't go above .5, but you can try it and see what the behavior is like. \n","\n","Once you have narrowed it down, make the window even smaller and try it again. If after running the above specification your model reports that .1 is the best optimizer, then you should probably try things like [.05, .08, .1, .12, .15] to try and narrow it down. \n","\n","It can also be good to tune the number of epochs in combination with the learning rate since the number of iterations that you allow the learning rate to reach the minimum can determine if you have let it run long enough to converge to the minimum. "]},{"cell_type":"code","metadata":{"id":"QLJaTm6NE-rY","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":666},"executionInfo":{"status":"ok","timestamp":1596188610082,"user_tz":240,"elapsed":165332,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"7e9a9662-7de3-4234-8db5-ee8086c5eb21"},"source":["from tensorflow.keras.optimizers import Adam\n","\n","# Function to create model, required for KerasClassifier\n","def create_model(lr):\n"," # create optimizer\n"," adam = Adam(learning_rate=lr)\n"," # create model\n"," model = Sequential()\n"," model.add(Dense(12, input_dim=8, activation='relu'))\n"," model.add(Dense(1, activation='sigmoid'))\n"," # Compile model\n"," model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n"," return model\n","\n","# create model\n","model = KerasClassifier(build_fn=create_model, verbose=0)\n","\n","# define the grid search parameters\n","# batch_size = [10, 20, 40, 60, 80, 100]\n","# param_grid = dict(batch_size=batch_size, epochs=epochs)\n","\n","# define the grid search parameters\n","param_grid = {'lr': [.001, .01, .1, .2, .3, .5],\n"," 'batch_size': [10, 20, 40, 60, 80, 100],\n"," 'epochs': [20]}\n","\n","# Create Grid Search\n","grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)\n","grid_result = grid.fit(X, Y)\n","\n","# Report Results\n","print(f\"Best: {grid_result.best_score_} using {grid_result.best_params_}\")\n","means = grid_result.cv_results_['mean_test_score']\n","stds = grid_result.cv_results_['std_test_score']\n","params = grid_result.cv_results_['params']\n","for mean, stdev, param in zip(means, stds, params):\n"," print(f\"Means: {mean}, Stdev: {stdev} with: {param}\") "],"execution_count":null,"outputs":[{"output_type":"stream","text":["Best: 0.670647656917572 using {'batch_size': 10, 'epochs': 20, 'lr': 0.3}\n","Means: 0.670622193813324, Stdev: 0.034055345246282176 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.001}\n","Means: 0.6666581869125366, Stdev: 0.03594786951719668 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.01}\n","Means: 0.6471861481666565, Stdev: 0.022662764625810836 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.1}\n","Means: 0.6589169025421142, Stdev: 0.030892849814578634 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.2}\n","Means: 0.670647656917572, Stdev: 0.028423874889773475 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.3}\n","Means: 0.6627875447273255, Stdev: 0.034789626968944866 with: {'batch_size': 10, 'epochs': 20, 'lr': 0.5}\n","Means: 0.625040328502655, Stdev: 0.023879240737213446 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.001}\n","Means: 0.6602240920066833, Stdev: 0.036695366614109 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.01}\n","Means: 0.5989644527435303, Stdev: 0.05464681996072435 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.1}\n","Means: 0.6483660221099854, Stdev: 0.025373358015569548 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.2}\n","Means: 0.653739070892334, Stdev: 0.045263564610636776 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.3}\n","Means: 0.6484508991241456, Stdev: 0.04095369245717369 with: {'batch_size': 20, 'epochs': 20, 'lr': 0.5}\n","Means: 0.5702911436557769, Stdev: 0.08065816036466286 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.001}\n","Means: 0.605670154094696, Stdev: 0.07785557987187622 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.01}\n","Means: 0.5351837754249573, Stdev: 0.0402444783049178 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.1}\n","Means: 0.6419743776321412, Stdev: 0.05035846735869015 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.2}\n","Means: 0.5665393590927124, Stdev: 0.044341777421539615 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.3}\n","Means: 0.5925982594490051, Stdev: 0.06802255382083158 with: {'batch_size': 40, 'epochs': 20, 'lr': 0.5}\n","Means: 0.6080298781394958, Stdev: 0.01986999799644162 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.001}\n","Means: 0.5925048828125, Stdev: 0.03098589834058058 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.01}\n","Means: 0.6198285460472107, Stdev: 0.033091026003625575 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.1}\n","Means: 0.6445802569389343, Stdev: 0.041037049178999155 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.2}\n","Means: 0.5636873066425323, Stdev: 0.07658745646679622 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.3}\n","Means: 0.5794924139976502, Stdev: 0.03117207372245685 with: {'batch_size': 60, 'epochs': 20, 'lr': 0.5}\n","Means: 0.5548255681991577, Stdev: 0.06417168938420525 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.001}\n","Means: 0.6120702862739563, Stdev: 0.042227130225532025 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.01}\n","Means: 0.4960699439048767, Stdev: 0.0891646310688851 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.1}\n","Means: 0.5742806434631348, Stdev: 0.024518855718530994 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.2}\n","Means: 0.5935404658317566, Stdev: 0.07797931925062536 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.3}\n","Means: 0.5167388141155242, Stdev: 0.09154661055777936 with: {'batch_size': 80, 'epochs': 20, 'lr': 0.5}\n","Means: 0.5624649882316589, Stdev: 0.050929645960734946 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.001}\n","Means: 0.6433749198913574, Stdev: 0.046313945062031306 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.01}\n","Means: 0.5611323416233063, Stdev: 0.06017028280820159 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.1}\n","Means: 0.5768440723419189, Stdev: 0.026621439592273848 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.2}\n","Means: 0.5442492127418518, Stdev: 0.08492793086368405 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.3}\n","Means: 0.6055767774581909, Stdev: 0.042685421384468195 with: {'batch_size': 100, 'epochs': 20, 'lr': 0.5}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"gNTBUWd1aLlA"},"source":["## Momentum\n","\n","Momentum is a hyperparameter that is more commonly associated with Stochastic Gradient Descent. SGD is a common optimizer because it's what people understand and know, but I doubt it will get you the best results, you can try hyperparameter tuning its attributes and see if you can beat the performance from adam. Momentum is a property that decides the willingness of an optimizer to overshoot the minimum. Imagine a ball rolling down one side of a bowl and then up the opposite side a little bit before settling back to the bottom. The purpose of momentum is to try and escale local minima."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"xnEG-bCJaLnZ"},"source":["## Activation Functions\n","\n","We've talked about this a little bit, typically you'l want to use ReLU for hidden layers and either Sigmoid, or Softmax for output layers of binary and multi-class classification implementations respectively, but try other activation functions and see if you can get any better results with sigmoid or tanh or something. There are a lot of activation functions that we haven't really talked about. Maybe you'll get good results with them. Maybe you won't. :) "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"oul9sPq-dU-h"},"source":["## Network Weight Initialization\n","\n","You saw how big of an effect the way that we initialize our network's weights can have on our results. There are **a lot** of what are called initialization modes. I don't understand all of them, but they can have a big affect on your model's initial accuracy. Your model will get further with less epochs if you initialize it with weights that are well suited to the problem you're trying to solve.\n","\n","Do proper weight initialization to prevent slow model\n","\n","`init_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']`"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"bqtEuxeQaLqE"},"source":["## Dropout Regularization and the Weight Constraint\n","\n","the Dropout Regularization value is a percentage of neurons that you want to be randomly deactivated during training. The weight constraint is a second regularization parameter that works in tandem with dropout regularization. You should tune these two values at the same time. \n","\n","Using dropout on visible vs hidden layers might have a different effect. Using dropout on hidden layers might not have any effect while using dropout on hidden layers might have a substantial effect. You don't necessarily need to turn use dropout unless you see that your model has overfitting and generalizability problems."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"P2c5Cv6oaLtO"},"source":["## Neurons in Hidden Layer \n","\n","Remember that when we only had a single perceptron our model was only able to fit to linearly separable data, but as we have added layers and nodes to those layers our network has become a powerhouse of fitting nonlinearity in data. The larger the network and the more nodes generally the stronger the network's capacity to fit nonlinear patterns in data. The more nodes and layers the longer it will take to train a network, and higher the probability of overfitting. The larger your network gets the more you'll need dropout regularization or other regularization techniques to keep it in check. \n","\n","Typically depth (more layers) is more important than width (more nodes) for neural networks. This is part of why Deep Learning is so highly touted. Certain deep learning architectures have truly been huge breakthroughs for certain machine learning tasks. \n","\n","You might borrow ideas from other network architectures. For example if I was doing image recognition and I wasn't taking cues from state of the art architectures like resnet, alexnet, googlenet, etc. Then I'm probably going to have to do a lot more experimentation on my own before I find something that works.\n","\n","There are some heuristics, but I am highly skeptical of them. I think you're better off experimenting on your own and forming your own intuition for these kinds of problems. \n","\n","- https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/"]},{"cell_type":"markdown","metadata":{"id":"Xoe0HIPdE-rd","colab_type":"text"},"source":["## Challenge\n","You will be expected to tune several hyperparameters in today's module project. "]},{"cell_type":"markdown","metadata":{"id":"fEseAXTYE-rd","colab_type":"text"},"source":["# Experiment Tracking Framework (Learn)\n",""]},{"cell_type":"markdown","metadata":{"id":"YEtOeWcmE-re","colab_type":"text"},"source":["## Overview\n","\n","You will notice quickly that managing the results of all the experiments you are running becomes challenging. Which set of parameters did the best? Are my results today different than my results yesterday? Although we use Ipython Notebooks to work, the format is not well suited to logging experimental results. Enter experiment tracking frameworks like [Comet.ml](https://comet.ml) and [Weights and Biases](https://wandb.ai/).\n","\n","Those tools will help you track your experiments, store the results, and the code associated with those experiments. Experimental results can also be readily visualized to see changes in performance across any metric you care about. Data is sent to the tool as each epoch is completed, so you can also see if your model is converging. Let's check out Weights & Biases today. "]},{"cell_type":"markdown","metadata":{"id":"YSIJgEjtE-re","colab_type":"text"},"source":["## Follow Along\n","\n","Make sure you login into `wandb` in the terminal before running the next cell. "]},{"cell_type":"code","metadata":{"id":"edFJB_p5E-rf","colab_type":"code","colab":{}},"source":["import wandb\n","from wandb.keras import WandbCallback"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"x8mDQVuZzf0s","colab_type":"code","colab":{}},"source":["wandb_project = 'dspt5-proj'"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"_QZ4WRQkzlQD","colab_type":"code","colab":{}},"source":["wand_group = 'alexkim'"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"nGb5lkBlE-rh","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596080493325,"user_tz":240,"elapsed":11916,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"08fa4545-6547-4f64-ef70-6dedcfbec83f"},"source":["wandb.init(project=wandb_project, entity=wandb_group) #Initializes and Experiment\n","\n","# Important Hyperparameters\n","\n","X = x_train\n","y = y_train\n","\n","inputs = X.shape[1]\n","wandb.config.epochs = 50\n","wandb.config.batch_size = 10\n","wandb.config.n_neurons = 128\n","\n","# Create Model\n","\n","model = Sequential()\n","model.add(Dense(wandb.config.n_neurons, activation='relu', input_shape=(inputs,)))\n","model.add(Dense(wandb.config.n_neurons, activation='relu'))\n","model.add(Dense(wandb.config.n_neurons, activation='relu'))\n","model.add(Dense(1))\n","\n","# Compile Model\n","\n","model.compile(optimizer='adam', loss='mse', metrics=['mse', 'mae'])\n","\n","# Fit Model\n","\n","model.fit(X, y, \n"," validation_split=0.33, \n"," epochs=wandb.config.epochs, \n"," batch_size=wandb.config.batch_size, \n"," callbacks=[WandbCallback()]\n"," )"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/html":["\n"," Logging results to Weights & Biases (Documentation).
\n"," Project page: https://app.wandb.ai/alexkim/dspt5-proj
\n"," Run page: https://app.wandb.ai/alexkim/dspt5-proj/runs/19qh1jrw
\n"," "],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/50\n","27/27 [==============================] - 0s 9ms/step - loss: 391.1430 - mse: 391.1430 - mae: 17.4173 - val_loss: 167.7349 - val_mse: 167.7349 - val_mae: 10.4675\n","Epoch 2/50\n","27/27 [==============================] - 0s 7ms/step - loss: 66.5965 - mse: 66.5965 - mae: 6.1247 - val_loss: 67.9068 - val_mse: 67.9068 - val_mae: 5.8267\n","Epoch 3/50\n","27/27 [==============================] - 0s 8ms/step - loss: 28.2629 - mse: 28.2629 - mae: 3.9662 - val_loss: 28.8423 - val_mse: 28.8423 - val_mae: 3.5844\n","Epoch 4/50\n","27/27 [==============================] - 0s 6ms/step - loss: 16.7242 - mse: 16.7242 - mae: 3.0382 - val_loss: 25.1663 - val_mse: 25.1663 - val_mae: 3.2271\n","Epoch 5/50\n","27/27 [==============================] - 0s 6ms/step - loss: 14.0112 - mse: 14.0112 - mae: 2.8022 - val_loss: 22.0214 - val_mse: 22.0214 - val_mae: 3.0296\n","Epoch 6/50\n","27/27 [==============================] - 0s 6ms/step - loss: 11.9238 - mse: 11.9238 - mae: 2.5811 - val_loss: 20.5218 - val_mse: 20.5218 - val_mae: 2.9348\n","Epoch 7/50\n","27/27 [==============================] - 0s 5ms/step - loss: 10.3844 - mse: 10.3844 - mae: 2.4451 - val_loss: 19.0816 - val_mse: 19.0816 - val_mae: 2.8696\n","Epoch 8/50\n","27/27 [==============================] - 0s 5ms/step - loss: 11.1359 - mse: 11.1359 - mae: 2.5724 - val_loss: 19.6867 - val_mse: 19.6867 - val_mae: 2.9614\n","Epoch 9/50\n","27/27 [==============================] - 0s 5ms/step - loss: 10.1459 - mse: 10.1459 - mae: 2.3832 - val_loss: 18.1334 - val_mse: 18.1334 - val_mae: 2.8562\n","Epoch 10/50\n","27/27 [==============================] - 0s 4ms/step - loss: 8.4068 - mse: 8.4068 - mae: 2.2671 - val_loss: 18.9494 - val_mse: 18.9494 - val_mae: 2.9336\n","Epoch 11/50\n","27/27 [==============================] - 0s 5ms/step - loss: 8.7361 - mse: 8.7361 - mae: 2.2407 - val_loss: 18.8876 - val_mse: 18.8876 - val_mae: 3.0149\n","Epoch 12/50\n","27/27 [==============================] - 0s 6ms/step - loss: 7.9194 - mse: 7.9194 - mae: 2.1462 - val_loss: 16.6289 - val_mse: 16.6289 - val_mae: 2.7676\n","Epoch 13/50\n","27/27 [==============================] - 0s 5ms/step - loss: 7.4178 - mse: 7.4178 - mae: 2.0611 - val_loss: 16.1517 - val_mse: 16.1517 - val_mae: 2.6978\n","Epoch 14/50\n","27/27 [==============================] - 0s 4ms/step - loss: 7.0305 - mse: 7.0305 - mae: 2.0162 - val_loss: 16.3927 - val_mse: 16.3927 - val_mae: 2.7069\n","Epoch 15/50\n","27/27 [==============================] - 0s 4ms/step - loss: 7.0741 - mse: 7.0741 - mae: 2.0188 - val_loss: 17.2692 - val_mse: 17.2692 - val_mae: 2.7936\n","Epoch 16/50\n","27/27 [==============================] - 0s 5ms/step - loss: 6.7651 - mse: 6.7651 - mae: 1.9338 - val_loss: 16.0255 - val_mse: 16.0255 - val_mae: 2.6737\n","Epoch 17/50\n","27/27 [==============================] - 0s 5ms/step - loss: 6.4261 - mse: 6.4261 - mae: 1.9180 - val_loss: 16.6136 - val_mse: 16.6136 - val_mae: 2.7637\n","Epoch 18/50\n","27/27 [==============================] - 0s 4ms/step - loss: 6.0420 - mse: 6.0420 - mae: 1.8508 - val_loss: 16.5037 - val_mse: 16.5037 - val_mae: 2.7686\n","Epoch 19/50\n","27/27 [==============================] - 0s 6ms/step - loss: 5.8512 - mse: 5.8512 - mae: 1.7919 - val_loss: 15.4612 - val_mse: 15.4612 - val_mae: 2.6377\n","Epoch 20/50\n","27/27 [==============================] - 0s 5ms/step - loss: 5.6173 - mse: 5.6173 - mae: 1.7363 - val_loss: 16.1367 - val_mse: 16.1367 - val_mae: 2.7273\n","Epoch 21/50\n","27/27 [==============================] - 0s 4ms/step - loss: 5.2380 - mse: 5.2380 - mae: 1.7024 - val_loss: 17.0036 - val_mse: 17.0036 - val_mae: 2.8135\n","Epoch 22/50\n","27/27 [==============================] - 0s 4ms/step - loss: 5.7001 - mse: 5.7001 - mae: 1.7733 - val_loss: 15.6823 - val_mse: 15.6823 - val_mae: 2.6842\n","Epoch 23/50\n","27/27 [==============================] - 0s 5ms/step - loss: 5.2340 - mse: 5.2340 - mae: 1.7234 - val_loss: 15.2864 - val_mse: 15.2864 - val_mae: 2.5884\n","Epoch 24/50\n","27/27 [==============================] - 0s 5ms/step - loss: 5.3445 - mse: 5.3445 - mae: 1.7179 - val_loss: 15.2897 - val_mse: 15.2897 - val_mae: 2.6455\n","Epoch 25/50\n","27/27 [==============================] - 0s 6ms/step - loss: 4.9448 - mse: 4.9448 - mae: 1.6732 - val_loss: 14.9823 - val_mse: 14.9823 - val_mae: 2.6131\n","Epoch 26/50\n","27/27 [==============================] - 0s 4ms/step - loss: 4.9711 - mse: 4.9711 - mae: 1.6476 - val_loss: 15.9045 - val_mse: 15.9045 - val_mae: 2.7565\n","Epoch 27/50\n","27/27 [==============================] - 0s 5ms/step - loss: 4.8460 - mse: 4.8460 - mae: 1.6556 - val_loss: 15.4617 - val_mse: 15.4617 - val_mae: 2.6994\n","Epoch 28/50\n","27/27 [==============================] - 0s 5ms/step - loss: 4.6180 - mse: 4.6180 - mae: 1.5818 - val_loss: 15.4385 - val_mse: 15.4385 - val_mae: 2.6746\n","Epoch 29/50\n","27/27 [==============================] - 0s 6ms/step - loss: 4.3651 - mse: 4.3651 - mae: 1.5423 - val_loss: 14.6620 - val_mse: 14.6620 - val_mae: 2.5743\n","Epoch 30/50\n","27/27 [==============================] - 0s 6ms/step - loss: 4.2810 - mse: 4.2810 - mae: 1.5440 - val_loss: 14.3034 - val_mse: 14.3034 - val_mae: 2.5697\n","Epoch 31/50\n","27/27 [==============================] - 0s 6ms/step - loss: 4.7221 - mse: 4.7221 - mae: 1.6354 - val_loss: 14.1337 - val_mse: 14.1337 - val_mae: 2.5260\n","Epoch 32/50\n","27/27 [==============================] - 0s 5ms/step - loss: 4.0898 - mse: 4.0898 - mae: 1.4952 - val_loss: 14.8753 - val_mse: 14.8753 - val_mae: 2.6116\n","Epoch 33/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.8771 - mse: 3.8771 - mae: 1.4714 - val_loss: 14.6752 - val_mse: 14.6752 - val_mae: 2.5962\n","Epoch 34/50\n","27/27 [==============================] - 0s 4ms/step - loss: 4.3426 - mse: 4.3426 - mae: 1.5982 - val_loss: 14.7166 - val_mse: 14.7166 - val_mae: 2.5539\n","Epoch 35/50\n","27/27 [==============================] - 0s 4ms/step - loss: 4.2177 - mse: 4.2177 - mae: 1.5448 - val_loss: 15.0853 - val_mse: 15.0853 - val_mae: 2.6453\n","Epoch 36/50\n","27/27 [==============================] - 0s 5ms/step - loss: 3.6385 - mse: 3.6385 - mae: 1.4104 - val_loss: 15.0372 - val_mse: 15.0372 - val_mae: 2.6117\n","Epoch 37/50\n","27/27 [==============================] - 0s 5ms/step - loss: 3.4844 - mse: 3.4844 - mae: 1.4032 - val_loss: 15.1504 - val_mse: 15.1504 - val_mae: 2.7293\n","Epoch 38/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.9794 - mse: 3.9794 - mae: 1.4966 - val_loss: 14.2705 - val_mse: 14.2705 - val_mae: 2.5583\n","Epoch 39/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.7010 - mse: 3.7010 - mae: 1.4516 - val_loss: 15.0156 - val_mse: 15.0156 - val_mae: 2.6889\n","Epoch 40/50\n","27/27 [==============================] - 0s 5ms/step - loss: 3.6161 - mse: 3.6161 - mae: 1.4723 - val_loss: 13.9314 - val_mse: 13.9314 - val_mae: 2.5308\n","Epoch 41/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.5704 - mse: 3.5704 - mae: 1.4434 - val_loss: 14.6295 - val_mse: 14.6295 - val_mae: 2.6004\n","Epoch 42/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.9330 - mse: 3.9330 - mae: 1.5342 - val_loss: 13.9997 - val_mse: 13.9997 - val_mae: 2.5261\n","Epoch 43/50\n","27/27 [==============================] - 0s 4ms/step - loss: 3.6389 - mse: 3.6389 - mae: 1.4680 - val_loss: 14.8900 - val_mse: 14.8900 - val_mae: 2.6775\n","Epoch 44/50\n","27/27 [==============================] - 0s 5ms/step - loss: 3.5906 - mse: 3.5906 - mae: 1.3690 - val_loss: 14.9394 - val_mse: 14.9394 - val_mae: 2.6415\n","Epoch 45/50\n","27/27 [==============================] - 0s 6ms/step - loss: 3.2782 - mse: 3.2782 - mae: 1.3676 - val_loss: 13.6757 - val_mse: 13.6757 - val_mae: 2.5094\n","Epoch 46/50\n","27/27 [==============================] - 0s 4ms/step - loss: 2.8351 - mse: 2.8351 - mae: 1.2536 - val_loss: 14.2248 - val_mse: 14.2248 - val_mae: 2.5976\n","Epoch 47/50\n","27/27 [==============================] - 0s 4ms/step - loss: 2.9305 - mse: 2.9305 - mae: 1.2887 - val_loss: 13.9465 - val_mse: 13.9465 - val_mae: 2.5031\n","Epoch 48/50\n","27/27 [==============================] - 0s 6ms/step - loss: 3.0577 - mse: 3.0577 - mae: 1.2600 - val_loss: 13.6310 - val_mse: 13.6310 - val_mae: 2.4892\n","Epoch 49/50\n","27/27 [==============================] - 0s 5ms/step - loss: 2.8667 - mse: 2.8667 - mae: 1.2690 - val_loss: 14.3752 - val_mse: 14.3752 - val_mae: 2.6033\n","Epoch 50/50\n","27/27 [==============================] - 0s 4ms/step - loss: 2.7187 - mse: 2.7187 - mae: 1.2176 - val_loss: 13.9669 - val_mse: 13.9669 - val_mae: 2.5294\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":55}]},{"cell_type":"code","metadata":{"id":"7QvmswkorW6K","colab_type":"code","colab":{}},"source":["model.save('my_model.hdf5')"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"wbL8YfhZrwM5","colab_type":"code","colab":{}},"source":["model = keras.models.load_model('my_model.hdf5')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CIMyIiuTE-rj","colab_type":"text"},"source":["### Your Turn\n","\n","Pick a few hyparameters that we *have not* tuned. Using the same code above, try changing a few parameters you're interested in and submitting the results to weights & biases. :) "]},{"cell_type":"code","metadata":{"id":"iezy_uHK3RYM","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596081538788,"user_tz":240,"elapsed":338,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"805bdba4-ad27-4d34-b122-93683518386a"},"source":["# Check shape of X and y data\n","\n","print(X.shape, y.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(404, 13) (404,)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"23uB2k8mE-rk","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596085216460,"user_tz":240,"elapsed":45917,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"ed4eda55-5f11-4903-89ce-0c814d5e2feb"},"source":["from tensorflow.keras import regularizers\n","from tensorflow.keras.layers import Dropout\n","from sklearn.preprocessing import MinMaxScaler\n","\n","wandb.init(project=wandb_project, entity=wandb_group) #Initializes and Experiment\n","\n","inputs = X.shape[1]\n","\n","wandb.config.epochs = 100\n","wandb.config.batch_size = 50\n","wandb.config.n_neurons = 1000\n","#wandb.config.metric = 'LeakyReLu'\n","#wandb.config.dropout = 0.5\n","wandb.config.l1_reg = 1e-5\n","wandb.config.l2_reg = 1e-4\n","\n","# Create Model\n","\n","model = Sequential()\n","model.add(Dense(wandb.config.n_neurons, \n"," kernel_regularizer=regularizers.l1_l2(l1=wandb.config.l1_reg, l2=wandb.config.l2_reg),\n"," activation='relu', \n"," input_shape=(inputs,)))\n","model.add(Dense(wandb.config.n_neurons, \n"," kernel_regularizer=regularizers.l1_l2(l1=wandb.config.l1_reg, l2=wandb.config.l2_reg),\n"," activation='relu'))\n","model.add(Dropout(0.1))\n","model.add(Dense(wandb.config.n_neurons, \n"," kernel_regularizer=regularizers.l1_l2(l1=wandb.config.l1_reg, l2=wandb.config.l2_reg),\n"," activation='relu'))\n","model.add(Dense(1))\n","\n","# Compile Model\n","\n","model.compile(optimizer='adam', loss='mse', metrics=['mse', 'mae'])\n","\n","# Fit Model\n","\n","model.fit(X, y, \n"," validation_split=0.33, \n"," epochs=wandb.config.epochs, \n"," batch_size=wandb.config.batch_size, \n"," callbacks=[WandbCallback()]\n"," )\n","\n","# https://stackoverflow.com/questions/48285129/saving-best-model-in-keras/48286003\n","\n","#model = getModel()\n","#model.summary()\n","#\n","#batch_size = 32\n","#\n","#earlyStopping = EarlyStopping(monitor='val_loss', patience=10, verbose=0, mode='min')\n","#mcp_save = ModelCheckpoint('.mdl_wts.hdf5', save_best_only=True, monitor='val_loss', mode='min')\n","#reduce_lr_loss = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=7, verbose=1, epsilon=1e-4, mode='min')\n","#\n","#model.fit(Xtr_more, Ytr_more, batch_size=batch_size, epochs=50, verbose=0, callbacks=[earlyStopping, mcp_save, reduce_lr_loss], validation_split=0.25)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/html":["\n"," Logging results to Weights & Biases (Documentation).
\n"," Project page: https://app.wandb.ai/alexkim/dspt5-proj
\n"," Run page: https://app.wandb.ai/alexkim/dspt5-proj/runs/1rmwawlv
\n"," "],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/100\n","6/6 [==============================] - 1s 84ms/step - loss: 415.9001 - mse: 415.1631 - mae: 18.0641 - val_loss: 151.3558 - val_mse: 150.6453 - val_mae: 9.4821\n","Epoch 2/100\n","6/6 [==============================] - 0s 65ms/step - loss: 160.0870 - mse: 159.3913 - mae: 10.1330 - val_loss: 105.4164 - val_mse: 104.7470 - val_mae: 7.3974\n","Epoch 3/100\n","6/6 [==============================] - 0s 82ms/step - loss: 91.2099 - mse: 90.5544 - mae: 6.9546 - val_loss: 100.5448 - val_mse: 99.9102 - val_mae: 6.5940\n","Epoch 4/100\n","6/6 [==============================] - 0s 81ms/step - loss: 58.5447 - mse: 57.9202 - mae: 5.6685 - val_loss: 70.6261 - val_mse: 70.0184 - val_mae: 6.5905\n","Epoch 5/100\n","6/6 [==============================] - 0s 66ms/step - loss: 47.7862 - mse: 47.1881 - mae: 4.9895 - val_loss: 76.4609 - val_mse: 75.8775 - val_mae: 5.5804\n","Epoch 6/100\n","6/6 [==============================] - 0s 64ms/step - loss: 41.9856 - mse: 41.4090 - mae: 4.4007 - val_loss: 53.2353 - val_mse: 52.6693 - val_mae: 5.2324\n","Epoch 7/100\n","6/6 [==============================] - 0s 79ms/step - loss: 34.8964 - mse: 34.3361 - mae: 4.2897 - val_loss: 52.3345 - val_mse: 51.7829 - val_mae: 4.5045\n","Epoch 8/100\n","6/6 [==============================] - 1s 84ms/step - loss: 29.3410 - mse: 28.7931 - mae: 3.6431 - val_loss: 42.1115 - val_mse: 41.5691 - val_mae: 4.3156\n","Epoch 9/100\n","6/6 [==============================] - 0s 77ms/step - loss: 25.0639 - mse: 24.5244 - mae: 3.4437 - val_loss: 40.0008 - val_mse: 39.4651 - val_mae: 3.9119\n","Epoch 10/100\n","6/6 [==============================] - 0s 72ms/step - loss: 22.6731 - mse: 22.1387 - mae: 3.3671 - val_loss: 34.8510 - val_mse: 34.3186 - val_mae: 3.6946\n","Epoch 11/100\n","6/6 [==============================] - 1s 85ms/step - loss: 20.8485 - mse: 20.3169 - mae: 3.0880 - val_loss: 31.2815 - val_mse: 30.7507 - val_mae: 3.6261\n","Epoch 12/100\n","6/6 [==============================] - 0s 65ms/step - loss: 18.9193 - mse: 18.3893 - mae: 3.0939 - val_loss: 33.8560 - val_mse: 33.3277 - val_mae: 3.6460\n","Epoch 13/100\n","6/6 [==============================] - 0s 66ms/step - loss: 18.0021 - mse: 17.4746 - mae: 2.8542 - val_loss: 28.5601 - val_mse: 28.0337 - val_mae: 3.5363\n","Epoch 14/100\n","6/6 [==============================] - 0s 77ms/step - loss: 16.3037 - mse: 15.7782 - mae: 2.7372 - val_loss: 27.8515 - val_mse: 27.3276 - val_mae: 3.2259\n","Epoch 15/100\n","6/6 [==============================] - 0s 77ms/step - loss: 15.0569 - mse: 14.5336 - mae: 2.6504 - val_loss: 26.1673 - val_mse: 25.6452 - val_mae: 3.1281\n","Epoch 16/100\n","6/6 [==============================] - 0s 81ms/step - loss: 15.1082 - mse: 14.5870 - mae: 2.5589 - val_loss: 24.5428 - val_mse: 24.0227 - val_mae: 3.2456\n","Epoch 17/100\n","6/6 [==============================] - 0s 78ms/step - loss: 13.8455 - mse: 13.3261 - mae: 2.5188 - val_loss: 24.1558 - val_mse: 23.6375 - val_mae: 2.9939\n","Epoch 18/100\n","6/6 [==============================] - 0s 74ms/step - loss: 12.7867 - mse: 12.2691 - mae: 2.3589 - val_loss: 22.3356 - val_mse: 21.8188 - val_mae: 3.1389\n","Epoch 19/100\n","6/6 [==============================] - 0s 74ms/step - loss: 12.8617 - mse: 12.3457 - mae: 2.4537 - val_loss: 22.8371 - val_mse: 22.3222 - val_mae: 2.9424\n","Epoch 20/100\n","6/6 [==============================] - 0s 58ms/step - loss: 12.2171 - mse: 11.7026 - mae: 2.3686 - val_loss: 22.0743 - val_mse: 21.5609 - val_mae: 2.9241\n","Epoch 21/100\n","6/6 [==============================] - 0s 77ms/step - loss: 11.6559 - mse: 11.1430 - mae: 2.2166 - val_loss: 20.6766 - val_mse: 20.1645 - val_mae: 3.0359\n","Epoch 22/100\n","6/6 [==============================] - 0s 71ms/step - loss: 12.0389 - mse: 11.5273 - mae: 2.4695 - val_loss: 20.2554 - val_mse: 19.7447 - val_mae: 2.8170\n","Epoch 23/100\n","6/6 [==============================] - 0s 74ms/step - loss: 11.5262 - mse: 11.0157 - mae: 2.3782 - val_loss: 19.8900 - val_mse: 19.3804 - val_mae: 2.8153\n","Epoch 24/100\n","6/6 [==============================] - 0s 70ms/step - loss: 10.9772 - mse: 10.4680 - mae: 2.3128 - val_loss: 19.7194 - val_mse: 19.2109 - val_mae: 3.0378\n","Epoch 25/100\n","6/6 [==============================] - 0s 73ms/step - loss: 9.5842 - mse: 9.0764 - mae: 2.1347 - val_loss: 18.7319 - val_mse: 18.2251 - val_mae: 2.7271\n","Epoch 26/100\n","6/6 [==============================] - 0s 76ms/step - loss: 9.1395 - mse: 8.6331 - mae: 2.0648 - val_loss: 17.9697 - val_mse: 17.4640 - val_mae: 2.7582\n","Epoch 27/100\n","6/6 [==============================] - 0s 73ms/step - loss: 8.6974 - mse: 8.1921 - mae: 1.9828 - val_loss: 17.0544 - val_mse: 16.5497 - val_mae: 2.6707\n","Epoch 28/100\n","6/6 [==============================] - 0s 61ms/step - loss: 8.2756 - mse: 7.7714 - mae: 1.9826 - val_loss: 17.0853 - val_mse: 16.5820 - val_mae: 2.6267\n","Epoch 29/100\n","6/6 [==============================] - 0s 46ms/step - loss: 8.4591 - mse: 7.9561 - mae: 1.9616 - val_loss: 17.1791 - val_mse: 16.6768 - val_mae: 2.7617\n","Epoch 30/100\n","6/6 [==============================] - 0s 48ms/step - loss: 7.9801 - mse: 7.4782 - mae: 1.9712 - val_loss: 17.2661 - val_mse: 16.7649 - val_mae: 2.6627\n","Epoch 31/100\n","6/6 [==============================] - 0s 59ms/step - loss: 7.7457 - mse: 7.2450 - mae: 1.9620 - val_loss: 16.2991 - val_mse: 15.7991 - val_mae: 2.6684\n","Epoch 32/100\n","6/6 [==============================] - 0s 76ms/step - loss: 7.4162 - mse: 6.9166 - mae: 1.8992 - val_loss: 15.8290 - val_mse: 15.3300 - val_mae: 2.5350\n","Epoch 33/100\n","6/6 [==============================] - 0s 74ms/step - loss: 7.2451 - mse: 6.7465 - mae: 1.8844 - val_loss: 15.4361 - val_mse: 14.9381 - val_mae: 2.5211\n","Epoch 34/100\n","6/6 [==============================] - 0s 70ms/step - loss: 6.9342 - mse: 6.4366 - mae: 1.8247 - val_loss: 15.6035 - val_mse: 15.1065 - val_mae: 2.6346\n","Epoch 35/100\n","6/6 [==============================] - 0s 47ms/step - loss: 6.9457 - mse: 6.4491 - mae: 1.8591 - val_loss: 16.9042 - val_mse: 16.4084 - val_mae: 2.6409\n","Epoch 36/100\n","6/6 [==============================] - 0s 49ms/step - loss: 7.4301 - mse: 6.9346 - mae: 1.9341 - val_loss: 16.6481 - val_mse: 16.1535 - val_mae: 2.6180\n","Epoch 37/100\n","6/6 [==============================] - 0s 53ms/step - loss: 6.8160 - mse: 6.3218 - mae: 1.8418 - val_loss: 15.5851 - val_mse: 15.0915 - val_mae: 2.6370\n","Epoch 38/100\n","6/6 [==============================] - 0s 55ms/step - loss: 7.1809 - mse: 6.6878 - mae: 1.8544 - val_loss: 14.8071 - val_mse: 14.3148 - val_mae: 2.4819\n","Epoch 39/100\n","6/6 [==============================] - 0s 66ms/step - loss: 6.4473 - mse: 5.9553 - mae: 1.7505 - val_loss: 15.2306 - val_mse: 14.7391 - val_mae: 2.6353\n","Epoch 40/100\n","6/6 [==============================] - 0s 58ms/step - loss: 6.6332 - mse: 6.1423 - mae: 1.8351 - val_loss: 14.3380 - val_mse: 13.8477 - val_mae: 2.4445\n","Epoch 41/100\n","6/6 [==============================] - 0s 66ms/step - loss: 6.6360 - mse: 6.1459 - mae: 1.8368 - val_loss: 14.6287 - val_mse: 14.1392 - val_mae: 2.5059\n","Epoch 42/100\n","6/6 [==============================] - 0s 49ms/step - loss: 6.4592 - mse: 5.9700 - mae: 1.8195 - val_loss: 14.6140 - val_mse: 14.1257 - val_mae: 2.4317\n","Epoch 43/100\n","6/6 [==============================] - 0s 46ms/step - loss: 6.2608 - mse: 5.7727 - mae: 1.7767 - val_loss: 15.4184 - val_mse: 14.9311 - val_mae: 2.5678\n","Epoch 44/100\n","6/6 [==============================] - 0s 50ms/step - loss: 6.0452 - mse: 5.5582 - mae: 1.7782 - val_loss: 15.7560 - val_mse: 15.2695 - val_mae: 2.7896\n","Epoch 45/100\n","6/6 [==============================] - 0s 47ms/step - loss: 6.7162 - mse: 6.2303 - mae: 1.8240 - val_loss: 14.4419 - val_mse: 13.9564 - val_mae: 2.4974\n","Epoch 46/100\n","6/6 [==============================] - 0s 50ms/step - loss: 6.0130 - mse: 5.5281 - mae: 1.7504 - val_loss: 14.8483 - val_mse: 14.3639 - val_mae: 2.5917\n","Epoch 47/100\n","6/6 [==============================] - 0s 47ms/step - loss: 7.2153 - mse: 6.7316 - mae: 1.9867 - val_loss: 17.5554 - val_mse: 17.0722 - val_mae: 3.0980\n","Epoch 48/100\n","6/6 [==============================] - 0s 46ms/step - loss: 7.5722 - mse: 7.0897 - mae: 1.9929 - val_loss: 14.7735 - val_mse: 14.2918 - val_mae: 2.4983\n","Epoch 49/100\n","6/6 [==============================] - 0s 60ms/step - loss: 5.7241 - mse: 5.2426 - mae: 1.7193 - val_loss: 13.7060 - val_mse: 13.2251 - val_mae: 2.3638\n","Epoch 50/100\n","6/6 [==============================] - 0s 69ms/step - loss: 5.3412 - mse: 4.8609 - mae: 1.6472 - val_loss: 14.0014 - val_mse: 13.5216 - val_mae: 2.4696\n","Epoch 51/100\n","6/6 [==============================] - 0s 48ms/step - loss: 5.6979 - mse: 5.2185 - mae: 1.7124 - val_loss: 15.5082 - val_mse: 15.0296 - val_mae: 2.5609\n","Epoch 52/100\n","6/6 [==============================] - 0s 46ms/step - loss: 6.2437 - mse: 5.7656 - mae: 1.7987 - val_loss: 14.5199 - val_mse: 14.0426 - val_mae: 2.4229\n","Epoch 53/100\n","6/6 [==============================] - 0s 60ms/step - loss: 6.0427 - mse: 5.5655 - mae: 1.6968 - val_loss: 13.2632 - val_mse: 12.7863 - val_mae: 2.2999\n","Epoch 54/100\n","6/6 [==============================] - 0s 66ms/step - loss: 5.6284 - mse: 5.1518 - mae: 1.7051 - val_loss: 13.4641 - val_mse: 12.9879 - val_mae: 2.4236\n","Epoch 55/100\n","6/6 [==============================] - 0s 50ms/step - loss: 5.9864 - mse: 5.5107 - mae: 1.7609 - val_loss: 13.9510 - val_mse: 13.4759 - val_mae: 2.4254\n","Epoch 56/100\n","6/6 [==============================] - 0s 47ms/step - loss: 6.4820 - mse: 6.0073 - mae: 1.8481 - val_loss: 13.9438 - val_mse: 13.4698 - val_mae: 2.4251\n","Epoch 57/100\n","6/6 [==============================] - 0s 49ms/step - loss: 6.2230 - mse: 5.7492 - mae: 1.8689 - val_loss: 14.8618 - val_mse: 14.3887 - val_mae: 2.4893\n","Epoch 58/100\n","6/6 [==============================] - 0s 48ms/step - loss: 6.1582 - mse: 5.6852 - mae: 1.8229 - val_loss: 15.5412 - val_mse: 15.0690 - val_mae: 2.5869\n","Epoch 59/100\n","6/6 [==============================] - 0s 56ms/step - loss: 5.7823 - mse: 5.3102 - mae: 1.7531 - val_loss: 12.9502 - val_mse: 12.4788 - val_mae: 2.2730\n","Epoch 60/100\n","6/6 [==============================] - 0s 65ms/step - loss: 5.4028 - mse: 4.9318 - mae: 1.6457 - val_loss: 13.8831 - val_mse: 13.4126 - val_mae: 2.4664\n","Epoch 61/100\n","6/6 [==============================] - 0s 53ms/step - loss: 5.3882 - mse: 4.9181 - mae: 1.6137 - val_loss: 13.0397 - val_mse: 12.5699 - val_mae: 2.3645\n","Epoch 62/100\n","6/6 [==============================] - 0s 63ms/step - loss: 5.2043 - mse: 4.7349 - mae: 1.6305 - val_loss: 12.4378 - val_mse: 11.9688 - val_mae: 2.2271\n","Epoch 63/100\n","6/6 [==============================] - 0s 66ms/step - loss: 4.7936 - mse: 4.3249 - mae: 1.5940 - val_loss: 15.0833 - val_mse: 14.6152 - val_mae: 2.5339\n","Epoch 64/100\n","6/6 [==============================] - 0s 48ms/step - loss: 6.1788 - mse: 5.7110 - mae: 1.8242 - val_loss: 14.8800 - val_mse: 14.4129 - val_mae: 2.4269\n","Epoch 65/100\n","6/6 [==============================] - 0s 50ms/step - loss: 5.2343 - mse: 4.7673 - mae: 1.6203 - val_loss: 13.5325 - val_mse: 13.0659 - val_mae: 2.3742\n","Epoch 66/100\n","6/6 [==============================] - 0s 53ms/step - loss: 4.9433 - mse: 4.4770 - mae: 1.6076 - val_loss: 13.6695 - val_mse: 13.2037 - val_mae: 2.2924\n","Epoch 67/100\n","6/6 [==============================] - 0s 53ms/step - loss: 5.1128 - mse: 4.6472 - mae: 1.5830 - val_loss: 14.2412 - val_mse: 13.7762 - val_mae: 2.4425\n","Epoch 68/100\n","6/6 [==============================] - 0s 50ms/step - loss: 4.7764 - mse: 4.3115 - mae: 1.5427 - val_loss: 13.0398 - val_mse: 12.5754 - val_mae: 2.3491\n","Epoch 69/100\n","6/6 [==============================] - 0s 56ms/step - loss: 4.6816 - mse: 4.2174 - mae: 1.5222 - val_loss: 13.0879 - val_mse: 12.6242 - val_mae: 2.2734\n","Epoch 70/100\n","6/6 [==============================] - 0s 55ms/step - loss: 4.7123 - mse: 4.2487 - mae: 1.5258 - val_loss: 12.7048 - val_mse: 12.2416 - val_mae: 2.2637\n","Epoch 71/100\n","6/6 [==============================] - 0s 54ms/step - loss: 4.7008 - mse: 4.2378 - mae: 1.5363 - val_loss: 13.4734 - val_mse: 13.0108 - val_mae: 2.3438\n","Epoch 72/100\n","6/6 [==============================] - 0s 55ms/step - loss: 4.9094 - mse: 4.4470 - mae: 1.5758 - val_loss: 13.2634 - val_mse: 12.8015 - val_mae: 2.3278\n","Epoch 73/100\n","6/6 [==============================] - 0s 54ms/step - loss: 4.7034 - mse: 4.2418 - mae: 1.5428 - val_loss: 13.2030 - val_mse: 12.7415 - val_mae: 2.4031\n","Epoch 74/100\n","6/6 [==============================] - 0s 58ms/step - loss: 6.0843 - mse: 5.6232 - mae: 1.7829 - val_loss: 12.4737 - val_mse: 12.0127 - val_mae: 2.2562\n","Epoch 75/100\n","6/6 [==============================] - 0s 54ms/step - loss: 4.7993 - mse: 4.3386 - mae: 1.5608 - val_loss: 13.1595 - val_mse: 12.6993 - val_mae: 2.4079\n","Epoch 76/100\n","6/6 [==============================] - 0s 50ms/step - loss: 4.3658 - mse: 3.9058 - mae: 1.5183 - val_loss: 12.7009 - val_mse: 12.2413 - val_mae: 2.2968\n","Epoch 77/100\n","6/6 [==============================] - 0s 55ms/step - loss: 4.5781 - mse: 4.1188 - mae: 1.4839 - val_loss: 13.1440 - val_mse: 12.6850 - val_mae: 2.3672\n","Epoch 78/100\n","6/6 [==============================] - 0s 53ms/step - loss: 5.3628 - mse: 4.9039 - mae: 1.6825 - val_loss: 12.5943 - val_mse: 12.1357 - val_mae: 2.2855\n","Epoch 79/100\n","6/6 [==============================] - 0s 52ms/step - loss: 5.9023 - mse: 5.4439 - mae: 1.7509 - val_loss: 16.0827 - val_mse: 15.6252 - val_mae: 2.4740\n","Epoch 80/100\n","6/6 [==============================] - 0s 54ms/step - loss: 7.2038 - mse: 6.7464 - mae: 1.9804 - val_loss: 21.8703 - val_mse: 21.4141 - val_mae: 3.3039\n","Epoch 81/100\n","6/6 [==============================] - 0s 52ms/step - loss: 6.7059 - mse: 6.2495 - mae: 1.9480 - val_loss: 17.2533 - val_mse: 16.7977 - val_mae: 2.7146\n","Epoch 82/100\n","6/6 [==============================] - 0s 53ms/step - loss: 6.9529 - mse: 6.4976 - mae: 1.9495 - val_loss: 13.8723 - val_mse: 13.4177 - val_mae: 2.4468\n","Epoch 83/100\n","6/6 [==============================] - 0s 49ms/step - loss: 5.6533 - mse: 5.1990 - mae: 1.7949 - val_loss: 13.6761 - val_mse: 13.2218 - val_mae: 2.5461\n","Epoch 84/100\n","6/6 [==============================] - 0s 51ms/step - loss: 5.6482 - mse: 5.1942 - mae: 1.7874 - val_loss: 14.6211 - val_mse: 14.1672 - val_mae: 2.6895\n","Epoch 85/100\n","6/6 [==============================] - 0s 51ms/step - loss: 6.3609 - mse: 5.9075 - mae: 1.9224 - val_loss: 13.9526 - val_mse: 13.4995 - val_mae: 2.6103\n","Epoch 86/100\n","6/6 [==============================] - 0s 50ms/step - loss: 6.0167 - mse: 5.5641 - mae: 1.8407 - val_loss: 14.9606 - val_mse: 14.5084 - val_mae: 2.4605\n","Epoch 87/100\n","6/6 [==============================] - 0s 56ms/step - loss: 5.6405 - mse: 5.1883 - mae: 1.7226 - val_loss: 16.0856 - val_mse: 15.6342 - val_mae: 2.7384\n","Epoch 88/100\n","6/6 [==============================] - 0s 52ms/step - loss: 5.3779 - mse: 4.9264 - mae: 1.6900 - val_loss: 14.1069 - val_mse: 13.6559 - val_mae: 2.5132\n","Epoch 89/100\n","6/6 [==============================] - 0s 52ms/step - loss: 4.4941 - mse: 4.0430 - mae: 1.5672 - val_loss: 13.8816 - val_mse: 13.4309 - val_mae: 2.4463\n","Epoch 90/100\n","6/6 [==============================] - 0s 51ms/step - loss: 4.6582 - mse: 4.2076 - mae: 1.5259 - val_loss: 12.8600 - val_mse: 12.4098 - val_mae: 2.3174\n","Epoch 91/100\n","6/6 [==============================] - 0s 61ms/step - loss: 4.5750 - mse: 4.1249 - mae: 1.5347 - val_loss: 11.9359 - val_mse: 11.4859 - val_mae: 2.2523\n","Epoch 92/100\n","6/6 [==============================] - 0s 64ms/step - loss: 4.1469 - mse: 3.6972 - mae: 1.4534 - val_loss: 13.6604 - val_mse: 13.2111 - val_mae: 2.3543\n","Epoch 93/100\n","6/6 [==============================] - 0s 49ms/step - loss: 4.0976 - mse: 3.6486 - mae: 1.3897 - val_loss: 12.9087 - val_mse: 12.4600 - val_mae: 2.2790\n","Epoch 94/100\n","6/6 [==============================] - 0s 49ms/step - loss: 3.9156 - mse: 3.4670 - mae: 1.4030 - val_loss: 12.3666 - val_mse: 11.9183 - val_mae: 2.2499\n","Epoch 95/100\n","6/6 [==============================] - 0s 54ms/step - loss: 3.7305 - mse: 3.2823 - mae: 1.3490 - val_loss: 11.9405 - val_mse: 11.4927 - val_mae: 2.1922\n","Epoch 96/100\n","6/6 [==============================] - 0s 64ms/step - loss: 4.3112 - mse: 3.8636 - mae: 1.5267 - val_loss: 11.9212 - val_mse: 11.4739 - val_mae: 2.2011\n","Epoch 97/100\n","6/6 [==============================] - 0s 67ms/step - loss: 3.9122 - mse: 3.4651 - mae: 1.3912 - val_loss: 13.4123 - val_mse: 12.9656 - val_mae: 2.3429\n","Epoch 98/100\n","6/6 [==============================] - 0s 53ms/step - loss: 3.7634 - mse: 3.3168 - mae: 1.3278 - val_loss: 12.7914 - val_mse: 12.3452 - val_mae: 2.3691\n","Epoch 99/100\n","6/6 [==============================] - 0s 49ms/step - loss: 3.6774 - mse: 3.2313 - mae: 1.3932 - val_loss: 13.8439 - val_mse: 13.3982 - val_mae: 2.3398\n","Epoch 100/100\n","6/6 [==============================] - 0s 49ms/step - loss: 4.1135 - mse: 3.6681 - mae: 1.4362 - val_loss: 12.8841 - val_mse: 12.4390 - val_mae: 2.2964\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":74}]},{"cell_type":"code","metadata":{"id":"HFuNUscku1MJ","colab_type":"code","colab":{}},"source":["model.save('my_model.hdf5')"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"i0vdKl7cHPbb","colab_type":"code","colab":{}},"source":["model = keras.models.load_model('my_model.hdf5')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"TRcCliWfE-rl","colab_type":"text"},"source":["## Challenge\n","\n","You will be expected to use Weights & Biases to try to tune your model during your module assignment today. "]},{"cell_type":"markdown","metadata":{"id":"86WcPI1mE-rm","colab_type":"text"},"source":["# Hyperparameters with RandomSearchCV (Learn)"]},{"cell_type":"markdown","metadata":{"id":"Dqgxv2MBE-rm","colab_type":"text"},"source":["## Overview\n","\n","Basically `GridSearchCV` takes forever. You'll want to adopt a slightly more sophiscated strategy.\n","\n","Let's also take a look at an alternative with Keras-Tuner."]},{"cell_type":"code","metadata":{"id":"-B7Kue2PE-rm","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":510},"executionInfo":{"status":"ok","timestamp":1596079178179,"user_tz":240,"elapsed":6355,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"58b1bb44-7e01-4be2-8840-92738626352a"},"source":["!pip install keras-tuner"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Collecting keras-tuner\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/a7/f7/4b41b6832abf4c9bef71a664dc563adb25afc5812831667c6db572b1a261/keras-tuner-1.0.1.tar.gz (54kB)\n","\r\u001b[K |██████ | 10kB 18.7MB/s eta 0:00:01\r\u001b[K |████████████ | 20kB 1.7MB/s eta 0:00:01\r\u001b[K |██████████████████ | 30kB 2.3MB/s eta 0:00:01\r\u001b[K |████████████████████████ | 40kB 2.6MB/s eta 0:00:01\r\u001b[K |██████████████████████████████ | 51kB 2.0MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 61kB 1.9MB/s \n","\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.16.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.18.5)\n","Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.8.7)\n","Collecting terminaltables\n"," Downloading https://files.pythonhosted.org/packages/9b/c4/4a21174f32f8a7e1104798c445dacdc1d4df86f2f26722767034e4de4bff/terminaltables-3.1.0.tar.gz\n","Collecting colorama\n"," Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl\n","Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (4.41.1)\n","Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (2.23.0)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.4.1)\n","Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.22.2.post1)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2020.6.20)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (3.0.4)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (1.24.3)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2.10)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->keras-tuner) (0.16.0)\n","Building wheels for collected packages: keras-tuner, terminaltables\n"," Building wheel for keras-tuner (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for keras-tuner: filename=keras_tuner-1.0.1-cp36-none-any.whl size=73200 sha256=48d479cd0797e270637f82f5d34ecd1648a63a39fc4028a9cac16214e1ab6c93\n"," Stored in directory: /root/.cache/pip/wheels/b9/cc/62/52716b70dd90f3db12519233c3a93a5360bc672da1a10ded43\n"," Building wheel for terminaltables (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for terminaltables: filename=terminaltables-3.1.0-cp36-none-any.whl size=15356 sha256=f77b68f187e22a4859cfcd859182d0cb556943b2b5be99ea7351dc47ca2829da\n"," Stored in directory: /root/.cache/pip/wheels/30/6b/50/6c75775b681fb36cdfac7f19799888ef9d8813aff9e379663e\n","Successfully built keras-tuner terminaltables\n","Installing collected packages: terminaltables, colorama, keras-tuner\n","Successfully installed colorama-0.4.3 keras-tuner-1.0.1 terminaltables-3.1.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"t00TeAp2E-ro","colab_type":"text"},"source":["## Follow Along"]},{"cell_type":"code","metadata":{"id":"NSKCO7xxE-rp","colab_type":"code","colab":{}},"source":["from tensorflow import keras\n","from tensorflow.keras import layers\n","from kerastuner.tuners import RandomSearch\n","\n","\"\"\"\n","This model Tunes:\n","- Number of Neurons in the Hidden Layer\n","- Learning Rate in Adam\n","\n","\"\"\"\n","\n","def build_model(hp):\n"," model = keras.Sequential()\n"," model.add(layers.Dense(units=hp.Int('units', # uniform distribution of integers\n"," min_value=32,\n"," max_value=512,\n"," step=32),\n"," activation='relu'))\n"," model.add(layers.Dense(10, activation='softmax'))\n"," model.compile(\n"," optimizer=keras.optimizers.Adam(\n"," hp.Choice('learning_rate', \n"," values=[1e-2, 1e-3, 1e-4])),\n"," loss='mse',\n"," metrics=['mse', 'mae'])\n"," \n"," return model"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"s2cZeBhsE-rq","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596080656800,"user_tz":240,"elapsed":419,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"12b1c051-549f-4e21-f025-e2d34bfb700a"},"source":["tuner = RandomSearch(\n"," build_model,\n"," objective='mse',\n"," max_trials=5,\n"," executions_per_trial=3,\n"," directory='./keras-tuner-trial',\n"," project_name='helloworld')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:Reloading Oracle from existing project ./keras-tuner-trial/helloworld/oracle.json\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"uh2JpmvpE-rs","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":281},"executionInfo":{"status":"ok","timestamp":1596080658691,"user_tz":240,"elapsed":314,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"e6a14b59-c13a-4533-c83b-9854e541c563"},"source":["tuner.search_space_summary()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/html":["

Search space summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Default search space size: 2"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

units (Int)

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-default: None"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-max_value: 512"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-min_value: 32"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-sampling: None"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-step: 32"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

learning_rate (Choice)

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-default: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-ordered: True"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-values: [0.01, 0.001, 0.0001]"],"text/plain":[""]},"metadata":{"tags":[]}}]},{"cell_type":"code","metadata":{"id":"h9eSjSwjE-ru","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1596080680403,"user_tz":240,"elapsed":16431,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"96aa5a57-2823-4dcc-97ba-9bdabea0957a"},"source":["tuner.search(X, y, epochs=5, validation_split=0.3)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 19ms/step - loss: 553.9500 - mse: 553.9500 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9504 - mse: 553.9504 - mae: 21.8688 - val_loss: 645.8178 - val_mse: 645.8178 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9495 - mse: 553.9495 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9500 - mse: 553.9500 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["

Trial complete

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 055c21e3210e11336ee60b43c1944863"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491577148438"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 64"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 16ms/step - loss: 553.9497 - mse: 553.9497 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 8ms/step - loss: 553.9494 - mse: 553.9494 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 8ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9498 - mse: 553.9498 - mae: 21.8688 - val_loss: 645.8179 - val_mse: 645.8179 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9494 - mse: 553.9494 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 8ms/step - loss: 553.9491 - mse: 553.9491 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9491 - mse: 553.9491 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 17ms/step - loss: 553.9497 - mse: 553.9497 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["

Trial complete

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 08ba1904a6cfa846c660846dc4bb91be"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491373697916"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 512"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 17ms/step - loss: 553.9496 - mse: 553.9496 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 16ms/step - loss: 553.9496 - mse: 553.9496 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9497 - mse: 553.9497 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["

Trial complete

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 3c6ffca63cd1aea28c0a66eaaafc049b"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491577148438"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 416"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 16ms/step - loss: 553.9495 - mse: 553.9495 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 8ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9491 - mse: 553.9491 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 9ms/step - loss: 553.9490 - mse: 553.9490 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9491 - mse: 553.9491 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["

Trial complete

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 051f0a91b3283deececfa60139eb9f0f"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9490966796875"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.001"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 352"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 16ms/step - loss: 553.9498 - mse: 553.9498 - mae: 21.8688 - val_loss: 645.8176 - val_mse: 645.8176 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 8ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9498 - mse: 553.9498 - mae: 21.8688 - val_loss: 645.8177 - val_mse: 645.8177 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9490 - mse: 553.9490 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 1/5\n","WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx.\n","\n","If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n","\n","To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n","\n","9/9 [==============================] - 0s 15ms/step - loss: 553.9498 - mse: 553.9498 - mae: 21.8688 - val_loss: 645.8178 - val_mse: 645.8178 - val_mae: 23.2803\n","Epoch 2/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9493 - mse: 553.9493 - mae: 21.8688 - val_loss: 645.8175 - val_mse: 645.8175 - val_mae: 23.2803\n","Epoch 3/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 4/5\n","9/9 [==============================] - 0s 7ms/step - loss: 553.9492 - mse: 553.9492 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n","Epoch 5/5\n","9/9 [==============================] - 0s 6ms/step - loss: 553.9491 - mse: 553.9491 - mae: 21.8688 - val_loss: 645.8174 - val_mse: 645.8174 - val_mae: 23.2803\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["

Trial complete

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: a475da3e929738c81144600859fa34d6"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9490966796875"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 480"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"stream","text":["INFO:tensorflow:Oracle triggered exit\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"F9JI94fJE-rw","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":903},"executionInfo":{"status":"ok","timestamp":1596080685383,"user_tz":240,"elapsed":304,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"1331ff21-b308-43fe-cffb-4aaf827abcbb"},"source":["tuner.results_summary()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/html":["

Results summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Results in ./keras-tuner-trial/helloworld"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Showing 10 best trials"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Objective(name='mse', direction='min')"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 051f0a91b3283deececfa60139eb9f0f"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9490966796875"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.001"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 352"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: a475da3e929738c81144600859fa34d6"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9490966796875"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 480"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 08ba1904a6cfa846c660846dc4bb91be"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491373697916"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 512"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 055c21e3210e11336ee60b43c1944863"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491577148438"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 64"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Trial summary

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Trial ID: 3c6ffca63cd1aea28c0a66eaaafc049b"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Score: 553.9491577148438"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-Best step: 0"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["

Hyperparameters:

"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-learning_rate: 0.01"],"text/plain":[""]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":[" |-units: 416"],"text/plain":[""]},"metadata":{"tags":[]}}]},{"cell_type":"code","metadata":{"id":"o7in7rHrwFIg","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1596080731556,"user_tz":240,"elapsed":510,"user":{"displayName":"John Dailey","photoUrl":"","userId":"05832358191778965953"}},"outputId":"e8f3e67b-c6e3-4814-f50e-48cced4e6e43"},"source":["tuner.get_best_hyperparameters(1)[0].values"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'learning_rate': 0.001, 'units': 352}"]},"metadata":{"tags":[]},"execution_count":64}]},{"cell_type":"markdown","metadata":{"id":"g1NnsarwE-ry","colab_type":"text"},"source":["## Challenge\n","\n","Try to apply RandomSearchCV to your module project today. "]},{"cell_type":"markdown","metadata":{"id":"tFzNcABME-ry","colab_type":"text"},"source":["# Review\n","* Part 1: Describe the major hyperparemeters to tune\n"," - Activation Functions\n"," - Optimizer\n"," - Number of Layers\n"," - Number of Neurons\n"," - Batch Size\n"," - Dropout Regulaization\n"," - Learning Rate\n"," - Number of Epochs\n"," - and many more\n","* Part 2: Implement an experiment tracking framework\n"," - Weights & Biases\n"," - Comet.ml\n"," - By Hand / GridSearch\n","* Part 3: Search the hyperparameter space using RandomSearch\n"," - Sklearn still useful (haha)\n"," - Integration with Wieghts & Biases\n","* Part 4: Discuss emerging hyperparameter tuning strategies\n"," - Bayesian Optimization\n"," - Hyperopt\n"," - Genetic Evolution"]},{"cell_type":"markdown","metadata":{"id":"kv2C8aVME-rz","colab_type":"text"},"source":["# Sources"]},{"cell_type":"markdown","metadata":{"id":"f1u0YL0ME-rz","colab_type":"text"},"source":["## Additional Reading\n","- https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/\n","- https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/\n","- https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/\n","- https://machinelearningmastery.com/introduction-to-weight-constraints-to-reduce-generalization-error-in-deep-learning/\n","- https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/"]}]} \ No newline at end of file