binary.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "binary.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/chokkan/deeplearning/blob/master/binary.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "CX_9BuPB_hA-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Feedforward Neural Networks\n",
        "\n",
        "This Jupyter notebook explains various ways of implementing single-layer and multi-layer neural networks. The implementations are arranged by concrete (explicit) to abstract order so that one can understand the black-boxed processing in deep learning frameworks.\n",
        "\n",
        "In order to focus on understanding the internals of training, this notebook uses a simple and classic example: *threshold logic units*.\n",
        "Supposing $x=0$ as *false* and $x=1$ as *true*, single-layer neural networks can realize logic units such as AND ($\\wedge$), OR ($\\vee$), NOT ($\\lnot$), and NAND ($|$). Multi-layer neural networks can realize logical compounds such as XOR.\n",
        "\n",
        "| $x_1$ | $x_2$ | AND | OR | NAND | XOR |\n",
        "| :---: |:-----:|:---:|:--:|:----:|:---:|\n",
        "| 0 | 0 | 0 | 0 | 1 | 0 |\n",
        "| 0 | 1 | 0 | 1 | 1 | 1 |\n",
        "| 1 | 0 | 0 | 1 | 1 | 1 |\n",
        "| 1 | 1 | 1 | 1 | 0 | 0 |\n"
      ]
    },
    {
      "metadata": {
        "id": "qxdFG8U2Net8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Using numpy"
      ]
    },
    {
      "metadata": {
        "id": "Ea5cM4JEsENr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Single-layer perceptron\n",
        "\n",
        "A single layer perceptron predicts a binary label $\\hat{y} \\in \\{0, 1\\}$ for a given input vector $\\boldsymbol{x} \\in \\mathbb{R}^d$ ($d$ presents the number of dimensions of inputs) by using the following formula,\n",
        "$$\n",
        "\\hat{y} = g(\\boldsymbol{w} \\cdot \\boldsymbol{x} + b) = g(w_1 x_1 + w_2 x_2 + ... + w_d x_d + b)\n",
        "$$\n",
        "\n",
        "Here, $\\boldsymbol{w} \\in \\mathbb{R}^d$ is a weight vector; $b \\in \\mathbb{R}$ is a bias weight; and $g(.)$ denotes a Heaviside step function (we assume $g(0)=0$).\n",
        "\n",
        "Let's train a NAND gate with two inputs ($d = 2$). More specifically, we want to find a weight vector $\\boldsymbol{w}$ and a bias weight $b$ of a single-layer perceptron that realizes the truth table of the NAND gate: $\\{0,1\\}^2 \\to \\{0,1\\}$.\n",
        "\n",
        "| $x_1$ | $x_2$ | $y$  |\n",
        "| :---: |:-----:|:----:|\n",
        "| 0 | 0 | 1|\n",
        "| 0 | 1 | 1|\n",
        "| 1 | 0 | 1|\n",
        "| 1 | 1 | 0|\n",
        "\n",
        "We convert the truth table into a training set consisting of all mappings of the NAND gate,\n",
        "$$\n",
        "\\boldsymbol{x}_1 = (0, 0), y_1 = 1 \\\\\n",
        "\\boldsymbol{x}_2 = (0, 1), y_2 = 1 \\\\\n",
        "\\boldsymbol{x}_3 = (1, 0), y_3 = 1 \\\\\n",
        "\\boldsymbol{x}_4 = (1, 1), y_4 = 0 \\\\\n",
        "$$\n",
        "\n",
        "In order to train a weight vector and bias weight in a unified code, we include a bias term as an additional dimension to inputs. More concretely, we append $1$ to each input,\n",
        "$$\n",
        "\\boldsymbol{x}'_1 = (0, 0, 1), y_1 = 1 \\\\\n",
        "\\boldsymbol{x}'_2 = (0, 1, 1), y_2 = 1 \\\\\n",
        "\\boldsymbol{x}'_3 = (1, 0, 1), y_3 = 1 \\\\\n",
        "\\boldsymbol{x}'_4 = (1, 1, 1), y_4 = 0 \\\\\n",
        "$$\n",
        "\n",
        "Then, the formula of the single-layer perceptron becomes,\n",
        "$$\n",
        "\\hat{y} = g((w_1, w_2, w_3) \\cdot \\boldsymbol{x}') = g(w_1 x_1 + w_2 x_2 + w_3)\n",
        "$$\n",
        "In other words, $w_1$ and $w_2$ present weights for $x_1$ and $x_2$, respectively, and $w_3$ does a bias weight.\n",
        "\n",
        "The code below implements Rosenblatt's perceptron algorithm with a fixed number of iterations (100 times). We use a constant learning rate 0.5 for simplicity.\n"
      ]
    },
    {
      "metadata": {
        "id": "2ygoUjQYrPoj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "# Training data for NAND.\n",
        "x = np.array([\n",
        "    [0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]\n",
        "    ])\n",
        "y = np.array([0, 0, 0, 1])\n",
        "w = np.array([0.0, 0.0, 0.0])\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    for i in range(len(y)):\n",
        "        y_pred = np.heaviside(np.dot(x[i], w), 0)\n",
        "        w += (y[i] - y_pred) * eta * x[i]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TYoeshu2rXdK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b625033e-81cd-486b-b7e7-05d80a270b32"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 1. ,  0.5, -1. ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "metadata": {
        "id": "hOFgUFojraFA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "88394c13-b3b5-4df8-be13-5b2a404966c0"
      },
      "cell_type": "code",
      "source": [
        "np.heaviside(np.dot(x, w), 0)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 0., 0., 1.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "metadata": {
        "id": "bl_ZAEguNzgU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Single-layer perceptron with mini-batch\n",
        "\n",
        "It is desireable to reduce the execusion run by the Python interpreter, which is relatively slow. The common technique to speed up a machine-learning code written in Python is to to execute computations within the matrix library (e.g., numpy).\n",
        "\n",
        "The single-layer perceptron makes predictions for four inputs,\n",
        "$$\n",
        "\\hat{y}_1 = g(\\boldsymbol{x}_1 \\cdot \\boldsymbol{w}) \\\\\n",
        "\\hat{y}_2 = g(\\boldsymbol{x}_2 \\cdot \\boldsymbol{w}) \\\\\n",
        "\\hat{y}_3 = g(\\boldsymbol{x}_3 \\cdot \\boldsymbol{w}) \\\\\n",
        "\\hat{y}_4 = g(\\boldsymbol{x}_4 \\cdot \\boldsymbol{w}) \\\\\n",
        "$$\n",
        "\n",
        "Here, we define $\\hat{Y} \\in \\mathbb{R}^{4 \\times 1}$ and $X \\in \\mathbb{R}^{4 \\times d}$ as,\n",
        "$$\n",
        "\\hat{Y} = \\begin{pmatrix} \n",
        "  \\hat{y}_1 \\\\ \n",
        "  \\hat{y}_2 \\\\ \n",
        "  \\hat{y}_3 \\\\ \n",
        "  \\hat{y}_4 \\\\ \n",
        "\\end{pmatrix},\n",
        "X = \\begin{pmatrix} \n",
        "  \\boldsymbol{x}_1 \\\\ \n",
        "  \\boldsymbol{x}_2 \\\\ \n",
        "  \\boldsymbol{x}_3 \\\\ \n",
        "  \\boldsymbol{x}_4 \\\\ \n",
        "\\end{pmatrix}\n",
        "$$\n",
        "\n",
        "Then, we can write the four predictions in one dot-product computation,\n",
        "$$\n",
        "\\hat{Y} = X \\cdot \\boldsymbol{w}\n",
        "$$\n",
        "\n",
        "The code below implements this idea. The function `np.heaviside()` yields a vector corresponding to the four predictions, applying the step function for every element of the argument.\n",
        "\n",
        "This technique is frequently used in mini-batch training."
      ]
    },
    {
      "metadata": {
        "id": "2fK-_WimtPwb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "# Training data for NAND.\n",
        "x = np.array([\n",
        "    [0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]\n",
        "    ])\n",
        "y = np.array([1, 1, 1, 0])\n",
        "w = np.array([0.0, 0.0, 0.0])\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    y_pred = np.heaviside(np.dot(x, w), 0)\n",
        "    w += np.dot((y - y_pred), x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_x4p1BldtQ-K",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "14f51ece-f621-4381-8ccf-b25497b495de"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-1., -1.,  2.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "metadata": {
        "id": "E-P2RpWrtVyf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2b0c1c82-6d50-4454-cf10-3f3ecff138cf"
      },
      "cell_type": "code",
      "source": [
        "np.heaviside(np.dot(x, w), 0)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 1., 1., 0.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "metadata": {
        "id": "nkxBcCSTtDvm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Stochastic gradient descent (SGD) with mini-batch"
      ]
    },
    {
      "metadata": {
        "id": "bltpfNRctjV5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def sigmoid(v):\n",
        "    return 1.0 / (1 + np.exp(-v))\n",
        "\n",
        "# Training data for NAND.\n",
        "x = np.array([\n",
        "    [0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]\n",
        "    ])\n",
        "y = np.array([1, 1, 1, 0])\n",
        "w = np.array([0.0, 0.0, 0.0])\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    y_pred = sigmoid(np.dot(x, w))\n",
        "    w -= np.dot((y_pred - y), x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9bASDMfhtm-I",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b9970d78-2df3-4d7f-8807-153b2d2a945e"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-5.59504346, -5.59504346,  8.57206068])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "-69r0c4KtqlW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c32f12f1-dc08-4def-d3f6-9086daa35584"
      },
      "cell_type": "code",
      "source": [
        "sigmoid(np.dot(x, w))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.99981071, 0.95152498, 0.95152498, 0.06798725])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "ELUeNFRFuJv3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Automatic differentiation"
      ]
    },
    {
      "metadata": {
        "id": "aB0DwVOyuXP_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### autograd\n",
        "\n",
        "Installing [autograd](https://github.com/HIPS/autograd) (do this once)."
      ]
    },
    {
      "metadata": {
        "id": "kpt7DJ7a-qov",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "b5224832-95f6-4374-d694-074046af3914"
      },
      "cell_type": "code",
      "source": [
        "!pip install autograd"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting autograd\n",
            "  Downloading https://files.pythonhosted.org/packages/08/7a/1ccee2a929d806ba3dbe632a196ad6a3f1423d6e261ae887e5fef2011420/autograd-1.2.tar.gz\n",
            "Requirement already satisfied: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from autograd) (1.14.5)\n",
            "Requirement already satisfied: future>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from autograd) (0.16.0)\n",
            "Building wheels for collected packages: autograd\n",
            "  Running setup.py bdist_wheel for autograd ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/72/6f/c2/40f130cca2c91f31d354bf72de282922479c09ce0b7853c4c5\n",
            "Successfully built autograd\n",
            "Installing collected packages: autograd\n",
            "Successfully installed autograd-1.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DyC9iOFO-1cd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "8446e847-ac07-4a24-b575-1ac992755aa0"
      },
      "cell_type": "code",
      "source": [
        "import autograd\n",
        "import autograd.numpy as np\n",
        "\n",
        "def loss(w, x):\n",
        "    return -np.log(1.0 / (1 + np.exp(-np.dot(x, w))))\n",
        "\n",
        "x = np.array([1, 1, 1])\n",
        "w = np.array([1.0, 1.0, -1.5])\n",
        "\n",
        "grad_loss = autograd.grad(loss)\n",
        "print(loss(w, x))\n",
        "print(grad_loss(w, x))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.47407698418010663\n",
            "[-0.37754067 -0.37754067 -0.37754067]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "59D2aARTuQDL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "VOoZXvXP-6b2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336
        },
        "outputId": "988e55e6-3988-4d9b-f29d-bd71146edbf2"
      },
      "cell_type": "code",
      "source": [
        "!pip install torch torchvision"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (519.5MB)\n",
            "\u001b[K    100% |████████████████████████████████| 519.5MB 26kB/s \n",
            "tcmalloc: large alloc 1073750016 bytes == 0x586c2000 @  0x7f423cd5f1c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54e4c8\n",
            "\u001b[?25hCollecting torchvision\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 1.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.5)\n",
            "Collecting pillow>=4.1.1 (from torchvision)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d1/24/f53ff6b61b3d728b90934bddb4f03f8ab584a7f49299bf3bde56e2952612/Pillow-5.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 3.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n",
            "Installing collected packages: torch, pillow, torchvision\n",
            "  Found existing installation: Pillow 4.0.0\n",
            "    Uninstalling Pillow-4.0.0:\n",
            "      Successfully uninstalled Pillow-4.0.0\n",
            "Successfully installed pillow-5.2.0 torch-0.4.1 torchvision-0.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "nHixTUut_LIm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "3a75b20d-1648-4654-c32a-3dbbe1009c7d"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "x = torch.tensor([1, 1, 1], dtype=dtype)\n",
        "w = torch.tensor([1.0, 1.0, -1.5], dtype=dtype, requires_grad=True)\n",
        "\n",
        "loss = -torch.dot(x, w).sigmoid().log()\n",
        "loss.backward()\n",
        "print(loss.item())\n",
        "print(w.grad)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.4740769565105438\n",
            "tensor([-0.3775, -0.3775, -0.3775])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "sen63x1lvQq9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "IfYnaLS-vU6j",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        },
        "outputId": "aae97f5f-e925-48c1-bb59-4ec312151086"
      },
      "cell_type": "code",
      "source": [
        "!pip install chainer"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting chainer\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/c6/61ff9041ea7427fc1e39768f740ab8b880f8ef20960a5f791e978e8d81c0/chainer-4.3.1.tar.gz (400kB)\n",
            "\u001b[K    100% |████████████████████████████████| 409kB 4.7MB/s \n",
            "\u001b[?25hCollecting filelock (from chainer)\n",
            "  Downloading https://files.pythonhosted.org/packages/2d/ba/db7e0717368958827fa97af0b8acafd983ac3a6ecd679f60f3ccd6e5b16e/filelock-3.0.4.tar.gz\n",
            "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from chainer) (1.14.5)\n",
            "Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from chainer) (3.6.1)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from chainer) (1.11.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.0.0->chainer) (39.1.0)\n",
            "Building wheels for collected packages: chainer, filelock\n",
            "  Running setup.py bdist_wheel for chainer ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/8a/ef/b0/e67e0555c4d520566d6565d9634ecb7fbb1594758236bb7b40\n",
            "  Running setup.py bdist_wheel for filelock ... \u001b[?25l-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/35/ba/67/4cc48738870c3b54f9e3b5d78bf9de130befb70c1d359faf8b\n",
            "Successfully built chainer filelock\n",
            "Installing collected packages: filelock, chainer\n",
            "Successfully installed chainer-4.3.1 filelock-3.0.4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yhcXvLS4vld2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "5be9f91b-b651-4820-d5c2-41d69d766ccd"
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from chainer import Variable\n",
        "import chainer.functions as F\n",
        "\n",
        "dtype = np.float32\n",
        "\n",
        "x = np.array([1,1,1], dtype=dtype)\n",
        "w = Variable(np.array([1.0,1.0,-1.5], dtype=dtype), requires_grad=True)\n",
        "\n",
        "loss = -F.log(F.sigmoid(np.dot(x,w)))\n",
        "loss.backward()\n",
        "print(loss.data)\n",
        "print(w.grad)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.47407696\n",
            "[-0.37754062 -0.37754062 -0.37754062]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ywdJcyJIwgjo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "95JvtYS6wjfu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "d35ce9b4-82c3-49aa-dbe2-e4647d8c3036"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "x = tf.constant([1., 1., 1.])\n",
        "w = tf.Variable([1.0, 1.0, -1.5])\n",
        "\n",
        "loss = -tf.log(tf.sigmoid(tf.tensordot(x, w, axes=1)))\n",
        "grad = tf.gradients(loss, w)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    loss_value = sess.run(loss)\n",
        "    grad_value = sess.run(grad)\n",
        "    print(loss_value)\n",
        "    print(grad_value)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.47407696\n",
            "[array([-0.37754062, -0.37754062, -0.37754062], dtype=float32)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Z9xsthOcH6Go",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### MXNet"
      ]
    },
    {
      "metadata": {
        "id": "lv5pWNhLH9oK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "4ee807c5-1dc8-4bf7-ad16-4493cf1636a2"
      },
      "cell_type": "code",
      "source": [
        "!pip install mxnet"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting mxnet\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bb/53/5d33f71c5224a676112679458714eb728f6db8cae15f39fcdf27226f6e41/mxnet-1.2.1.post1-py2.py3-none-manylinux1_x86_64.whl (24.2MB)\n",
            "\u001b[K    100% |████████████████████████████████| 24.2MB 1.6MB/s \n",
            "\u001b[?25hCollecting graphviz<0.9.0,>=0.8.1 (from mxnet)\n",
            "  Downloading https://files.pythonhosted.org/packages/53/39/4ab213673844e0c004bed8a0781a0721a3f6bb23eb8854ee75c236428892/graphviz-0.8.4-py2.py3-none-any.whl\n",
            "Requirement already satisfied: numpy<1.15.0,>=1.8.2 in /usr/local/lib/python3.6/dist-packages (from mxnet) (1.14.5)\n",
            "Requirement already satisfied: requests<2.19.0,>=2.18.4 in /usr/local/lib/python3.6/dist-packages (from mxnet) (2.18.4)\n",
            "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<2.19.0,>=2.18.4->mxnet) (1.22)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<2.19.0,>=2.18.4->mxnet) (3.0.4)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<2.19.0,>=2.18.4->mxnet) (2.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<2.19.0,>=2.18.4->mxnet) (2018.8.13)\n",
            "Installing collected packages: graphviz, mxnet\n",
            "Successfully installed graphviz-0.8.4 mxnet-1.2.1.post1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-a9oa0n7IDpJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "93c35407-48d0-41cb-99b9-fdbd24135a22"
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd, gluon\n",
        "\n",
        "x = nd.array([1., 1., 1.])\n",
        "w = nd.array([1.0, 1.0, -1.5])\n",
        "w.attach_grad()\n",
        "\n",
        "with autograd.record():\n",
        "    loss = -nd.dot(x, w).sigmoid().log()\n",
        "loss.backward()\n",
        "print(loss)\n",
        "print(w.grad)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "[0.47407696]\n",
            "<NDArray 1 @cpu(0)>\n",
            "\n",
            "[-0.37754065 -0.37754065 -0.37754065]\n",
            "<NDArray 3 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "STwWdvJCva4G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Single-layer neural network using automatic differentiation"
      ]
    },
    {
      "metadata": {
        "id": "KFUhyjhc0uCu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "xOJHDGYKIgsm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for NAND.\n",
        "x = torch.tensor([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[1], [1], [1], [0]], dtype=dtype)\n",
        "w = torch.randn(3, 1, dtype=dtype, requires_grad=True)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    # y_pred = \\sigma(x \\cdot w)\n",
        "    y_pred = x.mm(w).sigmoid()\n",
        "    ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "    loss = -ll.log().sum()      # The loss value.\n",
        "    #print(t, loss.item())\n",
        "    loss.backward()             # Compute the gradients of the loss.\n",
        "\n",
        "    with torch.no_grad():\n",
        "        w -= eta * w.grad       # Update weights using SGD.        \n",
        "        w.grad.zero_()          # Clear the gradients for the next iteration."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FyoS5iP6n7Ru",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "760a3c43-a4d0-4d38-cc72-c52f3cd493c1"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-4.2454],\n",
              "        [-4.2453],\n",
              "        [ 6.5599]], requires_grad=True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "metadata": {
        "id": "Jfp604gFn9Yw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "6c4447c9-dd56-4cf4-877c-d09cf1ee0239"
      },
      "cell_type": "code",
      "source": [
        "x.mm(w).sigmoid()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9986],\n",
              "        [0.9101],\n",
              "        [0.9101],\n",
              "        [0.1267]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "metadata": {
        "id": "rJ6AH9MP01qo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "RyWQMzPx0A-S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import chainer\n",
        "from chainer import Variable\n",
        "import chainer.functions as F\n",
        "\n",
        "dtype = np.float32\n",
        "\n",
        "# Training data for NAND\n",
        "x = Variable(np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]], dtype=dtype))\n",
        "y = Variable(np.array([[1], [1], [1], [0]], dtype=dtype))\n",
        "w = Variable(np.random.rand(3, 1).astype(dtype=dtype), requires_grad=True)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    # y_pred = \\sigma(x \\cdot w)\n",
        "    y_pred = F.sigmoid(F.matmul(x, w))\n",
        "    ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "    loss = -F.sum(F.log(ll))    # The loss value.\n",
        "    #print(t, loss)\n",
        "    loss.backward()             # Compute the gradients of the loss.\n",
        "\n",
        "    with chainer.no_backprop_mode():\n",
        "        w -= eta * w.grad       # Update weights using SGD.\n",
        "        w.cleargrad()           # Clear the gradients for the next iteration."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jFT26C730Z1t",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "4e5192c7-b277-459e-f8ad-dbc1428e1922"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[-4.245978 ],\n",
              "          [-4.2458024],\n",
              "          [ 6.5606837]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "metadata": {
        "id": "O4v2mCnI0edo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "37888695-7033-4572-cf65-a985fc65ceb3"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(F.matmul(x, w))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.9985871 ],\n",
              "          [0.910102  ],\n",
              "          [0.9100877 ],\n",
              "          [0.12662926]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "metadata": {
        "id": "2n01NoDj11iG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "v2UHkctj14f3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        },
        "outputId": "2aa8611e-74a6-4275-8e67-68b138f69ede"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Training data for NAND\n",
        "x_data = [[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]]\n",
        "y_data = [[1], [1], [1], [0]]\n",
        "\n",
        "x = tf.placeholder(tf.float32, [4, 3])\n",
        "y = tf.placeholder(tf.float32, [4, 1])\n",
        "w = tf.Variable(tf.random_normal([3,1]))\n",
        "\n",
        "# y_pred = \\sigma(x \\cdot w)\n",
        "y_pred = tf.sigmoid(tf.matmul(x, w))\n",
        "ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "loss = -tf.reduce_sum(tf.log(ll))\n",
        "grad = tf.gradients(loss, w)\n",
        "\n",
        "eta = 0.5\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "\n",
        "    for t in range(100):\n",
        "        grads = sess.run(grad, feed_dict={x: x_data, y: y_data})\n",
        "        sess.run(w.assign_sub(eta * grads[0]))\n",
        "    print(sess.run(w))\n",
        "    print(sess.run(y_pred, feed_dict={x: x_data, y: y_data}))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[-4.1939316]\n",
            " [-4.193681 ]\n",
            " [ 6.483271 ]]\n",
            "[[0.99847347]\n",
            " [0.9080112 ]\n",
            " [0.9079903 ]\n",
            " [0.12961791]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KDIxs_lZIiY9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### MXNet"
      ]
    },
    {
      "metadata": {
        "id": "PBAknfkvIqHD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd\n",
        "\n",
        "# Training data for NAND.\n",
        "x = nd.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])\n",
        "y = nd.array([[1], [1], [1], [0]])\n",
        "w = nd.random.normal(0, 1, shape=(3, 1))\n",
        "w.attach_grad()\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    with autograd.record():\n",
        "        # y_pred = \\sigma(x \\cdot w).\n",
        "        y_pred = nd.dot(x, w).sigmoid()\n",
        "        ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "        loss = -ll.log().sum()      # The loss value.\n",
        "        #print(t, loss)\n",
        "    loss.backward()                 # Compute the gradients of the loss.\n",
        "    w -= eta * w.grad               # Update weights using SGD."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "x77z9r85I_QE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "48822b0b-c6ab-4708-c53f-33c8cd204596"
      },
      "cell_type": "code",
      "source": [
        "w"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\n",
              "[[-4.2020216]\n",
              " [-4.20314  ]\n",
              " [ 6.4963117]]\n",
              "<NDArray 3x1 @cpu(0)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "metadata": {
        "id": "GQh5ABrCJEyR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "a777eb9d-90b4-4dc8-ad76-911cb3837a0e"
      },
      "cell_type": "code",
      "source": [
        "nd.dot(x, w).sigmoid()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\n",
              "[[0.9984933 ]\n",
              " [0.90831   ]\n",
              " [0.90840304]\n",
              " [0.12911019]]\n",
              "<NDArray 4x1 @cpu(0)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "metadata": {
        "id": "w6eu-AuDvl9J",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Multi-layer neural network using automatic differentiation"
      ]
    },
    {
      "metadata": {
        "id": "M6HF5OmoJ3cN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "ts2RTKVPn_xk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for XOR.\n",
        "x = torch.tensor([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[0], [1], [1], [0]], dtype=dtype)\n",
        "w1 = torch.randn(3, 2, dtype=dtype, requires_grad=True)\n",
        "w2 = torch.randn(2, 1, dtype=dtype, requires_grad=True)\n",
        "b2 = torch.randn(1, 1, dtype=dtype, requires_grad=True)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(1000):\n",
        "    # y_pred = \\sigma(w_2 \\cdot \\sigma(x \\cdot w_1) + b_2)\n",
        "    y_pred = x.mm(w1).sigmoid().mm(w2).add(b2).sigmoid()\n",
        "    ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "    loss = -ll.log().sum()\n",
        "    #print(t, loss.item())\n",
        "    loss.backward()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "        # Update weights using SGD.\n",
        "        w1 -= eta * w1.grad\n",
        "        w2 -= eta * w2.grad\n",
        "        b2 -= eta * b2.grad\n",
        "        \n",
        "        # Clear the gradients for the next iteration.\n",
        "        w1.grad.zero_()\n",
        "        w2.grad.zero_()\n",
        "        b2.grad.zero_()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FWgbqAXawEof",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "1e777d4c-5afb-488f-a2dc-3a7260240050"
      },
      "cell_type": "code",
      "source": [
        "print(w1)\n",
        "print(w2)\n",
        "print(b2)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 6.8666, -6.5697],\n",
            "        [-7.0005,  6.2410],\n",
            "        [-3.7858, -3.4025]], requires_grad=True)\n",
            "tensor([[11.2838],\n",
            "        [11.3932]], requires_grad=True)\n",
            "tensor([[-5.5641]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yS4bql3foxB5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "3c9139f8-2d68-4767-d611-f8b25a482ec6"
      },
      "cell_type": "code",
      "source": [
        "x.mm(w1).sigmoid().mm(w2).add(b2).sigmoid()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.0071],\n",
              "        [0.9945],\n",
              "        [0.9946],\n",
              "        [0.0062]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "metadata": {
        "id": "tL7eH-4CKRrd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "HGDfRFVOKXE2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import chainer\n",
        "from chainer import Variable\n",
        "import chainer.functions as F\n",
        "\n",
        "dtype = np.float32\n",
        "\n",
        "# Training data for XOR.\n",
        "x = np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]], dtype=dtype)\n",
        "y = np.array([[0], [1], [1], [0]],dtype=dtype)\n",
        "w1 = Variable(np.random.randn(3, 2).astype(dtype),requires_grad=True)\n",
        "w2 = Variable(np.random.randn(2, 1).astype(dtype),requires_grad=True)\n",
        "b2 = Variable(np.random.randn(1).astype(dtype), requires_grad=True)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(1000):\n",
        "    # y_pred = \\sigma(w_2 \\cdot \\sigma(x \\cdot w_1) + b_2)\n",
        "    y_pred = F.sigmoid(F.bias(F.matmul(F.sigmoid(F.matmul(x, w1)), w2), b2))\n",
        "    ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "    loss = -F.sum(F.log(ll))\n",
        "    #print(t, loss.data)\n",
        "    loss.backward()\n",
        "    with chainer.no_backprop_mode():\n",
        "        # Update weights using SGD.\n",
        "        w1 -= eta * w1.grad\n",
        "        w2 -= eta * w2.grad\n",
        "        b2 -= eta * b2.grad\n",
        "\n",
        "        # Clear the gradients for the next iteration.\n",
        "        w1.cleargrad()\n",
        "        w2.cleargrad()\n",
        "        b2.cleargrad()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6zXS3DQJN501",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "3b1b431f-ba33-4670-d590-85ea06a40f0b"
      },
      "cell_type": "code",
      "source": [
        "print(w1)\n",
        "print(w2)\n",
        "print(b2)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "variable([[ 6.359068  -7.071981 ]\n",
            "          [-6.6552386  6.8526063]\n",
            "          [-3.472819  -3.7479024]])\n",
            "variable([[11.586926]\n",
            "          [11.482569]])\n",
            "variable([-5.663216])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wNCDnvIZN_6j",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "bb853cad-9015-4269-b893-ac2654a6b059"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(F.bias(F.matmul(F.sigmoid(F.matmul(x,w1)) ,w2), b2))"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.00636774],\n",
              "          [0.9951651 ],\n",
              "          [0.9950907 ],\n",
              "          [0.00554883]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "metadata": {
        "id": "SR1jaeOtOcYE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "ko4WhSSDOev4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "5d000600-0156-47cc-b1ff-56ae866fc130"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Training data for XOR.\n",
        "x_data = [[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]]\n",
        "y_data = [[0], [1], [1], [0]]\n",
        "\n",
        "x = tf.placeholder(tf.float32, [4, 3])\n",
        "y = tf.placeholder(tf.float32, [4, 1])\n",
        "w1 = tf.Variable(tf.random_normal([3, 2]))\n",
        "w2 = tf.Variable(tf.random_normal([2, 1]))\n",
        "b2 = tf.Variable(tf.random_normal([1, 1]))\n",
        "\n",
        "y_pred = tf.sigmoid(tf.add(tf.matmul(tf.sigmoid(tf.matmul(x, w1)), w2), b2))\n",
        "ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "log = tf.log(ll)\n",
        "loss = -tf.reduce_sum(log)\n",
        "grad = tf.gradients(loss, [w1, w2, b2])\n",
        "\n",
        "eta = 0.5\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    for t in range(1000):\n",
        "        w1_grad, w2_grad, b2_grad = sess.run(grad, feed_dict={x: x_data, y: y_data})\n",
        "        sess.run(tf.assign_sub(w1, eta * w1_grad))\n",
        "        sess.run(tf.assign_sub(w2, eta * w2_grad))\n",
        "        sess.run(tf.assign_sub(b2, eta * b2_grad))\n",
        "        \n",
        "    print(sess.run(w1))\n",
        "    print(sess.run(w2))\n",
        "    print(sess.run(b2))\n",
        "    print(sess.run(y_pred, feed_dict={x: x_data, y: y_data}))"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ 7.0503726  6.132738 ]\n",
            " [-6.8502035 -6.4016724]\n",
            " [ 3.4437704 -3.2621946]]\n",
            "[[-10.97688 ]\n",
            " [ 11.618423]]\n",
            "[[5.131303]]\n",
            "[[0.00619888]\n",
            " [0.99167174]\n",
            " [0.9942344 ]\n",
            " [0.0052963 ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CTg-EJBlFqJE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### MXNet"
      ]
    },
    {
      "metadata": {
        "id": "DO_j4DyjFo1r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd\n",
        "\n",
        "# Training data for XOR.\n",
        "x = nd.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])\n",
        "y = nd.array([[0], [1], [1], [0]])\n",
        "\n",
        "w1 = nd.random.normal(0, 1, shape=(3, 2))\n",
        "w2 = nd.random.normal(0, 1, shape=(2, 1))\n",
        "b2 = nd.random.normal(0, 1, shape=(1, 1))\n",
        "w1.attach_grad()\n",
        "w2.attach_grad()\n",
        "b2.attach_grad()\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(1000):\n",
        "    with autograd.record():\n",
        "        # y_pred = \\sigma(w_2 \\cdot \\sigma(x \\cdot w_1) + b_2)\n",
        "        y_pred = (nd.dot(nd.dot(x, w1).sigmoid(), w2) + b2).sigmoid()\n",
        "        ll = y * y_pred + (1 - y) * (1 - y_pred)\n",
        "        loss = -ll.log().sum()\n",
        "    loss.backward()\n",
        "    \n",
        "    # Update weights using SGD.\n",
        "    w1 -= eta * w1.grad\n",
        "    w2 -= eta * w2.grad\n",
        "    b2 -= eta * b2.grad"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "f9al8eIVHGQM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 228
        },
        "outputId": "9d732e77-d8ce-4d80-db22-9fe4f1e63113"
      },
      "cell_type": "code",
      "source": [
        "print(w1)\n",
        "print(w2)\n",
        "print(b2)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "[[-6.5379534  8.085663 ]\n",
            " [ 6.8499675 -7.9502664]\n",
            " [ 3.2666583  4.045889 ]]\n",
            "<NDArray 3x2 @cpu(0)>\n",
            "\n",
            "[[-9.515034]\n",
            " [-9.384542]]\n",
            "<NDArray 2x1 @cpu(0)>\n",
            "\n",
            "[[13.912197]]\n",
            "<NDArray 1x1 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SA6mRL4pHOuo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "e036058a-9395-4ffa-dee8-e72147a28f7d"
      },
      "cell_type": "code",
      "source": [
        "print((nd.dot(nd.dot(x, w1).sigmoid(), w2) + b2).sigmoid())"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "[[0.01124511]\n",
            " [0.98540175]\n",
            " [0.9849283 ]\n",
            " [0.01007298]]\n",
            "<NDArray 4x1 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "OGS23bDSazMJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Single-layer neural network with high-level NN modules\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "OG447nIlHwt7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "Jt9eizLFo1iN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "575a43ab-ee08-4996-c9c7-544703ca30fe"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for NAND.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[1], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network using high-level modules.\n",
        "model = torch.nn.Sequential(\n",
        "    torch.nn.Linear(2, 1, bias=True),   # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    y_pred = model(x)                   # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)           # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    model.zero_grad()                   # Zero-clear the gradients.\n",
        "    loss.backward()                     # Compute the gradients.\n",
        "        \n",
        "    with torch.no_grad():\n",
        "        for param in model.parameters():\n",
        "            param -= eta * param.grad   # Update the parameters using SGD."
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "Zq6oqLmIENFa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "14ea61d4-4ac4-4502-cb52-87da942808e5"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('0.weight', tensor([[-4.2161, -4.2162]])),\n",
              "             ('0.bias', tensor([6.5164]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "metadata": {
        "id": "QdPOdNgO840b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "7ddc028c-e714-4437-a10b-2a892f968ef5"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9985],\n",
              "        [0.9089],\n",
              "        [0.9089],\n",
              "        [0.1283]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "metadata": {
        "id": "MYkgcs9CIQxD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "LSVBhNQFIThA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import chainer\n",
        "import numpy as np\n",
        "from chainer import Variable, Function\n",
        "import chainer.functions as F\n",
        "import chainer.links as L\n",
        "\n",
        "dtype = np.float32\n",
        "\n",
        "# Training data for NAND\n",
        "x = Variable(np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype))\n",
        "y = Variable(np.array([[1], [1], [1], [0]], dtype=np.int32))\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "model = chainer.Sequential(\n",
        "    L.Linear(2, 1, nobias=False)            # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn=F.sigmoid_cross_entropy\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(100):\n",
        "    y_pred = model(x)                       # Make predictions.\n",
        "    loss = loss_fn(y_pred, y, normalize=False)\n",
        "    # print(t, loss.data)\n",
        "    model.cleargrads()                      # Zero-clear the gradients.\n",
        "    loss.backward()                         # Compute the gradients.\n",
        "\n",
        "    with chainer.no_backprop_mode():\n",
        "        for para in model.params():\n",
        "            para.data -= eta * para.grad    # Update the parameters using SGD."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ELDrFNAjLld2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "fc46fc74-79be-40be-872e-76202bfe7f84"
      },
      "cell_type": "code",
      "source": [
        "for para in model.params():\n",
        "    print(para)"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "variable W([[-2.1200492 -2.1241121]])\n",
            "variable b([3.4434984])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "GyP2DgEzLnO9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "0de31f9c-152f-496b-da8a-24264dab474b"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.9690367 ],\n",
              "          [0.78907955],\n",
              "          [0.789755  ],\n",
              "          [0.30988365]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "metadata": {
        "id": "KfuoJMeqbClA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Multi-layer neural network with high-level NN modules"
      ]
    },
    {
      "metadata": {
        "id": "WSMdmdxVM0v-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "D6ss25zA9nPk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "942a48ad-3476-4fc7-aac2-7d848be2e82e"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for XOR.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[0], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network using high-level modules.\n",
        "model = torch.nn.Sequential(\n",
        "    torch.nn.Linear(2, 2, bias=True),   # 2 dims (with bias) -> 2 dims\n",
        "    torch.nn.Sigmoid(),                 # Sigmoid function\n",
        "    torch.nn.Linear(2, 1, bias=True),   # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)                   # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)           # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    model.zero_grad()                   # Zero-clear the gradients.\n",
        "    loss.backward()                     # Compute the gradients.\n",
        "        \n",
        "    with torch.no_grad():\n",
        "        for param in model.parameters():\n",
        "            param -= eta * param.grad   # Update the parameters using SGD."
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "iLFGZWL0--2n",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "af35efe8-d919-4676-a758-09515ec8dbc5"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('0.weight', tensor([[-5.8098,  5.7082],\n",
              "                      [ 3.7659, -4.1443]])),\n",
              "             ('0.bias', tensor([-3.5452, -2.0044])),\n",
              "             ('2.weight', tensor([[6.4493, 5.9124]])),\n",
              "             ('2.bias', tensor([-2.9767]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "metadata": {
        "id": "1BF6-W_J-82M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "1c194187-047a-4da5-d54a-343e63713a8f"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.1097],\n",
              "        [0.9438],\n",
              "        [0.8879],\n",
              "        [0.0900]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "metadata": {
        "id": "1FQ-xBYYM3ua",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "v5V_RgGaM6Rw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import chainer\n",
        "import numpy as np\n",
        "from chainer import Variable, Function\n",
        "import chainer.functions as F\n",
        "import chainer.links as L\n",
        "\n",
        "dtype = np.float32\n",
        "\n",
        "# Training data for XOR.\n",
        "x = chainer.Variable(np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype))\n",
        "y = chainer.Variable(np.array([[0], [1], [1], [0]], dtype=np.int32))\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "init=chainer.initializers.HeNormal()\n",
        "model = chainer.Sequential(\n",
        "    L.Linear(2, 2, nobias=False, initialW=init), # 2 dims (with bias) -> 2 dims\n",
        "    F.sigmoid,                                   # Sigmoid function\n",
        "    L.Linear(2, 1, nobias=False, initialW=init), # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn=F.sigmoid_cross_entropy\n",
        "\n",
        "eta = 0.5\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)                            # Make predictions.\n",
        "    loss = loss_fn(y_pred, y, normalize=False)\n",
        "    # print(t, loss.data)\n",
        "    model.cleargrads()                           # Zero-clear the gradients.\n",
        "    loss.backward()                              # Compute the gradients.\n",
        "\n",
        "    with chainer.no_backprop_mode():\n",
        "        for para in model.params():\n",
        "            para.data -= eta * para.grad     # Update the parameters using SGD."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "R6UCLsxZNbUj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "73c9d8a6-8d4a-41fe-a784-f497fa612454"
      },
      "cell_type": "code",
      "source": [
        "for para in model.params():\n",
        "    print(para)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "variable W([[-5.715909  -3.6899567]\n",
            "            [ 6.2749352 -4.1101246]])\n",
            "variable b([0.88635695 2.5805264 ])\n",
            "variable W([[-5.1726246 -4.258353 ]])\n",
            "variable b([4.2472806])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zZzG-fzRNiFU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "12da40e2-4cf7-4598-e671-adb0b7665734"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.03311139],\n",
              "          [0.96059114],\n",
              "          [0.48713586],\n",
              "          [0.5061473 ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "metadata": {
        "id": "gGK3DJBubb0f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Single-layer neural network with an optimizer."
      ]
    },
    {
      "metadata": {
        "id": "z0PCruKKN1ER",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "puqbP4F9bidv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "114925bc-74b2-46d9-ad4d-2a5eb5a6fbeb"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for NAND.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[1], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network using high-level modules.\n",
        "model = torch.nn.Sequential(\n",
        "    torch.nn.Linear(2, 1, bias=True),   # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=0.5)\n",
        "\n",
        "for t in range(100):\n",
        "    y_pred = model(x)           # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)   # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    optimizer.zero_grad()       # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.step()            # Update the parameters using the gradients."
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "RBUX1BUhcDK4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "b1b0fc37-c68e-4316-c1fb-476035a080d6"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('0.weight', tensor([[-4.2385, -4.2383]])),\n",
              "             ('0.bias', tensor([6.5496]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "metadata": {
        "id": "nJWcDaPpcKCB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "861a70e7-918a-41d6-92e1-adde4d0f7198"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9986],\n",
              "        [0.9098],\n",
              "        [0.9098],\n",
              "        [0.1271]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "metadata": {
        "id": "5BHpqwdZN8tr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "f9ua2_09N_tv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import chainer\n",
        "import numpy as np\n",
        "from chainer import functions as F\n",
        "from chainer import links as L\n",
        "chainer.config.train = True\n",
        "\n",
        "dtype=np.float32\n",
        "\n",
        "# Training data for NAND.\n",
        "x = chainer.Variable(np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype))\n",
        "y = chainer.Variable(np.array([[1], [1], [1], [0]], dtype=np.int32))\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "model = chainer.Sequential(\n",
        "    L.Linear(2, 1, nobias=False),   # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = F.sigmoid_cross_entropy\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = chainer.optimizers.SGD(lr=0.5)\n",
        "optimizer.setup(model)\n",
        "\n",
        "for t in range(100):\n",
        "    y_pred = model(x)                   # Make predictions.\n",
        "    loss = loss_fn(y_pred, y, normalize=False)   # Compute the loss.\n",
        "    #print(t, loss.data)\n",
        "    \n",
        "    model.cleargrads()          # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.update()          # Update the parameters using the gradients."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SX5G1gCUObKP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "74b42b73-65f8-4fa2-8882-c98faaaf40c3"
      },
      "cell_type": "code",
      "source": [
        "for para in model.params():\n",
        "    print(para)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "variable W([[-2.2661328 -2.1941178]])\n",
            "variable b([3.5988352])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SuGkwIFwOfWS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "10d5a432-a10a-488e-a0b0-b6e9097b5601"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.9733728 ],\n",
              "          [0.8029314 ],\n",
              "          [0.7912873 ],\n",
              "          [0.29704365]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "metadata": {
        "id": "hzJjvZ7XOn6b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TensorFlow (Keras)"
      ]
    },
    {
      "metadata": {
        "id": "VAKV169IOqvH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3605
        },
        "outputId": "d70d665e-c3fd-4e5e-c3db-a08010c0428d"
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Flatten, Dense, Activation\n",
        "from tensorflow.keras import optimizers\n",
        "\n",
        "# Training data for NAND.\n",
        "x_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
        "y_data = np.array([[1], [1], [1], [0]])\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "model = Sequential([\n",
        "    Flatten(),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "    optimizer=optimizers.SGD(lr=0.5),\n",
        "    loss='binary_crossentropy',\n",
        "    metrics=['accuracy']\n",
        "    )\n",
        "\n",
        "model.fit(x_data, y_data, epochs=100)"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()\n",
            "  return _inspect.getargspec(target)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "4/4 [==============================] - 1s 218ms/step - loss: 0.7724 - acc: 0.5000\n",
            "Epoch 2/100\n",
            "4/4 [==============================] - 0s 1ms/step - loss: 0.7112 - acc: 0.7500\n",
            "Epoch 3/100\n",
            "4/4 [==============================] - 0s 932us/step - loss: 0.6657 - acc: 0.7500\n",
            "Epoch 4/100\n",
            "4/4 [==============================] - 0s 339us/step - loss: 0.6314 - acc: 0.7500\n",
            "Epoch 5/100\n",
            "4/4 [==============================] - 0s 506us/step - loss: 0.6050 - acc: 0.7500\n",
            "Epoch 6/100\n",
            "4/4 [==============================] - 0s 430us/step - loss: 0.5840 - acc: 0.7500\n",
            "Epoch 7/100\n",
            "4/4 [==============================] - 0s 372us/step - loss: 0.5667 - acc: 0.5000\n",
            "Epoch 8/100\n",
            "4/4 [==============================] - 0s 341us/step - loss: 0.5522 - acc: 0.5000\n",
            "Epoch 9/100\n",
            "4/4 [==============================] - 0s 474us/step - loss: 0.5394 - acc: 0.5000\n",
            "Epoch 10/100\n",
            "4/4 [==============================] - 0s 330us/step - loss: 0.5281 - acc: 0.5000\n",
            "Epoch 11/100\n",
            "4/4 [==============================] - 0s 347us/step - loss: 0.5177 - acc: 0.5000\n",
            "Epoch 12/100\n",
            "4/4 [==============================] - 0s 474us/step - loss: 0.5081 - acc: 0.5000\n",
            "Epoch 13/100\n",
            "4/4 [==============================] - 0s 416us/step - loss: 0.4992 - acc: 0.7500\n",
            "Epoch 14/100\n",
            "4/4 [==============================] - 0s 413us/step - loss: 0.4907 - acc: 0.7500\n",
            "Epoch 15/100\n",
            "4/4 [==============================] - 0s 341us/step - loss: 0.4826 - acc: 0.7500\n",
            "Epoch 16/100\n",
            "4/4 [==============================] - 0s 273us/step - loss: 0.4749 - acc: 0.7500\n",
            "Epoch 17/100\n",
            "4/4 [==============================] - 0s 276us/step - loss: 0.4675 - acc: 0.7500\n",
            "Epoch 18/100\n",
            "4/4 [==============================] - 0s 430us/step - loss: 0.4603 - acc: 0.7500\n",
            "Epoch 19/100\n",
            "4/4 [==============================] - 0s 269us/step - loss: 0.4535 - acc: 0.7500\n",
            "Epoch 20/100\n",
            "4/4 [==============================] - 0s 284us/step - loss: 0.4469 - acc: 0.7500\n",
            "Epoch 21/100\n",
            "4/4 [==============================] - 0s 513us/step - loss: 0.4405 - acc: 0.7500\n",
            "Epoch 22/100\n",
            "4/4 [==============================] - 0s 345us/step - loss: 0.4343 - acc: 0.7500\n",
            "Epoch 23/100\n",
            "4/4 [==============================] - 0s 376us/step - loss: 0.4283 - acc: 0.7500\n",
            "Epoch 24/100\n",
            "4/4 [==============================] - 0s 522us/step - loss: 0.4225 - acc: 0.7500\n",
            "Epoch 25/100\n",
            "4/4 [==============================] - 0s 406us/step - loss: 0.4168 - acc: 0.7500\n",
            "Epoch 26/100\n",
            "4/4 [==============================] - 0s 908us/step - loss: 0.4114 - acc: 0.7500\n",
            "Epoch 27/100\n",
            "4/4 [==============================] - 0s 484us/step - loss: 0.4061 - acc: 0.7500\n",
            "Epoch 28/100\n",
            "4/4 [==============================] - 0s 296us/step - loss: 0.4010 - acc: 0.7500\n",
            "Epoch 29/100\n",
            "4/4 [==============================] - 0s 674us/step - loss: 0.3960 - acc: 1.0000\n",
            "Epoch 30/100\n",
            "4/4 [==============================] - 0s 322us/step - loss: 0.3911 - acc: 1.0000\n",
            "Epoch 31/100\n",
            "4/4 [==============================] - 0s 312us/step - loss: 0.3864 - acc: 1.0000\n",
            "Epoch 32/100\n",
            "4/4 [==============================] - 0s 646us/step - loss: 0.3819 - acc: 1.0000\n",
            "Epoch 33/100\n",
            "4/4 [==============================] - 0s 285us/step - loss: 0.3774 - acc: 1.0000\n",
            "Epoch 34/100\n",
            "4/4 [==============================] - 0s 310us/step - loss: 0.3731 - acc: 1.0000\n",
            "Epoch 35/100\n",
            "4/4 [==============================] - 0s 548us/step - loss: 0.3689 - acc: 1.0000\n",
            "Epoch 36/100\n",
            "4/4 [==============================] - 0s 382us/step - loss: 0.3648 - acc: 1.0000\n",
            "Epoch 37/100\n",
            "4/4 [==============================] - 0s 289us/step - loss: 0.3608 - acc: 1.0000\n",
            "Epoch 38/100\n",
            "4/4 [==============================] - 0s 281us/step - loss: 0.3569 - acc: 1.0000\n",
            "Epoch 39/100\n",
            "4/4 [==============================] - 0s 386us/step - loss: 0.3531 - acc: 1.0000\n",
            "Epoch 40/100\n",
            "4/4 [==============================] - 0s 299us/step - loss: 0.3494 - acc: 1.0000\n",
            "Epoch 41/100\n",
            "4/4 [==============================] - 0s 260us/step - loss: 0.3458 - acc: 1.0000\n",
            "Epoch 42/100\n",
            "4/4 [==============================] - 0s 251us/step - loss: 0.3423 - acc: 1.0000\n",
            "Epoch 43/100\n",
            "4/4 [==============================] - 0s 272us/step - loss: 0.3388 - acc: 1.0000\n",
            "Epoch 44/100\n",
            "4/4 [==============================] - 0s 266us/step - loss: 0.3355 - acc: 1.0000\n",
            "Epoch 45/100\n",
            "4/4 [==============================] - 0s 256us/step - loss: 0.3322 - acc: 1.0000\n",
            "Epoch 46/100\n",
            "4/4 [==============================] - 0s 253us/step - loss: 0.3290 - acc: 1.0000\n",
            "Epoch 47/100\n",
            "4/4 [==============================] - 0s 275us/step - loss: 0.3258 - acc: 1.0000\n",
            "Epoch 48/100\n",
            "4/4 [==============================] - 0s 334us/step - loss: 0.3228 - acc: 1.0000\n",
            "Epoch 49/100\n",
            "4/4 [==============================] - 0s 351us/step - loss: 0.3198 - acc: 1.0000\n",
            "Epoch 50/100\n",
            "4/4 [==============================] - 0s 311us/step - loss: 0.3169 - acc: 1.0000\n",
            "Epoch 51/100\n",
            "4/4 [==============================] - 0s 323us/step - loss: 0.3140 - acc: 1.0000\n",
            "Epoch 52/100\n",
            "4/4 [==============================] - 0s 308us/step - loss: 0.3112 - acc: 1.0000\n",
            "Epoch 53/100\n",
            "4/4 [==============================] - 0s 318us/step - loss: 0.3084 - acc: 1.0000\n",
            "Epoch 54/100\n",
            "4/4 [==============================] - 0s 278us/step - loss: 0.3057 - acc: 1.0000\n",
            "Epoch 55/100\n",
            "4/4 [==============================] - 0s 321us/step - loss: 0.3031 - acc: 1.0000\n",
            "Epoch 56/100\n",
            "4/4 [==============================] - 0s 280us/step - loss: 0.3005 - acc: 1.0000\n",
            "Epoch 57/100\n",
            "4/4 [==============================] - 0s 258us/step - loss: 0.2980 - acc: 1.0000\n",
            "Epoch 58/100\n",
            "4/4 [==============================] - 0s 265us/step - loss: 0.2955 - acc: 1.0000\n",
            "Epoch 59/100\n",
            "4/4 [==============================] - 0s 318us/step - loss: 0.2930 - acc: 1.0000\n",
            "Epoch 60/100\n",
            "4/4 [==============================] - 0s 306us/step - loss: 0.2907 - acc: 1.0000\n",
            "Epoch 61/100\n",
            "4/4 [==============================] - 0s 317us/step - loss: 0.2883 - acc: 1.0000\n",
            "Epoch 62/100\n",
            "4/4 [==============================] - 0s 301us/step - loss: 0.2860 - acc: 1.0000\n",
            "Epoch 63/100\n",
            "4/4 [==============================] - 0s 259us/step - loss: 0.2837 - acc: 1.0000\n",
            "Epoch 64/100\n",
            "4/4 [==============================] - 0s 272us/step - loss: 0.2815 - acc: 1.0000\n",
            "Epoch 65/100\n",
            "4/4 [==============================] - 0s 256us/step - loss: 0.2793 - acc: 1.0000\n",
            "Epoch 66/100\n",
            "4/4 [==============================] - 0s 322us/step - loss: 0.2772 - acc: 1.0000\n",
            "Epoch 67/100\n",
            "4/4 [==============================] - 0s 294us/step - loss: 0.2751 - acc: 1.0000\n",
            "Epoch 68/100\n",
            "4/4 [==============================] - 0s 318us/step - loss: 0.2730 - acc: 1.0000\n",
            "Epoch 69/100\n",
            "4/4 [==============================] - 0s 352us/step - loss: 0.2710 - acc: 1.0000\n",
            "Epoch 70/100\n",
            "4/4 [==============================] - 0s 417us/step - loss: 0.2690 - acc: 1.0000\n",
            "Epoch 71/100\n",
            "4/4 [==============================] - 0s 291us/step - loss: 0.2670 - acc: 1.0000\n",
            "Epoch 72/100\n",
            "4/4 [==============================] - 0s 330us/step - loss: 0.2651 - acc: 1.0000\n",
            "Epoch 73/100\n",
            "4/4 [==============================] - 0s 306us/step - loss: 0.2632 - acc: 1.0000\n",
            "Epoch 74/100\n",
            "4/4 [==============================] - 0s 304us/step - loss: 0.2613 - acc: 1.0000\n",
            "Epoch 75/100\n",
            "4/4 [==============================] - 0s 327us/step - loss: 0.2595 - acc: 1.0000\n",
            "Epoch 76/100\n",
            "4/4 [==============================] - 0s 272us/step - loss: 0.2576 - acc: 1.0000\n",
            "Epoch 77/100\n",
            "4/4 [==============================] - 0s 270us/step - loss: 0.2559 - acc: 1.0000\n",
            "Epoch 78/100\n",
            "4/4 [==============================] - 0s 261us/step - loss: 0.2541 - acc: 1.0000\n",
            "Epoch 79/100\n",
            "4/4 [==============================] - 0s 267us/step - loss: 0.2524 - acc: 1.0000\n",
            "Epoch 80/100\n",
            "4/4 [==============================] - 0s 320us/step - loss: 0.2507 - acc: 1.0000\n",
            "Epoch 81/100\n",
            "4/4 [==============================] - 0s 347us/step - loss: 0.2490 - acc: 1.0000\n",
            "Epoch 82/100\n",
            "4/4 [==============================] - 0s 281us/step - loss: 0.2473 - acc: 1.0000\n",
            "Epoch 83/100\n",
            "4/4 [==============================] - 0s 281us/step - loss: 0.2457 - acc: 1.0000\n",
            "Epoch 84/100\n",
            "4/4 [==============================] - 0s 411us/step - loss: 0.2441 - acc: 1.0000\n",
            "Epoch 85/100\n",
            "4/4 [==============================] - 0s 419us/step - loss: 0.2425 - acc: 1.0000\n",
            "Epoch 86/100\n",
            "4/4 [==============================] - 0s 394us/step - loss: 0.2409 - acc: 1.0000\n",
            "Epoch 87/100\n",
            "4/4 [==============================] - 0s 242us/step - loss: 0.2394 - acc: 1.0000\n",
            "Epoch 88/100\n",
            "4/4 [==============================] - 0s 519us/step - loss: 0.2379 - acc: 1.0000\n",
            "Epoch 89/100\n",
            "4/4 [==============================] - 0s 256us/step - loss: 0.2364 - acc: 1.0000\n",
            "Epoch 90/100\n",
            "4/4 [==============================] - 0s 237us/step - loss: 0.2349 - acc: 1.0000\n",
            "Epoch 91/100\n",
            "4/4 [==============================] - 0s 293us/step - loss: 0.2334 - acc: 1.0000\n",
            "Epoch 92/100\n",
            "4/4 [==============================] - 0s 296us/step - loss: 0.2320 - acc: 1.0000\n",
            "Epoch 93/100\n",
            "4/4 [==============================] - 0s 309us/step - loss: 0.2306 - acc: 1.0000\n",
            "Epoch 94/100\n",
            "4/4 [==============================] - 0s 266us/step - loss: 0.2292 - acc: 1.0000\n",
            "Epoch 95/100\n",
            "4/4 [==============================] - 0s 220us/step - loss: 0.2278 - acc: 1.0000\n",
            "Epoch 96/100\n",
            "4/4 [==============================] - 0s 229us/step - loss: 0.2264 - acc: 1.0000\n",
            "Epoch 97/100\n",
            "4/4 [==============================] - 0s 235us/step - loss: 0.2251 - acc: 1.0000\n",
            "Epoch 98/100\n",
            "4/4 [==============================] - 0s 255us/step - loss: 0.2238 - acc: 1.0000\n",
            "Epoch 99/100\n",
            "4/4 [==============================] - 0s 214us/step - loss: 0.2224 - acc: 1.0000\n",
            "Epoch 100/100\n",
            "4/4 [==============================] - 0s 228us/step - loss: 0.2211 - acc: 1.0000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7faad2c70ef0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "metadata": {
        "id": "skQrTrapbkbL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "84716b8e-cc80-45e0-9003-83b9fd44dcdc"
      },
      "cell_type": "code",
      "source": [
        "model.get_weights()"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([[-2.03777  ],\n",
              "        [-2.1895845]], dtype=float32), array([3.432624], dtype=float32)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "metadata": {
        "id": "HUitJEpqPWke",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "3c9e4596-6e03-4841-bd52-afd23aa5b6bf"
      },
      "cell_type": "code",
      "source": [
        "model.predict(x_data)"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()\n",
            "  return _inspect.getargspec(target)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.96870875],\n",
              "       [0.77609265],\n",
              "       [0.80136603],\n",
              "       [0.31115386]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "metadata": {
        "id": "cFw4nCJbPmHr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### MXNet"
      ]
    },
    {
      "metadata": {
        "id": "Pf_u7JeePxAO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd, gluon\n",
        "\n",
        "# Training data for NAND.\n",
        "x = nd.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
        "y = nd.array([[1], [1], [1], [0]])\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "net = gluon.nn.Sequential()\n",
        "with net.name_scope():\n",
        "    net.add(gluon.nn.Dense(1))\n",
        "net.collect_params().initialize(mx.init.Normal(sigma=1.))\n",
        "  \n",
        "# Binary cross-entropy loss agter sigmoid function.\n",
        "loss_fn = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n",
        "\n",
        "# Optimizer based on SGD\n",
        "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.5})\n",
        "\n",
        "for t in range(100):\n",
        "    with autograd.record():\n",
        "        # Make predictions.\n",
        "        y_pred = net(x)\n",
        "        # Compute the loss.\n",
        "        loss = loss_fn(y_pred, y)\n",
        "    # Compute the gradients of the loss.\n",
        "    loss.backward()\n",
        "    # Update weights using SGD.\n",
        "    # the batch_size is set to one to be consistent with the slide.\n",
        "    trainer.step(batch_size=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ig5PjaWSQGjP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "1bf3e178-3b9e-4bb2-b0af-d55638da8600"
      },
      "cell_type": "code",
      "source": [
        "for v in net.collect_params().values():\n",
        "    print(v, v.data())"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Parameter sequential0_dense0_weight (shape=(1, 2), dtype=float32) \n",
            "[[-4.260149 -4.260375]]\n",
            "<NDArray 1x2 @cpu(0)>\n",
            "Parameter sequential0_dense0_bias (shape=(1,), dtype=float32) \n",
            "[6.582048]\n",
            "<NDArray 1 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qzjTXlAwQHWO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "1fd57cc6-032d-48da-d7f5-932487ce1705"
      },
      "cell_type": "code",
      "source": [
        "net(x).sigmoid()"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\n",
              "[[0.99861693]\n",
              " [0.9106561 ]\n",
              " [0.9106745 ]\n",
              " [0.12581538]]\n",
              "<NDArray 4x1 @cpu(0)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "metadata": {
        "id": "scIE8zZWdhLs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Multi-layer neural networks using an optimizer"
      ]
    },
    {
      "metadata": {
        "id": "_RKNkmZeQeqx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "J1BrWIxkcMfI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "35b94bc9-d8d6-419e-aceb-edca30b6d96c"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for XOR.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[0], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network using high-level modules.\n",
        "model = torch.nn.Sequential(\n",
        "    torch.nn.Linear(2, 2, bias=True),   # 2 dims (with bias) -> 2 dims\n",
        "    torch.nn.Sigmoid(),                 # Sigmoid function\n",
        "    torch.nn.Linear(2, 1, bias=True),   # 2 dims (with bias) -> 1 dim\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=0.5)\n",
        "\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)           # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)   # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    optimizer.zero_grad()       # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.step()            # Update the parameters using the gradients."
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "mp0sNnxhducs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "fe15165d-d758-43be-cb49-c36db3180c3c"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('0.weight', tensor([[ 5.9422, -6.2540],\n",
              "                      [-6.6625,  6.4892]])),\n",
              "             ('0.bias', tensor([-3.2577, -3.5896])),\n",
              "             ('2.weight', tensor([[10.4214, 10.3275]])),\n",
              "             ('2.bias', tensor([-5.0840]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "metadata": {
        "id": "AYgEteqydwt2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "c9f453f7-56d6-4959-c23c-960140716cf1"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.0119],\n",
              "        [0.9910],\n",
              "        [0.9907],\n",
              "        [0.0103]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "metadata": {
        "id": "4840LhzGQh99",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "Yg_9TF-AQkPv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import chainer\n",
        "import numpy as np\n",
        "from chainer import functions as F\n",
        "from chainer import links as L\n",
        "\n",
        "dtype=np.float32\n",
        "\n",
        "# Training data for XOR.\n",
        "x = chainer.Variable(np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype))\n",
        "y = chainer.Variable(np.array([[0], [1], [1], [0]], dtype=np.int32))\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "model = chainer.Sequential(\n",
        "    L.Linear(2, 2, nobias=False),\n",
        "    F.sigmoid,\n",
        "    L.Linear(2, 1, nobias=False),\n",
        ")\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn=F.sigmoid_cross_entropy\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = chainer.optimizers.SGD(lr=0.5)\n",
        "optimizer.setup(model)\n",
        "\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)                   # Make predictions.\n",
        "    loss = loss_fn(y_pred, y, normalize=False)  # Compute the loss.\n",
        "    #print(t, loss.data)\n",
        "    \n",
        "    model.cleargrads()          # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.update()          # Update the parameters using the gradients."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "njBneY_uRMmL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "e18dcbf0-0af9-4a13-cfed-de5a812e4e83"
      },
      "cell_type": "code",
      "source": [
        "for para in model.params():\n",
        "    print(para)"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "variable W([[ 1.7916094   0.20070258]\n",
            "            [-5.864411   -5.7855535 ]])\n",
            "variable b([-0.447787  1.010492])\n",
            "variable W([[-1.5884778 -5.9605026]])\n",
            "variable b([1.7095171])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EEk0i40yRVQi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "1307b631-05ca-4b49-a9f7-78611c2a7bd3"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.03627935],\n",
              "          [0.7237286 ],\n",
              "          [0.5995398 ],\n",
              "          [0.5987538 ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "metadata": {
        "id": "4E1ojAV5RgQg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TensorFlow (Keras)"
      ]
    },
    {
      "metadata": {
        "id": "B146BmixRfC7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "7aba9096-bc25-4084-822b-9cab4d138655"
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Flatten, Dense, Activation\n",
        "from tensorflow.nn import sigmoid_cross_entropy_with_logits\n",
        "from scipy.special import expit\n",
        "\n",
        "# Training data for XOR.\n",
        "x_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
        "y_data = np.array([[0], [1], [1], [0]])\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "model = Sequential([\n",
        "    Flatten(),\n",
        "    Dense(2, activation='sigmoid'),    # 2 dims (with bias) -> 2 dims\n",
        "    Dense(1, activation='sigmoid')     # 2 dims (with bias) -> 2 dims\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "    optimizer=optimizers.SGD(lr=0.5),\n",
        "    loss='binary_crossentropy',\n",
        "    metrics=['accuracy']\n",
        "    )\n",
        "\n",
        "model.fit(x_data, y_data, epochs=1000, verbose=0)"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()\n",
            "  return _inspect.getargspec(target)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7faad6c94cc0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "metadata": {
        "id": "oU4jxzPTdxnZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "ce9f677d-1c66-48ad-a383-0dea7dc07c90"
      },
      "cell_type": "code",
      "source": [
        "model.get_weights()"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([[5.910432 , 3.7894218],\n",
              "        [5.8785963, 3.7844138]], dtype=float32),\n",
              " array([-2.432777 , -5.7740874], dtype=float32),\n",
              " array([[ 7.7418504],\n",
              "        [-8.216927 ]], dtype=float32),\n",
              " array([-3.507641], dtype=float32)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "metadata": {
        "id": "UD-o4WqRR6-X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "89211d53-5505-4e9d-8c71-d2f132a98d11"
      },
      "cell_type": "code",
      "source": [
        "model.predict(x_data)"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()\n",
            "  return _inspect.getargspec(target)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.0517463 ],\n",
              "       [0.9528718 ],\n",
              "       [0.95300215],\n",
              "       [0.05638292]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "metadata": {
        "id": "6cW9oj0QSswT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### MXNet"
      ]
    },
    {
      "metadata": {
        "id": "yeakEHomSr4E",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd, gluon\n",
        "\n",
        "# Training data for XOR.\n",
        "x = nd.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])\n",
        "y = nd.array([[0], [1], [1], [0]])\n",
        "\n",
        "# Define a neural network using high-level modules.\n",
        "net = gluon.nn.Sequential()\n",
        "with net.name_scope():\n",
        "    net.add(gluon.nn.Dense(2))\n",
        "    net.add(gluon.nn.Activation('sigmoid'))\n",
        "    net.add(gluon.nn.Dense(1))\n",
        "net.collect_params().initialize(mx.init.Normal(sigma=1.))\n",
        "\n",
        "# Binary cross-entropy loss agter sigmoid function.\n",
        "loss_fn = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n",
        "\n",
        "# Optimizer based on SGD\n",
        "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.5})\n",
        "\n",
        "for t in range(1000):\n",
        "    with autograd.record():\n",
        "        # Make predictions.\n",
        "        y_pred = net(x)\n",
        "        # Compute the loss.\n",
        "        loss = loss_fn(y_pred, y)\n",
        "    # Compute the gradients of the loss.\n",
        "    loss.backward()\n",
        "    # Update weights using SGD.\n",
        "    # the batch_size is set to one to be consistent with the slide.\n",
        "    trainer.step(batch_size=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BaJOvCabS8XI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "3abd1658-a22b-4f84-bec1-b5224f1d5422"
      },
      "cell_type": "code",
      "source": [
        "for v in net.collect_params().values():\n",
        "    print(v, v.data())"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Parameter sequential1_dense0_weight (shape=(2, 3), dtype=float32) \n",
            "[[-6.3653517 -6.364059   5.089471 ]\n",
            " [ 7.591934   7.5853524 -1.585847 ]]\n",
            "<NDArray 2x3 @cpu(0)>\n",
            "Parameter sequential1_dense0_bias (shape=(2,), dtype=float32) \n",
            "[ 4.5148544 -1.9408077]\n",
            "<NDArray 2 @cpu(0)>\n",
            "Parameter sequential1_dense1_weight (shape=(1, 2), dtype=float32) \n",
            "[[9.933848 9.837673]]\n",
            "<NDArray 1x2 @cpu(0)>\n",
            "Parameter sequential1_dense1_bias (shape=(1,), dtype=float32) \n",
            "[-14.568179]\n",
            "<NDArray 1 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "IGhZkcopS9Rk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "019bfd82-801b-4497-80fe-0e969b40f403"
      },
      "cell_type": "code",
      "source": [
        "net(x).sigmoid()"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\n",
              "[[0.01269207]\n",
              " [0.99064106]\n",
              " [0.99064684]\n",
              " [0.0132224 ]]\n",
              "<NDArray 4x1 @cpu(0)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "metadata": {
        "id": "7x7Ed3SBGjPx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Single-layer neural network with a customizable NN class."
      ]
    },
    {
      "metadata": {
        "id": "klKcvlpVkpk9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "858b8965-dd7d-430d-e665-8d991174411c"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for NAND.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[1], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network model.\n",
        "class SingleLayerNN(torch.nn.Module):\n",
        "    def __init__(self, d_in, d_out):\n",
        "        super(SingleLayerNN, self).__init__()\n",
        "        self.linear1 = torch.nn.Linear(d_in, d_out, bias=True)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.linear1(x)\n",
        "\n",
        "model = SingleLayerNN(2, 1)\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=0.5)\n",
        "\n",
        "for t in range(100):\n",
        "    y_pred = model(x)           # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)   # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    optimizer.zero_grad()       # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.step()            # Update the parameters using the gradients."
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "ExxVfDnB5vPW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "a0e2c694-81ff-4397-b66d-02302493e597"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('linear1.weight', tensor([[-4.2283, -4.2277]])),\n",
              "             ('linear1.bias', tensor([6.5341]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "metadata": {
        "id": "xKczD7tZ518W",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "f8fcd54d-d190-4bfa-a375-92a3cc3edf4c"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9985],\n",
              "        [0.9094],\n",
              "        [0.9094],\n",
              "        [0.1276]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "metadata": {
        "id": "PvJRK0KOT-0V",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "noL4pXVLUAs8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from chainer import optimizers\n",
        "\n",
        "x = Variable(np.array([[0,0],[0,1],[1,0],[1,1]],dtype=np.float32))\n",
        "y = Variable(np.array([[1],[1],[1],[0]],dtype=np.int32))\n",
        "\n",
        "class Linear(chainer.Chain):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        with self.init_scope():\n",
        "            self.l1 = L.Linear(2,1)\n",
        "    def __call__(self,x):\n",
        "        return self.l1(x)\n",
        "\n",
        "model = Linear()\n",
        "\n",
        "optimizer = optimizers.SGD(lr=0.5).setup(model)\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)\n",
        "    loss = F.sigmoid_cross_entropy(y_pred,y)\n",
        "    #print(t,loss.data)\n",
        "    model.cleargrads()\n",
        "    loss.backward()\n",
        "    optimizer.update()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XsXwC56CULNw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "a587b6c8-52c6-43dd-a2d1-87593b2b8baf"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.9998992 ],\n",
              "          [0.96028996],\n",
              "          [0.9602896 ],\n",
              "          [0.0556649 ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "metadata": {
        "id": "jzTPZXUxGNso",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Multi-layer neural network with a customizable NN class.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "gSpf1qft53-O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "ac620421-87d0-437d-aac2-5a670dab7019"
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "dtype = torch.float\n",
        "\n",
        "# Training data for XOR.\n",
        "x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=dtype)\n",
        "y = torch.tensor([[0], [1], [1], [0]], dtype=dtype)\n",
        "                                        \n",
        "# Define a neural network model.\n",
        "class ThreeLayerNN(torch.nn.Module):\n",
        "    def __init__(self, d_in, d_hidden, d_out):\n",
        "        super(ThreeLayerNN, self).__init__()\n",
        "        self.linear1 = torch.nn.Linear(d_in, d_hidden, bias=True)\n",
        "        self.linear2 = torch.nn.Linear(d_hidden, d_out, bias=True)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.linear2(self.linear1(x).sigmoid())\n",
        "\n",
        "model = ThreeLayerNN(2, 2, 1)\n",
        "\n",
        "# Binary corss-entropy loss after sigmoid function.\n",
        "loss_fn = torch.nn.BCEWithLogitsLoss(size_average=False)\n",
        "\n",
        "# Optimizer based on SGD (change \"SGD\" to \"Adam\" to use Adam)\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=0.5)\n",
        "\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)           # Make predictions.\n",
        "    loss = loss_fn(y_pred, y)   # Compute the loss.\n",
        "    #print(t, loss.item())\n",
        "    \n",
        "    optimizer.zero_grad()       # Zero-clear gradients.\n",
        "    loss.backward()             # Compute the gradients.\n",
        "    optimizer.step()            # Update the parameters using the gradients."
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "k6LLbCf0684G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "0cc97d20-23e5-4786-c40a-39ac6f11798e"
      },
      "cell_type": "code",
      "source": [
        "model.state_dict()"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('linear1.weight', tensor([[-6.7652,  6.8954],\n",
              "                      [-6.3203,  6.0169]])),\n",
              "             ('linear1.bias', tensor([ 3.4285, -3.1931])),\n",
              "             ('linear2.weight', tensor([[-10.6559,  11.2825]])),\n",
              "             ('linear2.bias', tensor([4.9741]))])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "metadata": {
        "id": "iPhm3J4R61S2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "ed907501-77b3-4512-da54-48ca0fad54da"
      },
      "cell_type": "code",
      "source": [
        "model(x).sigmoid()"
      ],
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.0074],\n",
              "        [0.9931],\n",
              "        [0.9901],\n",
              "        [0.0063]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "metadata": {
        "id": "kDzRNjSQT0ym",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Chainer"
      ]
    },
    {
      "metadata": {
        "id": "JTkBzZteTb7r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import chainer\n",
        "import numpy as np\n",
        "from chainer import functions as F\n",
        "from chainer import links as L\n",
        "from chainer import optimizers\n",
        "\n",
        "x = Variable(np.array([[0,0],[0,1],[1,0],[1,1]],dtype=np.float32))\n",
        "y = Variable(np.array([[0],[1],[1],[0]],dtype=np.int32))\n",
        "\n",
        "class Linear(chainer.Chain):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        with self.init_scope():\n",
        "            self.l1 = L.Linear(2,2)\n",
        "            self.l2 = L.Linear(2,1)\n",
        "      \n",
        "    def __call__(self,x):\n",
        "        h = F.sigmoid(self.l1(x))\n",
        "        o = self.l2(h)\n",
        "        return o\n",
        "    \n",
        "model = Linear()\n",
        "optimizer = optimizers.SGD(lr=0.5).setup(model)\n",
        "for t in range(1000):\n",
        "    y_pred = model(x)\n",
        "    loss = F.sigmoid_cross_entropy(y_pred,y)\n",
        "    #print(t,loss.data)\n",
        "    model.cleargrads()\n",
        "    loss.backward()\n",
        "    optimizer.update()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OeOa6mt5Fgw4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "05c3921f-1467-44d9-c400-638ab8c9dfea"
      },
      "cell_type": "code",
      "source": [
        "F.sigmoid(model(x))"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "variable([[0.5009184 ],\n",
              "          [0.49899408],\n",
              "          [0.5010106 ],\n",
              "          [0.49909088]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "metadata": {
        "id": "KwOfELX7UTmQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}