Lesson8单层神经网络.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson8 单层神经网络\n",
    "\n",
    "## 一、单层神经网络：线性回归\n",
    "\n",
    "### 1.单层回归网络的理论基础\n",
    "\n",
    "在深度学习中，我们用$z$表示线性回归的结果。考虑$m$个样本的线性回归为$\\hat{z}=b+w_1x_1,+...+w_nx_n$，矩阵表示如下:\n",
    "$$\\hat{z}=Xw$$\n",
    "\n",
    "- 单层神经网络\n",
    "  - 在描述神经网络的层数的时候，我们不考虑输入层。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.tensor实现单层神经网络的正向传播\n",
    "\n",
    "- `torch.Tensor()` 无论输入数据的类型，都生成的是浮点型数据\n",
    "- `torch.tensor()` 根据输入的类型，来确定数据类型\n",
    "  \n",
    "\n",
    "**tensor计算中的坑**\n",
    "- PyTorch的静态性\n",
    "  - torch中要求分类标签必须是整形，不能是浮点型\n",
    "  - torch有许多函数要求真实标签的类型必须与预测值的类型一致\n",
    "  - PyTorch中许多函数不接受一维张量但同时也有许多函数不接受二维标签\n",
    "- 精度问题\n",
    "  - float32只要保留32位，存在精度问题\n",
    "  - torch.mv()在计算时，也容易出现精度问题\n",
    "  - 精度问题在tensor维度很高，数字很大时，也会变得更大\n",
    "  - PyTorch设置了64位浮点数来尽量减轻精度问题，但是很占内存计算很慢\n",
    "  - `allclose(x,x_1)` 忽视非常小的区别，判断是否相等\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0.],\n",
       "        [1., 1., 0.],\n",
       "        [1., 0., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————单层回归神经网络的简单例子————————————————————————————\n",
    "import torch\n",
    "\n",
    "X = torch.tensor([[1,0,0],[1,1,0],[1,0,1],[1,1,1]],dtype=torch.float32)  # 要记得定义标签类型，最安全的就是 先写floa32\n",
    "z = torch.tensor([-0.2,-0.05,-0.05,0.1])\n",
    "w = torch.tensor([-0.2,0.15,0.15])\n",
    "\n",
    "x # 第一列全是1 用与拟合参数b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2000,  0.0000,  0.0000],\n",
       "        [-0.2000,  0.1500,  0.0000],\n",
       "        [-0.2000,  0.0000,  0.1500],\n",
       "        [-0.2000,  0.1500,  0.1500]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X*w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LinearR(X,w):\n",
    "    zhat = torch.mv(X,w)  # mv(矩阵，向量)\n",
    "    return zhat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.2000, -0.0500, -0.0500,  0.1000])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zhat = LinearR(X,w)\n",
    "zhat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ True, False, False, False])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zhat == z # zhat是一维的，z是二维的\n",
    "\n",
    "z = z.view([4])\n",
    "z == zhat  \n",
    "\n",
    "# 为什么会出现不等于？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(8.3267e-17)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算SSE，zhat与z相等则SSE应该完全为0\n",
    "SSE = sum((zhat-z) ** 2)\n",
    "SSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.200000002980232238769531250000, -0.049999997019767761230468750000,\n",
       "        -0.049999997019767761230468750000,  0.100000008940696716308593750000])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#设置显示精度，再来看yhat与y_reg\n",
    "torch.set_printoptions(precision=30)  # 查看小数点后30位的情况\n",
    "zhat  \n",
    "\n",
    "# zhat 和 z 有精度的差异"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([300, 68, 64, 64])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 精度的问题的例子\n",
    "\n",
    "preds = torch.ones(300,68,64,64)\n",
    "preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "83558400"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = 300*68*64*64\n",
    "a   # 8kw多个数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(83558328.)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_ = torch.ones(300,68,64,64) * 0.1\n",
    "preds_.sum() *10\n",
    "\n",
    "# 83558400 与 tensor(83558328.) 精度出现了问题\n",
    "# PyTorch设置了64位浮点数来尽量减轻精度问题\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(83558400.000000059604644775390625000000, dtype=torch.float64)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#与Python中存在Decimal库不同，PyTorch设置了64位浮点数来尽量减轻精度问题\n",
    "preds__ = torch.ones(300, 68, 64, 64, dtype=torch.float64) * 0.1\n",
    "preds__.sum() * 10\n",
    "\n",
    "\n",
    "# float64 很占内存，计算速度很慢，有精度要求时使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#如果你希望能够无视掉非常小的区别，而让两个张量的比较结果展示为True，你可以使用下面的代码：\n",
    "torch.allclose(zhat,z)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.torch.nn.Linear实现单层回归神经网络的正向传播\n",
    "\n",
    "- `torch.nn.Linear`类来实现单层回归神经网络\n",
    "  - in_features: int 输入的特征数\n",
    "  - out_features: int 输出的特征数\n",
    "  - bias: bool = True 截距\n",
    "- 属性`.weight` 查看生成的w\n",
    "- 属性`.bias` 查看生成的截距\n",
    "\n",
    "- 人为设置随机数种子 保证每次运行的参数都相同torch.random.manual_seed(任意整数)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# 刷新前面设置的30位小数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0.],\n",
       "        [1., 0.],\n",
       "        [0., 1.],\n",
       "        [1., 1.]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.tensor([[0,0],[1,0],[0,1],[1,1]],dtype=torch.float32)\n",
    "X\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#————————————————————————————实例化nn.Linear————————————————————————————\n",
    "\n",
    "# 人为设置随机数种子 保证每次运行的参数都相同\n",
    "# 随机数种子要和nn.Linear在同一个cell中\n",
    "torch.random.manual_seed(420)\n",
    "\n",
    "\n",
    "output = torch.nn.Linear(2,1,bias=True)\n",
    "\n",
    "# 单层网络 也就输入层和输出层 输入层加权和的结果就是输出\n",
    "# 上一层的神经元个数（不需要考虑x0 只需要考虑真正的特征）\n",
    "# 这一层的神经元的个数\n",
    "# bias 控制是否生成截图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.431836068630218505859375000000, -0.425621807575225830078125000000]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.weight  # 查看生成的w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([0.672958195209503173828125000000], requires_grad=True)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.bias   #  查看截距"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.095281660556793212890625000000],\n",
       "        [ 0.224004089832305908203125000000],\n",
       "        [-0.516246497631072998046875000000],\n",
       "        [-0.387524068355560302734375000000]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zhat = output(X)\n",
    "zhat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 由于不需要定义常量b，因此在特征张量中，也**不需要留出与常数项相乘的x0**那一列。在输入数据时，我们只输入了两个特征x1与x2\n",
    "- 输入层只有一层，并且输入层的结构（神经元的个数）由输入的特征张量 决定，因此在PyTorch中构筑神经网络时，不需要定义输入层\n",
    "- 实例化之后，将特征张量输入到实例化后的类中，即可得出输出层的输出结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、二分类神经网络：逻辑回归\n",
    "\n",
    "### 1.二分类神经网络的理论基础\n",
    "\n",
    "对线性回归进行一定的变换，称为广义线性回归，例如等式两边同时取对数的对数函数回归、同时取指数的S形函数回归等。\n",
    "- **Sigmoid函数**$Sigmoid(z)=\\frac{1}{1+e^-z}=\\sigma$\n",
    "  - 对线性回归的结果$z$进行变换\n",
    "  - 有将连续性变量转化为离散型变量，化回归算法为分类算法\n",
    "\n",
    "\n",
    "- **逻辑回归(对数几率回归)**\n",
    "  1. 将结果$\\sigma$以几率($\\frac{\\sigma}{1-\\sigma}$)的形式展现\n",
    "  2. 在几率上秋$e$为底的对数\n",
    "   $\\ln{\\frac{\\sigma}{1-\\sigma}}=\\ln{e^{Xw}}={Xw}$\n",
    "  - 虽然叫逻辑回归，但常用于分类问题 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.tensor实现二分类神经网络的正向传播\n",
    "\n",
    "- 以与门为例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#————————————————————————————以与门为例————————————————————————————\n",
    "X = torch.tensor([[1,0,0],[1,1,0],[1,0,1],[1,1,1]],dtype=torch.float32)\n",
    "andgate = torch.tensor([[0],[0],[0],[1]],dtype=torch.float32)\n",
    "w = torch.tensor([-0.2,0.15,0.15], dtype = torch.float32)\n",
    "\n",
    "# 保险起见，生成二维的、float32类型的标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#————————————————————————————定义逻辑回归的函数————————————————————————————\n",
    "def LogisticR(X,w):\n",
    "    zhat = torch.mv(X,w)\n",
    "    sigma = 1/(1+torch.exp(-zhat))\n",
    "    # 设置阈值为0.5 大于0.5 为1 \n",
    "    andhat = torch.tensor([int(x) for x in sigma >= 0.5],dtype=torch.float32)\n",
    "    return sigma,andhat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.450166016817092895507812500000, 0.487502634525299072265625000000,\n",
       "        0.487502634525299072265625000000, 0.524979174137115478515625000000])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma, andhat = LogisticR(X,w)\n",
    "sigma  # sigma 可以看做事件发生的概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 1.])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " andhat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.符号函数sign, ReLU, Tanh\n",
    "\n",
    "这些函数是最常见的二分类转化函数，他们在神经网络的结构中有着不可替代的作用。符号和双曲正切函数都从torch中调用\n",
    "\n",
    "- **符号函数(阶跃函数)**\n",
    "$$y=\\begin{cases}\n",
    "-1,if \\quad x< 0 \\\\\\\\\n",
    "0,if\\quad x= 0 \\\\\\\\\n",
    "1,if \\quad x>0\n",
    "\\end{cases}$$\n",
    "\n",
    "- **ReLU(整流线型单元函数)**  \n",
    "  - ReLU函数现在神经网络领域中的宠儿，应用甚至比sigmoid更广泛。\n",
    "  - 当输入的自变量大于0时，直接输出该值，当输入的自变量小于等于0时，输出0。本质就是$\\max(0,z)$\n",
    "  - 从nn.functional中调用\n",
    "$$ReLU:\\sigma=\\begin{cases}\n",
    "0,if \\quad z\\leq 0 \\\\\\\\\n",
    "z,if\\quad z> 0 \n",
    "\\end{cases}$$\n",
    "\n",
    "- **tanh(双曲正切函数)**\n",
    "  - 双曲正切函数的性质与sigmoid相似，它能够将数值压缩到(-1,1)区间内。\n",
    "  $$tanh:\\sigma=\\frac{e^{2z}-1}{e^{2z}+1}$$\n",
    "\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.torch.functional实现单层二分类神经网络的正向传播\n",
    "\n",
    "逻辑回归与线性回归的唯一区别，就是在线性回归的结果之后套上了sigmoid函数。\n",
    "- 从`nn.functional`中调用相关函数\n",
    "  - 调取的都是函数 不是类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.nn import functional as F\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x28b7e034b30>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.tensor([[0,0],[1,0],[0,1],[1,1]], dtype = torch.float32)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 1, 1, 1]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.random.manual_seed(420) #人为设置随机数种子  必须和nn.Linear() 放在一个cell里\n",
    "dense = torch.nn.Linear(2,1)  # 上一层神经元的个数和下一层神经元的个数\n",
    "zhat = dense(X)\n",
    "sigma =  F.sigmoid(zhat)  # 调用nn.functionald中的sigmoid函数\n",
    "y = [int(x) for x in sigma>=0.5]\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.],\n",
       "        [1.],\n",
       "        [1.],\n",
       "        [1.]], grad_fn=<SignBackward0>)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————ReLU、Tanh以及Sign函数————————————————————————————\n",
    "# 符号和双曲正切函数都从torch中调用\n",
    "\n",
    "torch.sign(zhat)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.672958195209503173828125000000],\n",
       "        [1.104794263839721679687500000000],\n",
       "        [0.247336387634277343750000000000],\n",
       "        [0.679172456264495849609375000000]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.relu(zhat)  # max(0,zhat)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.586922407150268554687500000000],\n",
       "        [0.802214503288269042968750000000],\n",
       "        [0.242413192987442016601562500000],\n",
       "        [0.590981125831604003906250000000]], grad_fn=<TanhBackward0>)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.tanh(zhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、多分类神经网络：Softmax回归\n",
    "\n",
    "### 1.认识softmax函数\n",
    "\n",
    "在多分类中，神经元的个数与标签类别的个数是一致的。样本的预测标签就是**所有输出的概率中最大的概率对应的标签类别**\n",
    "$$\\sigma_k=Softmax(z_k)=\\frac{e^{z_k}}{\\sum^K{e^z}}$$\n",
    "\n",
    "很容易可以看出，Softmax函数的分子是多分类状况下某一个标签类别的回归结果的指数函数，分母是多分类状况下所有标签类别的回归结果的指数函数之和，因此**Softmax函数的结果代表了样本的结果为类别$k$的概率**。\n",
    "\n",
    "\n",
    "### 2.PyTorch中的softmax函数\n",
    "\n",
    "- `torch.softmax(张量z，n)`\n",
    "  - 第一个参数是我们输入的用来进行计算的张量z，\n",
    "  - 第二次参数代表你希望运行softmax计算的维度的索引。\n",
    "    - 0就是由shape返回的第0维\n",
    "\n",
    "- softmax函数将多分类问题转换为概率，但是计算量十分巨大\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([nan, nan, nan])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————例子：计量算过大 出现溢出————————————————————————————\n",
    "#对于单一样本，假定一组巨大的z\n",
    "z = torch.tensor([1010,1000,990], dtype=torch.float32)\n",
    "\n",
    "# softmax函数的运算\n",
    "torch.exp(z) / torch.sum(torch.exp(z))  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([9.999545812606811523437500000000e-01,\n",
       "        4.539786823443137109279632568359e-05,\n",
       "        2.061059989344471432559657841921e-09])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————直接调用softmax函数————————————————————————————\n",
    "torch.softmax(z,0)\n",
    "\n",
    "# 利用函数不容易溢出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.727475106716156005859375000000, 0.267623156309127807617187500000,\n",
       "        0.004901689011603593826293945312])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————调小z，查看两者区别————————————————————————————\n",
    "# 假设三个输出层神经元得到的z分别是10，9，5  \n",
    "# 调小z\n",
    "z = torch.tensor([10,9,5], dtype=torch.float32)\n",
    "torch.exp(z) / torch.sum(torch.exp(z)) # softmax函数的运算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.727475166320800781250000000000, 0.267623156309127807617187500000,\n",
       "        0.004901689011603593826293945312])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.softmax(z,0)\n",
    "\n",
    "# 两者结果一样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "softmax函数输出的是从0到1.0之间的实数，而且多个输出值的总和是1。因为有了这个性质，我们可以把softmax函数的输出解释为“概率”，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 2])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = torch.tensor([[[1,2],[3,4],[5,6]],[[5,6],[7,8],[9,10]]], dtype=torch.float32)\n",
    "s.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.shape[0]   #  前面softmax(z,0) 的0在z中就3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.练习：使用nn.Linear与functional实现多分类神经网络的正向传播\n",
    "\n",
    "你需要弄明白这些问题：\n",
    "1. nn.Linear中需要输入什么结构？\n",
    "2. softmax中的dim应该填写什么值？\n",
    "3. 最终输出的sigma的shape应该是几乘几才对？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.000000000000000000000000000000],\n",
       "        [0.077730834484100341796875000000],\n",
       "        [0.574015319347381591796875000000],\n",
       "        [0.651746153831481933593750000000]], grad_fn=<MmBackward0>)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————练习：我的答案————————————————————————————\n",
    "X = torch.tensor([[0,0],[1,0],[0,1],[1,1]], dtype = torch.float32)\n",
    "dense = torch.nn.Linear(2,1,bias=False)\n",
    "zhat = dense(X)\n",
    "sigma = torch.softmax(zhat,0)\n",
    "zhat  # 设置的nn.Linear结构不对，导致返回的zhat也没有可以求和的维度\n",
    "\n",
    "# 错误：\n",
    "# 1. 三分类问题 应该输出三个神经元 返回在3个分类上的概率\n",
    "# 2. 虽然特征矩阵中没有全为1的一列来拟合参数b，但是ANN中默认会拟合常数项，无须关闭bias\n",
    "# 3. softmax的维度错误 需要加和的维度是1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.462298959493637084960937500000, 0.349371731281280517578125000000,\n",
       "         0.188329339027404785156250000000],\n",
       "        [0.459773510694503784179687500000, 0.442208200693130493164062500000,\n",
       "         0.098018281161785125732421875000],\n",
       "        [0.489574432373046875000000000000, 0.322912752628326416015625000000,\n",
       "         0.187512844800949096679687500000],\n",
       "        [0.490227818489074707031250000000, 0.411511898040771484375000000000,\n",
       "         0.098260335624217987060546875000]], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#————————————————————————————正确答案————————————————————————————\n",
    "torch.random.manual_seed(420) \n",
    "dense = torch.nn.Linear(2,3) #此时，输出层上的神经元个数是3个，因此应该是（2，3）\n",
    "zhat = dense(X)\n",
    "sigma = F.softmax(zhat,dim=1) #此时需要进行加和的维度是1\n",
    "sigma\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 3])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zhat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 2])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape # 4个样本2个特征"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.4 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e9c95e2fca121e63b81ef3e38ca658e3e94770f8b66e1c1e59a7460ee39b328d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}