PyTorch_beginner/60分钟入门PyTorch-5.数据并行.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 60分钟入门深度学习工具-PyTorch(五、数据并行(选读))\n",
    "**作者**：Soumith Chintala\n",
    "\n",
    "\n",
    "原文翻译自：https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n",
    "    \n",
    "中文翻译、注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：PyTorch 1.0，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](images/gongzhong.jpg)\n",
    "### 目录\n",
    "* 1.[Pytorch是什么？](60分钟入门PyTorch-1.PyTorch是什么？.ipynb)\n",
    "* 2.[AUTOGRAD](60分钟入门PyTorch-2.AUTOGRAD.ipynb)\n",
    "* 3.[神经网络](60分钟入门PyTorch-3.神经网络.ipynb)\n",
    "* 4.[训练一个分类器](60分钟入门PyTorch-4.训练一个分类器.ipynb)\n",
    "* 5.[数据并行](60分钟入门PyTorch-5.数据并行.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 五、数据并行(选读)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**作者**:Sung Kim和Jenny Kang\n",
    "\n",
    "在这个教程里,我们将学习如何使用数据并行(DataParallel)来使用多GPU。\n",
    "\n",
    "PyTorch非常容易的就可以使用GPU,你可以用如下方式把一个模型放到GPU上:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`device = torch.device(\"cuda:0\")`\n",
    "\n",
    "`model.to(device)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后你可以复制所有的张量到GPU上:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`mytensor = my_tensor.to(device)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请注意,只调用`mytensor.gpu()`并没有复制张量到GPU上。你需要把它赋值给一个新的张量并在GPU上使用这个张量。\n",
    "\n",
    "在多GPU上执行前向和反向传播是自然而然的事。然而，PyTorch默认将只是用一个GPU。你可以使用`DataParallel`让模型并行运行来轻易的让你的操作在多个GPU上运行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`model = nn.DataParallel(model)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是这篇教程背后的核心，我们接下来将更详细的介绍它。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入和参数\n",
    "导入PyTorch模块和定义参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "# Parameters and DataLoaders\n",
    "input_size = 5\n",
    "output_size = 2\n",
    "\n",
    "batch_size = 30\n",
    "data_size = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设备："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 虚拟数据集\n",
    "制作一个虚拟（随机）数据集，你只需实现`__getitem__`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class RandomDataset(Dataset):\n",
    "\n",
    "    def __init__(self, size, length):\n",
    "        self.len = length\n",
    "        self.data = torch.randn(length, size)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.data[index]\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.len\n",
    "\n",
    "rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),\n",
    "                         batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简单模型\n",
    "作为演示，我们的模型只接受一个输入，执行一个线性操作，然后得到结果。然而，你能在任何模型（CNN，RNN，Capsule Net等）上使用`DataParallel`。\n",
    "\n",
    "我们在模型内部放置了一条打印语句来检测输入和输出向量的大小。请注意批等级为0时打印的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    # Our model\n",
    "\n",
    "    def __init__(self, input_size, output_size):\n",
    "        super(Model, self).__init__()\n",
    "        self.fc = nn.Linear(input_size, output_size)\n",
    "\n",
    "    def forward(self, input):\n",
    "        output = self.fc(input)\n",
    "        print(\"\\tIn Model: input size\", input.size(),\n",
    "              \"output size\", output.size())\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建一个模型和数据并行\n",
    "这是本教程的核心部分。首先，我们需要创建一个模型实例和检测我们是否有多个GPU。如果我们有多个GPU，我们使用`nn.DataParallel`来包装我们的模型。然后通过`model.to(device)`把模型放到GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "  (fc): Linear(in_features=5, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Model(input_size, output_size)\n",
    "if torch.cuda.device_count() > 1:\n",
    "  print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n",
    "  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs\n",
    "  model = nn.DataParallel(model)\n",
    "\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 运行模型\n",
    "现在我们可以看输入和输出张量的大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n",
      "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
      "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n",
      "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
      "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n",
      "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
      "\tIn Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
      "Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n"
     ]
    }
   ],
   "source": [
    "for data in rand_loader:\n",
    "    input = data.to(device)\n",
    "    output = model(input)\n",
    "    print(\"Outside: input size\", input.size(),\n",
    "          \"output_size\", output.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结果\n",
    "当我们对30个输入和输出进行批处理时，我们和期望的一样得到30个输入和30个输出，但是如果你有多个GPU，你得到如下的结果。\n",
    "\n",
    "## 2个GPU\n",
    "\n",
    "如果你有2个GPU，你将看到："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# on 2 GPUs\n",
    "Let's use 2 GPUs!\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n",
    "    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n",
    "Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3个GPU：\n",
    "如果你有3个GPU，你将看到："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Let's use 3 GPUs!\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8个GPU：\n",
    "如果你有8个GPU，你将看到："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Let's use 8 GPUs!\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n",
    "Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "`DataParallel`自动的划分数据，并将作业发送到多个GPU上的多个模型。在每个模型完成作业后，`DataParallel`收集并合并结果返回给你。\n",
    "\n",
    "更多信息请看这里：\n",
    "\n",
    "http://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本章的官方代码：\n",
    "* Python：[data_parallel_tutorial.py](download/data_parallel_tutorial.py)\n",
    "* Jupyter notebook:[data_parallel_tutorial.ipynb](download/data_parallel_tutorial.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}