homework/03_semanticSimilarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 作业\n",
    "\n",
    "完成预测环节预训练模型的调用代码，并跑通整个项目，成功提交千言文本相似度竞赛，按要求截图，提交作业即可。\n",
    "\n",
    "tips：\n",
    "\n",
    "- 预测可以使用自己训练的模型（训练时间较长），也可以直接使用提供下载的模型权重；\n",
    "- 报名千言文本相似度竞赛，并成功提交结果；\n",
    "- 并将如下图所示的结果截图，贴到本项目作业最后一行即完成作业。\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/cf119d3bc6504c098cc3cc58597b7061890d5fe915364f5fbd52341033307c7c)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 基于预训练模型 ERNIE-Gram 实现语义匹配\n",
    "\n",
    "\n",
    "6.7NLP直播打卡课即将开播，欢迎大家关注课程，有任何问题来评论区或**QQ群**（群号:758287592）交流吧~~\n",
    "\n",
    "**[直播链接请戳这里，每晚20:00-21:30👈](http://live.bilibili.com/21689802)**\n",
    "\n",
    "**[课程地址请戳这里👈](https://aistudio.baidu.com/aistudio/course/introduce/24177)**\n",
    "\n",
    "\n",
    "本案例介绍 NLP 最基本的任务类型之一 —— 文本语义匹配，并且基于 PaddleNLP 使用百度开源的预训练模型 ERNIE-Gram 搭建效果优异的语义匹配模型，来判断 2 段文本语义是否相同。\n",
    "\n",
    "## 1. 背景介绍\n",
    "文本语义匹配任务，简单来说就是给定两段文本，让模型来判断两段文本是不是语义相似。\n",
    "\n",
    "在本案例中以权威的语义匹配数据集 [LCQMC](http://icrc.hitsz.edu.cn/Article/show/171.html) 为例，[LCQMC](http://icrc.hitsz.edu.cn/Article/show/171.html) 数据集是基于百度知道相似问题推荐构造的通问句语义匹配数据集。训练集中的每两段文本都会被标记为 1（语义相似） 或者 0（语义不相似）。更多数据集可访问[千言](https://www.luge.ai/)获取哦。\n",
    "\n",
    "例如百度知道场景下，用户搜索一个问题，模型会计算这个问题与候选问题是否语义相似，语义匹配模型会找出与问题语义相似的候选问题返回给用户，加快用户提问-获取答案的效率。例如，当某用户在搜索引擎中搜索 “深度学习的教材有哪些？”，模型就自动找到了一些语义相似的问题展现给用户:\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/ecc1244685ec4476b869ce8a32d421c0ad530666e98d487da21fa4f61670544f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 2.快速实践\n",
    "\n",
    "介绍如何准备数据，基于 ERNIE-Gram 模型搭建匹配网络，然后快速进行语义匹配模型的训练、评估和预测。\n",
    "\n",
    "### 2.1 数据加载\n",
    "为了训练匹配模型，一般需要准备三个数据集：训练集 train.tsv、验证集dev.tsv、测试集test.tsv。此案例我们使用 PaddleNLP 内置的语义数据集 [LCQMC](http://icrc.hitsz.edu.cn/Article/show/171.html) 来进行训练、评估、预测。\n",
    "\n",
    "训练集: 用来训练模型参数的数据集，模型直接根据训练集来调整自身参数以获得更好的分类效果。\n",
    "\n",
    "验证集: 用于在训练过程中检验模型的状态，收敛情况。验证集通常用于调整超参数，根据几组模型验证集上的表现，决定采用哪组超参数。\n",
    "\n",
    "测试集: 用来计算模型的各项评估指标，验证模型泛化能力。\n",
    "\n",
    "[LCQMC](http://icrc.hitsz.edu.cn/Article/show/171.html) 数据集是公开的语义匹配权威数据集。PaddleNLP 已经内置该数据集，一键即可加载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting paddlenlp\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b1/e9/128dfc1371db3fc2fa883d8ef27ab6b21e3876e76750a43f58cf3c24e707/paddlenlp-2.0.2-py3-none-any.whl (426kB)\n",
      "\u001b[K     |████████████████████████████████| 430kB 901kB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: jieba in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.42.1)\n",
      "Requirement already satisfied, skipping upgrade: colorlog in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (4.1.0)\n",
      "Requirement already satisfied, skipping upgrade: colorama in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.4.4)\n",
      "Requirement already satisfied, skipping upgrade: seqeval in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (1.2.2)\n",
      "Requirement already satisfied, skipping upgrade: h5py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (2.9.0)\n",
      "Requirement already satisfied, skipping upgrade: visualdl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (2.1.1)\n",
      "Requirement already satisfied, skipping upgrade: multiprocess in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.70.11.1)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.14.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from seqeval->paddlenlp) (1.16.4)\n",
      "Requirement already satisfied, skipping upgrade: scikit-learn>=0.21.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from seqeval->paddlenlp) (0.22.1)\n",
      "Requirement already satisfied, skipping upgrade: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from h5py->paddlenlp) (1.15.0)\n",
      "Requirement already satisfied, skipping upgrade: Flask-Babel>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.0.0)\n",
      "Requirement already satisfied, skipping upgrade: requests in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (2.22.0)\n",
      "Requirement already satisfied, skipping upgrade: flake8>=3.7.9 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (3.8.2)\n",
      "Requirement already satisfied, skipping upgrade: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (0.8.53)\n",
      "Requirement already satisfied, skipping upgrade: protobuf>=3.11.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (3.14.0)\n",
      "Requirement already satisfied, skipping upgrade: Pillow>=7.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (7.1.2)\n",
      "Requirement already satisfied, skipping upgrade: shellcheck-py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (0.7.1.1)\n",
      "Requirement already satisfied, skipping upgrade: flask>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.1.1)\n",
      "Requirement already satisfied, skipping upgrade: pre-commit in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.21.0)\n",
      "Requirement already satisfied, skipping upgrade: dill>=0.3.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from multiprocess->paddlenlp) (0.3.3)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.17.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (0.14.1)\n",
      "Requirement already satisfied, skipping upgrade: pytz in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2019.3)\n",
      "Requirement already satisfied, skipping upgrade: Jinja2>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2.10.3)\n",
      "Requirement already satisfied, skipping upgrade: Babel>=2.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2.8.0)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->paddlenlp) (2019.9.11)\n",
      "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->paddlenlp) (2.8)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->paddlenlp) (1.25.6)\n",
      "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->paddlenlp) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: mccabe<0.7.0,>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlenlp) (0.6.1)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < \"3.8\" in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlenlp) (0.23)\n",
      "Requirement already satisfied, skipping upgrade: pycodestyle<2.7.0,>=2.6.0a1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlenlp) (2.6.0)\n",
      "Requirement already satisfied, skipping upgrade: pyflakes<2.3.0,>=2.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlenlp) (2.2.0)\n",
      "Requirement already satisfied, skipping upgrade: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlenlp) (3.9.9)\n",
      "Requirement already satisfied, skipping upgrade: future>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlenlp) (0.18.0)\n",
      "Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: click>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (7.0)\n",
      "Requirement already satisfied, skipping upgrade: nodeenv>=0.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (1.3.4)\n",
      "Requirement already satisfied, skipping upgrade: cfgv>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (2.0.1)\n",
      "Requirement already satisfied, skipping upgrade: virtualenv>=15.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (16.7.9)\n",
      "Requirement already satisfied, skipping upgrade: identify>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (1.4.10)\n",
      "Requirement already satisfied, skipping upgrade: aspy.yaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: toml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (0.10.0)\n",
      "Requirement already satisfied, skipping upgrade: pyyaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->paddlenlp) (5.1.2)\n",
      "Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Jinja2>=2.5->Flask-Babel>=1.0.0->visualdl->paddlenlp) (1.1.1)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from importlib-metadata; python_version < \"3.8\"->flake8>=3.7.9->visualdl->paddlenlp) (0.6.0)\n",
      "Requirement already satisfied, skipping upgrade: more-itertools in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata; python_version < \"3.8\"->flake8>=3.7.9->visualdl->paddlenlp) (7.2.0)\n",
      "Installing collected packages: paddlenlp\n",
      "  Found existing installation: paddlenlp 2.0.1\n",
      "    Uninstalling paddlenlp-2.0.1:\n",
      "      Successfully uninstalled paddlenlp-2.0.1\n",
      "Successfully installed paddlenlp-2.0.2\n"
     ]
    }
   ],
   "source": [
    "# 正式开始实验之前首先通过如下命令安装最新版本的 paddlenlp\n",
    "!pip install --upgrade paddlenlp -i https://pypi.org/simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 6827/6827 [00:00<00:00, 56365.16it/s]\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "import paddle\n",
    "import paddle.nn.functional as F\n",
    "from paddlenlp.datasets import load_dataset\n",
    "import paddlenlp\n",
    "\n",
    "# 一键加载 Lcqmc 的训练集、验证集\n",
    "train_ds, dev_ds = load_dataset(\"lcqmc\", splits=[\"train\", \"dev\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'query': '喜欢打篮球的男生喜欢什么样的女生', 'title': '爱打篮球的男生喜欢什么样的女生', 'label': 1}\n",
      "{'query': '我手机丢了，我想换个手机', 'title': '我想买个新手机，求推荐', 'label': 1}\n",
      "{'query': '大家觉得她好看吗', 'title': '大家觉得跑男好看吗？', 'label': 0}\n",
      "{'query': '求秋色之空漫画全集', 'title': '求秋色之空全集漫画', 'label': 1}\n"
     ]
    }
   ],
   "source": [
    "# 输出训练集的前 3 条样本\n",
    "for idx, example in enumerate(train_ds):\n",
    "    if idx <= 3:\n",
    "        print(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2.2 数据预处理\n",
    "\n",
    "通过 PaddleNLP 加载进来的 [LCQMC](http://icrc.hitsz.edu.cn/Article/show/171.html) 数据集是原始的明文数据集，这部分我们来实现组 batch、tokenize 等预处理逻辑，将原始明文数据转换成网络训练的输入数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 定义样本转换函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2021-06-10 03:46:12,265] [    INFO] - Downloading vocab.txt from https://paddlenlp.bj.bcebos.com/models/transformers/ernie_gram_zh/vocab.txt\n",
      "100%|██████████| 78/78 [00:00<00:00, 4273.92it/s]\n"
     ]
    }
   ],
   "source": [
    "# 因为是基于预训练模型 ERNIE-Gram 来进行，所以需要首先加载 ERNIE-Gram 的 tokenizer，\n",
    "# 后续样本转换函数基于 tokenizer 对文本进行切分\n",
    "\n",
    "tokenizer = paddlenlp.transformers.ErnieGramTokenizer.from_pretrained('ernie-gram-zh')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 将 1 条明文数据的 query、title 拼接起来，根据预训练模型的 tokenizer 将明文转换为 ID 数据\n",
    "# 返回 input_ids 和 token_type_ids\n",
    "\n",
    "def convert_example(example, tokenizer, max_seq_length=512, is_test=False):\n",
    "\n",
    "    query, title = example[\"query\"], example[\"title\"]\n",
    "\n",
    "    encoded_inputs = tokenizer(\n",
    "        text=query, text_pair=title, max_seq_len=max_seq_length)\n",
    "\n",
    "    input_ids = encoded_inputs[\"input_ids\"]\n",
    "    token_type_ids = encoded_inputs[\"token_type_ids\"]\n",
    "\n",
    "    if not is_test:\n",
    "        label = np.array([example[\"label\"]], dtype=\"int64\")\n",
    "        return input_ids, token_type_ids, label\n",
    "    # 在预测或者评估阶段，不返回 label 字段\n",
    "    else:\n",
    "        return input_ids, token_type_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### 对训练集的第 1 条数据进行转换\n",
    "input_ids, token_type_ids, label = convert_example(train_ds[0], tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'query': '喜欢打篮球的男生喜欢什么样的女生', 'title': '爱打篮球的男生喜欢什么样的女生', 'label': 1}\n"
     ]
    }
   ],
   "source": [
    "print(train_ds[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 692, 811, 445, 2001, 497, 5, 654, 21, 692, 811, 614, 356, 314, 5, 291, 21, 2, 329, 445, 2001, 497, 5, 654, 21, 692, 811, 614, 356, 314, 5, 291, 21, 2]\n"
     ]
    }
   ],
   "source": [
    "\r\n",
    "print(input_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "print(token_type_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\n"
     ]
    }
   ],
   "source": [
    "print(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 为了后续方便使用，我们使用python偏函数（partial）给 convert_example 赋予一些默认参数\n",
    "from functools import partial\n",
    "\n",
    "# 训练集和验证集的样本转换函数\n",
    "trans_func = partial(\n",
    "    convert_example,\n",
    "    tokenizer=tokenizer,\n",
    "    max_seq_length=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 组装 Batch 数据 & Padding\n",
    "\n",
    "上一小节，我们完成了对单条样本的转换，本节我们需要将样本组合成 Batch 数据，对于不等长的数据还需要进行 Padding 操作，便于 GPU 训练。\n",
    "\n",
    "PaddleNLP 提供了许多关于 NLP 任务中构建有效的数据 pipeline 的常用 API\n",
    "\n",
    "| API                             | 简介                                       |\n",
    "| ------------------------------- | :----------------------------------------- |\n",
    "| `paddlenlp.data.Stack`          | 堆叠N个具有相同shape的输入数据来构建一个batch |\n",
    "| `paddlenlp.data.Pad`            | 将长度不同的多个句子padding到统一长度，取N个输入数据中的最大长度 |\n",
    "| `paddlenlp.data.Tuple`          | 将多个batchify函数包装在一起 |\n",
    "\n",
    "更多数据处理操作详见： [https://paddlenlp.readthedocs.io/zh/latest/data_prepare/data_preprocess.html](https://paddlenlp.readthedocs.io/zh/latest/data_prepare/data_preprocess.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stacked Data: \n",
      " [[1 2 3 4]\n",
      " [3 4 5 6]\n",
      " [5 6 7 8]]\n",
      "\n",
      "Padded Data: \n",
      " [[1 2 3 4]\n",
      " [5 6 7 0]\n",
      " [8 9 0 0]]\n",
      "\n",
      "ids: \n",
      " [[1 2 3 4]\n",
      " [5 6 7 0]\n",
      " [8 9 0 0]]\n",
      "\n",
      "labels: \n",
      " [[1]\n",
      " [0]\n",
      " [1]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from paddlenlp.data import Stack, Pad, Tuple\n",
    "a = [1, 2, 3, 4]\n",
    "b = [3, 4, 5, 6]\n",
    "c = [5, 6, 7, 8]\n",
    "result = Stack()([a, b, c])\n",
    "print(\"Stacked Data: \\n\", result)\n",
    "print()\n",
    "\n",
    "a = [1, 2, 3, 4]\n",
    "b = [5, 6, 7]\n",
    "c = [8, 9]\n",
    "result = Pad(pad_val=0)([a, b, c])\n",
    "print(\"Padded Data: \\n\", result)\n",
    "print()\n",
    "\n",
    "data = [\n",
    "        [[1, 2, 3, 4], [1]],\n",
    "        [[5, 6, 7], [0]],\n",
    "        [[8, 9], [1]],\n",
    "       ]\n",
    "batchify_fn = Tuple(Pad(pad_val=0), Stack())\n",
    "ids, labels = batchify_fn(data)\n",
    "print(\"ids: \\n\", ids)\n",
    "print()\n",
    "print(\"labels: \\n\", labels)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 我们的训练数据会返回 input_ids, token_type_ids, labels 3 个字段\n",
    "# 因此针对这 3 个字段需要分别定义 3 个组 batch 操作\n",
    "batchify_fn = lambda samples, fn=Tuple(\n",
    "    Pad(axis=0, pad_val=tokenizer.pad_token_id),  # input_ids\n",
    "    Pad(axis=0, pad_val=tokenizer.pad_token_type_id),  # token_type_ids\n",
    "    Stack(dtype=\"int64\")  # label\n",
    "): [data for data in fn(samples)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 定义 Dataloader\n",
    "下面我们基于组 batchify_fn 函数和样本转换函数 trans_func 来构造训练集的 DataLoader, 支持多卡训练\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "# 定义分布式 Sampler: 自动对训练数据进行切分，支持多卡并行训练\n",
    "batch_sampler = paddle.io.DistributedBatchSampler(train_ds, batch_size=32, shuffle=True)\n",
    "\n",
    "# 基于 train_ds 定义 train_data_loader\n",
    "# 因为我们使用了分布式的 DistributedBatchSampler, train_data_loader 会自动对训练数据进行切分\n",
    "train_data_loader = paddle.io.DataLoader(\n",
    "        dataset=train_ds.map(trans_func),\n",
    "        batch_sampler=batch_sampler,\n",
    "        collate_fn=batchify_fn,\n",
    "        return_list=True)\n",
    "\n",
    "# 针对验证集数据加载，我们使用单卡进行评估，所以采用 paddle.io.BatchSampler 即可\n",
    "# 定义 dev_data_loader\n",
    "batch_sampler = paddle.io.BatchSampler(dev_ds, batch_size=32, shuffle=False)\n",
    "dev_data_loader = paddle.io.DataLoader(\n",
    "        dataset=dev_ds.map(trans_func),\n",
    "        batch_sampler=batch_sampler,\n",
    "        collate_fn=batchify_fn,\n",
    "        return_list=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2.3 模型搭建\n",
    "\n",
    "自从 2018 年 10 月以来，NLP 个领域的任务都通过 Pretrain + Finetune 的模式相比传统 DNN 方法在效果上取得了显著的提升，本节我们以百度开源的预训练模型 ERNIE-Gram 为基础模型，在此之上构建 Point-wise 语义匹配网络。\n",
    "\n",
    "首先我们来定义网络结构:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2021-06-10 03:49:47,386] [    INFO] - Downloading https://paddlenlp.bj.bcebos.com/models/transformers/ernie_gram_zh/ernie_gram_zh.pdparams and saved to /home/aistudio/.paddlenlp/models/ernie-gram-zh\n",
      "[2021-06-10 03:49:47,395] [    INFO] - Downloading ernie_gram_zh.pdparams from https://paddlenlp.bj.bcebos.com/models/transformers/ernie_gram_zh/ernie_gram_zh.pdparams\n",
      "100%|██████████| 583566/583566 [00:11<00:00, 52222.93it/s]\n"
     ]
    }
   ],
   "source": [
    "import paddle.nn as nn\n",
    "\n",
    "# 我们基于 ERNIE-Gram 模型结构搭建 Point-wise 语义匹配网络\n",
    "# 所以此处先定义 ERNIE-Gram 的 pretrained_model\n",
    "pretrained_model = paddlenlp.transformers.ErnieGramModel.from_pretrained('ernie-gram-zh')\n",
    "#pretrained_model = paddlenlp.transformers.ErnieModel.from_pretrained('ernie-1.0')\n",
    "\n",
    "\n",
    "class PointwiseMatching(nn.Layer):\n",
    "   \n",
    "    # 此处的 pretained_model 在本例中会被 ERNIE-Gram 预训练模型初始化\n",
    "    def __init__(self, pretrained_model, dropout=None):\n",
    "        super().__init__()\n",
    "        self.ptm = pretrained_model\n",
    "        self.dropout = nn.Dropout(dropout if dropout is not None else 0.1)\n",
    "\n",
    "        # 语义匹配任务: 相似、不相似 2 分类任务\n",
    "        self.classifier = nn.Linear(self.ptm.config[\"hidden_size\"], 2)\n",
    "\n",
    "    def forward(self,\n",
    "                input_ids,\n",
    "                token_type_ids=None,\n",
    "                position_ids=None,\n",
    "                attention_mask=None):\n",
    "\n",
    "        # 此处的 Input_ids 由两条文本的 token ids 拼接而成\n",
    "        # token_type_ids 表示两段文本的类型编码\n",
    "        # 返回的 cls_embedding 就表示这两段文本经过模型的计算之后而得到的语义表示向量\n",
    "        _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,\n",
    "                                    attention_mask)\n",
    "\n",
    "        cls_embedding = self.dropout(cls_embedding)\n",
    "\n",
    "        # 基于文本对的语义表示向量进行 2 分类任务\n",
    "        logits = self.classifier(cls_embedding)\n",
    "        probs = F.softmax(logits)\n",
    "\n",
    "        return probs\n",
    "\n",
    "# 定义 Point-wise 语义匹配网络\n",
    "model = PointwiseMatching(pretrained_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2.4 模型训练 & 评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from paddlenlp.transformers import LinearDecayWithWarmup\n",
    "\n",
    "epochs = 3\n",
    "num_training_steps = len(train_data_loader) * epochs\n",
    "\n",
    "# 定义 learning_rate_scheduler，负责在训练过程中对 lr 进行调度\n",
    "lr_scheduler = LinearDecayWithWarmup(5E-5, num_training_steps, 0.0)\n",
    "\n",
    "# Generate parameter names needed to perform weight decay.\n",
    "# All bias and LayerNorm parameters are excluded.\n",
    "decay_params = [\n",
    "    p.name for n, p in model.named_parameters()\n",
    "    if not any(nd in n for nd in [\"bias\", \"norm\"])\n",
    "]\n",
    "\n",
    "# 定义 Optimizer\n",
    "optimizer = paddle.optimizer.AdamW(\n",
    "    learning_rate=lr_scheduler,\n",
    "    parameters=model.parameters(),\n",
    "    weight_decay=0.0,\n",
    "    apply_decay_param_fun=lambda x: x in decay_params)\n",
    "\n",
    "# 采用交叉熵 损失函数\n",
    "criterion = paddle.nn.loss.CrossEntropyLoss()\n",
    "\n",
    "# 评估的时候采用准确率指标\n",
    "metric = paddle.metric.Accuracy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 因为训练过程中同时要在验证集进行模型评估，因此我们先定义评估函数\n",
    "\n",
    "@paddle.no_grad()\n",
    "def evaluate(model, criterion, metric, data_loader, phase=\"dev\"):\n",
    "    model.eval()\n",
    "    metric.reset()\n",
    "    losses = []\n",
    "    for batch in data_loader:\n",
    "        input_ids, token_type_ids, labels = batch\n",
    "        probs = model(input_ids=input_ids, token_type_ids=token_type_ids)\n",
    "        loss = criterion(probs, labels)\n",
    "        losses.append(loss.numpy())\n",
    "        correct = metric.compute(probs, labels)\n",
    "        metric.update(correct)\n",
    "        accu = metric.accumulate()\n",
    "    print(\"eval {} loss: {:.5}, accu: {:.5}\".format(phase,\n",
    "                                                    np.mean(losses), accu))\n",
    "    model.train()\n",
    "    metric.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 接下来，开始正式训练模型，训练时间较长，可注释掉这部分\n",
    "\n",
    "global_step = 0\n",
    "tic_train = time.time()\n",
    "\n",
    "for epoch in range(1, epochs + 1):\n",
    "    for step, batch in enumerate(train_data_loader, start=1):\n",
    "\n",
    "        input_ids, token_type_ids, labels = batch\n",
    "        probs = model(input_ids=input_ids, token_type_ids=token_type_ids)\n",
    "        loss = criterion(probs, labels)\n",
    "        correct = metric.compute(probs, labels)\n",
    "        metric.update(correct)\n",
    "        acc = metric.accumulate()\n",
    "\n",
    "        global_step += 1\n",
    "        \n",
    "        # 每间隔 10 step 输出训练指标\n",
    "        if global_step % 10 == 0:\n",
    "            print(\n",
    "                \"global step %d, epoch: %d, batch: %d, loss: %.5f, accu: %.5f, speed: %.2f step/s\"\n",
    "                % (global_step, epoch, step, loss, acc,\n",
    "                    10 / (time.time() - tic_train)))\n",
    "            tic_train = time.time()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        lr_scheduler.step()\n",
    "        optimizer.clear_grad()\n",
    "\n",
    "        # 每间隔 100 step 在验证集和测试集上进行评估\n",
    "        if global_step % 100 == 0:\n",
    "            evaluate(model, criterion, metric, dev_data_loader, \"dev\")\n",
    "            \n",
    "# 训练结束后，存储模型参数\n",
    "save_dir = os.path.join(\"checkpoint\", \"model_%d\" % global_step)\n",
    "os.makedirs(save_dir)\n",
    "\n",
    "save_param_path = os.path.join(save_dir, 'model_state.pdparams')\n",
    "paddle.save(model.state_dict(), save_param_path)\n",
    "tokenizer.save_pretrained(save_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "模型训练过程中会输出如下日志:\n",
    "```\n",
    "global step 5310, epoch: 3, batch: 1578, loss: 0.31671, accu: 0.95000, speed: 0.63 step/s\n",
    "global step 5320, epoch: 3, batch: 1588, loss: 0.36240, accu: 0.94063, speed: 6.98 step/s\n",
    "global step 5330, epoch: 3, batch: 1598, loss: 0.41451, accu: 0.93854, speed: 7.40 step/s\n",
    "global step 5340, epoch: 3, batch: 1608, loss: 0.31327, accu: 0.94063, speed: 7.01 step/s\n",
    "global step 5350, epoch: 3, batch: 1618, loss: 0.40664, accu: 0.93563, speed: 7.83 step/s\n",
    "global step 5360, epoch: 3, batch: 1628, loss: 0.33064, accu: 0.93958, speed: 7.34 step/s\n",
    "global step 5370, epoch: 3, batch: 1638, loss: 0.38411, accu: 0.93795, speed: 7.72 step/s\n",
    "global step 5380, epoch: 3, batch: 1648, loss: 0.35376, accu: 0.93906, speed: 7.92 step/s\n",
    "global step 5390, epoch: 3, batch: 1658, loss: 0.39706, accu: 0.93924, speed: 7.47 step/s\n",
    "global step 5400, epoch: 3, batch: 1668, loss: 0.41198, accu: 0.93781, speed: 7.41 step/s\n",
    "eval dev loss: 0.4177, accu: 0.89082\n",
    "global step 5410, epoch: 3, batch: 1678, loss: 0.34453, accu: 0.93125, speed: 0.63 step/s\n",
    "global step 5420, epoch: 3, batch: 1688, loss: 0.34569, accu: 0.93906, speed: 7.75 step/s\n",
    "global step 5430, epoch: 3, batch: 1698, loss: 0.39160, accu: 0.92917, speed: 7.54 step/s\n",
    "global step 5440, epoch: 3, batch: 1708, loss: 0.46002, accu: 0.93125, speed: 7.05 step/s\n",
    "global step 5450, epoch: 3, batch: 1718, loss: 0.32302, accu: 0.93188, speed: 7.14 step/s\n",
    "global step 5460, epoch: 3, batch: 1728, loss: 0.40802, accu: 0.93281, speed: 7.22 step/s\n",
    "global step 5470, epoch: 3, batch: 1738, loss: 0.34607, accu: 0.93348, speed: 7.44 step/s\n",
    "global step 5480, epoch: 3, batch: 1748, loss: 0.34709, accu: 0.93398, speed: 7.38 step/s\n",
    "global step 5490, epoch: 3, batch: 1758, loss: 0.31814, accu: 0.93437, speed: 7.39 step/s\n",
    "global step 5500, epoch: 3, batch: 1768, loss: 0.42689, accu: 0.93125, speed: 7.74 step/s\n",
    "eval dev loss: 0.41789, accu: 0.88968\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "基于默认参数配置进行单卡训练大概要持续 4 个小时左右，会训练完成 3 个 Epoch, 模型最终的收敛指标结果如下:\n",
    "\n",
    "\n",
    "| 数据集 | Accuracy |\n",
    "| -------- | -------- |\n",
    "| dev.tsv     | 89.62  |\n",
    "\n",
    "可以看到: 我们基于 PaddleNLP ，利用 ERNIE-Gram 预训练模型使用非常简洁的代码，就在权威语义匹配数据集上取得了很不错的效果."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2.5 模型预测\n",
    "\n",
    "接下来我们使用已经训练好的语义匹配模型对一些预测数据进行预测。待预测数据为每行都是文本对的 tsv 文件，我们使用 Lcqmc 数据集的测试集作为我们的预测数据，进行预测并提交预测结果到 [千言文本相似度竞赛](https://aistudio.baidu.com/aistudio/competition/detail/45)\n",
    "\n",
    "下载我们已经训练好的语义匹配模型, 并解压"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-06-10 03:51:34--  https://paddlenlp.bj.bcebos.com/models/text_matching/ernie_gram_zh_pointwise_matching_model.tar\n",
      "Resolving paddlenlp.bj.bcebos.com (paddlenlp.bj.bcebos.com)... 182.61.200.229, 182.61.200.195, 2409:8c00:6c21:10ad:0:ff:b00e:67d\n",
      "Connecting to paddlenlp.bj.bcebos.com (paddlenlp.bj.bcebos.com)|182.61.200.229|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 597667840 (570M) [application/x-tar]\n",
      "Saving to: ‘ernie_gram_zh_pointwise_matching_model.tar.1’\n",
      "\n",
      "ernie_gram_zh_point 100%[===================>] 569.98M  88.3MB/s    in 7.8s    \n",
      "\n",
      "2021-06-10 03:51:42 (73.4 MB/s) - ‘ernie_gram_zh_pointwise_matching_model.tar.1’ saved [597667840/597667840]\n",
      "\n",
      "ernie_gram_zh_pointwise_matching_model/\n",
      "ernie_gram_zh_pointwise_matching_model/model_state.pdparams\n",
      "ernie_gram_zh_pointwise_matching_model/vocab.txt\n",
      "ernie_gram_zh_pointwise_matching_model/tokenizer_config.json\n"
     ]
    }
   ],
   "source": [
    "# 下载我们基于 Lcqmc 事先训练好的语义匹配模型并解压\n",
    "! wget https://paddlenlp.bj.bcebos.com/models/text_matching/ernie_gram_zh_pointwise_matching_model.tar\n",
    "! tar -xvf ernie_gram_zh_pointwise_matching_model.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "谁有狂三这张高清的\t这张高清图，谁有\r\n",
      "英雄联盟什么英雄最好\t英雄联盟最好英雄是什么\r\n",
      "这是什么意思，被蹭网吗\t我也是醉了，这是什么意思\r\n"
     ]
    }
   ],
   "source": [
    "# 测试数据由 2 列文本构成 tab 分隔\n",
    "# Lcqmc 默认下载到如下路径\n",
    "! head -n3 \"${HOME}/.paddlenlp/datasets/LCQMC/lcqmc/lcqmc/test.tsv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 定义预测函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "def predict(model, data_loader):\n",
    "    \n",
    "    batch_probs = []\n",
    "\n",
    "    # 预测阶段打开 eval 模式，模型中的 dropout 等操作会关掉\n",
    "    model.eval()\n",
    "\n",
    "    with paddle.no_grad():\n",
    "        for batch_data in data_loader:\n",
    "            input_ids, token_type_ids = batch_data\n",
    "            input_ids = paddle.to_tensor(input_ids)\n",
    "            token_type_ids = paddle.to_tensor(token_type_ids)\n",
    "            \n",
    "            # 获取每个样本的预测概率: [batch_size, 2] 的矩阵\n",
    "            batch_prob = model(\n",
    "                input_ids=input_ids, token_type_ids=token_type_ids).numpy()\n",
    "\n",
    "            batch_probs.append(batch_prob)\n",
    "        batch_probs = np.concatenate(batch_probs, axis=0)\n",
    "\n",
    "        return batch_probs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 定义预测数据的 data_loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 预测数据的转换函数\n",
    "# predict 数据没有 label, 因此 convert_exmaple 的 is_test 参数设为 True\n",
    "trans_func = partial(\n",
    "    convert_example,\n",
    "    tokenizer=tokenizer,\n",
    "    max_seq_length=512,\n",
    "    is_test=True)\n",
    "\n",
    "# 预测数据的组 batch 操作\n",
    "# predict 数据只返回 input_ids 和 token_type_ids，因此只需要 2 个 Pad 对象作为 batchify_fn\n",
    "batchify_fn = lambda samples, fn=Tuple(\n",
    "    Pad(axis=0, pad_val=tokenizer.pad_token_id),  # input_ids\n",
    "    Pad(axis=0, pad_val=tokenizer.pad_token_type_id),  # segment_ids\n",
    "): [data for data in fn(samples)]\n",
    "\n",
    "# 加载预测数据\n",
    "test_ds = load_dataset(\"lcqmc\", splits=[\"test\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_sampler = paddle.io.BatchSampler(test_ds, batch_size=32, shuffle=False)\n",
    "\n",
    "# 生成预测数据 data_loader\n",
    "predict_data_loader =paddle.io.DataLoader(\n",
    "        dataset=test_ds.map(trans_func),\n",
    "        batch_sampler=batch_sampler,\n",
    "        collate_fn=batchify_fn,\n",
    "        return_list=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 定义预测模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2021-06-10 03:54:17,435] [    INFO] - Already cached /home/aistudio/.paddlenlp/models/ernie-gram-zh/ernie_gram_zh.pdparams\n"
     ]
    }
   ],
   "source": [
    "# 选择预训练ernie gram，填写自己的代码\n",
    "pretrained_model = paddlenlp.transformers.ErnieGramModel.from_pretrained('ernie-gram-zh')\n",
    "\n",
    "model = PointwiseMatching(pretrained_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 加载已训练好的模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 刚才下载的模型解压之后存储路径为 ./ernie_gram_zh_pointwise_matching_model/model_state.pdparams\n",
    "state_dict = paddle.load(\"./ernie_gram_zh_pointwise_matching_model/model_state.pdparams\")\n",
    "\n",
    "# 刚才下载的模型解压之后存储路径为 ./pointwise_matching_model/ernie1.0_base_pointwise_matching.pdparams\n",
    "# state_dict = paddle.load(\"pointwise_matching_model/ernie1.0_base_pointwise_matching.pdparams\")\n",
    "model.set_dict(state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 开始预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Tensor(shape=[32, 38], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[1   , 1022, 9   , ..., 0   , 0   , 0   ],\n",
      "        [1   , 514 , 904 , ..., 0   , 0   , 0   ],\n",
      "        [1   , 47  , 10  , ..., 0   , 0   , 0   ],\n",
      "        ...,\n",
      "        [1   , 733 , 404 , ..., 0   , 0   , 0   ],\n",
      "        [1   , 134 , 170 , ..., 0   , 0   , 0   ],\n",
      "        [1   , 379 , 3122, ..., 0   , 0   , 0   ]]), Tensor(shape=[32, 38], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[0, 0, 0, ..., 0, 0, 0],\n",
      "        [0, 0, 0, ..., 0, 0, 0],\n",
      "        [0, 0, 0, ..., 0, 0, 0],\n",
      "        ...,\n",
      "        [0, 0, 0, ..., 0, 0, 0],\n",
      "        [0, 0, 0, ..., 0, 0, 0],\n",
      "        [0, 0, 0, ..., 0, 0, 0]])]\n"
     ]
    }
   ],
   "source": [
    "for idx, batch in enumerate(predict_data_loader):\n",
    "    if idx < 1:\n",
    "        print(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 执行预测函数\n",
    "y_probs = predict(model, predict_data_loader)\n",
    "\n",
    "# 根据预测概率获取预测 label\n",
    "y_preds = np.argmax(y_probs, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 输出预测结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'query': '高中数学,数列问题', 'title': '高中数学，数列问题', 'label': 1}"
     ]
    }
   ],
   "source": [
    "# 我们按照千言文本相似度竞赛的提交格式将预测结果存储在 lcqmc.tsv 中，用来后续提交\n",
    "# 同时将预测结果输出到终端，便于大家直观感受模型预测效果\n",
    "\n",
    "test_ds = load_dataset(\"lcqmc\", splits=[\"test\"])\n",
    "\n",
    "with open(\"lcqmc.tsv\", 'w', encoding=\"utf-8\") as f:\n",
    "    f.write(\"index\\tprediction\\n\")    \n",
    "    for idx, y_pred in enumerate(y_preds):\n",
    "        f.write(\"{}\\t{}\\n\".format(idx, y_pred))\n",
    "        text_pair = test_ds[idx]\n",
    "        text_pair[\"label\"] = y_pred\n",
    "        print(text_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 提交 LCQMC 预测结果[千言文本相似度竞赛](https://aistudio.baidu.com/aistudio/competition/detail/45)\n",
    "\n",
    "千言文本相似度竞赛一共有 3 个数据集: lcqmc、bq_corpus、paws-x, 我们刚才生成了 lcqmc 的预测结果 lcqmc.tsv, 同时我们在项目内提供了 bq_corpus、paw-x 数据集的空预测结果，我们将这 3 个文件打包提交到千言文本相似度竞赛，即可看到自己的模型在 Lcqmc 数据集上的竞赛成绩。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  adding: lcqmc.tsv (deflated 65%)\r\n",
      "  adding: paws-x.tsv (deflated 61%)\r\n",
      "  adding: bq_corpus.tsv (deflated 62%)\r\n"
     ]
    }
   ],
   "source": [
    "# 打包预测结果\n",
    "!zip submit.zip lcqmc.tsv paws-x.tsv bq_corpus.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "##### 提交预测结果 submit.zip 到 [千言文本相似度竞赛](https://aistudio.baidu.com/aistudio/competition/detail/45)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 千言文本相似度竞赛结果截图\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/5f58463e537a404a932ac3c964ae7b2880c29cc98b88421a8e9ab89161ce8ed9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "- 作业GitHub地址：[https://github.com/michellexxz/Baidu_AI_Studio-NLP_DL-Note/blob/master/homework/03_semanticSimilarity.ipynb](https://github.com/michellexxz/Baidu_AI_Studio-NLP_DL-Note/blob/master/homework/03_semanticSimilarity.ipynb)\n",
    "- requirement截图\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/6a859cb4068c4b67991d4cda5e033ed5414809a328594526bcd353090a203315)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.1.0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}