usage-classificator/BERT Studies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Read the annotated data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import annotation_data\n",
    "pd.set_option('display.max_rows', 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 451 positive and 451 negative labels = 902 total\n"
     ]
    }
   ],
   "source": [
    "df = annotation_data.load_annotation_data('annotation_data_train_out.csv')\n",
    "train_set, dev_set = annotation_data.get_train_test_data(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ner</th>\n",
       "      <th>sentence</th>\n",
       "      <th>pre_sentence</th>\n",
       "      <th>post_sentence</th>\n",
       "      <th>used</th>\n",
       "      <th>section_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>790</th>\n",
       "      <td>associative vector embeddings</td>\n",
       "      <td>Newell et al . extended their SHG idea by outp...</td>\n",
       "      <td>This model runs in realtime .</td>\n",
       "      <td>They group keypoints with similar tags into in...</td>\n",
       "      <td>0</td>\n",
       "      <td>Bottom - up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>749</th>\n",
       "      <td>HG</td>\n",
       "      <td>Hourglass blocks , ( HG ) developed by Newell ...</td>\n",
       "      <td>Wei et al . were inspired by the pose machines...</td>\n",
       "      <td>Newell et al . stacked HG blocks to obtain an ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Single Person Pose Estimation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>566</th>\n",
       "      <td>pooling operations</td>\n",
       "      <td>The most basic version of this approach would ...</td>\n",
       "      <td>These pooling layers remain distinct from conv...</td>\n",
       "      <td>One step further brings us to what we refer to...</td>\n",
       "      <td>0</td>\n",
       "      <td>Tree pooling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>514</th>\n",
       "      <td>letter - based representations</td>\n",
       "      <td>Finally , there is currently a lot of interest...</td>\n",
       "      <td>ratinov2009design quantitatively compare sever...</td>\n",
       "      <td>gillick2015multilingual model the task of sequ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Related Work</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>879</th>\n",
       "      <td>pLSA</td>\n",
       "      <td>[ reference ] propose an iterative technique t...</td>\n",
       "      <td>Wang et al .</td>\n",
       "      <td>Bilen et al .</td>\n",
       "      <td>0</td>\n",
       "      <td>Related Work</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>computationally intensive neural network</td>\n",
       "      <td>This process is not parallelizable , and , in ...</td>\n",
       "      <td>Both model families use autoregressive decoder...</td>\n",
       "      <td>While several recently proposed models avoid r...</td>\n",
       "      <td>0</td>\n",
       "      <td>Introduction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>648</th>\n",
       "      <td>GMMs</td>\n",
       "      <td>With this in mind , it is easy to generalize G...</td>\n",
       "      <td>⇡ iN x|bi , AiATi .</td>\n",
       "      <td>Instead of sampling one transformation from a ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>347</th>\n",
       "      <td>sequence - based LSTM</td>\n",
       "      <td>As noted in , the simple BOW model performs ro...</td>\n",
       "      <td>Other question representations , such as an LS...</td>\n",
       "      <td>Specifically , we compute where represents the...</td>\n",
       "      <td>1</td>\n",
       "      <td>Word Guided Spatial Attention in One - Hop Model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>FM )</td>\n",
       "      <td>[ reference ] and factorization machines ( FM ...</td>\n",
       "      <td>[ reference ] . From the modeling perspective ...</td>\n",
       "      <td>However , these models are limited in mining h...</td>\n",
       "      <td>0</td>\n",
       "      <td>II . RELATED WORK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>sibling 1×1 convolutional layers</td>\n",
       "      <td>This RPN ends with two sibling 1×1 convolution...</td>\n",
       "      <td>Unlike the way in [ reference ] that is catego...</td>\n",
       "      <td>The cls and reg layers are both in a per - cla...</td>\n",
       "      <td>1</td>\n",
       "      <td>C. ImageNet Localization</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>298 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          ner  \\\n",
       "790             associative vector embeddings   \n",
       "749                                        HG   \n",
       "566                        pooling operations   \n",
       "514            letter - based representations   \n",
       "879                                      pLSA   \n",
       "..                                        ...   \n",
       "106  computationally intensive neural network   \n",
       "648                                      GMMs   \n",
       "347                     sequence - based LSTM   \n",
       "506                                      FM )   \n",
       "155          sibling 1×1 convolutional layers   \n",
       "\n",
       "                                              sentence  \\\n",
       "790  Newell et al . extended their SHG idea by outp...   \n",
       "749  Hourglass blocks , ( HG ) developed by Newell ...   \n",
       "566  The most basic version of this approach would ...   \n",
       "514  Finally , there is currently a lot of interest...   \n",
       "879  [ reference ] propose an iterative technique t...   \n",
       "..                                                 ...   \n",
       "106  This process is not parallelizable , and , in ...   \n",
       "648  With this in mind , it is easy to generalize G...   \n",
       "347  As noted in , the simple BOW model performs ro...   \n",
       "506  [ reference ] and factorization machines ( FM ...   \n",
       "155  This RPN ends with two sibling 1×1 convolution...   \n",
       "\n",
       "                                          pre_sentence  \\\n",
       "790                      This model runs in realtime .   \n",
       "749  Wei et al . were inspired by the pose machines...   \n",
       "566  These pooling layers remain distinct from conv...   \n",
       "514  ratinov2009design quantitatively compare sever...   \n",
       "879                                       Wang et al .   \n",
       "..                                                 ...   \n",
       "106  Both model families use autoregressive decoder...   \n",
       "648                                ⇡ iN x|bi , AiATi .   \n",
       "347  Other question representations , such as an LS...   \n",
       "506  [ reference ] . From the modeling perspective ...   \n",
       "155  Unlike the way in [ reference ] that is catego...   \n",
       "\n",
       "                                         post_sentence used  \\\n",
       "790  They group keypoints with similar tags into in...    0   \n",
       "749  Newell et al . stacked HG blocks to obtain an ...    0   \n",
       "566  One step further brings us to what we refer to...    0   \n",
       "514  gillick2015multilingual model the task of sequ...    0   \n",
       "879                                      Bilen et al .    0   \n",
       "..                                                 ...  ...   \n",
       "106  While several recently proposed models avoid r...    0   \n",
       "648  Instead of sampling one transformation from a ...    0   \n",
       "347  Specifically , we compute where represents the...    1   \n",
       "506  However , these models are limited in mining h...    0   \n",
       "155  The cls and reg layers are both in a per - cla...    1   \n",
       "\n",
       "                                         section_name  \n",
       "790                                       Bottom - up  \n",
       "749                     Single Person Pose Estimation  \n",
       "566                                      Tree pooling  \n",
       "514                                      Related Work  \n",
       "879                                      Related Work  \n",
       "..                                                ...  \n",
       "106                                      Introduction  \n",
       "648                                             Title  \n",
       "347  Word Guided Spatial Attention in One - Hop Model  \n",
       "506                                 II . RELATED WORK  \n",
       "155                          C. ImageNet Localization  \n",
       "\n",
       "[298 rows x 6 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labels and preds come from Bert prediction on dev set\n",
    "preds_bert = [0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0\n",
    ", 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0\n",
    ", 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0\n",
    ", 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1\n",
    ", 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0\n",
    ", 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1\n",
    ", 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1\n",
    ", 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0\n",
    ", 0, 1]\n",
    "preds_cnn = [0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0\n",
    ", 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0\n",
    ", 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1\n",
    ", 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0\n",
    ", 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0\n",
    ", 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1\n",
    ", 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0\n",
    ", 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1\n",
    ", 0, 1]\n",
    "labels = [0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0\n",
    ", 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0\n",
    ", 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0\n",
    ", 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1\n",
    ", 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0\n",
    ", 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0\n",
    ", 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0\n",
    ", 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1\n",
    ", 0, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-24-4a649e8560ed>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  dev_set['used'] = dev_set['used'].astype(int)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if we have the same labels\n",
    "dev_set['used'] = dev_set['used'].astype(int)\n",
    "dev_set['used'].values.tolist() == labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-38-228ac77ac9d5>:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  dev_set['preds_bert'] = preds_bert\n",
      "<ipython-input-38-228ac77ac9d5>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  dev_set['preds_cnn'] = preds_cnn\n"
     ]
    }
   ],
   "source": [
    "dev_set['preds_bert'] = preds_bert\n",
    "dev_set['preds_cnn'] = preds_cnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ner</th>\n",
       "      <th>sentence</th>\n",
       "      <th>pre_sentence</th>\n",
       "      <th>post_sentence</th>\n",
       "      <th>used</th>\n",
       "      <th>section_name</th>\n",
       "      <th>preds_bert</th>\n",
       "      <th>preds_cnn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>566</th>\n",
       "      <td>pooling operations</td>\n",
       "      <td>The most basic version of this approach would ...</td>\n",
       "      <td>These pooling layers remain distinct from conv...</td>\n",
       "      <td>One step further brings us to what we refer to...</td>\n",
       "      <td>0</td>\n",
       "      <td>Tree pooling</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>558</th>\n",
       "      <td>explicit attention mechanism</td>\n",
       "      <td>Based on this , we argue that MRN is an implic...</td>\n",
       "      <td>Since MRN does not depend on a few attention p...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Qualitative Analysis</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td>fully - convolutional baselines</td>\n",
       "      <td>the two fully - convolutional baselines .</td>\n",
       "      <td>w.r.t .</td>\n",
       "      <td>Interestingly , the gap significantly widens w...</td>\n",
       "      <td>1</td>\n",
       "      <td>Evaluation for visual object tracking</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>365</th>\n",
       "      <td>scale expansion algorithm</td>\n",
       "      <td>The detail of scale expansion algorithm is sum...</td>\n",
       "      <td>Thanks to the “ progressive ” expansion proced...</td>\n",
       "      <td>In the pseudocode , are the intermediate resul...</td>\n",
       "      <td>0</td>\n",
       "      <td>Progressive Scale Expansion Algorithm</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>931</th>\n",
       "      <td>multi - scale and sliding pooling</td>\n",
       "      <td>We start from a baseline setting of our FeatMa...</td>\n",
       "      <td>We present the results of adding different com...</td>\n",
       "      <td>This baseline setting is the conventional full...</td>\n",
       "      <td>0</td>\n",
       "      <td>Component Evaluation</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>654</th>\n",
       "      <td>FastEval14k</td>\n",
       "      <td>FastEval14k consists of 14000 images with labe...</td>\n",
       "      <td>FastEval14k ’ .</td>\n",
       "      <td>Unlike labels in JFT - 300 M , the images in F...</td>\n",
       "      <td>0</td>\n",
       "      <td>Monitoring Training Progress</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786</th>\n",
       "      <td>coding schemes</td>\n",
       "      <td>The assignment of codes is more balanced for l...</td>\n",
       "      <td>” for the first component , while the subcode ...</td>\n",
       "      <td>In any coding scheme , even the most unpopular...</td>\n",
       "      <td>1</td>\n",
       "      <td>Analysis of Code Efficiency</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938</th>\n",
       "      <td>bidirectional LSTM model</td>\n",
       "      <td>These vectors are concatenated ( into a vector...</td>\n",
       "      <td>Each model consumes the words in the sentence ...</td>\n",
       "      <td>The convolutional model uses 600 total filters...</td>\n",
       "      <td>0</td>\n",
       "      <td>C. Information Extraction Details</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>716</th>\n",
       "      <td>HFA based methods</td>\n",
       "      <td>The major advancements of the proposed approac...</td>\n",
       "      <td>Specifically , given a feature , the HFA based...</td>\n",
       "      <td>Secondly , we explicitly project the identity ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Discussion</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>475</th>\n",
       "      <td>RASP</td>\n",
       "      <td>The required POS tags were generated with RASP...</td>\n",
       "      <td>[ reference ] .</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Pattern Extraction</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>CHFusion</td>\n",
       "      <td>subsubsection : Context - Aware Hierarchical F...</td>\n",
       "      <td>Wherein our method learns the weights automati...</td>\n",
       "      <td>The results of this experiment are shown in ta...</td>\n",
       "      <td>1</td>\n",
       "      <td>Context - Aware Hierarchical Fusion ( CHFusion )</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>Ranking model</td>\n",
       "      <td>KEYWORDS Ranking model , weak supervision , de...</td>\n",
       "      <td>Our findings also suggest that supervised neur...</td>\n",
       "      <td>compat=1.11 ternary compat</td>\n",
       "      <td>0</td>\n",
       "      <td>Neural Ranking Models with Weak Supervision</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>567</th>\n",
       "      <td>VIS + BOW</td>\n",
       "      <td>VIS + BOW performs multinomial logistic regres...</td>\n",
       "      <td>GUESS simply selects the modal answer from the...</td>\n",
       "      <td>VIS + LSTM has one LSTM to encode the image an...</td>\n",
       "      <td>0</td>\n",
       "      <td>Results on DAQURA</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>Evolution</td>\n",
       "      <td>Evolution has discovered efficient feedforward...</td>\n",
       "      <td>Once trained , the CNN never changes its weigh...</td>\n",
       "      <td>However , an expert ornithologist , asked to c...</td>\n",
       "      <td>1</td>\n",
       "      <td>Title</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>469</th>\n",
       "      <td>Quantitative Analysis</td>\n",
       "      <td>subsubsection : Quantitative Analysis</td>\n",
       "      <td>It manifests that our different - resolution i...</td>\n",
       "      <td>To further understand accuracy gain in each br...</td>\n",
       "      <td>1</td>\n",
       "      <td>Quantitative Analysis</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>question and answer representations</td>\n",
       "      <td>The Neural Answer Selection Model ( Figure [ r...</td>\n",
       "      <td>This is a classification task where we treat e...</td>\n",
       "      <td>It employs two different LSTMs to embed raw qu...</td>\n",
       "      <td>0</td>\n",
       "      <td>Neural Answer Selection Model</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>classification function</td>\n",
       "      <td>IIC is a generic clustering algorithm that dir...</td>\n",
       "      <td>In this paper , we introduce Invariant Informa...</td>\n",
       "      <td>It involves a simple objective function , whic...</td>\n",
       "      <td>1</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>uniform weights</td>\n",
       "      <td>Also , regardless of the embedding method , le...</td>\n",
       "      <td>This supports the hypothesis that with the emb...</td>\n",
       "      <td>Although weight learning can significantly aff...</td>\n",
       "      <td>1</td>\n",
       "      <td>Results and Discussion</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>812</th>\n",
       "      <td>masked self - attentional layers</td>\n",
       "      <td>We present graph attention networks ( GATs ) ,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>By stacking layers in which nodes are able to ...</td>\n",
       "      <td>0</td>\n",
       "      <td>ABSTRACT</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>Google NLP pipeline</td>\n",
       "      <td>The text has been run through a Google NLP pip...</td>\n",
       "      <td>A news article is usually associated with a fe...</td>\n",
       "      <td>It it tokenized , lowercased , and named entit...</td>\n",
       "      <td>0</td>\n",
       "      <td>The Reading Comprehension Task</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>867</th>\n",
       "      <td>joint update</td>\n",
       "      <td>We can write the joint update for all as Restr...</td>\n",
       "      <td>Let , and .</td>\n",
       "      <td>This means that there is some such that for an...</td>\n",
       "      <td>0</td>\n",
       "      <td>Contraction Map Example</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>892</th>\n",
       "      <td>left eye model</td>\n",
       "      <td>Moreover , for the right eye cluster , the rig...</td>\n",
       "      <td>Taking the left eye model as an example , it a...</td>\n",
       "      <td>It can be concluded that each shape prediction...</td>\n",
       "      <td>1</td>\n",
       "      <td>Analysis of Shape Prediction Layers</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737</th>\n",
       "      <td>fusion sub - components</td>\n",
       "      <td>Below , we go into the details of our point cl...</td>\n",
       "      <td>[ reference ] D ) , in which we use a dense sp...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>PointFusion</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>generic face alignment algorithm</td>\n",
       "      <td>[ reference ] that even as a generic face alig...</td>\n",
       "      <td>It can be seen in Tabel .</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Medium Pose Face Alignment</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>matrix of convolutional filters</td>\n",
       "      <td>The result is reshaped to generate a matrix of...</td>\n",
       "      <td>The hypernetwork is a fully connected layer ( ...</td>\n",
       "      <td>Whilst the overall dimensionality of the filte...</td>\n",
       "      <td>0</td>\n",
       "      <td>Scoring Function and Model Architecture</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>downsamplers</td>\n",
       "      <td>Existing literatures have considered two types...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In this paper , we consider the bicubic downsa...</td>\n",
       "      <td>1</td>\n",
       "      <td>Downsampler .</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>deeper model</td>\n",
       "      <td>The existence of this constructed solution ind...</td>\n",
       "      <td>There exists a solution by construction to the...</td>\n",
       "      <td>But experiments show that our current solvers ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>684</th>\n",
       "      <td>LSTM Q + I</td>\n",
       "      <td>[ reference ] , Q + I vs Question , and LSTM Q...</td>\n",
       "      <td>No , as shown in Table .</td>\n",
       "      <td>Our results demonstrate clearly the positive i...</td>\n",
       "      <td>1</td>\n",
       "      <td>Results and analysis</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>986</th>\n",
       "      <td>label maps</td>\n",
       "      <td>We demonstrate that this approach is effective...</td>\n",
       "      <td>This makes it possible to apply the same gener...</td>\n",
       "      <td>Indeed , since the release of the pix2pix soft...</td>\n",
       "      <td>0</td>\n",
       "      <td>Image - to - Image Translation with Conditiona...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>982</th>\n",
       "      <td>mixture 10 Gaussians</td>\n",
       "      <td>In contrast , the VAE exhibit systematic diffe...</td>\n",
       "      <td>The adversarial autoencoder successfully match...</td>\n",
       "      <td>An important difference between VAEs and adver...</td>\n",
       "      <td>0</td>\n",
       "      <td>Relationship to Variational Autoencoders</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>CUDA implementations</td>\n",
       "      <td>While it is possible to have a 10 speed up for...</td>\n",
       "      <td>Table [ reference ] shows that methods using c...</td>\n",
       "      <td>The GRNNs method of tang2015document takes aro...</td>\n",
       "      <td>0</td>\n",
       "      <td>Training time .</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>881</th>\n",
       "      <td>vanilla Transformers</td>\n",
       "      <td>Specifically , Transformer - XL significantly ...</td>\n",
       "      <td>Although Transformer - XL is mainly designed t...</td>\n",
       "      <td>We also report the results on word - level Pen...</td>\n",
       "      <td>0</td>\n",
       "      <td>Main Results</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>963</th>\n",
       "      <td>probability distribution strategies</td>\n",
       "      <td>We also experiment with majority voting and av...</td>\n",
       "      <td>Formally , we replace Equations [ reference ] ...</td>\n",
       "      <td>Figure [ reference ] shows the behavior of the...</td>\n",
       "      <td>0</td>\n",
       "      <td>Ensemble Strategy Study</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>687</th>\n",
       "      <td>biasing process</td>\n",
       "      <td>Then , in the biasing process , for each layer...</td>\n",
       "      <td>In the upsampling process , one uses a convolu...</td>\n",
       "      <td>The larger image is then generated as usual .</td>\n",
       "      <td>0</td>\n",
       "      <td>Multi - Scale PixelRNN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>443</th>\n",
       "      <td>class embedding</td>\n",
       "      <td>It uses a simpler variant of skip - conditioni...</td>\n",
       "      <td>The BigGAN - deep model ( Figure [ reference ]...</td>\n",
       "      <td>BigGAN - deep is based on residual blocks with...</td>\n",
       "      <td>1</td>\n",
       "      <td>Architectural details</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>greedy search process</td>\n",
       "      <td>The greedy search process applies rules in a m...</td>\n",
       "      <td>op2 \" Korea \" ) , then determine if they are a...</td>\n",
       "      <td>The results are mutually exclusive which means...</td>\n",
       "      <td>0</td>\n",
       "      <td>JAMR Aligner .</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>deep network architectures</td>\n",
       "      <td>This process is similar to that of the origina...</td>\n",
       "      <td>That is , the dimension slowly increases in in...</td>\n",
       "      <td>The visual illustrations of additive and multi...</td>\n",
       "      <td>0</td>\n",
       "      <td>Feature Map Dimension Configuration</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>DSS</td>\n",
       "      <td>Deep Sliding Shapes ( DSS ) generates 3D regio...</td>\n",
       "      <td>Comparison with other methods We compare our m...</td>\n",
       "      <td>Our model outperforms DSS by 3 % mAP while bei...</td>\n",
       "      <td>1</td>\n",
       "      <td>Evaluation on SUN - RGBD</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>dynamic sentence representation</td>\n",
       "      <td>Given the caption representation from the lang...</td>\n",
       "      <td>[ reference ] ) , with and initialized to lear...</td>\n",
       "      <td>The function of Eq .</td>\n",
       "      <td>0</td>\n",
       "      <td>Image Model : the Conditional DRAW Network</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>capsules</td>\n",
       "      <td>It is inspired by the notion of capsules devel...</td>\n",
       "      <td>In particular , we propose a new model , refer...</td>\n",
       "      <td>Our graph capsule idea is quite general and ca...</td>\n",
       "      <td>1</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>evo - toy</td>\n",
       "      <td>evo - toy shows the evaluation of the accuracy...</td>\n",
       "      <td>fig :</td>\n",
       "      <td>It is clear that pairing the data - augmented ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Data - augmented batches : toy model</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>first order approximation</td>\n",
       "      <td>Motivated as a first order approximation of th...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adding the identity matrix corresponds to addi...</td>\n",
       "      <td>0</td>\n",
       "      <td>The special case of Kipf and Welling ( 2016 )</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>LSTM system</td>\n",
       "      <td>For the LSTM system , the conversation - side ...</td>\n",
       "      <td>A 100 - dimensional i - vector is generated fo...</td>\n",
       "      <td>For convolutional networks , this approach is ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Speaker Adaptive Modeling</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>399</th>\n",
       "      <td>person representation</td>\n",
       "      <td>At last , the features of global full body and...</td>\n",
       "      <td>The Dropout is adopted after each FC layer to ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Feature Extraction and Fusion</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>744</th>\n",
       "      <td>CNN - based depth estimation</td>\n",
       "      <td>CNN - based depth estimation .</td>\n",
       "      <td>Given training data like the one produced by ,...</td>\n",
       "      <td>Our work has been inspired by the work of who ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Closely related work</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>519</th>\n",
       "      <td>3D face regressor model</td>\n",
       "      <td>[ f , t , Π , α s , α exp ] to regress for the...</td>\n",
       "      <td>Putting them together , we have in total 62 pa...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3D morphable model</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>620</th>\n",
       "      <td>pose prediction</td>\n",
       "      <td>To achieve this , we refine our previous pose ...</td>\n",
       "      <td>Ultimately , our goal is to predict the locati...</td>\n",
       "      <td>Furthermore , we only learn correction offsets...</td>\n",
       "      <td>0</td>\n",
       "      <td>Multi - Task Loss</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>element - wise means</td>\n",
       "      <td>With element - wise means , this is trivial ; ...</td>\n",
       "      <td>As with dropout , signals may need appropriate...</td>\n",
       "      <td>In experiments , we train with dropout and a m...</td>\n",
       "      <td>1</td>\n",
       "      <td>Regularization via Drop - path</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>854</th>\n",
       "      <td>reading comprehension models</td>\n",
       "      <td>This is reminiscent of the ( soft ) attention ...</td>\n",
       "      <td>Pooling over lemma - occurrences effectively c...</td>\n",
       "      <td>However , our setup is more general as it allo...</td>\n",
       "      <td>0</td>\n",
       "      <td>Refined Word Embeddings ( )</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750</th>\n",
       "      <td>MAN - AGER</td>\n",
       "      <td>Next , given the goal embedding produced by th...</td>\n",
       "      <td>We thus call it a leakage of information from D.</td>\n",
       "      <td>As such , the guiding signals from D are not o...</td>\n",
       "      <td>0</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>747</th>\n",
       "      <td>Siamese nets</td>\n",
       "      <td>1 - shot 5 - shot 1 - shot 5 - shot MANN , no ...</td>\n",
       "      <td>Accuracy Omniglot [ reference ]</td>\n",
       "      <td>[ reference ] 98.4 % 99.6 % 95.0 % 98.6 % MAML...</td>\n",
       "      <td>1</td>\n",
       "      <td>Classification</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>concatenation of previous layer encodings</td>\n",
       "      <td>As is also typical , we can have multiple such...</td>\n",
       "      <td>a BiLSTM :</td>\n",
       "      <td>The last layer l has both forward ( f l c , 1 ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Sentence - based Character Model</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>644</th>\n",
       "      <td>spectral normalization method</td>\n",
       "      <td>Moreover , the RBF - B kernel managed to stabi...</td>\n",
       "      <td>In Appendix [ reference ] , we showed the prop...</td>\n",
       "      <td>In Appendix [ reference ] , we showed the grad...</td>\n",
       "      <td>0</td>\n",
       "      <td>Quantitative Analysis</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>980</th>\n",
       "      <td>AI techniques</td>\n",
       "      <td>We illustrate the promise of ALE by developing...</td>\n",
       "      <td>Most importantly , it provides a rigorous test...</td>\n",
       "      <td>In doing so , we also propose an evaluation me...</td>\n",
       "      <td>0</td>\n",
       "      <td>The Arcade Learning Environment : An Evaluatio...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>468</th>\n",
       "      <td>quaternion convolution</td>\n",
       "      <td>This section defines the internal quaternion r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Quaternion convolutional neural networks</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>DeepLab</td>\n",
       "      <td>It is more efficient and accurate than modern ...</td>\n",
       "      <td>ICNet still performs satisfyingly regarding co...</td>\n",
       "      <td>Compared to our baseline model , it achieves 5...</td>\n",
       "      <td>0</td>\n",
       "      <td>COCO - Stuff</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>classification layers</td>\n",
       "      <td>This feature vector is then passed through cla...</td>\n",
       "      <td>For each sliding window , we drop out the valu...</td>\n",
       "      <td>Based on the loss of all the windows , we sele...</td>\n",
       "      <td>1</td>\n",
       "      <td>Adversarial Spatial Dropout for Occlusion</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>deeper architecture</td>\n",
       "      <td>When using CaffeNet as the backbone , we direc...</td>\n",
       "      <td>We mainly use two networks pre - trained on Im...</td>\n",
       "      <td>When using ResNet - 50 as the backbone , we ha...</td>\n",
       "      <td>0</td>\n",
       "      <td>Datasets and Settings</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>fc 7 features</td>\n",
       "      <td>Nearest neighbors are computed using fc 7 feat...</td>\n",
       "      <td>We approach this difficult task in the style o...</td>\n",
       "      <td>Since we are specifically testing the quality ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Intraclass alignment</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>Non - approximate joint training</td>\n",
       "      <td>( iii ) Non - approximate joint training .</td>\n",
       "      <td>This solver is included in our released Python...</td>\n",
       "      <td>As discussed above , the bounding boxes predic...</td>\n",
       "      <td>0</td>\n",
       "      <td>Sharing Features for RPN and Fast R - CNN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>785</th>\n",
       "      <td>Rainbow</td>\n",
       "      <td>Additionally , unlike Rainbow , Reactor does n...</td>\n",
       "      <td>( see Figure [ reference ] , right ) .</td>\n",
       "      <td>When evaluating under the no - op starts regim...</td>\n",
       "      <td>0</td>\n",
       "      <td>Comparing to prior work</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>424</th>\n",
       "      <td>GK</td>\n",
       "      <td>, Graphlet Kernel ( GK ) , Weisfeiler - Lehman...</td>\n",
       "      <td>We adopted state - of - art graphs kernels for...</td>\n",
       "      <td>Baselines Settings :</td>\n",
       "      <td>1</td>\n",
       "      <td>Experiment and Results</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>798</th>\n",
       "      <td>log - likelihood training</td>\n",
       "      <td>When beam - search is used , the ranking of th...</td>\n",
       "      <td>Surprisingly , the best performing method is R...</td>\n",
       "      <td>The final performances of the actor - critic a...</td>\n",
       "      <td>0</td>\n",
       "      <td>IWSLT 2014 with a convolutional encoder</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>low - rank language models</td>\n",
       "      <td>Empirically , our high - rank language model o...</td>\n",
       "      <td>For example , semantic meanings might not be t...</td>\n",
       "      <td>We also provide evidences in Section [ referen...</td>\n",
       "      <td>0</td>\n",
       "      <td>Hypothesis : Natural Language is High - Rank</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>people detector output</td>\n",
       "      <td>ROI are either based on a ground truth ( GT RO...</td>\n",
       "      <td>This corresponds to unary only performance .</td>\n",
       "      <td>Results on WAF .</td>\n",
       "      <td>0</td>\n",
       "      <td>Multi Person Pose Estimation</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>pose estimation</td>\n",
       "      <td>Unlike our proposed pose estimation , they reg...</td>\n",
       "      <td>Some recently addressed faces in particular , ...</td>\n",
       "      <td>We regress 6DoF directly from image intensitie...</td>\n",
       "      <td>1</td>\n",
       "      <td>Related work</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>log - linear extractive summarization model</td>\n",
       "      <td>Since these corpora are too small to train lar...</td>\n",
       "      <td>The DUC corpus comes in two parts : the 2003 c...</td>\n",
       "      <td>They call the original neural attention model ...</td>\n",
       "      <td>0</td>\n",
       "      <td>DUC Corpus</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>823</th>\n",
       "      <td>3D human pose model</td>\n",
       "      <td>Information captured by the 3D human pose mode...</td>\n",
       "      <td>We propose a novel CNN architecture that learn...</td>\n",
       "      <td>The advantage of integrating the output propos...</td>\n",
       "      <td>1</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>945</th>\n",
       "      <td>analytically - differentiable rendering layer</td>\n",
       "      <td>With the projection parameter , 3D shape , and...</td>\n",
       "      <td>Two decoders serve as the nonlinear 3DMM to ma...</td>\n",
       "      <td>The entire network is end - to - end trainable...</td>\n",
       "      <td>0</td>\n",
       "      <td>Nonlinear 3D Face Morphable Model</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>RNN - based VQA approach</td>\n",
       "      <td>Secondly , in this paper we have shown that it...</td>\n",
       "      <td>Indeed , at the time of submitting this paper ...</td>\n",
       "      <td>The knowledge bases which are currently availa...</td>\n",
       "      <td>0</td>\n",
       "      <td>Conclusions</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>Class - Activation Mapping</td>\n",
       "      <td>Change in Class - Activation Mapping :</td>\n",
       "      <td>As summarized in Table 4 , we observe an avera...</td>\n",
       "      <td>To qualitatively study the improvement in loca...</td>\n",
       "      <td>1</td>\n",
       "      <td>Improvement in Localization Ability</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>770</th>\n",
       "      <td>convnet features</td>\n",
       "      <td>In this paper , we provide evidence that convn...</td>\n",
       "      <td>Or do large receptive fields mean that corresp...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>CTC - LSTM models</td>\n",
       "      <td>These features were also used in the original ...</td>\n",
       "      <td>The input features for these experiments were ...</td>\n",
       "      <td>The CTC layer itself was trained on the 61 lab...</td>\n",
       "      <td>0</td>\n",
       "      <td>CTC setup</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>573</th>\n",
       "      <td>convolutional image filters</td>\n",
       "      <td>The spatial term uses the FoE model from , whi...</td>\n",
       "      <td>They formulate a classical flow problem with a...</td>\n",
       "      <td>With limited training data and a small set of ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Related Work</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>fully - connected linear layer</td>\n",
       "      <td>Finally , a fully - connected linear layer pro...</td>\n",
       "      <td>All these layers are separated by Rectifier Li...</td>\n",
       "      <td>The optimization employed to train the network...</td>\n",
       "      <td>1</td>\n",
       "      <td>Network Architecture</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>linearized and anonymized AMR graph</td>\n",
       "      <td>Following konstas2017neural , the input sequen...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Linearization is used to convert the graph int...</td>\n",
       "      <td>1</td>\n",
       "      <td>Sequential AMRs</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>Inter - Algorithm Normalization</td>\n",
       "      <td>subsubsection : Inter - Algorithm Normalization</td>\n",
       "      <td>For example , humans often play games without ...</td>\n",
       "      <td>A third alternative is to normalize using the ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Inter - Algorithm Normalization</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>559</th>\n",
       "      <td>Autoregressive NMT without RNNs</td>\n",
       "      <td>paragraph : Autoregressive NMT without RNNs</td>\n",
       "      <td>Choosing to factorize the machine translation ...</td>\n",
       "      <td>Since the entire target translation is known a...</td>\n",
       "      <td>0</td>\n",
       "      <td>Autoregressive NMT without RNNs</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>474</th>\n",
       "      <td>stochastic approximations</td>\n",
       "      <td>Thus , the gradients and are stochastic approx...</td>\n",
       "      <td>If the true gradients are and , then we can de...</td>\n",
       "      <td>Consequently , we analyze convergence of GANs ...</td>\n",
       "      <td>1</td>\n",
       "      <td>Two Time - Scale Update Rule for GANs</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>RoI</td>\n",
       "      <td>For the RoI pooling layer , its inputs are a p...</td>\n",
       "      <td>( [ reference ] ) lies on the spatial transfor...</td>\n",
       "      <td>In Fast R - CNN , the box proposals are pre - ...</td>\n",
       "      <td>0</td>\n",
       "      <td>End - to - End Training</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>bimodal</td>\n",
       "      <td>Similarly to bimodal fusion ( sec : bimodal ) ...</td>\n",
       "      <td>So , we define the fused features as where , i...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Trimodal fusion</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>self - supervised way</td>\n",
       "      <td>Instead , it is trained to encode 3D model vie...</td>\n",
       "      <td>Finally , the AAE does not require any real po...</td>\n",
       "      <td>A schematic overview of the approach is shown ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>multi - layer bidirectional Transformer</td>\n",
       "      <td>For example , BERT is based on a multi - layer...</td>\n",
       "      <td>These are neural network language models train...</td>\n",
       "      <td>To apply a pre - trained model to specific NLU...</td>\n",
       "      <td>0</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>speech systems</td>\n",
       "      <td>Deep Speech also handles challenging noisy env...</td>\n",
       "      <td>Our system , called Deep Speech , outperforms ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Deep Speech : Scaling up end - to - end speech...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>dilated models</td>\n",
       "      <td>Also , the dilated models outperform their reg...</td>\n",
       "      <td>To the best of our knowledge , the dilated GRU...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Language modeling</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>MMD - rep - b</td>\n",
       "      <td>[ reference ] shows that : When PICO was used ...</td>\n",
       "      <td>Fig .</td>\n",
       "      <td>For hinge and MMD - rbf , higher may result in...</td>\n",
       "      <td>1</td>\n",
       "      <td>Experiments</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>657</th>\n",
       "      <td>VQA models</td>\n",
       "      <td>Visual features from CNN already have implicit...</td>\n",
       "      <td>The example in lower part of Figure [ referenc...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Understanding the Visual QA model</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>405</th>\n",
       "      <td>1 - hidden - layer mlp generative model</td>\n",
       "      <td>the overall architecture for a Mult - vae pr /...</td>\n",
       "      <td>As a concrete example , recall I is the total ...</td>\n",
       "      <td>→ 600 → I ] .</td>\n",
       "      <td>0</td>\n",
       "      <td>Experimental setup</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>Inverse of Block Matrices</td>\n",
       "      <td>appendix : Inverse of Block Matrices</td>\n",
       "      <td>bibliography : References</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Inverse of Block Matrices</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601</th>\n",
       "      <td>deep model</td>\n",
       "      <td>Unexpectedly , such degradation is not caused ...</td>\n",
       "      <td>When deeper networks are able to start converg...</td>\n",
       "      <td>Fig .</td>\n",
       "      <td>0</td>\n",
       "      <td>Introduction</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>702</th>\n",
       "      <td>domain discriminator</td>\n",
       "      <td>For example , we can set the architecture of t...</td>\n",
       "      <td>We assume that the family of domain classifier...</td>\n",
       "      <td>Given the assumption holds , one can easily sh...</td>\n",
       "      <td>0</td>\n",
       "      <td>Relation to - distance</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>SDE</td>\n",
       "      <td>The SDE dynamics are governed by a \" noise sca...</td>\n",
       "      <td>The authors argued that the SGD algorithm can ...</td>\n",
       "      <td>They conclude that a higher noise scale preven...</td>\n",
       "      <td>0</td>\n",
       "      <td>Related Work</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>648</th>\n",
       "      <td>GMMs</td>\n",
       "      <td>With this in mind , it is easy to generalize G...</td>\n",
       "      <td>⇡ iN x|bi , AiATi .</td>\n",
       "      <td>Instead of sampling one transformation from a ...</td>\n",
       "      <td>0</td>\n",
       "      <td>Title</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>347</th>\n",
       "      <td>sequence - based LSTM</td>\n",
       "      <td>As noted in , the simple BOW model performs ro...</td>\n",
       "      <td>Other question representations , such as an LS...</td>\n",
       "      <td>Specifically , we compute where represents the...</td>\n",
       "      <td>1</td>\n",
       "      <td>Word Guided Spatial Attention in One - Hop Model</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               ner  \\\n",
       "566                             pooling operations   \n",
       "558                   explicit attention mechanism   \n",
       "563                fully - convolutional baselines   \n",
       "365                      scale expansion algorithm   \n",
       "931              multi - scale and sliding pooling   \n",
       "654                                    FastEval14k   \n",
       "786                                 coding schemes   \n",
       "938                       bidirectional LSTM model   \n",
       "716                              HFA based methods   \n",
       "475                                           RASP   \n",
       "444                                       CHFusion   \n",
       "839                                  Ranking model   \n",
       "567                                      VIS + BOW   \n",
       "210                                      Evolution   \n",
       "469                          Quantitative Analysis   \n",
       "162            question and answer representations   \n",
       "499                        classification function   \n",
       "294                                uniform weights   \n",
       "812               masked self - attentional layers   \n",
       "136                            Google NLP pipeline   \n",
       "867                                   joint update   \n",
       "892                                 left eye model   \n",
       "737                        fusion sub - components   \n",
       "247               generic face alignment algorithm   \n",
       "315                matrix of convolutional filters   \n",
       "132                                   downsamplers   \n",
       "297                                   deeper model   \n",
       "684                                     LSTM Q + I   \n",
       "986                                     label maps   \n",
       "982                           mixture 10 Gaussians   \n",
       "246                           CUDA implementations   \n",
       "881                           vanilla Transformers   \n",
       "963            probability distribution strategies   \n",
       "687                                biasing process   \n",
       "443                                class embedding   \n",
       "181                          greedy search process   \n",
       "116                     deep network architectures   \n",
       "148                                            DSS   \n",
       "54                 dynamic sentence representation   \n",
       "873                                       capsules   \n",
       "102                                      evo - toy   \n",
       "285                      first order approximation   \n",
       "256                                    LSTM system   \n",
       "399                          person representation   \n",
       "744                   CNN - based depth estimation   \n",
       "519                        3D face regressor model   \n",
       "620                                pose prediction   \n",
       "704                           element - wise means   \n",
       "854                   reading comprehension models   \n",
       "750                                     MAN - AGER   \n",
       "747                                   Siamese nets   \n",
       "314      concatenation of previous layer encodings   \n",
       "644                  spectral normalization method   \n",
       "980                                  AI techniques   \n",
       "468                         quaternion convolution   \n",
       "357                                        DeepLab   \n",
       "851                          classification layers   \n",
       "133                            deeper architecture   \n",
       "248                                  fc 7 features   \n",
       "376               Non - approximate joint training   \n",
       "785                                        Rainbow   \n",
       "424                                             GK   \n",
       "798                      log - likelihood training   \n",
       "94                      low - rank language models   \n",
       "233                         people detector output   \n",
       "236                                pose estimation   \n",
       "53     log - linear extractive summarization model   \n",
       "823                            3D human pose model   \n",
       "945  analytically - differentiable rendering layer   \n",
       "872                       RNN - based VQA approach   \n",
       "372                     Class - Activation Mapping   \n",
       "770                               convnet features   \n",
       "186                              CTC - LSTM models   \n",
       "573                    convolutional image filters   \n",
       "301                 fully - connected linear layer   \n",
       "123            linearized and anonymized AMR graph   \n",
       "971                Inter - Algorithm Normalization   \n",
       "559                Autoregressive NMT without RNNs   \n",
       "474                      stochastic approximations   \n",
       "882                                            RoI   \n",
       "168                                        bimodal   \n",
       "167                          self - supervised way   \n",
       "369        multi - layer bidirectional Transformer   \n",
       "177                                 speech systems   \n",
       "26                                  dilated models   \n",
       "5                                    MMD - rep - b   \n",
       "657                                     VQA models   \n",
       "405        1 - hidden - layer mlp generative model   \n",
       "319                      Inverse of Block Matrices   \n",
       "601                                     deep model   \n",
       "702                           domain discriminator   \n",
       "238                                            SDE   \n",
       "648                                           GMMs   \n",
       "347                          sequence - based LSTM   \n",
       "\n",
       "                                              sentence  \\\n",
       "566  The most basic version of this approach would ...   \n",
       "558  Based on this , we argue that MRN is an implic...   \n",
       "563          the two fully - convolutional baselines .   \n",
       "365  The detail of scale expansion algorithm is sum...   \n",
       "931  We start from a baseline setting of our FeatMa...   \n",
       "654  FastEval14k consists of 14000 images with labe...   \n",
       "786  The assignment of codes is more balanced for l...   \n",
       "938  These vectors are concatenated ( into a vector...   \n",
       "716  The major advancements of the proposed approac...   \n",
       "475  The required POS tags were generated with RASP...   \n",
       "444  subsubsection : Context - Aware Hierarchical F...   \n",
       "839  KEYWORDS Ranking model , weak supervision , de...   \n",
       "567  VIS + BOW performs multinomial logistic regres...   \n",
       "210  Evolution has discovered efficient feedforward...   \n",
       "469              subsubsection : Quantitative Analysis   \n",
       "162  The Neural Answer Selection Model ( Figure [ r...   \n",
       "499  IIC is a generic clustering algorithm that dir...   \n",
       "294  Also , regardless of the embedding method , le...   \n",
       "812  We present graph attention networks ( GATs ) ,...   \n",
       "136  The text has been run through a Google NLP pip...   \n",
       "867  We can write the joint update for all as Restr...   \n",
       "892  Moreover , for the right eye cluster , the rig...   \n",
       "737  Below , we go into the details of our point cl...   \n",
       "247  [ reference ] that even as a generic face alig...   \n",
       "315  The result is reshaped to generate a matrix of...   \n",
       "132  Existing literatures have considered two types...   \n",
       "297  The existence of this constructed solution ind...   \n",
       "684  [ reference ] , Q + I vs Question , and LSTM Q...   \n",
       "986  We demonstrate that this approach is effective...   \n",
       "982  In contrast , the VAE exhibit systematic diffe...   \n",
       "246  While it is possible to have a 10 speed up for...   \n",
       "881  Specifically , Transformer - XL significantly ...   \n",
       "963  We also experiment with majority voting and av...   \n",
       "687  Then , in the biasing process , for each layer...   \n",
       "443  It uses a simpler variant of skip - conditioni...   \n",
       "181  The greedy search process applies rules in a m...   \n",
       "116  This process is similar to that of the origina...   \n",
       "148  Deep Sliding Shapes ( DSS ) generates 3D regio...   \n",
       "54   Given the caption representation from the lang...   \n",
       "873  It is inspired by the notion of capsules devel...   \n",
       "102  evo - toy shows the evaluation of the accuracy...   \n",
       "285  Motivated as a first order approximation of th...   \n",
       "256  For the LSTM system , the conversation - side ...   \n",
       "399  At last , the features of global full body and...   \n",
       "744                     CNN - based depth estimation .   \n",
       "519  [ f , t , Π , α s , α exp ] to regress for the...   \n",
       "620  To achieve this , we refine our previous pose ...   \n",
       "704  With element - wise means , this is trivial ; ...   \n",
       "854  This is reminiscent of the ( soft ) attention ...   \n",
       "750  Next , given the goal embedding produced by th...   \n",
       "747  1 - shot 5 - shot 1 - shot 5 - shot MANN , no ...   \n",
       "314  As is also typical , we can have multiple such...   \n",
       "644  Moreover , the RBF - B kernel managed to stabi...   \n",
       "980  We illustrate the promise of ALE by developing...   \n",
       "468  This section defines the internal quaternion r...   \n",
       "357  It is more efficient and accurate than modern ...   \n",
       "851  This feature vector is then passed through cla...   \n",
       "133  When using CaffeNet as the backbone , we direc...   \n",
       "248  Nearest neighbors are computed using fc 7 feat...   \n",
       "376         ( iii ) Non - approximate joint training .   \n",
       "785  Additionally , unlike Rainbow , Reactor does n...   \n",
       "424  , Graphlet Kernel ( GK ) , Weisfeiler - Lehman...   \n",
       "798  When beam - search is used , the ranking of th...   \n",
       "94   Empirically , our high - rank language model o...   \n",
       "233  ROI are either based on a ground truth ( GT RO...   \n",
       "236  Unlike our proposed pose estimation , they reg...   \n",
       "53   Since these corpora are too small to train lar...   \n",
       "823  Information captured by the 3D human pose mode...   \n",
       "945  With the projection parameter , 3D shape , and...   \n",
       "872  Secondly , in this paper we have shown that it...   \n",
       "372             Change in Class - Activation Mapping :   \n",
       "770  In this paper , we provide evidence that convn...   \n",
       "186  These features were also used in the original ...   \n",
       "573  The spatial term uses the FoE model from , whi...   \n",
       "301  Finally , a fully - connected linear layer pro...   \n",
       "123  Following konstas2017neural , the input sequen...   \n",
       "971    subsubsection : Inter - Algorithm Normalization   \n",
       "559        paragraph : Autoregressive NMT without RNNs   \n",
       "474  Thus , the gradients and are stochastic approx...   \n",
       "882  For the RoI pooling layer , its inputs are a p...   \n",
       "168  Similarly to bimodal fusion ( sec : bimodal ) ...   \n",
       "167  Instead , it is trained to encode 3D model vie...   \n",
       "369  For example , BERT is based on a multi - layer...   \n",
       "177  Deep Speech also handles challenging noisy env...   \n",
       "26   Also , the dilated models outperform their reg...   \n",
       "5    [ reference ] shows that : When PICO was used ...   \n",
       "657  Visual features from CNN already have implicit...   \n",
       "405  the overall architecture for a Mult - vae pr /...   \n",
       "319               appendix : Inverse of Block Matrices   \n",
       "601  Unexpectedly , such degradation is not caused ...   \n",
       "702  For example , we can set the architecture of t...   \n",
       "238  The SDE dynamics are governed by a \" noise sca...   \n",
       "648  With this in mind , it is easy to generalize G...   \n",
       "347  As noted in , the simple BOW model performs ro...   \n",
       "\n",
       "                                          pre_sentence  \\\n",
       "566  These pooling layers remain distinct from conv...   \n",
       "558  Since MRN does not depend on a few attention p...   \n",
       "563                                            w.r.t .   \n",
       "365  Thanks to the “ progressive ” expansion proced...   \n",
       "931  We present the results of adding different com...   \n",
       "654                                    FastEval14k ’ .   \n",
       "786  ” for the first component , while the subcode ...   \n",
       "938  Each model consumes the words in the sentence ...   \n",
       "716  Specifically , given a feature , the HFA based...   \n",
       "475                                    [ reference ] .   \n",
       "444  Wherein our method learns the weights automati...   \n",
       "839  Our findings also suggest that supervised neur...   \n",
       "567  GUESS simply selects the modal answer from the...   \n",
       "210  Once trained , the CNN never changes its weigh...   \n",
       "469  It manifests that our different - resolution i...   \n",
       "162  This is a classification task where we treat e...   \n",
       "499  In this paper , we introduce Invariant Informa...   \n",
       "294  This supports the hypothesis that with the emb...   \n",
       "812                                                NaN   \n",
       "136  A news article is usually associated with a fe...   \n",
       "867                                        Let , and .   \n",
       "892  Taking the left eye model as an example , it a...   \n",
       "737  [ reference ] D ) , in which we use a dense sp...   \n",
       "247                          It can be seen in Tabel .   \n",
       "315  The hypernetwork is a fully connected layer ( ...   \n",
       "132                                                NaN   \n",
       "297  There exists a solution by construction to the...   \n",
       "684                           No , as shown in Table .   \n",
       "986  This makes it possible to apply the same gener...   \n",
       "982  The adversarial autoencoder successfully match...   \n",
       "246  Table [ reference ] shows that methods using c...   \n",
       "881  Although Transformer - XL is mainly designed t...   \n",
       "963  Formally , we replace Equations [ reference ] ...   \n",
       "687  In the upsampling process , one uses a convolu...   \n",
       "443  The BigGAN - deep model ( Figure [ reference ]...   \n",
       "181  op2 \" Korea \" ) , then determine if they are a...   \n",
       "116  That is , the dimension slowly increases in in...   \n",
       "148  Comparison with other methods We compare our m...   \n",
       "54   [ reference ] ) , with and initialized to lear...   \n",
       "873  In particular , we propose a new model , refer...   \n",
       "102                                              fig :   \n",
       "285                                                NaN   \n",
       "256  A 100 - dimensional i - vector is generated fo...   \n",
       "399  The Dropout is adopted after each FC layer to ...   \n",
       "744  Given training data like the one produced by ,...   \n",
       "519  Putting them together , we have in total 62 pa...   \n",
       "620  Ultimately , our goal is to predict the locati...   \n",
       "704  As with dropout , signals may need appropriate...   \n",
       "854  Pooling over lemma - occurrences effectively c...   \n",
       "750   We thus call it a leakage of information from D.   \n",
       "747                    Accuracy Omniglot [ reference ]   \n",
       "314                                         a BiLSTM :   \n",
       "644  In Appendix [ reference ] , we showed the prop...   \n",
       "980  Most importantly , it provides a rigorous test...   \n",
       "468                                                NaN   \n",
       "357  ICNet still performs satisfyingly regarding co...   \n",
       "851  For each sliding window , we drop out the valu...   \n",
       "133  We mainly use two networks pre - trained on Im...   \n",
       "248  We approach this difficult task in the style o...   \n",
       "376  This solver is included in our released Python...   \n",
       "785             ( see Figure [ reference ] , right ) .   \n",
       "424  We adopted state - of - art graphs kernels for...   \n",
       "798  Surprisingly , the best performing method is R...   \n",
       "94   For example , semantic meanings might not be t...   \n",
       "233       This corresponds to unary only performance .   \n",
       "236  Some recently addressed faces in particular , ...   \n",
       "53   The DUC corpus comes in two parts : the 2003 c...   \n",
       "823  We propose a novel CNN architecture that learn...   \n",
       "945  Two decoders serve as the nonlinear 3DMM to ma...   \n",
       "872  Indeed , at the time of submitting this paper ...   \n",
       "372  As summarized in Table 4 , we observe an avera...   \n",
       "770  Or do large receptive fields mean that corresp...   \n",
       "186  The input features for these experiments were ...   \n",
       "573  They formulate a classical flow problem with a...   \n",
       "301  All these layers are separated by Rectifier Li...   \n",
       "123                                                NaN   \n",
       "971  For example , humans often play games without ...   \n",
       "559  Choosing to factorize the machine translation ...   \n",
       "474  If the true gradients are and , then we can de...   \n",
       "882  ( [ reference ] ) lies on the spatial transfor...   \n",
       "168  So , we define the fused features as where , i...   \n",
       "167  Finally , the AAE does not require any real po...   \n",
       "369  These are neural network language models train...   \n",
       "177  Our system , called Deep Speech , outperforms ...   \n",
       "26   To the best of our knowledge , the dilated GRU...   \n",
       "5                                                Fig .   \n",
       "657  The example in lower part of Figure [ referenc...   \n",
       "405  As a concrete example , recall I is the total ...   \n",
       "319                          bibliography : References   \n",
       "601  When deeper networks are able to start converg...   \n",
       "702  We assume that the family of domain classifier...   \n",
       "238  The authors argued that the SGD algorithm can ...   \n",
       "648                                ⇡ iN x|bi , AiATi .   \n",
       "347  Other question representations , such as an LS...   \n",
       "\n",
       "                                         post_sentence  used  \\\n",
       "566  One step further brings us to what we refer to...     0   \n",
       "558                                                NaN     0   \n",
       "563  Interestingly , the gap significantly widens w...     1   \n",
       "365  In the pseudocode , are the intermediate resul...     0   \n",
       "931  This baseline setting is the conventional full...     0   \n",
       "654  Unlike labels in JFT - 300 M , the images in F...     0   \n",
       "786  In any coding scheme , even the most unpopular...     1   \n",
       "938  The convolutional model uses 600 total filters...     0   \n",
       "716  Secondly , we explicitly project the identity ...     1   \n",
       "475                                                NaN     0   \n",
       "444  The results of this experiment are shown in ta...     1   \n",
       "839                         compat=1.11 ternary compat     0   \n",
       "567  VIS + LSTM has one LSTM to encode the image an...     0   \n",
       "210  However , an expert ornithologist , asked to c...     1   \n",
       "469  To further understand accuracy gain in each br...     1   \n",
       "162  It employs two different LSTMs to embed raw qu...     0   \n",
       "499  It involves a simple objective function , whic...     1   \n",
       "294  Although weight learning can significantly aff...     1   \n",
       "812  By stacking layers in which nodes are able to ...     0   \n",
       "136  It it tokenized , lowercased , and named entit...     0   \n",
       "867  This means that there is some such that for an...     0   \n",
       "892  It can be concluded that each shape prediction...     1   \n",
       "737                                                NaN     0   \n",
       "247                                                NaN     1   \n",
       "315  Whilst the overall dimensionality of the filte...     0   \n",
       "132  In this paper , we consider the bicubic downsa...     1   \n",
       "297  But experiments show that our current solvers ...     1   \n",
       "684  Our results demonstrate clearly the positive i...     1   \n",
       "986  Indeed , since the release of the pix2pix soft...     0   \n",
       "982  An important difference between VAEs and adver...     0   \n",
       "246  The GRNNs method of tang2015document takes aro...     0   \n",
       "881  We also report the results on word - level Pen...     0   \n",
       "963  Figure [ reference ] shows the behavior of the...     0   \n",
       "687      The larger image is then generated as usual .     0   \n",
       "443  BigGAN - deep is based on residual blocks with...     1   \n",
       "181  The results are mutually exclusive which means...     0   \n",
       "116  The visual illustrations of additive and multi...     0   \n",
       "148  Our model outperforms DSS by 3 % mAP while bei...     1   \n",
       "54                                The function of Eq .     0   \n",
       "873  Our graph capsule idea is quite general and ca...     1   \n",
       "102  It is clear that pairing the data - augmented ...     0   \n",
       "285  Adding the identity matrix corresponds to addi...     0   \n",
       "256  For convolutional networks , this approach is ...     0   \n",
       "399                                                NaN     0   \n",
       "744  Our work has been inspired by the work of who ...     0   \n",
       "519                                                NaN     0   \n",
       "620  Furthermore , we only learn correction offsets...     0   \n",
       "704  In experiments , we train with dropout and a m...     1   \n",
       "854  However , our setup is more general as it allo...     0   \n",
       "750  As such , the guiding signals from D are not o...     0   \n",
       "747  [ reference ] 98.4 % 99.6 % 95.0 % 98.6 % MAML...     1   \n",
       "314  The last layer l has both forward ( f l c , 1 ...     1   \n",
       "644  In Appendix [ reference ] , we showed the grad...     0   \n",
       "980  In doing so , we also propose an evaluation me...     0   \n",
       "468                                                NaN     0   \n",
       "357  Compared to our baseline model , it achieves 5...     0   \n",
       "851  Based on the loss of all the windows , we sele...     1   \n",
       "133  When using ResNet - 50 as the backbone , we ha...     0   \n",
       "248  Since we are specifically testing the quality ...     1   \n",
       "376  As discussed above , the bounding boxes predic...     0   \n",
       "785  When evaluating under the no - op starts regim...     0   \n",
       "424                               Baselines Settings :     1   \n",
       "798  The final performances of the actor - critic a...     0   \n",
       "94   We also provide evidences in Section [ referen...     0   \n",
       "233                                   Results on WAF .     0   \n",
       "236  We regress 6DoF directly from image intensitie...     1   \n",
       "53   They call the original neural attention model ...     0   \n",
       "823  The advantage of integrating the output propos...     1   \n",
       "945  The entire network is end - to - end trainable...     0   \n",
       "872  The knowledge bases which are currently availa...     0   \n",
       "372  To qualitatively study the improvement in loca...     1   \n",
       "770                                                NaN     1   \n",
       "186  The CTC layer itself was trained on the 61 lab...     0   \n",
       "573  With limited training data and a small set of ...     0   \n",
       "301  The optimization employed to train the network...     1   \n",
       "123  Linearization is used to convert the graph int...     1   \n",
       "971  A third alternative is to normalize using the ...     1   \n",
       "559  Since the entire target translation is known a...     0   \n",
       "474  Consequently , we analyze convergence of GANs ...     1   \n",
       "882  In Fast R - CNN , the box proposals are pre - ...     0   \n",
       "168                                                NaN     0   \n",
       "167  A schematic overview of the approach is shown ...     0   \n",
       "369  To apply a pre - trained model to specific NLU...     0   \n",
       "177                                                NaN     1   \n",
       "26                                                 NaN     1   \n",
       "5    For hinge and MMD - rbf , higher may result in...     1   \n",
       "657                                                NaN     0   \n",
       "405                                      → 600 → I ] .     0   \n",
       "319                                                NaN     1   \n",
       "601                                              Fig .     0   \n",
       "702  Given the assumption holds , one can easily sh...     0   \n",
       "238  They conclude that a higher noise scale preven...     0   \n",
       "648  Instead of sampling one transformation from a ...     0   \n",
       "347  Specifically , we compute where represents the...     1   \n",
       "\n",
       "                                          section_name  preds_bert  preds_cnn  \n",
       "566                                       Tree pooling           1          0  \n",
       "558                               Qualitative Analysis           0          1  \n",
       "563              Evaluation for visual object tracking           1          0  \n",
       "365              Progressive Scale Expansion Algorithm           1          1  \n",
       "931                               Component Evaluation           1          1  \n",
       "654                       Monitoring Training Progress           1          1  \n",
       "786                        Analysis of Code Efficiency           1          0  \n",
       "938                  C. Information Extraction Details           1          1  \n",
       "716                                         Discussion           1          0  \n",
       "475                                 Pattern Extraction           1          1  \n",
       "444   Context - Aware Hierarchical Fusion ( CHFusion )           1          0  \n",
       "839        Neural Ranking Models with Weak Supervision           1          0  \n",
       "567                                  Results on DAQURA           1          1  \n",
       "210                                              Title           0          0  \n",
       "469                              Quantitative Analysis           1          0  \n",
       "162                      Neural Answer Selection Model           1          1  \n",
       "499                                       Introduction           0          0  \n",
       "294                             Results and Discussion           1          0  \n",
       "812                                           ABSTRACT           1          1  \n",
       "136                     The Reading Comprehension Task           1          1  \n",
       "867                            Contraction Map Example           1          1  \n",
       "892                Analysis of Shape Prediction Layers           1          0  \n",
       "737                                        PointFusion           1          1  \n",
       "247                         Medium Pose Face Alignment           1          0  \n",
       "315            Scoring Function and Model Architecture           1          1  \n",
       "132                                      Downsampler .           1          0  \n",
       "297                                       Introduction           1          0  \n",
       "684                               Results and analysis           1          0  \n",
       "986  Image - to - Image Translation with Conditiona...           1          1  \n",
       "982           Relationship to Variational Autoencoders           0          1  \n",
       "246                                    Training time .           1          1  \n",
       "881                                       Main Results           1          0  \n",
       "963                            Ensemble Strategy Study           1          1  \n",
       "687                             Multi - Scale PixelRNN           1          1  \n",
       "443                              Architectural details           0          1  \n",
       "181                                     JAMR Aligner .           1          0  \n",
       "116                Feature Map Dimension Configuration           1          0  \n",
       "148                           Evaluation on SUN - RGBD           1          0  \n",
       "54          Image Model : the Conditional DRAW Network           0          1  \n",
       "873                                       Introduction           0          0  \n",
       "102               Data - augmented batches : toy model           1          1  \n",
       "285      The special case of Kipf and Welling ( 2016 )           0          1  \n",
       "256                          Speaker Adaptive Modeling           1          1  \n",
       "399                      Feature Extraction and Fusion           1          0  \n",
       "744                               Closely related work           1          0  \n",
       "519                                 3D morphable model           1          0  \n",
       "620                                  Multi - Task Loss           1          1  \n",
       "704                     Regularization via Drop - path           1          0  \n",
       "854                        Refined Word Embeddings ( )           1          0  \n",
       "750                                       Introduction           1          1  \n",
       "747                                     Classification           0          1  \n",
       "314                   Sentence - based Character Model           1          0  \n",
       "644                              Quantitative Analysis           1          0  \n",
       "980  The Arcade Learning Environment : An Evaluatio...           1          1  \n",
       "468           Quaternion convolutional neural networks           0          1  \n",
       "357                                       COCO - Stuff           1          1  \n",
       "851          Adversarial Spatial Dropout for Occlusion           1          0  \n",
       "133                              Datasets and Settings           1          1  \n",
       "248                               Intraclass alignment           1          0  \n",
       "376          Sharing Features for RPN and Fast R - CNN           1          0  \n",
       "785                            Comparing to prior work           1          0  \n",
       "424                             Experiment and Results           1          0  \n",
       "798            IWSLT 2014 with a convolutional encoder           1          1  \n",
       "94        Hypothesis : Natural Language is High - Rank           1          1  \n",
       "233                       Multi Person Pose Estimation           1          0  \n",
       "236                                       Related work           0          0  \n",
       "53                                          DUC Corpus           0          1  \n",
       "823                                       Introduction           0          0  \n",
       "945                  Nonlinear 3D Face Morphable Model           1          1  \n",
       "872                                        Conclusions           1          1  \n",
       "372                Improvement in Localization Ability           1          0  \n",
       "770                                       Introduction           0          1  \n",
       "186                                          CTC setup           1          0  \n",
       "573                                       Related Work           0          1  \n",
       "301                               Network Architecture           1          0  \n",
       "123                                    Sequential AMRs           0          0  \n",
       "971                    Inter - Algorithm Normalization           0          0  \n",
       "559                    Autoregressive NMT without RNNs           0          1  \n",
       "474              Two Time - Scale Update Rule for GANs           1          0  \n",
       "882                            End - to - End Training           1          1  \n",
       "168                                    Trimodal fusion           1          1  \n",
       "167                                       Introduction           1          0  \n",
       "369                                       Introduction           1          0  \n",
       "177  Deep Speech : Scaling up end - to - end speech...           1          0  \n",
       "26                                   Language modeling           0          1  \n",
       "5                                          Experiments           1          0  \n",
       "657                  Understanding the Visual QA model           1          0  \n",
       "405                                 Experimental setup           1          1  \n",
       "319                          Inverse of Block Matrices           0          1  \n",
       "601                                       Introduction           1          1  \n",
       "702                             Relation to - distance           1          1  \n",
       "238                                       Related Work           0          1  \n",
       "648                                              Title           1          0  \n",
       "347   Word Guided Spatial Attention in One - Hop Model           0          1  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = dev_set[(dev_set['used'] != dev_set['preds_bert']) | (dev_set['used'] != dev_set['preds_cnn'])]\n",
    "df2.to_csv('preds_labels.csv')\n",
    "df2"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}