ML_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3d894b98",
   "metadata": {},
   "source": [
    "The goal of this Notebook is to develop a systemic approach for testing various classifiers.<br>\n",
    "We will learn how to build a training and testing pipeline with various configurations. Then we will\n",
    "evaluate these models with simple classification metrics.<br>\n",
    "We will use:\n",
    "* the very useful model Pipeline, LabelEncoder, and dataset split features of sklearn\n",
    "* some ML classifiers and a first deep learning model\n",
    "* Document embeddings is not really the focus here, so will re-use the TFIDF embeddings for the moment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a0555bec",
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from config import RAW_DATA_PATH\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3b1b6b8",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Load the dataframe and tokenized documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7d6577d3",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/preprocessing/document_tokens_labelled.csv', sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bd6a0abd",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_tokens</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>défavorable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>défavorable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>défavorable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48741</th>\n",
       "      <td>monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>favorable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48742</th>\n",
       "      <td>monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>sans objet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48743</th>\n",
       "      <td>maître|x|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>sans objet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48744</th>\n",
       "      <td>Monsieur|x|x|saisir|commission|accès|document|...</td>\n",
       "      <td>sans objet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48745</th>\n",
       "      <td>monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>sans objet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48746 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         document_tokens        label\n",
       "0      commission|accès|document|administratif|examin...  défavorable\n",
       "1      commission|accès|document|administratif|examin...  défavorable\n",
       "2      commission|accès|document|administratif|examin...  défavorable\n",
       "3      commission|accès|document|administratif|examin...          NaN\n",
       "4      commission|accès|document|administratif|examin...          NaN\n",
       "...                                                  ...          ...\n",
       "48741  monsieur|x|saisir|commission|accès|document|ad...    favorable\n",
       "48742  monsieur|x|saisir|commission|accès|document|ad...   sans objet\n",
       "48743  maître|x|x|saisir|commission|accès|document|ad...   sans objet\n",
       "48744  Monsieur|x|x|saisir|commission|accès|document|...   sans objet\n",
       "48745  monsieur|x|saisir|commission|accès|document|ad...   sans objet\n",
       "\n",
       "[48746 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0875aeaf",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Remove unlabelled documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b01ea67e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "88f6970c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40454"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "00afa59e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "favorable      26940\n",
       "sans objet      9849\n",
       "défavorable     3665\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5243d48e",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b89438b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Create a pipeline that will be applied to Train and Test separately"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c2feeed",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Split into Train and Test\n",
    "Do it upfront in order to avoid data leakage. Also we need to stratify the test sample by label class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "cc96bcb9",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df_train, df_test, y_train, y_test = train_test_split(\n",
    "    df.document_tokens, df.label, test_size=0.30,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "906f1606",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " ## Encode the labels\n",
    " Important to encode the labels on training only (again, prevent leakage).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "dd81dc00",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "\n",
    "label_encoder.fit(y_train)\n",
    "\n",
    "y_train = label_encoder.transform(y_train)\n",
    "y_test = label_encoder.transform(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5856981a",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['défavorable', 'favorable', 'sans objet'], dtype=object)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_encoder.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "9f8fdd64",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "unique, counts = np.unique(y_train, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "d6d840ed",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 2529, 1: 18851, 2: 6937}"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(zip(unique, counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "eb51d2f0",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "MOST_POPULAR_CLASS = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "e199f8f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'défavorable', 1: 'favorable', 2: 'sans objet'}"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CLASS_INDEX = dict(zip(list(range(len(label_encoder.classes_))),label_encoder.classes_))\n",
    "CLASS_INDEX"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8965c993",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Document vectorizer\n",
    "\n",
    "Encode the document into vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdf3e77",
   "metadata": {},
   "source": [
    "### TFIDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05ec91f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf = TfidfVectorizer(\n",
    "    analyzer='word',\n",
    "    max_df=10000,  # ignore tokens that appear more than X times in the document collection\n",
    "    max_features=10000,   # capping on vocabulary size\n",
    "    sublinear_tf=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57e47a87",
   "metadata": {},
   "source": [
    "### GloVe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "2c23454b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "id": "a2aacf18",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GloVeVectorizer():\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        return"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e706f8",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Define the models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75512547",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Choose a baseline naive model\n",
    "\n",
    "What is the simplest model you could think of? Think of a model that always predicts the most popular class\n",
    "from the train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "02f3fd8b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class NaiveClassifier():\n",
    "    \"Always predict the most popular class\"\n",
    "    \n",
    "    # we know from label_encoder that most popular class Favorable takes index 1.\n",
    "    most_popular_class = MOST_POPULAR_CLASS\n",
    "    \n",
    "    def fit(self, X, y=None, **fit_params):\n",
    "        return\n",
    "    \n",
    "    def transform(self):\n",
    "        return\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return np.array([self.most_popular_class] * X.shape[0])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "816a9840",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "naive_classifier = NaiveClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad92ed9",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Choose a baseline ML model\n",
    "\n",
    "Now choose a ML classifier, the simplest one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "7221725b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "naive_bayes = MultinomialNB()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85f166a5",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "### Non parametric ML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "28b1fbb4",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "random_forest = RandomForestClassifier(n_estimators=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdd000bd",
   "metadata": {},
   "source": [
    "### Deep learning classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "72dc13f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import metrics\n",
    "from tensorflow.keras import losses\n",
    "import tensorflow as tf\n",
    "\n",
    "class DeepLearningClassifier():\n",
    "    \"\"\"\n",
    "    My first deep learning classifier.\n",
    "    Takes as input a word-gram matrix weighted by TFIDF.\n",
    "    The model learns a mapping of the input space to an dense space of size 100.\n",
    "    Then it outputs three probabilities, one for each class (hence a layer softmax of size 3).\n",
    "    \"\"\"\n",
    "    \n",
    "    EPOCHS=5\n",
    "    INPUT_DIMENSION = 10000\n",
    "    model=None\n",
    "    \n",
    "    def build_model(self):\n",
    "        self.model = Sequential()\n",
    "        self.model.add(layers.Dense(100, input_dim=self.INPUT_DIMENSION, activation='relu'))\n",
    "        self.model.add(layers.Dense(3, activation='softmax'))\n",
    "        self.model.compile(\n",
    "            loss=losses.CategoricalCrossentropy(),\n",
    "            optimizer='adam',\n",
    "            metrics=metrics.CategoricalCrossentropy()\n",
    "        )\n",
    "        \n",
    "    \n",
    "    def fit(self, X, y=None, **fit_params):\n",
    "        \n",
    "        self.build_model()\n",
    "        \n",
    "        X_tensor = tf.constant(X.toarray())\n",
    "        y_onehot = tf.keras.utils.to_categorical(y,3)\n",
    "        \n",
    "        history = self.model.fit(\n",
    "            x=X_tensor,\n",
    "            y=y_onehot,\n",
    "            epochs=self.EPOCHS,\n",
    "        )\n",
    "        return\n",
    "    \n",
    "    \n",
    "    def predict(self, X):\n",
    "        X_tensor = tf.constant(X.toarray())\n",
    "        return self.model.predict(X_tensor).argmax(axis=1)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "cc98a801",
   "metadata": {},
   "outputs": [],
   "source": [
    "deep_learning_classifier = DeepLearningClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e82eafa",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Assemble pipeline components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "9da6dfca",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline_naive_classifier = Pipeline(steps=[\n",
    "    ('tfidf', tfidf),\n",
    "    ('naive_classifier', naive_classifier),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ff21ff01",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;background-color: white;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_classifier&#x27;,\n",
       "                 &lt;__main__.NaiveClassifier object at 0x12fff9ed0&gt;)])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_classifier&#x27;,\n",
       "                 &lt;__main__.NaiveClassifier object at 0x12fff9ed0&gt;)])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=10000, max_features=10000, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NaiveClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;__main__.NaiveClassifier object at 0x12fff9ed0&gt;</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                ('naive_classifier',\n",
       "                 <__main__.NaiveClassifier object at 0x12fff9ed0>)])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_naive_classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "1e54c146",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline_baseline_ml = Pipeline(steps=[\n",
    "    ('tfidf', tfidf),\n",
    "    ('naive_bayes', naive_bayes),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "96903196",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-6 {color: black;background-color: white;}#sk-container-id-6 pre{padding: 0;}#sk-container-id-6 div.sk-toggleable {background-color: white;}#sk-container-id-6 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-6 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-6 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-6 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-6 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-6 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-6 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-6 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-6 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-6 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-6 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-6 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-6 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-6 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-6 div.sk-item {position: relative;z-index: 1;}#sk-container-id-6 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-6 div.sk-item::before, #sk-container-id-6 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-6 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-6 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-6 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-6 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-6 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-6 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-6 div.sk-label-container {text-align: center;}#sk-container-id-6 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-6 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-6\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_bayes&#x27;, MultinomialNB())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_bayes&#x27;, MultinomialNB())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=10000, max_features=10000, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MultinomialNB</label><div class=\"sk-toggleable__content\"><pre>MultinomialNB()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                ('naive_bayes', MultinomialNB())])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_baseline_ml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "6072f0ad",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline_random_forest = Pipeline(steps=[\n",
    "    ('tfidf', tfidf),\n",
    "    ('random_forest', random_forest),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "id": "8b3ce208",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline_deep_learning_baseline = Pipeline(steps=[\n",
    "    ('tfidf', tfidf),\n",
    "    ('deep_learning_classifier', deep_learning_classifier),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "id": "f8cc9609",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-11 {color: black;background-color: white;}#sk-container-id-11 pre{padding: 0;}#sk-container-id-11 div.sk-toggleable {background-color: white;}#sk-container-id-11 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-11 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-11 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-11 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-11 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-11 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-11 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-11 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-11 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-11 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-11 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-11 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-11 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-11 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-11 div.sk-item {position: relative;z-index: 1;}#sk-container-id-11 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-11 div.sk-item::before, #sk-container-id-11 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-11 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-11 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-11 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-11 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-11 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-11 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-11 div.sk-label-container {text-align: center;}#sk-container-id-11 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-11 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-11\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;deep_learning_classifier&#x27;,\n",
       "                 &lt;__main__.DeepLearningClassifier object at 0x148091600&gt;)])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;deep_learning_classifier&#x27;,\n",
       "                 &lt;__main__.DeepLearningClassifier object at 0x148091600&gt;)])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=10000, max_features=10000, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DeepLearningClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;__main__.DeepLearningClassifier object at 0x148091600&gt;</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                ('deep_learning_classifier',\n",
       "                 <__main__.DeepLearningClassifier object at 0x148091600>)])"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_deep_learning_baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "3bd6c308",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline_deep_learning_glove = Pipeline(steps=[\n",
    "    ('glove_vectorizer', glove_vectorizer),\n",
    "    ('deep_learning_glove', ),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12e3c25c",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Train the whole pipeline\n",
    "Remark: no need to fit the naive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "b0434222",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-7 {color: black;background-color: white;}#sk-container-id-7 pre{padding: 0;}#sk-container-id-7 div.sk-toggleable {background-color: white;}#sk-container-id-7 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-7 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-7 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-7 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-7 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-7 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-7 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-7 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-7 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-7 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-7 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-7 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-7 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-7 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-7 div.sk-item {position: relative;z-index: 1;}#sk-container-id-7 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-7 div.sk-item::before, #sk-container-id-7 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-7 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-7 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-7 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-7 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-7 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-7 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-7 div.sk-label-container {text-align: center;}#sk-container-id-7 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-7 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-7\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_bayes&#x27;, MultinomialNB())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;naive_bayes&#x27;, MultinomialNB())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=10000, max_features=10000, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MultinomialNB</label><div class=\"sk-toggleable__content\"><pre>MultinomialNB()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                ('naive_bayes', MultinomialNB())])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_baseline_ml.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "71b554c4",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-8 {color: black;background-color: white;}#sk-container-id-8 pre{padding: 0;}#sk-container-id-8 div.sk-toggleable {background-color: white;}#sk-container-id-8 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-8 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-8 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-8 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-8 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-8 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-8 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-8 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-8 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-8 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-8 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-8 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-8 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-8 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-8 div.sk-item {position: relative;z-index: 1;}#sk-container-id-8 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-8 div.sk-item::before, #sk-container-id-8 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-8 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-8 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-8 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-8 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-8 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-8 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-8 div.sk-label-container {text-align: center;}#sk-container-id-8 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-8 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-8\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;random_forest&#x27;, RandomForestClassifier(n_estimators=150))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                (&#x27;random_forest&#x27;, RandomForestClassifier(n_estimators=150))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=10000, max_features=10000, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_estimators=150)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=10000, max_features=10000,\n",
       "                                 sublinear_tf=True)),\n",
       "                ('random_forest', RandomForestClassifier(n_estimators=150))])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_random_forest.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "43e48438",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "885/885 [==============================] - 4s 5ms/step - loss: 0.3213 - categorical_crossentropy: 0.3213\n",
      "Epoch 2/5\n",
      "885/885 [==============================] - 4s 5ms/step - loss: 0.1093 - categorical_crossentropy: 0.1093\n",
      "Epoch 3/5\n",
      "885/885 [==============================] - 4s 5ms/step - loss: 0.0595 - categorical_crossentropy: 0.0595\n",
      "Epoch 4/5\n",
      "885/885 [==============================] - 4s 5ms/step - loss: 0.0371 - categorical_crossentropy: 0.0371\n",
      "Epoch 5/5\n",
      "885/885 [==============================] - 4s 5ms/step - loss: 0.0223 - categorical_crossentropy: 0.0223\n"
     ]
    }
   ],
   "source": [
    "deep_learning_classifier.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a2b13c",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Serve the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "b1fcf994",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "prediction_models = dict(naive=None, baseline_ml=None, random_forest=None, deep_learning=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "e4c73815",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "380/380 [==============================] - 1s 2ms/step\n"
     ]
    }
   ],
   "source": [
    "prediction_models['naive'] = pipeline_naive_classifier.predict(df_test)\n",
    "\n",
    "prediction_models['baseline_ml'] = pipeline_baseline_ml.predict(df_test)\n",
    "\n",
    "prediction_models['random_forest'] = pipeline_random_forest.predict(df_test)\n",
    "\n",
    "prediction_models['deep_learning'] = pipeline_deep_learning_baseline.predict(df_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a065c492",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "01c30f22",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "be352c64",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------\n",
      "naive\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.00      0.00      0.00      1136\n",
      "           1       0.67      1.00      0.80      8089\n",
      "           2       0.00      0.00      0.00      2912\n",
      "\n",
      "    accuracy                           0.67     12137\n",
      "   macro avg       0.22      0.33      0.27     12137\n",
      "weighted avg       0.44      0.67      0.53     12137\n",
      "\n",
      "------------------------------------------------------------\n",
      "baseline_ml\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.75      0.22      0.34      1136\n",
      "           1       0.83      0.94      0.88      8089\n",
      "           2       0.85      0.75      0.80      2912\n",
      "\n",
      "    accuracy                           0.83     12137\n",
      "   macro avg       0.81      0.64      0.67     12137\n",
      "weighted avg       0.83      0.83      0.81     12137\n",
      "\n",
      "------------------------------------------------------------\n",
      "random_forest\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.50      0.66      1136\n",
      "           1       0.90      0.99      0.94      8089\n",
      "           2       0.98      0.89      0.93      2912\n",
      "\n",
      "    accuracy                           0.92     12137\n",
      "   macro avg       0.95      0.79      0.84     12137\n",
      "weighted avg       0.93      0.92      0.91     12137\n",
      "\n",
      "------------------------------------------------------------\n",
      "deep_learning\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.83      0.83      1136\n",
      "           1       0.96      0.96      0.96      8089\n",
      "           2       0.94      0.94      0.94      2912\n",
      "\n",
      "    accuracy                           0.94     12137\n",
      "   macro avg       0.91      0.91      0.91     12137\n",
      "weighted avg       0.94      0.94      0.94     12137\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Antoinegrelety_1/PycharmProjects/NLP-coursework/venv/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/Antoinegrelety_1/PycharmProjects/NLP-coursework/venv/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/Antoinegrelety_1/PycharmProjects/NLP-coursework/venv/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "for k,v in prediction_models.items():\n",
    "    print('------------------------------------------------------------')\n",
    "    print(k)\n",
    "    print(classification_report(y_test, v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "e937b5c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_df_results(y_true, y_pred, documents):\n",
    "    \"Package results into a dataframe for better visualisation\"\n",
    "    result = pd.DataFrame({'true_class':y_true, 'predicted_class':y_pred, 'document':documents})\n",
    "    result['correct'] = (y_true - y_pred) == 0\n",
    "    result['true_class'] = result['true_class'].apply(lambda x: CLASS_INDEX[x])\n",
    "    result['predicted_class'] = result['predicted_class'].apply(lambda x: CLASS_INDEX[x])\n",
    "    return result\n",
    "\n",
    "\n",
    "df_result = make_df_results(y_true=y_test, y_pred=prediction_models['random_forest'], documents=df_test.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "e4273b1b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true_class</th>\n",
       "      <th>predicted_class</th>\n",
       "      <th>document</th>\n",
       "      <th>correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>Monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>favorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>monsieur|x|x|saisir|commission|accès|document|...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sans objet</td>\n",
       "      <td>sans objet</td>\n",
       "      <td>maître|x|conseil|Monsieur|x|saisir|commission|...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>favorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>défavorable</td>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    true_class predicted_class  \\\n",
       "0  défavorable       favorable   \n",
       "1    favorable       favorable   \n",
       "2   sans objet      sans objet   \n",
       "3    favorable       favorable   \n",
       "4  défavorable     défavorable   \n",
       "\n",
       "                                            document  correct  \n",
       "0  Monsieur|x|saisir|commission|accès|document|ad...    False  \n",
       "1  monsieur|x|x|saisir|commission|accès|document|...     True  \n",
       "2  maître|x|conseil|Monsieur|x|saisir|commission|...     True  \n",
       "3  monsieur|x|saisir|commission|accès|document|ad...     True  \n",
       "4  commission|accès|document|administratif|examin...     True  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_result.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "b53c3a25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true_class</th>\n",
       "      <th>predicted_class</th>\n",
       "      <th>document</th>\n",
       "      <th>correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>Monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>Monsieur|x|monsieur|x|Monsieur|x|Monsieur|x|sa...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>Monsieur|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>maître|x|x|saisir|commission|accès|document|ad...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>maître|t.|c.|conseil|Monsieur|b.|saisir|commis...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12004</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>Monsieur|xx|xxx|saisir|commission|accès|docume...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12010</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>maître|x.|conseil|Monsieur|monsieur|x.|saisir|...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12064</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>commission|examiner|séance|décembre|demande|co...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12082</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>maître|x|v.|conseil|boulangerie|cardinet|saisi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12133</th>\n",
       "      <td>défavorable</td>\n",
       "      <td>favorable</td>\n",
       "      <td>commission|accès|document|administratif|examin...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>571 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        true_class predicted_class  \\\n",
       "0      défavorable       favorable   \n",
       "10     défavorable       favorable   \n",
       "11     défavorable       favorable   \n",
       "20     défavorable       favorable   \n",
       "114    défavorable       favorable   \n",
       "...            ...             ...   \n",
       "12004  défavorable       favorable   \n",
       "12010  défavorable       favorable   \n",
       "12064  défavorable       favorable   \n",
       "12082  défavorable       favorable   \n",
       "12133  défavorable       favorable   \n",
       "\n",
       "                                                document  correct  \n",
       "0      Monsieur|x|saisir|commission|accès|document|ad...    False  \n",
       "10     Monsieur|x|monsieur|x|Monsieur|x|Monsieur|x|sa...    False  \n",
       "11     Monsieur|x|saisir|commission|accès|document|ad...    False  \n",
       "20     maître|x|x|saisir|commission|accès|document|ad...    False  \n",
       "114    maître|t.|c.|conseil|Monsieur|b.|saisir|commis...    False  \n",
       "...                                                  ...      ...  \n",
       "12004  Monsieur|xx|xxx|saisir|commission|accès|docume...    False  \n",
       "12010  maître|x.|conseil|Monsieur|monsieur|x.|saisir|...    False  \n",
       "12064  commission|examiner|séance|décembre|demande|co...    False  \n",
       "12082  maître|x|v.|conseil|boulangerie|cardinet|saisi...    False  \n",
       "12133  commission|accès|document|administratif|examin...    False  \n",
       "\n",
       "[571 rows x 4 columns]"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_result[(df_result.correct==False) & (df_result.true_class == 'défavorable')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da133222",
   "metadata": {},
   "source": [
    "Let's analyze which words have led to the wrong category."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940189ca",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Notes\n",
    "\n",
    "### 20/12/22\n",
    "Baseline ML and Random Forest have pretty much the same accuracy !\n",
    "\n",
    "We are probably reaching a complexity ceiling. We won't improve much with more fancy models (e.g. deep learning).\n",
    "\n",
    "Before that, we should fine tune the feature space. Maybe use a better word representation ?\n",
    "\n",
    "To go further:\n",
    "* fine tune the TfIdf embeddings\n",
    "* use to pre-trained word vectors e.g. GloVe\n",
    "\n",
    "\n",
    "### 31/12/22\n",
    "We have largely improved the results by adding a parameter of sublinear TF scaling to TFIDF vectorizer.<br>\n",
    "Now the RF is much better than the Naive Bayes.\n",
    "\n",
    "Let's try a deep learner model with dense layers. Then will we use a deep learner model with pre-trained\n",
    "embeddings (GloVe) as input.\n",
    "\n",
    "\n",
    "### 01/01/23\n",
    "I have build a first DL model with simple dense layer. Then I have added it to the sklearn Pipeline structure.\n",
    "The models are quite comparable to the Random Forest. Globally the recall is better but precision is lower.\n",
    "\n",
    "Next steps:\n",
    "* New Notebook dedicated to more advanced deep learning\n",
    "    * Better embedding space: vectorize words with pre-trained embeddings such as GloVe. And then learn a document\n",
    "    embeddings space with Doc2Vec framework.\n",
    "    * cross validation, printing loss/accuracy over the epochs,...\n",
    "    * trying different model architectures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "938d374f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}