Assignment-1/programming/spamassassin/spam-bayes-multinomial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Класифікація спаму: Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Мультиноміальна модель подій**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ми застосуємо наївний Баєсівський класифікатор зі згладжуванням Лапласа для навчання спам-фільтру на основі даних [SpamAssassin Public Corpus](http://spamassassin.apache.org/publiccorpus/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заповніть пропущений код (позначено коментарями) та визначте точність передбачення. У вас повинен вийти кращий результат, ніж при моделі багатовимірного розподілу Бернуллі."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tqdm import tqdm_notebook as progressbar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Завантаження даних"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Щоб краще зрозуміти, яким чином були очищені дані, див. [`spam-data-preparation.ipynb`](spam-data-preparation.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_json_from_file(filename):\n",
    "    with open(filename, \"r\", encoding=\"utf-8\") as f:\n",
    "        return json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "emails_tokenized_ham = load_json_from_file(\"emails-tokenized-ham.json\")\n",
    "emails_tokenized_spam = load_json_from_file(\"emails-tokenized-spam.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = load_json_from_file(\"vocab.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Кодування даних"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Представте кожен лист як $n$-вимірний вектор $\\left[ x_1, x_2, ..., x_n \\right]$, де $x_i$ — це індекс $i$-го слова даного листа у словнику $V$, а $n$ — кількість слів у листі.\n",
    "\n",
    "Наприклад, лист _\"Buy gold watches. Buy now.\"_ міг би бути закодований так: $\\left[ 3953, 11890, 32213, 3953, 20330 \\right]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def email_to_vector_multinomial(email_words, vocab):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Build a feature vector for a single email using the\n",
    "    # multinomial event model.\n",
    "    return np.array(list(map(vocab.get, email_words)))\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Тепер закодуємо всі листи:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = [\n",
    "    email_to_vector_multinomial(email, vocab)\n",
    "    for email in emails_tokenized_ham + emails_tokenized_spam\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.array([0] * len(emails_tokenized_ham) + \\\n",
    "             [1] * len(emails_tokenized_spam))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Поглянемо на кілька випадкових листів:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_emails = [emails_tokenized_ham[10], emails_tokenized_ham[70]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello', 'seen', 'discuss', 'articl', 'approach', 'thank', 'httpaddress', 'hell', 'rule', 'tri', 'accomplish', 'someth', 'thoma', 'alva', 'edison', 'sf', 'net', 'email', 'sponsor', 'osdn', 'tire', 'old', 'cell', 'phone', 'get', 'new', 'free', 'httpaddress', 'spamassassin', 'devel', 'mail', 'list', 'emailaddress', 'httpaddress']\n",
      "\n",
      "['fri', 'number', 'aug', 'number', 'tom', 'wrote', 'xvid', 'number', 'project', 'make', 'gpl', 'divx', 'codec', 'sigma', 'design', 'number', 'sorri', 'sigma', 'design', 'number', 'number', 'httpaddress']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for email in sample_emails:\n",
    "    print(email)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Email vector: [12866 26186  7632  1574  1361 29410 13468 12862 25408 30214   173 27396\n",
      " 29564   872  8567 26411 19758  8849 27722 21116 29770 20748  4549 22215\n",
      " 11528 19855 10871 13468 27540  7314 17535 16947  8851 13468]\n",
      "Dimensionality: (34,)\n",
      "\n",
      "Email vector: [10946 20419  1845 20419 29891 33008 33256 20419 23298 17594 12021  7779\n",
      "  5370 26756  7232 20419 27443 26756  7232 20419 20419 13468]\n",
      "Dimensionality: (22,)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for email in sample_emails:\n",
    "    email_vec = email_to_vector_multinomial(email, vocab)\n",
    "    \n",
    "    print(\"Email vector:\", email_vec)\n",
    "    print(\"Dimensionality:\", email_vec.shape)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Розділення вибірок"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1,\n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Train: 4987\n",
      "# Test:  555\n"
     ]
    }
   ],
   "source": [
    "print(\"# Train:\", len(X_train))\n",
    "print(\"# Test: \", len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Навчання наївного Баєсового класифікатора"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Підрахуйте сумарну кількість слів у ham- і spam-листах відповідно."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from operator import itemgetter \n",
    "# =============== TODO: Your code here ===============\n",
    "# Count the total number of words in ham and spam emails.\n",
    "# Store these counts into the two variables defined below.\n",
    "ham_indices = np.where(y_train == 0)[0]\n",
    "spam_indices = np.where(y_train == 1)[0]\n",
    "X_train_ham = itemgetter(*ham_indices)(X_train)\n",
    "X_train_spam = itemgetter(*spam_indices)(X_train)\n",
    "ham_total_words_train = len(np.hstack(X_train_ham))\n",
    "spam_total_words_train = len(np.hstack(X_train_spam))\n",
    "# ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Тепер обчисліть апріорні імовірності для класів ham і spam. Зауважте, що добуток імовірностей може переповнити тип даних змінної, тому ми будемо використовувати логарифми."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# =============== TODO: Your code here ===============\n",
    "# Compute the class priors for ham and spam emails.\n",
    "ham_log_prior = np.log(np.sum(y_train == 0) / len(y_train))\n",
    "spam_log_prior = np.log(np.sum(y_train == 1) / len(y_train))\n",
    "# total_words_train = ham_total_words_train + spam_total_words_train\n",
    "# ham_log_prior = np.log(ham_total_words_train / total_words_train)\n",
    "# spam_log_prior = np.log(spam_total_words_train / total_words_train)\n",
    "# ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обчисліть правдоподібності (likelihood) для кожного слова. Також, застосуйте згладжування Лапласа, щоб уникнути ділення на нуль."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Створимо порожні вектори $\\log{\\phi_{word \\, | \\, ham}}$ та $\\log{\\phi_{word \\, | \\, spam}}$ і заповнимо їх для кожного слова зі словника."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "ham_log_phi = np.zeros(len(vocab), dtype=\"float64\")\n",
    "spam_log_phi = np.zeros(len(vocab), dtype=\"float64\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "ham_word_counts = np.zeros(len(vocab))\n",
    "spam_word_counts = np.zeros(len(vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5c9e585ae254c14be2066ebc2db83f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d24f52a2ce25409598ace34d758ea3c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm_notebook as tqdm\n",
    "from collections import defaultdict\n",
    "# =============== TODO: Your code here ===============\n",
    "# Compute log phi(word | class) for each word in the vocabulary.\n",
    "# Fill out the `ham_log_phi` and `spam_log_phi` arrays below.\n",
    "V = len(vocab)\n",
    "alpha = 1e-8\n",
    "\n",
    "X_train_ham_freq = []\n",
    "X_train_ham_n = []\n",
    "for email in X_train_ham:\n",
    "    ham_freq = defaultdict(int)\n",
    "    for w in email:\n",
    "        ham_freq[w] += 1\n",
    "    X_train_ham_freq.append(dict(ham_freq))\n",
    "    X_train_ham_n.append(len(email))\n",
    "        \n",
    "X_train_spam_freq = []\n",
    "X_train_spam_n = []\n",
    "for email in X_train_spam:\n",
    "    spam_freq = defaultdict(int)\n",
    "    for w in email:\n",
    "        spam_freq[w] += 1\n",
    "    X_train_spam_freq.append(dict(spam_freq))\n",
    "    X_train_spam_n.append(len(email))\n",
    "\n",
    "for ham_freq, n in tqdm(zip(X_train_ham_freq, X_train_ham_n)):\n",
    "    ham_word_counts[list(ham_freq.keys())] += \\\n",
    "                    np.array(list(ham_freq.values())) / n\n",
    "\n",
    "for spam_freq, n in tqdm(zip(X_train_spam_freq, X_train_spam_n)):\n",
    "    spam_word_counts[list(spam_freq.keys())] += \\\n",
    "                    np.array(list(spam_freq.values())) / n\n",
    "    \n",
    "ham_log_phi = np.log((ham_word_counts + alpha) / \\\n",
    "                     (np.sum(y_train == 0) + alpha*V))\n",
    "spam_log_phi = np.log((spam_word_counts + alpha) / \\\n",
    "                      (np.sum(y_train == 1) + alpha*V)) \n",
    "# ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Передбачення"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте функцію передбачення. Пригадайте, що знаменник $P(words)$ — один і той самий для обох класів, тому для передбачення його можна проігнорувати."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Implement the prediction of target classes, given\n",
    "    # a feature dataset X. You should return a response\n",
    "    # vector containing n {0, 1} values, where n is the\n",
    "    # number of examples in X.\n",
    "    y_pred = []\n",
    "    for email in X:\n",
    "        log_prob_ham = 0\n",
    "        log_prob_spam = 0\n",
    "        for w in email:\n",
    "            log_prob_ham += ham_log_phi[w]\n",
    "            log_prob_spam += spam_log_phi[w]\n",
    "        log_prob_ham += ham_log_prior\n",
    "        log_prob_spam += spam_log_prior\n",
    "        pred = 1 if log_prob_spam > log_prob_ham else 0\n",
    "        y_pred.append(pred)\n",
    "    return y_pred\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Оцінка точності передбачення"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_train = predict(X_train)\n",
    "pred_test = predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_train = 1 - np.sum(pred_train != y_train) / len(y_train)\n",
    "accuracy_test = 1 - np.sum(pred_test != y_test) / len(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy:   98.276%\n",
      "Test accuracy:       97.477%\n"
     ]
    }
   ],
   "source": [
    "print(\"Training accuracy:   {0:.3f}%\".format(accuracy_train * 100))\n",
    "print(\"Test accuracy:       {0:.3f}%\".format(accuracy_test * 100))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}