3. Fake News Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4af5236",
   "metadata": {},
   "source": [
    "# Introduction to NLP in Python\n",
    "## Quest 3: Creating your own fake news classifier\n",
    "\n",
    "The rise of social media and the proliferation of digital news sources have made it easier than ever to spread misinformation and fake news. As a result, this prompts the need to be able to distinguish between real and fake news stories. Machine learning techniques offer a promising solution to this, by allowing us to classify news articles based on their content and other features.\n",
    "\n",
    "In this quest, we explore the use of machine learning classification methods to classify news articles as either real or fake. We will analyse the text of the news articles based on the natural language processing (NLP) methods learnt in previous quests, and evaluate the performance of our classifier using a variety of metrics. \n",
    "\n",
    "**Do take note:** Lines that contain underscores are for you to fill in with your code! Do remove the underscores before running the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbe4d3d",
   "metadata": {},
   "source": [
    "##### Part 1: Data Preprocessing using NLP Techniques\n",
    "\n",
    "In this first section, we will first perform some exploratory data analysis on the data provided, and preprocess the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2052f9c",
   "metadata": {},
   "source": [
    "1. Import the necesary libraries and methods. The libraries listed below are the basic data science libraries that you can use, but feel free to import other libraries that you feel will help you along this quest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "dbf12dfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk \n",
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab298ea5",
   "metadata": {},
   "source": [
    "2. Read the dataset from the CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a8d8c26b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"/Users/thomas/Documents/StackUp/NLP Fake News Classifier/fake_news_train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5caffac5",
   "metadata": {},
   "source": [
    "3. After loading the dataset, perform some primary exploratory data analysis to understand the dataset provided. You can use simple pandas methods and attributes such as `head()`, `shape` and `info()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0ffa7b73",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id                                              title              author  \\\n",
      "0   0  House Dem Aide: We Didn’t Even See Comey’s Let...       Darrell Lucus   \n",
      "1   1  FLYNN: Hillary Clinton, Big Woman on Campus - ...     Daniel J. Flynn   \n",
      "2   2                  Why the Truth Might Get You Fired  Consortiumnews.com   \n",
      "3   3  15 Civilians Killed In Single US Airstrike Hav...     Jessica Purkiss   \n",
      "4   4  Iranian woman jailed for fictional unpublished...      Howard Portnoy   \n",
      "\n",
      "                                                text  label  \n",
      "0  House Dem Aide: We Didn’t Even See Comey’s Let...      1  \n",
      "1  Ever get the feeling your life circles the rou...      0  \n",
      "2  Why the Truth Might Get You Fired October 29, ...      1  \n",
      "3  Videos 15 Civilians Killed In Single US Airstr...      1  \n",
      "4  Print \\nAn Iranian woman has been sentenced to...      1  \n",
      "(20800, 5)\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 20800 entries, 0 to 20799\n",
      "Data columns (total 5 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   id      20800 non-null  int64 \n",
      " 1   title   20242 non-null  object\n",
      " 2   author  18843 non-null  object\n",
      " 3   text    20761 non-null  object\n",
      " 4   label   20800 non-null  int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 812.6+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Exploratory data analysis to familiarize yourself with the data\n",
    "print(df.head())\n",
    "print(df.shape)\n",
    "print(df.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db0979e",
   "metadata": {},
   "source": [
    "4. Before proceeding, it is always a good measure to check if null values are present in the dataset or not. \n",
    "\n",
    "If there are null values in the DataFrame, use the `fillna` method to fill the null values with an empty string (i.e. \"\") Remember to specify the `axis=1` parameter to fill the missing values along the columns of the DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "aa589935",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>author</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>House Dem Aide: We Didn’t Even See Comey’s Let...</td>\n",
       "      <td>Darrell Lucus</td>\n",
       "      <td>House Dem Aide: We Didn’t Even See Comey’s Let...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>FLYNN: Hillary Clinton, Big Woman on Campus - ...</td>\n",
       "      <td>Daniel J. Flynn</td>\n",
       "      <td>Ever get the feeling your life circles the rou...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Why the Truth Might Get You Fired</td>\n",
       "      <td>Consortiumnews.com</td>\n",
       "      <td>Why the Truth Might Get You Fired October 29, ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>15 Civilians Killed In Single US Airstrike Hav...</td>\n",
       "      <td>Jessica Purkiss</td>\n",
       "      <td>Videos 15 Civilians Killed In Single US Airstr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Iranian woman jailed for fictional unpublished...</td>\n",
       "      <td>Howard Portnoy</td>\n",
       "      <td>Print \\nAn Iranian woman has been sentenced to...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20795</th>\n",
       "      <td>20795</td>\n",
       "      <td>Rapper T.I.: Trump a ’Poster Child For White S...</td>\n",
       "      <td>Jerome Hudson</td>\n",
       "      <td>Rapper T. I. unloaded on black celebrities who...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20796</th>\n",
       "      <td>20796</td>\n",
       "      <td>N.F.L. Playoffs: Schedule, Matchups and Odds -...</td>\n",
       "      <td>Benjamin Hoffman</td>\n",
       "      <td>When the Green Bay Packers lost to the Washing...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20797</th>\n",
       "      <td>20797</td>\n",
       "      <td>Macy’s Is Said to Receive Takeover Approach by...</td>\n",
       "      <td>Michael J. de la Merced and Rachel Abrams</td>\n",
       "      <td>The Macy’s of today grew from the union of sev...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20798</th>\n",
       "      <td>20798</td>\n",
       "      <td>NATO, Russia To Hold Parallel Exercises In Bal...</td>\n",
       "      <td>Alex Ansary</td>\n",
       "      <td>NATO, Russia To Hold Parallel Exercises In Bal...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20799</th>\n",
       "      <td>20799</td>\n",
       "      <td>What Keeps the F-35 Alive</td>\n",
       "      <td>David Swanson</td>\n",
       "      <td>David Swanson is an author, activist, journa...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20800 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                              title  \\\n",
       "0          0  House Dem Aide: We Didn’t Even See Comey’s Let...   \n",
       "1          1  FLYNN: Hillary Clinton, Big Woman on Campus - ...   \n",
       "2          2                  Why the Truth Might Get You Fired   \n",
       "3          3  15 Civilians Killed In Single US Airstrike Hav...   \n",
       "4          4  Iranian woman jailed for fictional unpublished...   \n",
       "...      ...                                                ...   \n",
       "20795  20795  Rapper T.I.: Trump a ’Poster Child For White S...   \n",
       "20796  20796  N.F.L. Playoffs: Schedule, Matchups and Odds -...   \n",
       "20797  20797  Macy’s Is Said to Receive Takeover Approach by...   \n",
       "20798  20798  NATO, Russia To Hold Parallel Exercises In Bal...   \n",
       "20799  20799                          What Keeps the F-35 Alive   \n",
       "\n",
       "                                          author  \\\n",
       "0                                  Darrell Lucus   \n",
       "1                                Daniel J. Flynn   \n",
       "2                             Consortiumnews.com   \n",
       "3                                Jessica Purkiss   \n",
       "4                                 Howard Portnoy   \n",
       "...                                          ...   \n",
       "20795                              Jerome Hudson   \n",
       "20796                           Benjamin Hoffman   \n",
       "20797  Michael J. de la Merced and Rachel Abrams   \n",
       "20798                                Alex Ansary   \n",
       "20799                              David Swanson   \n",
       "\n",
       "                                                    text label  \n",
       "0      House Dem Aide: We Didn’t Even See Comey’s Let...     1  \n",
       "1      Ever get the feeling your life circles the rou...     0  \n",
       "2      Why the Truth Might Get You Fired October 29, ...     1  \n",
       "3      Videos 15 Civilians Killed In Single US Airstr...     1  \n",
       "4      Print \\nAn Iranian woman has been sentenced to...     1  \n",
       "...                                                  ...   ...  \n",
       "20795  Rapper T. I. unloaded on black celebrities who...     0  \n",
       "20796  When the Green Bay Packers lost to the Washing...     0  \n",
       "20797  The Macy’s of today grew from the union of sev...     0  \n",
       "20798  NATO, Russia To Hold Parallel Exercises In Bal...     1  \n",
       "20799    David Swanson is an author, activist, journa...     1  \n",
       "\n",
       "[20800 rows x 5 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for null values and if any, fill them with an empty string \n",
    "df.isnull().sum()\n",
    "df.fillna(value=\"\", axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "6b65fea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text'] = df['text'].astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca23c16",
   "metadata": {},
   "source": [
    "5. For data preprocessing, we will focus on the 'text' column of the DataFrame, which contains the content of each news article. We will apply tokenization, the first text preprocessing method covered in Quest 1.\n",
    "\n",
    "Since the dataset has over 20,000 rows of data, it might take a while for the tokenization to finish running, depending on your machine specifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9e14b15c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function to tokenize the text given\n",
    "def tokenize_text(text):\n",
    "    return word_tokenize(text)\n",
    "\n",
    "# Apply the tokenize_text function to the 'text' column of the DataFrame and create a new column 'tokenized_text'\n",
    "df['tokenized_text'] = df[\"text\"].apply(tokenize_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9b0fb96",
   "metadata": {},
   "source": [
    "6. With our new column containing the tokens of the text, we will dive into the second preprocessing step, which is to remove the stop words from the tokens.\n",
    "\n",
    "Create a list `stop_words` that contains the NLTK predefined stopwords. Recall that the stopwords module was imported in the first cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ec139f81",
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = set(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b13f3e2",
   "metadata": {},
   "source": [
    "7. Define a function that removes stop words from a list of tokens. Take note that the NLTK predefined stopwords are in lowercase, while some of the tokens in your current DataFrame contain uppercase alphabets. \n",
    "\n",
    "Apply the method you have defined to the `tokenized_text` column. You can choose to create a new column for these tokens without stopwords, or replace the `tokenized_text` column entirely to only contain tokens without stopwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e6e06c12",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function to remove stopwords from a list of tokens\n",
    "def remove_stopwords(tokens):\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    tokens_without_stopwords = [word for word in tokens if word.lower() not in stop_words]     # replace and fill in underscores\n",
    "    return tokens_without_stopwords\n",
    "\n",
    "# Apply the remove_stopwords function to the 'tokenized_text' column\n",
    "df['stopwords_removed'] = df['tokenized_text'].apply(remove_stopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f0083e9",
   "metadata": {},
   "source": [
    "##### Part 2: Separating the dataset and Vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b3fba8",
   "metadata": {},
   "source": [
    "8. Before we proceed, we will separate the dataset into features and targets. This allows us to clearly define the inputs and outputs of our model. \n",
    "\n",
    "The features are the independent variables that we use to predict the target variable, which is the dependent variable we want to predict. In this case, we are using the text from the article to determine if the article is reliable or unreliable. Reliable articles are labelled '0' in the `label` column while unreliable articles are labelled '1'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "8928d91f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Separate the data into features and targets\n",
    "X_df = df['tokenized_text']\n",
    "y_df = df['label']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8270e470",
   "metadata": {},
   "source": [
    "9. Analyse the `y_df` data. Notice that the data type appears to be a `str` or `object` data type. \n",
    "\n",
    "Recall that we are looking to have a binary output which should only include numerical values. To train the model, we need to convert the label column into a numerical one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "70162a20",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_df = y_df.astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b360c894",
   "metadata": {},
   "source": [
    "10. As machine learning models take in numerical values for their inputs, we have to convert our feature data into numerical format as well. This is where we can incorporate our vectorization skills covered in Quest 2!\n",
    "\n",
    "Import the `TfidfVectorizer` and create a TfidfVectorizer object. Since the features we are working with are in tokens, we have to specify this in the parameter as the vectorizer takes in strings by default. \n",
    "\n",
    "We set the `tokenizer` parameter to a lambda function that simply returns each document as-is. We also set `lowercase=False` to ensure that the tokenization is not modified.\n",
    "\n",
    "After this, fit and transform the vectorizer on the tokenized documents `x_df`. This produces the TFIDF matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e065ab89",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:528: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Perform vectorization using the TFIDF Vectorizer and fit and transform the tokenized documents\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidf_vectorizer = TfidfVectorizer(tokenizer=lambda doc: doc, lowercase=False)\n",
    "tfidf = tfidf_vectorizer.fit_transform(X_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1569c369",
   "metadata": {},
   "source": [
    "##### Part 3: Training and testing the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bec5c74",
   "metadata": {},
   "source": [
    "11. Excellent! For the last part of this quest, we will be making use of a `LogisticRegression` model to create our fake news classifier. Logistic Regression has been covered in previous campaigns, but if you are new to this, don't worry as it is a relatively simple model to pick up. You can find out more about the different methods and attributes of Logistic Regression [here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html).\n",
    "\n",
    "We will split the data into a training set and a testing set to evaluate the performance of our Logistic Regression model. The training set is used to train the model, while the testing set is used to evaluate the model's performance on new data that is has not seen before. This helps us to determine how well the model will generalize to new data and avoid overfitting. \n",
    "\n",
    "This is important because the ultimate goal of a machine learning model is to make accurate predictions on new, real-world data. Without a testing set, we would have no way to evaluate the performance of our model on new data. \n",
    "\n",
    "Now with our TFIDF matrix and target data, we can split the data into testing and training sets using `train_test_split`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "002fa916",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into test and train\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(tfidf, y_df, random_state=0)     # replace and fill in underscores     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "83974285",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<15600x257154 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 5687445 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bac8fff",
   "metadata": {},
   "source": [
    "12. Import the necessary modules and create a LogisticRegression object. Fit the model according to the X and y training data produced above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8254cf81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)     # replace and fill in underscores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87090ab2",
   "metadata": {},
   "source": [
    "13. Now that the model has been trained, obtain the predictions of the model using the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e6a75376",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d2333ef",
   "metadata": {},
   "source": [
    "14. Now we need to evaluate how well the model did. Here, we use three evaluation metrics to assess the performance of our model. These are the metrics we will be working with:\n",
    "\n",
    "+ **Accuracy**: Accuracy is the proportion of correct predictions made by the model out of all the predictions made. It is calculated as the ratio of the number of correct predictions to the total number of predictions.\n",
    "+ **Precision**: Precision is the proportion of true positives out of all the positive predictions made by the model. It is calculated as the ratio of the number of true positives to the total number of positive predictions.\n",
    "+ **Recall**: Recall is the proportion of true positives out of all the actual positive cases in the dataset. It is calculated as the ratio of the number of true positives to the total number of actual positive cases.\n",
    "\n",
    "Import the following metrics and calculate the scores by comparing the test targets to the predicted targets of the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "334f02c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "precision = precision_score(y_test, y_pred)\n",
    "recall = recall_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9ce86b",
   "metadata": {},
   "source": [
    "In machine learning, multiple metrics for evaluation is typically used as a single metric may not provide a complete picture of the model's performance. Different metrics capture different aspects of model performance, and evaluating a model using multiple metrics helps to provide a better understanding of how well the model is performing.\n",
    "\n",
    "For example, a model with high accuracy may have poor performance for a specific class, or may be overfitting the training data. In this case, we may need to look at other metrics, such as precision and recall, to better understand the model's performance. Similarly, a model with high precision may have low recall, indicating that it is good at identifying positive cases, but is missing some of the actual positive cases.\n",
    "\n",
    "By using multiple metrics for evaluation, we can identify the strengths and weaknesses of the model to make informed decisions about how to improve its performance. It is important to choose evaluation metrics that are relevant to the data you are dealing with, and consider the trade-offs between different metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f25aaca2",
   "metadata": {},
   "source": [
    "15. Print out each of the scores for your model below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a782fbf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy score is  0.9659615384615384\n",
      "The precision score is  0.9705319556065825\n",
      "The recall score is  0.9620637329286799\n"
     ]
    }
   ],
   "source": [
    "print(\"The accuracy score is \", accuracy)\n",
    "print(\"The precision score is \", precision)\n",
    "print(\"The recall score is \", recall)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81e38df",
   "metadata": {},
   "source": [
    "Congratulations! You have made it to the end of the quest! Now you have learnt to use your various NLP skills in machine learning projects.\n",
    "\n",
    "**Head back to the StackUp platform** to view the instructions for your submission."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}