data_centric_model_centric/Lab - Data-Centric AI vs Model-Centric AI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e5916b2",
   "metadata": {},
   "source": [
    "# Lab: Data-Centric vs Model-Centric approaches\n",
    "\n",
    "This lab gives an introduction to data-centric vs model-centric approaches to machine learning problems, showing how data-centric approaches can outperform purely model-centric approaches.\n",
    "\n",
    "In this lab, we'll build a classifier for product reviews (restricted to the magazine category), like:\n",
    "\n",
    "> Excellent! I look forward to every issue. I had no idea just how much I didn't know.  The letters from the subscribers are educational, too.\n",
    "\n",
    "Label: ⭐️⭐️⭐️⭐️⭐️ (good)\n",
    "\n",
    "> My son waited and waited, it took the 6 weeks to get delivered that they said it would but when it got here he was so dissapointed, it only took him a few minutes to read it.\n",
    "\n",
    "Label: ⭐️ (bad)\n",
    "\n",
    "We'll work with a dataset that has some issues, and we'll see how we can squeeze only so much performance out of the model by being clever about model choice, searching for better hyperparameters, etc. Then, we'll take a look at the data (as any good data scientist should), develop an understanding of the issues, and use simple approaches to improve the data. Finally, we'll see how improving the data can improve results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5874cd",
   "metadata": {},
   "source": [
    "## Installing software\n",
    "\n",
    "For this lab, you'll need to install [scikit-learn](https://scikit-learn.org/) and [pandas](https://pandas.pydata.org/). If you don't have them installed already, you can install them by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3e0ee93",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install scikit-learn pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27dbfce6",
   "metadata": {},
   "source": [
    "# Loading the data\n",
    "\n",
    "First, let's load the train/test sets and take a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7405df2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5a633542",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>405</th>\n",
       "      <td>My husband is really enjoying his subscription...</td>\n",
       "      <td>good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>my favorite magazine for a really great price</td>\n",
       "      <td>good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>We love this magazine.\\nGreat choice.\\nMore fo...</td>\n",
       "      <td>good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>527</th>\n",
       "      <td>far too many adds  not enought info</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>821</th>\n",
       "      <td>It you live in Canada be warned,magazine is al...</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                review label\n",
       "405  My husband is really enjoying his subscription...  good\n",
       "281      my favorite magazine for a really great price  good\n",
       "391  We love this magazine.\\nGreat choice.\\nMore fo...  good\n",
       "527                far too many adds  not enought info   bad\n",
       "821  It you live in Canada be warned,magazine is al...   bad"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('reviews_train.csv')\n",
    "test = pd.read_csv('reviews_test.csv')\n",
    "\n",
    "test.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6446a894",
   "metadata": {},
   "source": [
    "# Training a baseline model\n",
    "\n",
    "There are many approaches for training a sequence classification model for text data. In this lab, we're giving you code that mirrors what you find if you look up [how to train a text classifier](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html), where we'll train an SVM on [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) features (numeric representations of each text field based on word occurrences)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "26e13e03",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "afcb7dbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd_clf = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier()),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "14a09cc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = sgd_clf.fit(train['review'], train['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8885717d",
   "metadata": {},
   "source": [
    "## Evaluating model accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2850df30",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "60677a08",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(clf):\n",
    "    pred = clf.predict(test['review'])\n",
    "    acc = metrics.accuracy_score(test['label'], pred)\n",
    "    print(f'Accuracy: {100*acc:.1f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f77729fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 76.5%\n"
     ]
    }
   ],
   "source": [
    "evaluate(sgd_clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3880fa",
   "metadata": {},
   "source": [
    "## Trying another model\n",
    "\n",
    "76% accuracy is not great for this binary classification problem. Can you do better with a different model, or by tuning hyperparameters for the SVM trained with SGD?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70bdf2c7",
   "metadata": {},
   "source": [
    "# Exercise 1\n",
    "\n",
    "Can you train a more accurate model on the dataset (without changing the dataset)? You might find this [scikit-learn classifier comparison](https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html) handy, as well as the [documentation for supervised learning in scikit-learn](https://scikit-learn.org/stable/supervised_learning.html).\n",
    "\n",
    "One idea for a model you could try is a [naive Bayes classifier](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html).\n",
    "\n",
    "You could also try experimenting with different values of the model hyperparameters, perhaps tuning them via a [grid search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html). \n",
    "\n",
    "Or you can even try training multiple different models and [ensembling their predictions](https://scikit-learn.org/stable/modules/ensemble.html#voting-classifier), a strategy often used to win prediction competitions like Kaggle.\n",
    "\n",
    "**Advanced:** If you want to be more ambitious, you could try an even fancier model, like training a Transformer neural network. If you go with that, you'll want to fine-tune a pre-trained model. This [guide from HuggingFace](https://huggingface.co/docs/transformers/training) may be helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ca681e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "\n",
    "# evaluate your model and see if it does better\n",
    "# than the ones we provided"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a530ee43",
   "metadata": {},
   "source": [
    "## Taking a closer look at the training data\n",
    "\n",
    "Let's actually take a look at some of the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ab34483c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Based on all the negative comments about Taste...</td>\n",
       "      <td>good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I still have not received this.  Obviously I c...</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;/tr&gt;The magazine is not worth the cost of sub...</td>\n",
       "      <td>good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>This magazine is basically ads. Kindve worthle...</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The only thing I've recieved, so far, is the b...</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review label\n",
       "0  Based on all the negative comments about Taste...  good\n",
       "1  I still have not received this.  Obviously I c...   bad\n",
       "2  </tr>The magazine is not worth the cost of sub...  good\n",
       "3  This magazine is basically ads. Kindve worthle...   bad\n",
       "4  The only thing I've recieved, so far, is the b...   bad"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3c84e2",
   "metadata": {},
   "source": [
    "Zooming in on one particular data point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6ebf3a84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'review': \"Based on all the negative comments about Taste of Home, I will not subscribeto the magazine. In the past it was a great read.\\nSorry it, too, has gone the 'way of the wind'.<br>o-p28pass4 </br>\", 'label': 'good'}\n"
     ]
    }
   ],
   "source": [
    "print(train.iloc[0].to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b839b33f",
   "metadata": {},
   "source": [
    "This data point is labeled \"good\", but it's clearly a negative review. Also, it looks like there's some funny HTML stuff at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e608bbc6",
   "metadata": {},
   "source": [
    "# Exercise 2\n",
    "\n",
    "Take a look at some more examples in the dataset. Do you notice any patterns with bad data points?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82e43ff7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae38448f",
   "metadata": {},
   "source": [
    "## Issues in the data\n",
    "\n",
    "It looks like there's some funny HTML tags in our dataset, and those datapoints have nonsense labels. Maybe this dataset was collected by scraping the internet, and the HTML wasn't quite parsed correctly in all cases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "664845e4",
   "metadata": {},
   "source": [
    "# Exercise 3\n",
    "\n",
    "To address this, a simple approach we might try is to throw out the bad data points, and train our model on only the \"clean\" data.\n",
    "\n",
    "Come up with a simple heuristic to identify data points containing HTML, and filter out the bad data points to create a cleaned training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5990cdcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_bad_data(review: str) -> bool:\n",
    "    # YOUR CODE HERE\n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "092849c1",
   "metadata": {},
   "source": [
    "## Creating the cleaned training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c7671e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_clean = train[~train['review'].map(is_bad_data)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1740bf3b",
   "metadata": {},
   "source": [
    "## Evaluating a model trained on the clean training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e83abad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import clone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5e3c7d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd_clf_clean = clone(sgd_clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f72b0db",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = sgd_clf_clean.fit(train_clean['review'], train_clean['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "775ebff4",
   "metadata": {},
   "source": [
    "This model should do significantly better:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80b78100",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate(sgd_clf_clean)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}