Text_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Text-Classification.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Shantanu9326/Data-Science-Portfolio/blob/master/Text_Classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgrlWtuEBEf0",
        "colab_type": "text"
      },
      "source": [
        "# Text Classification\n",
        "\n",
        "The **moviereviews2.tsv** dataset contains the text of 6000 movie reviews. 3000 are positive, 3000 are negative, and the text has been preprocessed as a tab-delimited file. As before, labels are given as `pos` and `neg`. \n",
        "\n",
        "We've included 20 reviews that contain either `NaN` data, or have strings made up of whitespace.\n",
        "\n",
        "For more information on this dataset visit http://ai.stanford.edu/~amaas/data/sentiment/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDAWIe2fBEf1",
        "colab_type": "text"
      },
      "source": [
        "### Task #1: Perform imports and load the dataset into a pandas DataFrame\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ziny-LPpB2h6",
        "colab_type": "code",
        "outputId": "d2b41879-c1eb-431a-debb-bead1cccbebc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#Running or Importing .py Files with Google Colab\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o2QKBJhCBEf1",
        "colab_type": "code",
        "outputId": "0d1dbc4f-9e2f-417e-a415-ba4c95ad3fe0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv('/content/drive/My Drive/app/moviereviews2.tsv', sep='\\t')\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>review</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>pos</td>\n",
              "      <td>I loved this movie and will watch it again. Or...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>pos</td>\n",
              "      <td>A warm, touching movie that has a fantasy-like...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>pos</td>\n",
              "      <td>I was not expecting the powerful filmmaking ex...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>neg</td>\n",
              "      <td>This so-called \"documentary\" tries to tell tha...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>pos</td>\n",
              "      <td>This show has been my escape from reality for ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label                                             review\n",
              "0   pos  I loved this movie and will watch it again. Or...\n",
              "1   pos  A warm, touching movie that has a fantasy-like...\n",
              "2   pos  I was not expecting the powerful filmmaking ex...\n",
              "3   neg  This so-called \"documentary\" tries to tell tha...\n",
              "4   pos  This show has been my escape from reality for ..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YEvK3g2dBEf4",
        "colab_type": "text"
      },
      "source": [
        "### Task #2: Check for missing values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z0cbjqfTBEf5",
        "colab_type": "code",
        "outputId": "f4f5326e-5d58-43dc-a676-6a4526d0610e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Check for NaN values:\n",
        "df.isnull().sum()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "label      0\n",
              "review    20\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kq--840lBEf7",
        "colab_type": "code",
        "outputId": "07604200-dd1f-4d3e-bb22-45bbf4fbaa30",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Check for whitespace strings (it's OK if there aren't any!):\n",
        "blanks = []  # start with an empty list\n",
        "\n",
        "for i,lb,rv in df.itertuples():  # iterate over the DataFrame\n",
        "    if type(rv)==str:            # avoid NaN values\n",
        "        if rv.isspace():         # test 'review' for whitespace\n",
        "            blanks.append(i)     # add matching index numbers to the list\n",
        "        \n",
        "len(blanks)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBUB9BGVBEf9",
        "colab_type": "text"
      },
      "source": [
        "### Task #3:  Remove NaN values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "McIVuiwWBEf9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.dropna(inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W9Y827jPBEf_",
        "colab_type": "text"
      },
      "source": [
        "### Task #4: Take a quick look at the `label` column:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QFW8FCuSBEgA",
        "colab_type": "code",
        "outputId": "169ef27c-71c7-4030-cbfc-3bbc4aefb7df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "df['label'].value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pos    2990\n",
              "neg    2990\n",
              "Name: label, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSJiR8BgBEgD",
        "colab_type": "text"
      },
      "source": [
        "### Task #5: Split the data into train & test sets:\n",
        "You may use whatever settings you like. To compare your results to the solution notebook, use `test_size=0.33, random_state=42`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8UTmpopBEgE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X = df['review']\n",
        "y = df['label']\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0iWbjsj6BEgF",
        "colab_type": "text"
      },
      "source": [
        "### Task #6: Build a pipeline to vectorize the date, then train and fit a model\n",
        "You may use whatever model you like. To compare your results to the solution notebook, use `LinearSVC`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "myVrEev0BEgG",
        "colab_type": "code",
        "outputId": "e9f96687-a102-4d85-c1de-96b6c2672ac1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.svm import LinearSVC\n",
        "\n",
        "text_clf = Pipeline([('tfidf', TfidfVectorizer()),\n",
        "                     ('clf', LinearSVC()),\n",
        "])\n",
        "\n",
        "# Feed the training data through the pipeline\n",
        "text_clf.fit(X_train, y_train)  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('tfidf',\n",
              "                 TfidfVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.float64'>,\n",
              "                                 encoding='utf-8', input='content',\n",
              "                                 lowercase=True, max_df=1.0, max_features=None,\n",
              "                                 min_df=1, ngram_range=(1, 1), norm='l2',\n",
              "                                 preprocessor=None, smooth_idf=True,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 sublinear_tf=False,\n",
              "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                                 tokenizer=None, use_idf=True,\n",
              "                                 vocabulary=None)),\n",
              "                ('clf',\n",
              "                 LinearSVC(C=1.0, class_weight=None, dual=True,\n",
              "                           fit_intercept=True, intercept_scaling=1,\n",
              "                           loss='squared_hinge', max_iter=1000,\n",
              "                           multi_class='ovr', penalty='l2', random_state=None,\n",
              "                           tol=0.0001, verbose=0))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1kBBJl-BEgI",
        "colab_type": "text"
      },
      "source": [
        "### Task #7: Run predictions and analyze the results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAFM8Y12BEgJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Form a prediction set\n",
        "predictions = text_clf.predict(X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qjqXt-ssBEgK",
        "colab_type": "code",
        "outputId": "507947a6-d8ec-4514-d933-3b864e80c920",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Report the confusion matrix\n",
        "from sklearn import metrics\n",
        "print(metrics.confusion_matrix(y_test,predictions))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[900  91]\n",
            " [ 63 920]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IxEzexQZBEgM",
        "colab_type": "code",
        "outputId": "7f51b465-f969-42d7-f511-f55bb7cd0f6b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "# Print a classification report\n",
        "print(metrics.classification_report(y_test,predictions))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         neg       0.93      0.91      0.92       991\n",
            "         pos       0.91      0.94      0.92       983\n",
            "\n",
            "    accuracy                           0.92      1974\n",
            "   macro avg       0.92      0.92      0.92      1974\n",
            "weighted avg       0.92      0.92      0.92      1974\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nABEntVJBEgO",
        "colab_type": "code",
        "outputId": "ce1a33b2-c6a6-42ed-d56b-c8469947d8b7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Print the overall accuracy\n",
        "print(metrics.accuracy_score(y_test,predictions))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.9219858156028369\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BMeCl2jABEgQ",
        "colab_type": "text"
      },
      "source": [
        "## Great job!"
      ]
    }
  ]
}