exercise_topicmodeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling Exercise\n",
    "- Stephen W. Thomas\n",
    "- Queen's MMAI 891\n",
    "\n",
    "In this exercise, we will use LDA to create topics from a twitter dataset. Namely, the dataset contains tweets from Donald Trump and Hillary Clinton leading up to the 2016 US presidential election.\n",
    "\n",
    "**Your mission**: Try to create better topics. The main ways to do this are to enhance the preprocessing steps and to tweak the LDA parameters. (See code snippets marked with `## EDIT CODE HERE ##`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "# Make sure the text doesn't get truncated when printed\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read in the Data\n",
    "\n",
    "Let's read in the data and look at its shape and some sample rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['IsRetweet',\n",
       " 'Time',\n",
       " 'Language',\n",
       " 'RetweetCount',\n",
       " 'FavoriteCount',\n",
       " 'Longitude',\n",
       " 'Latitude',\n",
       " 'Author',\n",
       " 'SourceURL',\n",
       " 'Content',\n",
       " 'OriginalAuthor',\n",
       " 'Place']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 6444 entries, 0 to 6443\n",
      "Data columns (total 12 columns):\n",
      "IsRetweet         6444 non-null bool\n",
      "Time              6444 non-null object\n",
      "Language          6444 non-null object\n",
      "RetweetCount      6444 non-null float64\n",
      "FavoriteCount     6444 non-null float64\n",
      "Longitude         12 non-null float64\n",
      "Latitude          12 non-null float64\n",
      "Author            6444 non-null object\n",
      "SourceURL         6444 non-null object\n",
      "Content           6444 non-null object\n",
      "OriginalAuthor    722 non-null object\n",
      "Place             204 non-null object\n",
      "dtypes: bool(1), float64(4), object(7)\n",
      "memory usage: 560.2+ KB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(6444, 12)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IsRetweet</th>\n",
       "      <th>Time</th>\n",
       "      <th>Language</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>FavoriteCount</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Author</th>\n",
       "      <th>SourceURL</th>\n",
       "      <th>Content</th>\n",
       "      <th>OriginalAuthor</th>\n",
       "      <th>Place</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-28 00:22:34</td>\n",
       "      <td>en</td>\n",
       "      <td>218.0</td>\n",
       "      <td>651.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://studio.twitter.com</td>\n",
       "      <td>The question in this election: Who can put the plans into action that will make your life better? https://t.co/XreEY9OicG</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-27 23:45:00</td>\n",
       "      <td>en</td>\n",
       "      <td>2445.0</td>\n",
       "      <td>5308.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>http://twitter.com</td>\n",
       "      <td>Last night, Donald Trump said not paying taxes was \"smart.\" You know what I call it? Unpatriotic. https://t.co/t0xmBfj7zF</td>\n",
       "      <td>timkaine</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-27 23:26:40</td>\n",
       "      <td>en</td>\n",
       "      <td>7834.0</td>\n",
       "      <td>27234.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://about.twitter.com/products/tweetdeck</td>\n",
       "      <td>Couldn't be more proud of @HillaryClinton. Her vision and command during last night's debate showed that she's ready to be our next @POTUS.</td>\n",
       "      <td>POTUS</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-27 23:08:41</td>\n",
       "      <td>en</td>\n",
       "      <td>916.0</td>\n",
       "      <td>2542.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://studio.twitter.com</td>\n",
       "      <td>If we stand together, there's nothing we can't do.   Make sure you're ready to vote: https://t.co/tTgeqxNqYm https://t.co/Q3Ymbb7UNy</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-27 22:30:27</td>\n",
       "      <td>en</td>\n",
       "      <td>859.0</td>\n",
       "      <td>2882.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://about.twitter.com/products/tweetdeck</td>\n",
       "      <td>Both candidates were asked about how they'd confront racial injustice. Only one had a real answer. https://t.co/sjnEokckis</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   IsRetweet                 Time Language  RetweetCount  FavoriteCount  \\\n",
       "0  False      2016-09-28 00:22:34  en       218.0         651.0           \n",
       "1  True       2016-09-27 23:45:00  en       2445.0        5308.0          \n",
       "2  True       2016-09-27 23:26:40  en       7834.0        27234.0         \n",
       "3  False      2016-09-27 23:08:41  en       916.0         2542.0          \n",
       "4  False      2016-09-27 22:30:27  en       859.0         2882.0          \n",
       "\n",
       "   Longitude  Latitude          Author  \\\n",
       "0 NaN        NaN        HillaryClinton   \n",
       "1 NaN        NaN        HillaryClinton   \n",
       "2 NaN        NaN        HillaryClinton   \n",
       "3 NaN        NaN        HillaryClinton   \n",
       "4 NaN        NaN        HillaryClinton   \n",
       "\n",
       "                                      SourceURL  \\\n",
       "0  https://studio.twitter.com                     \n",
       "1  http://twitter.com                             \n",
       "2  https://about.twitter.com/products/tweetdeck   \n",
       "3  https://studio.twitter.com                     \n",
       "4  https://about.twitter.com/products/tweetdeck   \n",
       "\n",
       "                                                                                                                                       Content  \\\n",
       "0  The question in this election: Who can put the plans into action that will make your life better? https://t.co/XreEY9OicG                     \n",
       "1  Last night, Donald Trump said not paying taxes was \"smart.\" You know what I call it? Unpatriotic. https://t.co/t0xmBfj7zF                     \n",
       "2  Couldn't be more proud of @HillaryClinton. Her vision and command during last night's debate showed that she's ready to be our next @POTUS.   \n",
       "3  If we stand together, there's nothing we can't do.   Make sure you're ready to vote: https://t.co/tTgeqxNqYm https://t.co/Q3Ymbb7UNy          \n",
       "4  Both candidates were asked about how they'd confront racial injustice. Only one had a real answer. https://t.co/sjnEokckis                    \n",
       "\n",
       "  OriginalAuthor Place  \n",
       "0  NaN            NaN   \n",
       "1  timkaine       NaN   \n",
       "2  POTUS          NaN   \n",
       "3  NaN            NaN   \n",
       "4  NaN            NaN   "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IsRetweet</th>\n",
       "      <th>Time</th>\n",
       "      <th>Language</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>FavoriteCount</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Author</th>\n",
       "      <th>SourceURL</th>\n",
       "      <th>Content</th>\n",
       "      <th>OriginalAuthor</th>\n",
       "      <th>Place</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6439</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:47:14</td>\n",
       "      <td>en</td>\n",
       "      <td>1110.0</td>\n",
       "      <td>4024.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@lilredfrmkokomo: @realDonaldTrump My Facebook Groups are all voting TRUMP /4000 people! !!\" Great!</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6440</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:44:17</td>\n",
       "      <td>en</td>\n",
       "      <td>855.0</td>\n",
       "      <td>3181.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@marybnall01: @realDonaldTrump watched lowell mass speech. Awesome. Great crowd. Make America Great Again!!!!!!\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6441</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:42:10</td>\n",
       "      <td>en</td>\n",
       "      <td>2315.0</td>\n",
       "      <td>5992.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@ghosthunter_lol: Iowa key endorsement for @realDonaldTrump Can't wait for the Iowa caucus in 4 weeks! #Trump2016 https://t.co/JBfyFrZfFb\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6442</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:39:11</td>\n",
       "      <td>en</td>\n",
       "      <td>1054.0</td>\n",
       "      <td>3258.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@iLoveiDevices: @EdwinRo47796972 @happyjack225 @FoxNews @krauthammer  Minimizing dependency on China is crucial.Only Trump talks about that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6443</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:36:53</td>\n",
       "      <td>en</td>\n",
       "      <td>748.0</td>\n",
       "      <td>2658.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@SalRiccobono: @realDonaldTrump @troyconway Donald get big business back and# MAKE AMERICA GREAT AGAIN FOR 2016\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      IsRetweet                 Time Language  RetweetCount  FavoriteCount  \\\n",
       "6439  False      2016-01-05 03:47:14  en       1110.0        4024.0          \n",
       "6440  False      2016-01-05 03:44:17  en       855.0         3181.0          \n",
       "6441  False      2016-01-05 03:42:10  en       2315.0        5992.0          \n",
       "6442  False      2016-01-05 03:39:11  en       1054.0        3258.0          \n",
       "6443  False      2016-01-05 03:36:53  en       748.0         2658.0          \n",
       "\n",
       "      Longitude  Latitude           Author  \\\n",
       "6439 NaN        NaN        realDonaldTrump   \n",
       "6440 NaN        NaN        realDonaldTrump   \n",
       "6441 NaN        NaN        realDonaldTrump   \n",
       "6442 NaN        NaN        realDonaldTrump   \n",
       "6443 NaN        NaN        realDonaldTrump   \n",
       "\n",
       "                                SourceURL  \\\n",
       "6439  http://twitter.com/download/android   \n",
       "6440  http://twitter.com/download/android   \n",
       "6441  http://twitter.com/download/android   \n",
       "6442  http://twitter.com/download/android   \n",
       "6443  http://twitter.com/download/android   \n",
       "\n",
       "                                                                                                                                           Content  \\\n",
       "6439  \"@lilredfrmkokomo: @realDonaldTrump My Facebook Groups are all voting TRUMP /4000 people! !!\" Great!                                           \n",
       "6440  \"@marybnall01: @realDonaldTrump watched lowell mass speech. Awesome. Great crowd. Make America Great Again!!!!!!\"                              \n",
       "6441  \"@ghosthunter_lol: Iowa key endorsement for @realDonaldTrump Can't wait for the Iowa caucus in 4 weeks! #Trump2016 https://t.co/JBfyFrZfFb\"    \n",
       "6442  \"@iLoveiDevices: @EdwinRo47796972 @happyjack225 @FoxNews @krauthammer  Minimizing dependency on China is crucial.Only Trump talks about that   \n",
       "6443  \"@SalRiccobono: @realDonaldTrump @troyconway Donald get big business back and# MAKE AMERICA GREAT AGAIN FOR 2016\"                              \n",
       "\n",
       "     OriginalAuthor Place  \n",
       "6439  NaN            NaN   \n",
       "6440  NaN            NaN   \n",
       "6441  NaN            NaN   \n",
       "6442  NaN            NaN   \n",
       "6443  NaN            NaN   "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## EDIT CODE HERE ##\n",
    "# Change these paths to where you downloaded the data\n",
    "in_dir = \"C:/Users/st50/Documents/sandbox/data\"\n",
    "out_dir = \"C:/Users/st50/Documents/sandbox/out\"\n",
    "\n",
    "df = pd.read_csv(os.path.join(in_dir, \"election-tweets-2016.csv\"))\n",
    "\n",
    "list(df)\n",
    "df.info()\n",
    "df.shape\n",
    "df.head()\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Preprocessing\n",
    "\n",
    "Here is where the preprocess magic happens. In the below, I've written a function called `preprocess` that does a bunch of standard steps: lower casing, removing puncuation, etc. You can comment some of these steps out, or add more of your own. Up to you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'steve man plan.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'arsonist oddly shaped feet.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "\"@patbatement i'm really hungry, i'd like reservation someplace.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'goooooaaaaaaallllll!!!!!'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import unidecode\n",
    "import re\n",
    "\n",
    "stop_words = set(stopwords.words('english') + stopwords.words('spanish'))\n",
    "\n",
    "lemmer = WordNetLemmatizer()\n",
    "\n",
    "# This functon take as input an entire document, preprocesses each word in the \n",
    "# document, and returns the preprocessed document.\n",
    "def preprocess(x):\n",
    "    # Lower case\n",
    "    x = x.lower()\n",
    "    \n",
    "    # Remove stop words\n",
    "    x = ' '.join([w for w in x.split() if w not in stop_words])\n",
    "    \n",
    "    ## EDIT CODE HERE ##\n",
    "    \n",
    "    \n",
    "    return x\n",
    "\n",
    "# Some test cases of our function\n",
    "preprocess(\"Steve is the man with the plan.\")\n",
    "preprocess(\"The arsonist had oddly shaped feet.\")\n",
    "preprocess(\"@PatBatement I'm not really hungry, but I'd like to have a reservation someplace.\")\n",
    "preprocess(\"GOOOOOAAAAAAALLLLLL!!!!!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's apply the `preprocess` function to the tweets in our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Content_Clean'] = df['Content'].apply(preprocess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look at some of the preprocessed documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IsRetweet</th>\n",
       "      <th>Time</th>\n",
       "      <th>Language</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>FavoriteCount</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Author</th>\n",
       "      <th>SourceURL</th>\n",
       "      <th>Content</th>\n",
       "      <th>OriginalAuthor</th>\n",
       "      <th>Place</th>\n",
       "      <th>Content_Clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-28 00:22:34</td>\n",
       "      <td>en</td>\n",
       "      <td>218.0</td>\n",
       "      <td>651.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://studio.twitter.com</td>\n",
       "      <td>The question in this election: Who can put the plans into action that will make your life better? https://t.co/XreEY9OicG</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>question election: put plans action make life better? https://t.co/xreey9oicg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-27 23:45:00</td>\n",
       "      <td>en</td>\n",
       "      <td>2445.0</td>\n",
       "      <td>5308.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>http://twitter.com</td>\n",
       "      <td>Last night, Donald Trump said not paying taxes was \"smart.\" You know what I call it? Unpatriotic. https://t.co/t0xmBfj7zF</td>\n",
       "      <td>timkaine</td>\n",
       "      <td>NaN</td>\n",
       "      <td>last night, donald trump said paying taxes \"smart.\" know call it? unpatriotic. https://t.co/t0xmbfj7zf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-27 23:26:40</td>\n",
       "      <td>en</td>\n",
       "      <td>7834.0</td>\n",
       "      <td>27234.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://about.twitter.com/products/tweetdeck</td>\n",
       "      <td>Couldn't be more proud of @HillaryClinton. Her vision and command during last night's debate showed that she's ready to be our next @POTUS.</td>\n",
       "      <td>POTUS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>proud @hillaryclinton. vision command last night's debate showed ready next @potus.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-27 23:08:41</td>\n",
       "      <td>en</td>\n",
       "      <td>916.0</td>\n",
       "      <td>2542.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://studio.twitter.com</td>\n",
       "      <td>If we stand together, there's nothing we can't do.   Make sure you're ready to vote: https://t.co/tTgeqxNqYm https://t.co/Q3Ymbb7UNy</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>stand together, there's nothing can't do. make sure ready vote: https://t.co/ttgeqxnqym https://t.co/q3ymbb7uny</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-09-27 22:30:27</td>\n",
       "      <td>en</td>\n",
       "      <td>859.0</td>\n",
       "      <td>2882.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HillaryClinton</td>\n",
       "      <td>https://about.twitter.com/products/tweetdeck</td>\n",
       "      <td>Both candidates were asked about how they'd confront racial injustice. Only one had a real answer. https://t.co/sjnEokckis</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>candidates asked they'd confront racial injustice. one real answer. https://t.co/sjneokckis</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   IsRetweet                 Time Language  RetweetCount  FavoriteCount  \\\n",
       "0  False      2016-09-28 00:22:34  en       218.0         651.0           \n",
       "1  True       2016-09-27 23:45:00  en       2445.0        5308.0          \n",
       "2  True       2016-09-27 23:26:40  en       7834.0        27234.0         \n",
       "3  False      2016-09-27 23:08:41  en       916.0         2542.0          \n",
       "4  False      2016-09-27 22:30:27  en       859.0         2882.0          \n",
       "\n",
       "   Longitude  Latitude          Author  \\\n",
       "0 NaN        NaN        HillaryClinton   \n",
       "1 NaN        NaN        HillaryClinton   \n",
       "2 NaN        NaN        HillaryClinton   \n",
       "3 NaN        NaN        HillaryClinton   \n",
       "4 NaN        NaN        HillaryClinton   \n",
       "\n",
       "                                      SourceURL  \\\n",
       "0  https://studio.twitter.com                     \n",
       "1  http://twitter.com                             \n",
       "2  https://about.twitter.com/products/tweetdeck   \n",
       "3  https://studio.twitter.com                     \n",
       "4  https://about.twitter.com/products/tweetdeck   \n",
       "\n",
       "                                                                                                                                       Content  \\\n",
       "0  The question in this election: Who can put the plans into action that will make your life better? https://t.co/XreEY9OicG                     \n",
       "1  Last night, Donald Trump said not paying taxes was \"smart.\" You know what I call it? Unpatriotic. https://t.co/t0xmBfj7zF                     \n",
       "2  Couldn't be more proud of @HillaryClinton. Her vision and command during last night's debate showed that she's ready to be our next @POTUS.   \n",
       "3  If we stand together, there's nothing we can't do.   Make sure you're ready to vote: https://t.co/tTgeqxNqYm https://t.co/Q3Ymbb7UNy          \n",
       "4  Both candidates were asked about how they'd confront racial injustice. Only one had a real answer. https://t.co/sjnEokckis                    \n",
       "\n",
       "  OriginalAuthor Place  \\\n",
       "0  NaN            NaN    \n",
       "1  timkaine       NaN    \n",
       "2  POTUS          NaN    \n",
       "3  NaN            NaN    \n",
       "4  NaN            NaN    \n",
       "\n",
       "                                                                                                     Content_Clean  \n",
       "0  question election: put plans action make life better? https://t.co/xreey9oicg                                    \n",
       "1  last night, donald trump said paying taxes \"smart.\" know call it? unpatriotic. https://t.co/t0xmbfj7zf           \n",
       "2  proud @hillaryclinton. vision command last night's debate showed ready next @potus.                              \n",
       "3  stand together, there's nothing can't do. make sure ready vote: https://t.co/ttgeqxnqym https://t.co/q3ymbb7uny  \n",
       "4  candidates asked they'd confront racial injustice. one real answer. https://t.co/sjneokckis                      "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IsRetweet</th>\n",
       "      <th>Time</th>\n",
       "      <th>Language</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>FavoriteCount</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Author</th>\n",
       "      <th>SourceURL</th>\n",
       "      <th>Content</th>\n",
       "      <th>OriginalAuthor</th>\n",
       "      <th>Place</th>\n",
       "      <th>Content_Clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6439</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:47:14</td>\n",
       "      <td>en</td>\n",
       "      <td>1110.0</td>\n",
       "      <td>4024.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@lilredfrmkokomo: @realDonaldTrump My Facebook Groups are all voting TRUMP /4000 people! !!\" Great!</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"@lilredfrmkokomo: @realdonaldtrump facebook groups voting trump /4000 people! !!\" great!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6440</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:44:17</td>\n",
       "      <td>en</td>\n",
       "      <td>855.0</td>\n",
       "      <td>3181.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@marybnall01: @realDonaldTrump watched lowell mass speech. Awesome. Great crowd. Make America Great Again!!!!!!\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"@marybnall01: @realdonaldtrump watched lowell mass speech. awesome. great crowd. make america great again!!!!!!\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6441</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:42:10</td>\n",
       "      <td>en</td>\n",
       "      <td>2315.0</td>\n",
       "      <td>5992.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@ghosthunter_lol: Iowa key endorsement for @realDonaldTrump Can't wait for the Iowa caucus in 4 weeks! #Trump2016 https://t.co/JBfyFrZfFb\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"@ghosthunter_lol: iowa key endorsement @realdonaldtrump can't wait iowa caucus 4 weeks! #trump2016 https://t.co/jbfyfrzffb\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6442</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:39:11</td>\n",
       "      <td>en</td>\n",
       "      <td>1054.0</td>\n",
       "      <td>3258.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@iLoveiDevices: @EdwinRo47796972 @happyjack225 @FoxNews @krauthammer  Minimizing dependency on China is crucial.Only Trump talks about that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"@iloveidevices: @edwinro47796972 @happyjack225 @foxnews @krauthammer minimizing dependency china crucial.only trump talks</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6443</th>\n",
       "      <td>False</td>\n",
       "      <td>2016-01-05 03:36:53</td>\n",
       "      <td>en</td>\n",
       "      <td>748.0</td>\n",
       "      <td>2658.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>http://twitter.com/download/android</td>\n",
       "      <td>\"@SalRiccobono: @realDonaldTrump @troyconway Donald get big business back and# MAKE AMERICA GREAT AGAIN FOR 2016\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"@salriccobono: @realdonaldtrump @troyconway donald get big business back and# make america great 2016\"</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      IsRetweet                 Time Language  RetweetCount  FavoriteCount  \\\n",
       "6439  False      2016-01-05 03:47:14  en       1110.0        4024.0          \n",
       "6440  False      2016-01-05 03:44:17  en       855.0         3181.0          \n",
       "6441  False      2016-01-05 03:42:10  en       2315.0        5992.0          \n",
       "6442  False      2016-01-05 03:39:11  en       1054.0        3258.0          \n",
       "6443  False      2016-01-05 03:36:53  en       748.0         2658.0          \n",
       "\n",
       "      Longitude  Latitude           Author  \\\n",
       "6439 NaN        NaN        realDonaldTrump   \n",
       "6440 NaN        NaN        realDonaldTrump   \n",
       "6441 NaN        NaN        realDonaldTrump   \n",
       "6442 NaN        NaN        realDonaldTrump   \n",
       "6443 NaN        NaN        realDonaldTrump   \n",
       "\n",
       "                                SourceURL  \\\n",
       "6439  http://twitter.com/download/android   \n",
       "6440  http://twitter.com/download/android   \n",
       "6441  http://twitter.com/download/android   \n",
       "6442  http://twitter.com/download/android   \n",
       "6443  http://twitter.com/download/android   \n",
       "\n",
       "                                                                                                                                           Content  \\\n",
       "6439  \"@lilredfrmkokomo: @realDonaldTrump My Facebook Groups are all voting TRUMP /4000 people! !!\" Great!                                           \n",
       "6440  \"@marybnall01: @realDonaldTrump watched lowell mass speech. Awesome. Great crowd. Make America Great Again!!!!!!\"                              \n",
       "6441  \"@ghosthunter_lol: Iowa key endorsement for @realDonaldTrump Can't wait for the Iowa caucus in 4 weeks! #Trump2016 https://t.co/JBfyFrZfFb\"    \n",
       "6442  \"@iLoveiDevices: @EdwinRo47796972 @happyjack225 @FoxNews @krauthammer  Minimizing dependency on China is crucial.Only Trump talks about that   \n",
       "6443  \"@SalRiccobono: @realDonaldTrump @troyconway Donald get big business back and# MAKE AMERICA GREAT AGAIN FOR 2016\"                              \n",
       "\n",
       "     OriginalAuthor Place  \\\n",
       "6439  NaN            NaN    \n",
       "6440  NaN            NaN    \n",
       "6441  NaN            NaN    \n",
       "6442  NaN            NaN    \n",
       "6443  NaN            NaN    \n",
       "\n",
       "                                                                                                                     Content_Clean  \n",
       "6439  \"@lilredfrmkokomo: @realdonaldtrump facebook groups voting trump /4000 people! !!\" great!                                     \n",
       "6440  \"@marybnall01: @realdonaldtrump watched lowell mass speech. awesome. great crowd. make america great again!!!!!!\"             \n",
       "6441  \"@ghosthunter_lol: iowa key endorsement @realdonaldtrump can't wait iowa caucus 4 weeks! #trump2016 https://t.co/jbfyfrzffb\"  \n",
       "6442  \"@iloveidevices: @edwinro47796972 @happyjack225 @foxnews @krauthammer minimizing dependency china crucial.only trump talks    \n",
       "6443  \"@salriccobono: @realdonaldtrump @troyconway donald get big business back and# make america great 2016\"                       "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling with Sci-kit Learn\n",
    "\n",
    "In sklearn, before we runb LDA, we need to vectorize the text. The good news is, we have a nice opportunity to remove rare and common words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 510 ms\n",
      "(6444, 1000)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "\n",
    "## EDIT CODE HERE\n",
    "\n",
    "# The number of \"features\" (i.e., tokens) we want to keep in the BOW.\n",
    "no_features = 1000\n",
    "\n",
    "vectorizer = CountVectorizer(max_df=0.99, min_df=0.0001, \n",
    "                                max_features=no_features, ngram_range=[1,3])\n",
    "%time dtm = vectorizer.fit_transform(df['Content_Clean'])\n",
    "print(dtm.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And just for fun, let's look at the top 30 tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrequencyVisualizer(ax=<matplotlib.axes._subplots.AxesSubplot object at 0x000001A0E7A96978>,\n",
       "          color=None,\n",
       "          features=['00', '000', '000 000', '10', '100', '11', '12', '15', '16', '20', '2016', '30', '47246', '50', '7pm', 'able', 'access', 'across', 'act', 'action', 'actually', 'ad', 'address', 'ads', 'afford', 'affordable', 'african', 'african american', 'african americans', 'again', 'again https', 'again... 'you', 'you https', 'you https co', 'you makeamericagreatagain', 'you re', 'young', 'zero', 'zika'],\n",
       "          n=None, orient='h')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 800x550 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from yellowbrick.text import FreqDistVisualizer\n",
    "\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "visualizer = FreqDistVisualizer(features=feature_names, n=30)\n",
    "visualizer.fit(dtm)\n",
    "visualizer.poof()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we run LDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 10.1 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-369992.9272526116"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "330.3561452032795"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "\n",
    "## EDIT CODE HERE ##\n",
    "\n",
    "lda_model = LatentDirichletAllocation(n_components=5,\n",
    "                                      doc_topic_prior=None,\n",
    "                                      topic_word_prior=None,\n",
    "                                      max_iter=20, \n",
    "                                      learning_method='batch', \n",
    "                                      random_state=123,\n",
    "                                      n_jobs=2,\n",
    "                                      verbose=0)\n",
    "%time lda_output = lda_model.fit(dtm)\n",
    "\n",
    "# Log Likelihood: Higher the better\n",
    "lda_model.score(dtm)\n",
    "\n",
    "# Perplexity: Lower the better. Perplexity = exp(-1. * log-likelihood per word)\n",
    "lda_model.perplexity(dtm)\n",
    "\n",
    "# Theta = document-topic matrix\n",
    "# Beta = components_ = topic-term matrix\n",
    "theta = pd.DataFrame(lda_model.transform(dtm))\n",
    "beta = pd.DataFrame(lda_model.components_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Look at the Topics\n",
    "\n",
    "Let's inspect the topics. Let's figure out the top words. Also, let's compute some simple metrics on each topic, like how many documents contain it (_support_) and what the total size of the topic is (_weight_)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TopicID</th>\n",
       "      <th>Support50</th>\n",
       "      <th>Support10</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Terms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1657</td>\n",
       "      <td>2386</td>\n",
       "      <td>1597.632208</td>\n",
       "      <td>co https co https trump donald donald trump hillary he realdonaldtrump it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1353</td>\n",
       "      <td>2416</td>\n",
       "      <td>1492.622506</td>\n",
       "      <td>co https co https thank trump2016 you makeamericagreatagain thank you trump2016 https trump2016 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1223</td>\n",
       "      <td>2450</td>\n",
       "      <td>1373.784758</td>\n",
       "      <td>people cruz amp great big trump realdonaldtrump ted like many</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>894</td>\n",
       "      <td>2387</td>\n",
       "      <td>1174.337207</td>\n",
       "      <td>hillary clinton we hillary clinton crooked crooked hillary re president women together</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>528</td>\n",
       "      <td>1722</td>\n",
       "      <td>805.623322</td>\n",
       "      <td>america great make again president make america first last america great great again</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TopicID                                                 ...                                                                                                                                                 Terms\n",
       "2        2                                                 ...                                                                             co https co https trump donald donald trump hillary he realdonaldtrump it\n",
       "3        3                                                 ...                                                   co https co https thank trump2016 you makeamericagreatagain thank you trump2016 https trump2016 ...\n",
       "0        0                                                 ...                                                                                         people cruz amp great big trump realdonaldtrump ted like many\n",
       "4        4                                                 ...                                                                hillary clinton we hillary clinton crooked crooked hillary re president women together\n",
       "1        1                                                 ...                                                                  america great make again president make america first last america great great again\n",
       "\n",
       "[5 rows x 5 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', 100)\n",
    "pd.set_option('display.max_columns', 0)\n",
    "\n",
    "# How many words to display for each topic\n",
    "no_top_words = 10\n",
    "weight = theta.sum(axis=0)\n",
    "\n",
    "# Number of documents that contain a topic more than 50%\n",
    "support50 = (theta > 0.5).sum(axis=0)\n",
    "\n",
    "# Number of documents that contain a document more than 10%\n",
    "support10 = (theta > 0.1).sum(axis=0)\n",
    "termss = list()\n",
    "for topic_id, topic in enumerate(lda_model.components_):\n",
    "    terms = \" \".join([feature_names[i] for i in topic.argsort()[:-no_top_words - 1:-1]])\n",
    "    termss.append(terms)\n",
    "topic_summary = pd.DataFrame({'TopicID': range(0, len(termss)), \"Support50\": support50, \"Support10\": support10, \"Weight\": weight, \"Terms\": termss})\n",
    "\n",
    "# Display the topics; sort by Weight\n",
    "topic_summary.sort_values('Weight', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show some Documents that Match a Given Topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "260     0.963223\n",
       "1658    0.962588\n",
       "532     0.959687\n",
       "6276    0.959604\n",
       "2716    0.959503\n",
       "Name: 3, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>FavoriteCount</th>\n",
       "      <th>Author</th>\n",
       "      <th>Content</th>\n",
       "      <th>Content_Clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>2016-09-21 22:35:19</td>\n",
       "      <td>10434.0</td>\n",
       "      <td>27350.0</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>Great new polls! Thank you Nevada, North Carolina &amp;amp; Ohio. Join the MOVEMENT today &amp;amp; lets #MAGA!… https://t.co/Y8Sb8MNyXA</td>\n",
       "      <td>great new polls! thank nevada, north carolina &amp;amp; ohio. join movement today &amp;amp; lets #maga!… https://t.co/y8sb8mnyxa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1658</th>\n",
       "      <td>2016-08-04 15:17:01</td>\n",
       "      <td>5814.0</td>\n",
       "      <td>18830.0</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>Looking forward to IA &amp;amp; WI with Gov. Pence, tomorrow. Join us! #MAGA https://t.co/3Hcnzj0Slx https://t.co/sEwLWkn1Sz https://t.co/0Ei3EdQdXB</td>\n",
       "      <td>looking forward ia &amp;amp; wi gov. pence, tomorrow. join us! #maga https://t.co/3hcnzj0slx https://t.co/sewlwkn1sz https://t.co/0ei3edqdxb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>2016-09-15 02:33:39</td>\n",
       "      <td>6649.0</td>\n",
       "      <td>20014.0</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>Great poll out of Nevada- thank you! See you soon. #MAGA #AmericaFirst https://t.co/3KWOl2ibaW https://t.co/27sR3MjjXc</td>\n",
       "      <td>great poll nevada- thank you! see soon. #maga #americafirst https://t.co/3kwol2ibaw https://t.co/27sr3mjjxc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6276</th>\n",
       "      <td>2016-01-18 17:48:53</td>\n",
       "      <td>2088.0</td>\n",
       "      <td>5831.0</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>A great morning with everyone @LibertyU! Thank you! Off to New Hampshire now. #Trump2016  https://t.co/XUWGANbq8k https://t.co/aEMUMqSoWm</td>\n",
       "      <td>great morning everyone @libertyu! thank you! new hampshire now. #trump2016 https://t.co/xuwganbq8k https://t.co/aemumqsowm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2716</th>\n",
       "      <td>2016-07-14 14:53:46</td>\n",
       "      <td>8970.0</td>\n",
       "      <td>23690.0</td>\n",
       "      <td>realDonaldTrump</td>\n",
       "      <td>Another new poll. Thank you for your support! Join the MOVEMENT today!  #ImWithYou https://t.co/3KWOl2ibaW https://t.co/miT4atHxQz</td>\n",
       "      <td>another new poll. thank support! join movement today! #imwithyou https://t.co/3kwol2ibaw https://t.co/mit4athxqz</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Time                                                                    ...                                                                                                                                                                                                Content_Clean\n",
       "260   2016-09-21 22:35:19                                                                    ...                                                                     great new polls! thank nevada, north carolina &amp; ohio. join movement today &amp; lets #maga!… https://t.co/y8sb8mnyxa                \n",
       "1658  2016-08-04 15:17:01                                                                    ...                                                                     looking forward ia &amp; wi gov. pence, tomorrow. join us! #maga https://t.co/3hcnzj0slx https://t.co/sewlwkn1sz https://t.co/0ei3edqdxb\n",
       "532   2016-09-15 02:33:39                                                                    ...                                                                     great poll nevada- thank you! see soon. #maga #americafirst https://t.co/3kwol2ibaw https://t.co/27sr3mjjxc                             \n",
       "6276  2016-01-18 17:48:53                                                                    ...                                                                     great morning everyone @libertyu! thank you! new hampshire now. #trump2016 https://t.co/xuwganbq8k https://t.co/aemumqsowm              \n",
       "2716  2016-07-14 14:53:46                                                                    ...                                                                     another new poll. thank support! join movement today! #imwithyou https://t.co/3kwol2ibaw https://t.co/mit4athxqz                        \n",
       "\n",
       "[5 rows x 6 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make sure the text doesn't get truncated when printed\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "\n",
    "# Which topic are you interested in?\n",
    "## EDIT CODE HERE ##\n",
    "topic_id = 3\n",
    "\n",
    "# Find the 5 documents with the largets membership (i.e., theta) for this topic\n",
    "Memberships = theta.iloc[:, topic_id].nlargest(5)\n",
    "\n",
    "Memberships\n",
    "\n",
    "# Display those documents (and getting rid of some of the columns)\n",
    "df[['Time', 'RetweetCount', 'FavoriteCount', 'Author', 'Content', 'Content_Clean']].iloc[Memberships.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize Topics with LDAVis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 12.1 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\st50\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\pyLDAvis\\_prepare.py:257: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  return pd.concat([default_term_info] + list(topic_dfs))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el2009217906065621927236996363\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el2009217906065621927236996363_data = {\"mdsDat\": {\"x\": [98.77940368652344, -239.0792694091797, -267.7159118652344, 101.72245025634766, 379.63836669921875], \"y\": [144.92453002929688, -216.68690490722656, 161.8026885986328, -349.68621826171875, -91.17501831054688], \"topics\": [1, 2, 3, 4, 5], \"cluster\": [1, 1, 1, 1, 1], \"Freq\": [26.22311190975494, 22.96559312725883, 19.829621895504072, 18.235844925032865, 12.745828142449279]}, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\"], \"Freq\": [3912.0, 3908.0, 3908.0, 1284.0, 555.0, 1120.0, 480.0, 442.0, 539.0, 447.0, 332.0, 341.0, 284.0, 365.0, 249.0, 249.0, 166.0, 205.0, 395.0, 185.0, 144.0, 176.0, 202.0, 138.0, 192.0, 192.0, 417.0, 161.0, 362.0, 120.0, 479.91200122514897, 446.96235997831445, 92.25642593056968, 91.28639147945434, 91.28639147945434, 65.12147982974838, 47.67930282965481, 47.67930282965481, 47.67914899335892, 46.704961900259825, 45.74164968675462, 45.74164968675462, 42.83396448218457, 40.89607712878712, 38.957728642132494, 38.957728642132494, 36.04993028732348, 36.04993028732348, 36.04993028732348, 36.04898585328632, 36.04898585328632, 33.14133606344197, 33.14133606344197, 33.140106418707134, 33.140106418707134, 32.16925575481187, 30.234259004762716, 29.265476260559897, 29.265476260559897, 29.2436981548472, 1037.337516334867, 86.14540727696867, 139.98755912389137, 58.37690417436519, 47.29884108689395, 1665.635231037121, 1665.6352310319892, 1666.5397571009016, 55.6057344096585, 120.36918765986218, 67.81037229943132, 125.333355421574, 54.41779048797506, 99.41160591744178, 63.48601356524104, 133.86100639325852, 204.55268803676435, 81.35209314623098, 112.09546025069687, 94.75331736842085, 72.07417875005699, 93.98704709003287, 76.71463453968688, 64.53532827971054, 60.20120722059616, 61.349206455830476, 58.07523858676141, 340.94872290759656, 248.45768316560762, 201.72500667872333, 191.99170759876392, 191.99170759876392, 89.7646080963374, 84.89680851072438, 84.89680851072438, 78.08195030510416, 71.26069817287527, 67.3659326621061, 66.39459659060013, 65.4214672496866, 58.60179354569265, 51.78172579374638, 51.78172579374636, 546.5621067766828, 47.89667550023851, 36.21648863475156, 36.21648863475156, 36.213374383224576, 36.213374383224505, 35.24092479596276, 35.23241446864447, 34.268904375125295, 34.268904375125295, 30.375010054330534, 30.374698334147933, 30.374698334147933, 30.372138275012855, 30.372138275012855, 134.22459865713518, 56.08090025570216, 287.082422727349, 2244.8919223791195, 2241.906225196914, 2241.9062251837827, 92.71975778271438, 179.5644703769246, 73.53851965376124, 69.04363955838433, 92.39033403769805, 43.605921780332864, 92.57854953928232, 84.41928734635405, 104.69459583627307, 54.24008507849638, 60.685924731916934, 58.8318888434424, 204.79564143139586, 137.24848429050772, 110.81137399812674, 108.85871542610913, 88.29549403883604, 81.44621421747937, 53.05655762468611, 53.05561705496901, 51.095735273214146, 53.9799130548324, 50.118116311899314, 48.16150208550321, 42.28830566090006, 40.32992741683657, 39.35234648991171, 39.348582137422575, 36.41644159636858, 36.406075250383516, 35.42943627755927, 35.428101424162435, 34.45785573979794, 34.45589458492612, 34.45557732421424, 33.47396479235833, 33.473670267212846, 32.494357309178284, 31.520674515362206, 31.519530677433007, 31.517591280405686, 29.557526653046168, 150.5002536137202, 61.12687602556871, 74.0312476816583, 43.65098224448845, 119.8510064977606, 75.44315761779602, 246.6144424569066, 86.86003114952487, 192.74075540205484, 68.81820697217913, 86.21169534849307, 127.81876013455339, 49.30962609619681, 104.04460810789931, 172.6950829072898, 63.580257517958586, 144.92213289572197, 96.67200834248115, 93.1771987676912, 66.74170308897205, 146.5426865236855, 63.46859055417418, 67.6950610048314, 69.81906343382036, 62.945675874743124, 58.7032076198432, 331.41267815914676, 283.6724699238128, 248.8593590048186, 185.20244837944708, 175.25645760914298, 160.3341025516494, 106.62411685312088, 91.70124998266829, 72.79922337764214, 70.8152952868714, 69.81706501374575, 67.83442100765453, 63.8533161801718, 60.86805539442246, 57.88593328459916, 57.883821919759, 48.93196458082947, 45.94852564165758, 41.97209052342557, 41.9673480199862, 41.96552619337925, 40.97380170948802, 38.98838079754764, 38.98274488202902, 37.988660934780135, 37.97797423116887, 35.00976290234764, 35.00822401755716, 33.02078720928734, 33.01877773610847, 715.4456647978797, 89.42474780017362, 115.11604684691477, 91.29827275873669, 114.84922330705108, 119.54479924157951, 100.85954153216441, 81.11681290918752, 70.32604345798481, 108.04496813418638, 97.07600737442196, 87.90593496885016, 133.9245385173056, 70.80708582723967, 86.47583445328166, 81.32852101491609, 78.49211216019593, 75.81000925330626, 76.28989877055197, 165.57962836624424, 143.87694024887077, 137.66656598351895, 120.10403122809193, 119.07052356188782, 116.99157137616346, 111.83530817286984, 99.43214761729261, 78.75638530952722, 74.62097261459938, 72.55148401133692, 69.45338749106428, 59.11994013976262, 49.81187253160892, 46.71172173267413, 38.44153885570436, 35.34804598936911, 35.34804598936911, 31.210298645298874, 29.146580334907693, 28.110609199517132, 28.09875229895254, 27.075469881665096, 27.074182232647683, 26.045029112283103, 26.045029112283103, 26.044777399135597, 26.043053015839117, 25.011354438006894, 25.008500602814607, 109.78551146125862, 338.5954711416357, 121.23522050346284, 36.89088407471347, 44.09065864403237, 227.47329156012893, 305.8020911849014, 75.51159273185729, 40.77524025782525, 149.1313152297173, 42.095817263452936, 48.725109656394686, 106.67690494458101, 77.35420901629125, 62.55592278017491, 99.77521176311456, 57.924050780884876, 56.0600439062331, 84.52097742343177, 47.136949585512454], \"Term\": [\"co\", \"https co\", \"https\", \"trump\", \"thank\", \"hillary\", \"donald\", \"america\", \"great\", \"donald trump\", \"clinton\", \"trump2016\", \"we\", \"make\", \"hillary clinton\", \"makeamericagreatagain\", \"again\", \"cruz\", \"president\", \"crooked\", \"make america\", \"crooked hillary\", \"thank you\", \"first\", \"trump2016 https co\", \"trump2016 https\", \"people\", \"re\", \"you\", \"america great\", \"donald\", \"donald trump\", \"plan\", \"hillary https co\", \"hillary https\", \"rncincle\", \"rncincle https co\", \"rncincle https\", \"chief\", \"make sure\", \"trump https co\", \"trump https\", \"commander\", \"in chief\", \"commander in\", \"commander in chief\", \"ttgeqxnqym\", \"co ttgeqxnqym\", \"https co ttgeqxnqym\", \"it https\", \"it https co\", \"realdonaldtrump https co\", \"realdonaldtrump https\", \"potus https co\", \"potus https\", \"trump says\", \"returns\", \"debatenight https\", \"debatenight https co\", \"share\", \"trump\", \"tax\", \"he\", \"business\", \"debatenight\", \"https co\", \"https\", \"co\", \"sure\", \"would\", \"in\", \"it\", \"best\", \"let\", \"says\", \"realdonaldtrump\", \"hillary\", \"can\", \"president\", \"one\", \"potus\", \"people\", \"like\", \"americans\", \"know\", \"make\", \"vote\", \"trump2016\", \"makeamericagreatagain\", \"thank you\", \"trump2016 https co\", \"trump2016 https\", \"makeamericagreatagain trump2016\", \"makeamericagreatagain https\", \"makeamericagreatagain https co\", \"makeamericagreatagain trump2016 https\", \"york\", \"new york\", \"votetrump\", \"americafirst\", \"maga\", \"president https\", \"president https co\", \"thank\", \"video\", \"imwithyou https\", \"imwithyou https co\", \"americafirst https co\", \"americafirst https\", \"tickets\", \"trumptrain\", \"maga https\", \"maga https co\", \"trump2016 makeamericagreatagain\", \"video https co\", \"video https\", \"you https\", \"you https co\", \"join\", \"imwithyou\", \"you\", \"co\", \"https co\", \"https\", \"hillaryclinton\", \"new\", \"live\", \"tomorrow\", \"watch\", \"new hampshire\", \"today\", \"amp\", \"hillary\", \"poll\", \"great\", \"realdonaldtrump\", \"cruz\", \"ted\", \"cnn\", \"ted cruz\", \"media\", \"rubio\", \"failing\", \"megynkelly\", \"beat\", \"nytimes\", \"dishonest\", \"jeb\", \"marco\", \"lyin\", \"kasich\", \"failed\", \"lyin ted\", \"police\", \"yesterday\", \"votes\", \"marco rubio\", \"ads\", \"ratings\", \"massive\", \"lies\", \"gave\", \"bush\", \"phony\", \"warren\", \"john\", \"big\", \"wow\", \"show\", \"total\", \"many\", \"even\", \"people\", \"jobs\", \"amp\", \"foxnews\", \"me\", \"like\", \"news\", \"vote\", \"great\", \"000\", \"realdonaldtrump\", \"get\", \"would\", \"win\", \"trump\", \"much\", \"never\", \"one\", \"today\", \"said\", \"clinton\", \"we\", \"hillary clinton\", \"crooked\", \"crooked hillary\", \"re\", \"we re\", \"enjoy\", \"wall\", \"rights\", \"care\", \"crooked hillary clinton\", \"interviewed\", \"gun\", \"we ve\", \"violence\", \"health\", \"stronger\", \"re going\", \"president obama\", \"future\", \"child\", \"we re going\", \"children\", \"free\", \"lives\", \"ve got\", \"street\", \"we ve got\", \"wall street\", \"hillary\", \"ve\", \"together\", \"obama\", \"families\", \"women\", \"need\", \"every\", \"work\", \"let\", \"country\", \"going\", \"president\", \"americans\", \"one\", \"it\", \"us\", \"make\", \"people\", \"again\", \"make america\", \"first\", \"america great\", \"great again\", \"job\", \"make america great\", \"america great again\", \"last night\", \"states\", \"party\", \"united\", \"united states\", \"white\", \"house\", \"knows\", \"again https\", \"again https co\", \"years ago\", \"great again https\", \"mr\", \"talks\", \"republican party\", \"democratic\", \"president united\", \"president united states\", \"america safe\", \"white house\", \"make america safe\", \"mitt\", \"night\", \"america\", \"last\", \"become\", \"convention\", \"make\", \"great\", \"republican\", \"ago\", \"president\", \"safe\", \"woman\", \"realdonaldtrump\", \"us\", \"good\", \"trump\", \"time\", \"potus\", \"hillary\", \"years\"], \"Total\": [3912.0, 3908.0, 3908.0, 1284.0, 555.0, 1120.0, 480.0, 442.0, 539.0, 447.0, 332.0, 341.0, 284.0, 365.0, 249.0, 249.0, 166.0, 205.0, 395.0, 185.0, 144.0, 176.0, 202.0, 138.0, 192.0, 192.0, 417.0, 161.0, 362.0, 120.0, 480.7128011336647, 447.76211329272775, 93.05810671673626, 92.0890041477661, 92.0890041477661, 65.92224201073986, 48.47775602352839, 48.47775602352839, 48.47774082585436, 47.50883214714357, 46.53945973468668, 46.53945973468668, 43.63204694871289, 41.69377050179531, 39.755499285816924, 39.755499285816924, 36.848103940041774, 36.848103940041774, 36.848103940041774, 36.84811658624076, 36.84811658624076, 33.94069551422041, 33.94069551422041, 33.94080337727836, 33.94080337727836, 32.97183288152091, 31.033340018974734, 30.064142705047857, 30.064142705047857, 30.06469815354758, 1284.0526954750324, 94.2137204852185, 174.63865936513608, 65.99063629392542, 52.374279707446355, 3908.150693258695, 3908.1506932598454, 3912.0409070706532, 66.1729860747031, 228.0894623217669, 102.58295228361673, 281.50610154017244, 76.59344811591748, 227.78932124312698, 102.0912815878342, 444.4939319961473, 1120.3165180653389, 181.2427006284609, 395.5510153127493, 292.3488400349412, 169.29917963919564, 417.2979655153957, 225.69957012303374, 155.1492991450683, 128.15395292397912, 365.02651961951744, 216.76570391954692, 341.74782182468334, 249.2564268213074, 202.52397165322083, 192.78794724314858, 192.78794724314858, 90.56061629807431, 85.69263674178318, 85.69263674178318, 78.87747862780657, 72.06245336060879, 68.16808764357742, 67.19448280076801, 66.22092217085553, 59.40576516096782, 52.5906268847418, 52.59062688474181, 555.433445891572, 48.69613924072103, 37.0129951064581, 37.0129951064581, 37.01319147544955, 37.01319147544955, 36.03941258534387, 36.03989767752123, 35.06580714518977, 35.06580714518977, 31.171423264252194, 31.171435470524344, 31.171435470524344, 31.171412780342926, 31.171412780342926, 143.31025815049784, 59.56134305916389, 362.69962882115317, 3912.0409070706532, 3908.150693258695, 3908.1506932598454, 113.81076836235341, 275.45266395350524, 92.90108386836226, 87.4677581600109, 138.51526932016176, 51.656695056508134, 203.96221321577076, 355.7484719739253, 1120.3165180653389, 100.54594328648403, 539.5783135846805, 444.4939319961473, 205.59209994712947, 138.04403191935532, 111.61229083997497, 109.6542715950991, 89.09626838679964, 82.24348110028028, 53.85373944780199, 53.85380967744832, 51.89596042933966, 54.8335502038245, 50.9168851900926, 48.958953187221695, 43.08519648895376, 41.127283687078254, 40.14834505697643, 40.14841441906353, 37.21142431308573, 37.21159754044891, 36.23259475302872, 36.23260813558861, 35.25353275773605, 35.25355200774755, 35.253635573831886, 34.27462782879495, 34.2745830028536, 33.29567143583279, 32.31666017063494, 32.31665774289077, 32.316695352513555, 30.35874525559447, 171.41299611267698, 67.52827991293078, 84.10750156449981, 47.042029528868184, 160.69171715649176, 92.84405142206025, 417.2979655153957, 113.98241139574998, 355.7484719739253, 92.3926828625823, 127.48333005014936, 225.69957012303374, 58.70126854885409, 216.76570391954692, 539.5783135846805, 95.90583204609062, 444.4939319961473, 241.50554098639722, 228.0894623217669, 135.3453112099652, 1284.0526954750324, 131.51450330107556, 179.58192304140618, 292.3488400349412, 203.96221321577076, 127.35279679028456, 332.21245909806004, 284.4687273325802, 249.65555377931094, 185.99714577408295, 176.05052365591976, 161.1305741526822, 107.41882645035042, 92.49884049748802, 73.60019806425443, 71.6109502860064, 70.61629810283053, 68.62700415912487, 64.64836410790157, 61.66432325507951, 58.6803569732303, 58.6804134767899, 49.728381935942195, 46.74442386281886, 42.76579429647002, 42.76580534688539, 42.76579011910621, 41.77111584329515, 39.78180567291966, 39.78181850513215, 38.78703299935311, 38.786968775928145, 35.80312313751875, 35.8030993309618, 33.81380755139663, 33.81377223273377, 1120.3165180653389, 108.91581596689882, 146.6976512428597, 114.85012267508425, 156.0867785206692, 183.43264880703683, 157.39368510574144, 124.39629192490787, 102.50831233427733, 227.78932124312698, 199.9833763428733, 168.08675793257785, 395.5510153127493, 155.1492991450683, 292.3488400349412, 281.50610154017244, 239.61724396309654, 365.02651961951744, 417.2979655153957, 166.36726229868444, 144.6616267410121, 138.45950482539166, 120.88861393810832, 119.85500951493829, 117.78719495411832, 112.61974662685023, 100.21644099535887, 79.54405600383326, 75.40958143407146, 73.34225349131182, 70.24157198283218, 59.90559878588283, 50.60281247656558, 47.50203102188375, 39.2330947510526, 36.13263141715523, 36.13263141715523, 31.998037168009105, 29.930986923103955, 28.897247749932696, 28.896578484339347, 27.86356618067566, 27.863513724772798, 26.830133510659994, 26.830133510659994, 26.83010943734675, 26.830009398887853, 25.79650655513034, 25.796358646477245, 121.26308437044644, 442.63485302197165, 151.29610269165175, 40.169243501551406, 50.30030306950105, 365.02651961951744, 539.5783135846805, 105.80433131635169, 51.10397567752067, 395.5510153127493, 57.274672812345145, 77.19238546768511, 444.4939319961473, 239.61724396309654, 150.1369358091191, 1284.0526954750324, 183.79936301312546, 169.29917963919564, 1120.3165180653389, 117.34388906455455], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.3369, 1.3367, 1.3299, 1.3298, 1.3298, 1.3263, 1.3219, 1.3219, 1.3219, 1.3215, 1.3212, 1.3212, 1.3201, 1.3192, 1.3183, 1.3183, 1.3166, 1.3166, 1.3166, 1.3166, 1.3166, 1.3147, 1.3147, 1.3147, 1.3147, 1.3139, 1.3124, 1.3116, 1.3116, 1.3108, 1.1252, 1.249, 1.1174, 1.2159, 1.2366, 0.4857, 0.4857, 0.4852, 1.1645, 0.6994, 0.9246, 0.5294, 0.9967, 0.5094, 0.8635, 0.1384, -0.362, 0.5375, 0.0776, 0.2119, 0.4846, -0.1521, 0.2594, 0.4614, 0.583, -0.4449, 0.0215, 1.4688, 1.468, 1.4672, 1.467, 1.467, 1.4623, 1.4618, 1.4618, 1.461, 1.46, 1.4593, 1.4592, 1.459, 1.4575, 1.4557, 1.4557, 1.4551, 1.4546, 1.4494, 1.4494, 1.4493, 1.4493, 1.4488, 1.4485, 1.4482, 1.4482, 1.4453, 1.4453, 1.4453, 1.4452, 1.4452, 1.4057, 1.411, 1.2374, 0.9158, 0.9154, 0.9154, 1.2662, 1.0433, 1.2374, 1.2346, 1.0662, 1.3017, 0.6813, 0.0327, -0.8991, 0.854, -0.7139, -0.5511, 1.6141, 1.6122, 1.6108, 1.6107, 1.609, 1.6083, 1.6031, 1.6031, 1.6025, 1.6023, 1.6022, 1.6016, 1.5993, 1.5984, 1.598, 1.5979, 1.5964, 1.5961, 1.5956, 1.5955, 1.5952, 1.5951, 1.5951, 1.5944, 1.5943, 1.5936, 1.5931, 1.593, 1.593, 1.5912, 1.4879, 1.5184, 1.4904, 1.5432, 1.3248, 1.4105, 1.092, 1.3462, 1.0051, 1.3234, 1.2268, 1.0494, 1.4437, 0.884, 0.4787, 1.2069, 0.4973, 0.7024, 0.7228, 0.911, -0.5525, 0.8894, 0.6424, 0.186, 0.4423, 0.8435, 1.6994, 1.699, 1.6986, 1.6975, 1.6973, 1.6968, 1.6944, 1.6931, 1.6908, 1.6906, 1.6904, 1.6902, 1.6894, 1.6888, 1.6882, 1.6881, 1.6856, 1.6846, 1.683, 1.6829, 1.6829, 1.6825, 1.6816, 1.6815, 1.681, 1.6807, 1.6794, 1.6793, 1.678, 1.678, 1.2533, 1.5046, 1.4593, 1.4723, 1.395, 1.2736, 1.2568, 1.2742, 1.325, 0.9559, 0.979, 1.0536, 0.6188, 0.9174, 0.4837, 0.4601, 0.5857, 0.13, 0.0025, 2.0552, 2.0545, 2.0542, 2.0535, 2.0534, 2.0532, 2.053, 2.0521, 2.05, 2.0495, 2.0491, 2.0487, 2.0468, 2.0442, 2.0432, 2.0396, 2.038, 2.038, 2.035, 2.0334, 2.0324, 2.032, 2.0313, 2.0312, 2.0303, 2.0303, 2.0303, 2.0302, 2.0291, 2.0289, 1.9605, 1.792, 1.8385, 1.9748, 1.9282, 1.587, 1.4921, 1.7227, 1.8342, 1.0845, 1.7521, 1.5999, 0.6328, 0.9293, 1.1845, -0.4949, 0.9053, 0.9547, -0.5244, 1.1479], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.5512, -3.6223, -5.2003, -5.2108, -5.2108, -5.5486, -5.8603, -5.8603, -5.8603, -5.881, -5.9018, -5.9018, -5.9675, -6.0138, -6.0623, -6.0623, -6.1399, -6.1399, -6.1399, -6.1399, -6.1399, -6.224, -6.224, -6.2241, -6.2241, -6.2538, -6.3158, -6.3484, -6.3484, -6.3492, -2.7804, -5.2688, -4.7833, -5.6579, -5.8683, -2.3069, -2.3069, -2.3063, -5.7065, -4.9343, -5.5081, -4.8938, -5.7281, -5.1256, -5.574, -4.828, -4.404, -5.326, -5.0055, -5.1735, -5.4471, -5.1817, -5.3847, -5.5576, -5.6271, -5.6082, -5.6631, -3.7604, -4.0769, -4.2853, -4.3347, -4.3347, -5.095, -5.1507, -5.1507, -5.2344, -5.3258, -5.382, -5.3966, -5.4113, -5.5214, -5.6451, -5.6451, -3.2885, -5.7231, -6.0027, -6.0027, -6.0028, -6.0028, -6.03, -6.0302, -6.0579, -6.0579, -6.1786, -6.1786, -6.1786, -6.1787, -6.1787, -4.6927, -5.5654, -3.9324, -1.8758, -1.8771, -1.8771, -5.0626, -4.4016, -5.2944, -5.3574, -5.0662, -5.817, -5.0641, -5.1564, -4.9411, -5.5988, -5.4865, -5.5175, -4.1233, -4.5236, -4.7375, -4.7553, -4.9647, -5.0454, -5.474, -5.474, -5.5117, -5.4567, -5.531, -5.5708, -5.7008, -5.7483, -5.7728, -5.7729, -5.8503, -5.8506, -5.8778, -5.8779, -5.9056, -5.9057, -5.9057, -5.9346, -5.9346, -5.9643, -5.9947, -5.9948, -5.9948, -6.059, -4.4314, -5.3324, -5.1409, -5.6691, -4.6591, -5.122, -3.9375, -4.9811, -4.184, -5.2139, -4.9886, -4.5947, -5.5472, -4.8005, -4.2938, -5.2931, -4.4692, -4.874, -4.9109, -5.2445, -4.458, -5.2948, -5.2303, -5.1995, -5.3031, -5.3729, -3.5582, -3.7138, -3.8447, -4.1401, -4.1953, -4.2843, -4.6923, -4.843, -5.0739, -5.1015, -5.1157, -5.1445, -5.205, -5.2529, -5.3031, -5.3031, -5.4711, -5.534, -5.6246, -5.6247, -5.6247, -5.6486, -5.6983, -5.6985, -5.7243, -5.7246, -5.8059, -5.806, -5.8644, -5.8645, -2.7887, -4.8682, -4.6156, -4.8474, -4.618, -4.5779, -4.7478, -4.9657, -5.1084, -4.679, -4.7861, -4.8853, -4.4643, -5.1016, -4.9017, -4.9631, -4.9986, -5.0333, -5.027, -3.8939, -4.0344, -4.0786, -4.215, -4.2237, -4.2413, -4.2864, -4.4039, -4.637, -4.691, -4.7191, -4.7627, -4.9238, -5.0951, -5.1594, -5.3542, -5.4381, -5.4381, -5.5626, -5.631, -5.6672, -5.6677, -5.7048, -5.7048, -5.7436, -5.7436, -5.7436, -5.7436, -5.7841, -5.7842, -4.3049, -3.1786, -4.2057, -5.3954, -5.2171, -3.5764, -3.2804, -4.6791, -5.2953, -3.9986, -5.2634, -5.1172, -4.3336, -4.655, -4.8673, -4.4005, -4.9443, -4.977, -4.5664, -5.1503]}, \"token.table\": {\"Topic\": [1, 2, 3, 4, 3, 5, 5, 5, 3, 5, 1, 2, 4, 5, 5, 5, 5, 2, 2, 2, 1, 3, 4, 2, 3, 4, 5, 3, 4, 5, 1, 3, 5, 3, 5, 3, 1, 3, 1, 3, 4, 5, 4, 1, 4, 4, 4, 3, 1, 2, 1, 1, 1, 1, 3, 5, 1, 3, 4, 5, 4, 4, 4, 3, 1, 2, 1, 1, 5, 3, 1, 1, 4, 1, 3, 1, 3, 4, 3, 3, 1, 4, 5, 3, 4, 4, 4, 3, 1, 2, 3, 4, 5, 1, 3, 4, 5, 1, 3, 4, 5, 2, 3, 5, 5, 5, 4, 1, 5, 4, 1, 2, 3, 4, 5, 4, 1, 1, 2, 4, 5, 5, 1, 2, 1, 2, 1, 2, 5, 2, 2, 1, 3, 4, 1, 4, 1, 2, 3, 4, 5, 1, 1, 3, 5, 3, 5, 3, 2, 4, 3, 1, 3, 4, 5, 5, 3, 5, 5, 1, 3, 4, 5, 3, 1, 3, 4, 2, 4, 4, 3, 3, 2, 2, 2, 1, 4, 5, 5, 5, 5, 1, 2, 2, 2, 2, 2, 1, 3, 4, 3, 3, 3, 2, 3, 4, 5, 3, 3, 5, 5, 1, 3, 5, 1, 3, 4, 5, 1, 3, 4, 5, 1, 2, 3, 4, 5, 2, 3, 2, 2, 3, 2, 5, 3, 1, 3, 4, 1, 3, 4, 5, 5, 1, 3, 4, 3, 1, 3, 2, 3, 1, 4, 5, 1, 1, 1, 4, 5, 2, 2, 4, 5, 5, 3, 4, 4, 1, 2, 3, 5, 1, 1, 3, 5, 5, 1, 4, 1, 1, 1, 3, 4, 5, 1, 3, 4, 1, 3, 4, 5, 1, 1, 3, 5, 4, 4, 1, 4, 5, 1, 4, 3, 3, 2, 5, 2, 2, 1, 2, 3, 4, 5, 2, 3, 4, 5, 2, 4, 5, 2, 4, 5, 3, 4, 1, 3, 5, 1, 1, 1, 2, 2, 2, 2, 2, 1, 5, 5, 1, 2, 3, 4, 5, 1, 4, 4, 2, 2, 2, 4, 1, 2, 3, 5, 3, 2, 4, 4, 3, 2, 3, 4, 4, 4, 4, 4, 5, 5, 2, 3, 4, 5, 1, 4, 5, 1, 2, 4, 1, 3, 4, 5, 1, 3, 4, 2, 3, 1, 3, 4, 5, 5, 3, 2, 1, 2, 4, 5, 2, 2], \"Freq\": [0.12512273491598525, 0.11469584033965315, 0.6673212528852547, 0.09384205118698893, 0.9644418239764304, 0.9977924605261276, 0.9686535031429382, 0.9686535031429382, 0.19567949200474324, 0.8022859172194473, 0.10166393290716824, 0.06325755825334912, 0.06777595527144549, 0.7658682945673341, 0.9926493165140989, 0.9878618619532178, 0.9690605273420443, 0.9815628938584477, 0.97262620608867, 0.97262620608867, 0.4189512963202202, 0.12246268661667975, 0.4576237236728559, 0.23612188559493463, 0.5425181419026475, 0.10119509382640056, 0.11806094279746732, 0.9827354495045991, 0.07468400543525633, 0.9211027337014946, 0.7050211386002067, 0.09139162907780457, 0.1958392051667241, 0.8809133696067091, 0.11667726749757736, 0.9902013336476312, 0.8789125739243557, 0.10607565547362914, 0.4469145500432937, 0.18207629816578635, 0.23173347039281897, 0.1324191259387537, 0.9912725798521316, 0.9901451507905341, 0.9815394961870781, 0.9803473412098723, 0.996350350310907, 0.9945141271147919, 0.4261202885141235, 0.5738692547775688, 0.976983783441835, 0.9855141577598725, 0.9809963577520342, 0.9809963577520342, 0.11928357552259013, 0.8747462204989942, 0.20001662503897144, 0.21001745629092, 0.48504031571950573, 0.10500872814546, 0.9946389189472074, 0.9940328285647537, 0.9908635941957915, 0.9971200257826943, 0.8973870430778972, 0.07637336536833168, 0.9646042557910961, 0.9646042557910961, 0.9690091589559623, 0.9819925121760785, 0.9985171996002942, 0.9982979504739171, 0.9946070621555352, 0.1831027377588211, 0.8078061959947991, 0.19293179586484507, 0.15273767172633568, 0.6511448110438521, 0.971395771522219, 0.9841470721150296, 0.2626743942605666, 0.7367696424381746, 0.9966813052959338, 0.7468123866759583, 0.24893746222531943, 0.9797088630273361, 0.9820933948145605, 0.9610858895478411, 0.15734628632036374, 0.1242207523581819, 0.4016470992914548, 0.2028938955183638, 0.11179867712236372, 0.011898617265271012, 0.3212626661623173, 0.5235391596719245, 0.14278340718325214, 0.13321172363226844, 0.36633223998873815, 0.07992703417936106, 0.4196169294416455, 0.11305124476695035, 0.3206207433554493, 0.5671095229292918, 0.9928663013886647, 0.9688955487670432, 0.9892267810621794, 0.8016552606905137, 0.1946877061676962, 0.9853527923574819, 0.1829840020157982, 0.09372351322760396, 0.009818653766701367, 0.6382124948355888, 0.0758714154699651, 0.9973741670498129, 0.9881744388719994, 0.9881744388719994, 0.8171458758973, 0.07907863315135161, 0.10543817753513549, 0.9894313777519855, 0.426288577580504, 0.5736728637067766, 0.4262885775806295, 0.5736728637069456, 0.976983783441835, 0.9402071397949118, 0.050368239631870274, 0.9726313662662401, 0.9726313662662401, 0.6628781730905605, 0.0487410421390118, 0.2924462528340708, 0.9833603319285925, 0.9899709123834995, 0.4440401089571475, 0.01420928348662872, 0.14919747660960156, 0.2877379906042316, 0.10301730527805822, 0.9769834481429792, 0.9769834481429792, 0.9804131190559854, 0.9933168036268717, 0.7632756574866071, 0.23687865232342978, 0.9881831329795041, 0.9350342517650018, 0.0558229404038807, 0.9713974497492547, 0.46818688484460547, 0.14825918020079173, 0.22629032767489263, 0.1560622949482018, 0.9685700361167783, 0.19167711186257919, 0.799756225357658, 0.9931603185559579, 0.43461212079531186, 0.008780042844349734, 0.47412231359488566, 0.0790203855991476, 0.9628125890620614, 0.3411614827534923, 0.5671255817200911, 0.09304404075095245, 0.7965461426139606, 0.20451860418466555, 0.9797104852283133, 0.9725903685821966, 0.9674448281556446, 0.9931695996193578, 0.9696055151168543, 0.9696055151168543, 0.16711114596162185, 0.20820405070628295, 0.6218726251358715, 0.9954263839283612, 0.9944969985689661, 0.9691234720706036, 0.9892897357365548, 0.9949593001980722, 0.9919171965279784, 0.9919171965279784, 0.993809491134324, 0.988875422451736, 0.12446192220674782, 0.7467715332404868, 0.1306850183170852, 0.9748127761415226, 0.9644423506049623, 0.9628113298512871, 0.06275330270124699, 0.6745980040384052, 0.10197411688952636, 0.15688325675311748, 0.9876956868491918, 0.9841457887090603, 0.9691290287365424, 0.968950408090861, 0.2737340680790571, 0.47903461913834994, 0.24331917162582856, 0.2223723269233199, 0.07624194065942397, 0.6417030005501517, 0.06353495054951996, 0.25615050346350166, 0.37865726598952415, 0.23387654664058846, 0.12807525173175083, 0.06534698100809906, 0.6534698100809906, 0.19967133085808048, 0.06534698100809906, 0.018151939168916408, 0.8517772953122079, 0.1355100242542149, 0.9828645971457368, 0.1533186628242924, 0.8347349420433697, 0.09071186055598021, 0.9071186055598021, 0.9847985366490758, 0.13931199747400064, 0.060948998894875275, 0.7923369856333786, 0.3249542566635315, 0.23943997859418112, 0.29416911655856537, 0.14024341603373466, 0.9953334745658945, 0.22525870665078043, 0.5919031972632209, 0.18212406069637566, 0.9902014080351355, 0.9886296126789138, 0.9674403245081883, 0.5370679137808536, 0.45750229692443084, 0.4252826277920769, 0.24217482971493268, 0.33077537717161537, 0.9722810516056265, 0.9722810516056265, 0.2831493174437822, 0.3387679333702394, 0.3766897169564602, 0.9887693507431233, 0.9887693507431232, 0.982093045116917, 0.9690596578533547, 0.9690596578533547, 0.964439537839824, 0.9929834908202406, 0.982093298883654, 0.3014664326197404, 0.13273522033257226, 0.32621367708852506, 0.24072319619635985, 0.9722841415012761, 0.9722841415012761, 0.28354226737940363, 0.7183070773611558, 0.9690073347009481, 0.9667022622011385, 0.9914684795612078, 0.986010154044979, 0.9901448403821226, 0.9901448403821226, 0.9848804904213126, 0.2618958653704759, 0.7333084230373325, 0.30623591301432035, 0.4632799709703821, 0.2355660869340926, 0.617094809861878, 0.17631280281767944, 0.03918062284837321, 0.15672249139349284, 0.9645864346247578, 0.10700591305875538, 0.8798263962608777, 0.9945685756865056, 0.9775689997243536, 0.9840745953997951, 0.8462667822906049, 0.15111906826617943, 0.9689728496809664, 0.9128182132823511, 0.07429915689507509, 0.9924369644609827, 0.9940333232296229, 0.9848164600926493, 0.014403165778320282, 0.9974127919330062, 0.9711590031362896, 0.09793287476580964, 0.010881430529534405, 0.2992393395621961, 0.27747647850312734, 0.31556148535649775, 0.4559668113701811, 0.3088807431862517, 0.09315450984982195, 0.14218319924446507, 0.08180090068472778, 0.7839252982286412, 0.1363348344745463, 0.7888621070380409, 0.06859670495982965, 0.1371934099196593, 0.9353337949205319, 0.06377275874458171, 0.8075992548081247, 0.11448128298630118, 0.07787842380020489, 0.9884085518447776, 0.9884085518447776, 0.9705253606915625, 0.9978117729596914, 0.9959128812022943, 0.9959128812022943, 0.9624199622095665, 0.9711459314666746, 0.976983783441835, 0.9823242568783108, 0.9848829023624376, 0.20031947286478466, 0.13354631524318977, 0.01669328940539872, 0.32551914340527505, 0.3213458210539254, 0.17444665709316626, 0.8171448674364103, 0.977568349709773, 0.9857044264375912, 0.962419585340173, 0.962419585340173, 0.9884047600131681, 0.2675700027783308, 0.22605051958858982, 0.47978069463700695, 0.027679655459827323, 0.9659806953179859, 0.9822234988502008, 0.9918451569419631, 0.9759337045529044, 0.9902002556555052, 0.6641867026757377, 0.3320933513378688, 0.9983522711372339, 0.9961009958478367, 0.980347657435473, 0.9884057117522193, 0.9759326851860841, 0.9880873720834242, 0.9690641405842274, 0.20687824165968166, 0.4950300782570954, 0.15515868124476126, 0.14777017261405834, 0.11659181077863763, 0.24613826719934612, 0.6347776364614716, 0.23987005740884268, 0.10903184427674667, 0.65419106566048, 0.087797758006699, 0.09755306445188779, 0.6828714511632145, 0.12681898378745413, 0.5261093554191268, 0.4077347504498233, 0.06137942479889814, 0.08885166344731785, 0.9033252450477314, 0.1704392121263117, 0.15339529091368054, 0.2727027394020987, 0.4005321484968325, 0.9688094253166591, 0.965981052104316, 0.9852564919585495, 0.07995596823260018, 0.791288375267457, 0.0771988658797519, 0.049627842351269075, 0.9624202859011372, 0.9624202859011372], \"Term\": [\"000\", \"000\", \"000\", \"000\", \"ads\", \"again\", \"again https\", \"again https co\", \"ago\", \"ago\", \"america\", \"america\", \"america\", \"america\", \"america great\", \"america great again\", \"america safe\", \"americafirst\", \"americafirst https\", \"americafirst https co\", \"americans\", \"americans\", \"americans\", \"amp\", \"amp\", \"amp\", \"amp\", \"beat\", \"become\", \"become\", \"best\", \"best\", \"best\", \"big\", \"big\", \"bush\", \"business\", \"business\", \"can\", \"can\", \"can\", \"can\", \"care\", \"chief\", \"child\", \"children\", \"clinton\", \"cnn\", \"co\", \"co\", \"co ttgeqxnqym\", \"commander\", \"commander in\", \"commander in chief\", \"convention\", \"convention\", \"country\", \"country\", \"country\", \"country\", \"crooked\", \"crooked hillary\", \"crooked hillary clinton\", \"cruz\", \"debatenight\", \"debatenight\", \"debatenight https\", \"debatenight https co\", \"democratic\", \"dishonest\", \"donald\", \"donald trump\", \"enjoy\", \"even\", \"even\", \"every\", \"every\", \"every\", \"failed\", \"failing\", \"families\", \"families\", \"first\", \"foxnews\", \"foxnews\", \"free\", \"future\", \"gave\", \"get\", \"get\", \"get\", \"get\", \"get\", \"going\", \"going\", \"going\", \"going\", \"good\", \"good\", \"good\", \"good\", \"great\", \"great\", \"great\", \"great again\", \"great again https\", \"gun\", \"he\", \"he\", \"health\", \"hillary\", \"hillary\", \"hillary\", \"hillary\", \"hillary\", \"hillary clinton\", \"hillary https\", \"hillary https co\", \"hillaryclinton\", \"hillaryclinton\", \"hillaryclinton\", \"house\", \"https\", \"https\", \"https co\", \"https co\", \"https co ttgeqxnqym\", \"imwithyou\", \"imwithyou\", \"imwithyou https\", \"imwithyou https co\", \"in\", \"in\", \"in\", \"in chief\", \"interviewed\", \"it\", \"it\", \"it\", \"it\", \"it\", \"it https\", \"it https co\", \"jeb\", \"job\", \"jobs\", \"jobs\", \"john\", \"join\", \"join\", \"kasich\", \"know\", \"know\", \"know\", \"know\", \"knows\", \"last\", \"last\", \"last night\", \"let\", \"let\", \"let\", \"let\", \"lies\", \"like\", \"like\", \"like\", \"live\", \"live\", \"lives\", \"lyin\", \"lyin ted\", \"maga\", \"maga https\", \"maga https co\", \"make\", \"make\", \"make\", \"make america\", \"make america great\", \"make america safe\", \"make sure\", \"makeamericagreatagain\", \"makeamericagreatagain https\", \"makeamericagreatagain https co\", \"makeamericagreatagain trump2016\", \"makeamericagreatagain trump2016 https\", \"many\", \"many\", \"many\", \"marco\", \"marco rubio\", \"massive\", \"me\", \"me\", \"me\", \"me\", \"media\", \"megynkelly\", \"mitt\", \"mr\", \"much\", \"much\", \"much\", \"need\", \"need\", \"need\", \"need\", \"never\", \"never\", \"never\", \"never\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new hampshire\", \"new hampshire\", \"new york\", \"news\", \"news\", \"night\", \"night\", \"nytimes\", \"obama\", \"obama\", \"obama\", \"one\", \"one\", \"one\", \"one\", \"party\", \"people\", \"people\", \"people\", \"phony\", \"plan\", \"police\", \"poll\", \"poll\", \"potus\", \"potus\", \"potus\", \"potus https\", \"potus https co\", \"president\", \"president\", \"president\", \"president https\", \"president https co\", \"president obama\", \"president united\", \"president united states\", \"ratings\", \"re\", \"re going\", \"realdonaldtrump\", \"realdonaldtrump\", \"realdonaldtrump\", \"realdonaldtrump\", \"realdonaldtrump https\", \"realdonaldtrump https co\", \"republican\", \"republican\", \"republican party\", \"returns\", \"rights\", \"rncincle\", \"rncincle https\", \"rncincle https co\", \"rubio\", \"safe\", \"safe\", \"said\", \"said\", \"said\", \"says\", \"says\", \"says\", \"says\", \"share\", \"show\", \"show\", \"states\", \"street\", \"stronger\", \"sure\", \"sure\", \"talks\", \"tax\", \"tax\", \"ted\", \"ted cruz\", \"thank\", \"thank\", \"thank you\", \"tickets\", \"time\", \"time\", \"time\", \"time\", \"time\", \"today\", \"today\", \"today\", \"today\", \"together\", \"together\", \"together\", \"tomorrow\", \"tomorrow\", \"tomorrow\", \"total\", \"total\", \"trump\", \"trump\", \"trump\", \"trump https\", \"trump https co\", \"trump says\", \"trump2016\", \"trump2016 https\", \"trump2016 https co\", \"trump2016 makeamericagreatagain\", \"trumptrain\", \"ttgeqxnqym\", \"united\", \"united states\", \"us\", \"us\", \"us\", \"us\", \"us\", \"ve\", \"ve\", \"ve got\", \"video\", \"video https\", \"video https co\", \"violence\", \"vote\", \"vote\", \"vote\", \"vote\", \"votes\", \"votetrump\", \"wall\", \"wall street\", \"warren\", \"watch\", \"watch\", \"we\", \"we re\", \"we re going\", \"we ve\", \"we ve got\", \"white\", \"white house\", \"win\", \"win\", \"win\", \"win\", \"woman\", \"woman\", \"woman\", \"women\", \"women\", \"women\", \"work\", \"work\", \"work\", \"work\", \"would\", \"would\", \"would\", \"wow\", \"wow\", \"years\", \"years\", \"years\", \"years\", \"years ago\", \"yesterday\", \"york\", \"you\", \"you\", \"you\", \"you\", \"you https\", \"you https co\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [3, 4, 1, 5, 2]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el2009217906065621927236996363\", ldavis_el2009217906065621927236996363_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el2009217906065621927236996363\", ldavis_el2009217906065621927236996363_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el2009217906065621927236996363\", ldavis_el2009217906065621927236996363_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=                x           y  topics  cluster       Freq\n",
       "topic                                                    \n",
       "2      98.779404   144.924530  1       1        26.223112\n",
       "3     -239.079269 -216.686905  2       1        22.965593\n",
       "0     -267.715912  161.802689  3       1        19.829622\n",
       "4      101.722450 -349.686218  4       1        18.235845\n",
       "1      379.638367 -91.175018   5       1        12.745828, topic_info=     Category         Freq   ...     loglift  logprob\n",
       "term                         ...                     \n",
       "151   Default  3912.000000   ...     30.0000  30.0000\n",
       "379   Default  3908.000000   ...     29.0000  29.0000\n",
       "378   Default  3908.000000   ...     28.0000  28.0000\n",
       "861   Default  1284.000000   ...     27.0000  27.0000\n",
       "810   Default  555.000000    ...     26.0000  26.0000\n",
       "360   Default  1120.000000   ...     25.0000  25.0000\n",
       "214   Default  480.000000    ...     24.0000  24.0000\n",
       "46    Default  442.000000    ...     23.0000  23.0000\n",
       "322   Default  539.000000    ...     22.0000  22.0000\n",
       "215   Default  447.000000    ...     21.0000  21.0000\n",
       "148   Default  332.000000    ...     20.0000  20.0000\n",
       "867   Default  341.000000    ...     19.0000  19.0000\n",
       "940   Default  284.000000    ...     18.0000  18.0000\n",
       "494   Default  365.000000    ...     17.0000  17.0000\n",
       "361   Default  249.000000    ...     16.0000  16.0000\n",
       "499   Default  249.000000    ...     15.0000  15.0000\n",
       "29    Default  166.000000    ...     14.0000  14.0000\n",
       "183   Default  205.000000    ...     13.0000  13.0000\n",
       "642   Default  395.000000    ...     12.0000  12.0000\n",
       "178   Default  185.000000    ...     11.0000  11.0000\n",
       "495   Default  144.000000    ...     10.0000  10.0000\n",
       "179   Default  176.000000    ...     9.0000   9.0000 \n",
       "813   Default  202.000000    ...     8.0000   8.0000 \n",
       "276   Default  138.000000    ...     7.0000   7.0000 \n",
       "869   Default  192.000000    ...     6.0000   6.0000 \n",
       "868   Default  192.000000    ...     5.0000   5.0000 \n",
       "613   Default  417.000000    ...     4.0000   4.0000 \n",
       "673   Default  161.000000    ...     3.0000   3.0000 \n",
       "992   Default  362.000000    ...     2.0000   2.0000 \n",
       "47    Default  120.000000    ...     1.0000   1.0000 \n",
       "...       ...         ...    ...        ...      ... \n",
       "548   Topic5   28.110609     ...     2.0324  -5.6672 \n",
       "794   Topic5   28.098752     ...     2.0320  -5.6677 \n",
       "693   Topic5   27.075470     ...     2.0313  -5.7048 \n",
       "201   Topic5   27.074182     ...     2.0312  -5.7048 \n",
       "646   Topic5   26.045029     ...     2.0303  -5.7436 \n",
       "647   Topic5   26.045029     ...     2.0303  -5.7436 \n",
       "51    Topic5   26.044777     ...     2.0303  -5.7436 \n",
       "955   Topic5   26.043053     ...     2.0302  -5.7436 \n",
       "497   Topic5   25.011354     ...     2.0291  -5.7841 \n",
       "538   Topic5   25.008501     ...     2.0289  -5.7842 \n",
       "570   Topic5   109.785511    ...     1.9605  -4.3049 \n",
       "46    Topic5   338.595471    ...     1.7920  -3.1786 \n",
       "437   Topic5   121.235221    ...     1.8385  -4.2057 \n",
       "84    Topic5   36.890884     ...     1.9748  -5.3954 \n",
       "168   Topic5   44.090659     ...     1.9282  -5.2171 \n",
       "494   Topic5   227.473292    ...     1.5870  -3.5764 \n",
       "322   Topic5   305.802091    ...     1.4921  -3.2804 \n",
       "692   Topic5   75.511593     ...     1.7227  -4.6791 \n",
       "32    Topic5   40.775240     ...     1.8342  -5.2953 \n",
       "642   Topic5   149.131315    ...     1.0845  -3.9986 \n",
       "716   Topic5   42.095817     ...     1.7521  -5.2634 \n",
       "965   Topic5   48.725110     ...     1.5999  -5.1172 \n",
       "678   Topic5   106.676905    ...     0.6328  -4.3336 \n",
       "893   Topic5   77.354209     ...     0.9293  -4.6550 \n",
       "314   Topic5   62.555923     ...     1.1845  -4.8673 \n",
       "861   Topic5   99.775212     ...    -0.4949  -4.4005 \n",
       "835   Topic5   57.924051     ...     0.9053  -4.9443 \n",
       "636   Topic5   56.060044     ...     0.9547  -4.9770 \n",
       "360   Topic5   84.520977     ...    -0.5244  -4.5664 \n",
       "986   Topic5   47.136950     ...     1.1479  -5.1503 \n",
       "\n",
       "[291 rows x 6 columns], token_table=      Topic      Freq                   Term\n",
       "term                                        \n",
       "1     1      0.125123  000                  \n",
       "1     2      0.114696  000                  \n",
       "1     3      0.667321  000                  \n",
       "1     4      0.093842  000                  \n",
       "23    3      0.964442  ads                  \n",
       "29    5      0.997792  again                \n",
       "30    5      0.968654  again https          \n",
       "31    5      0.968654  again https co       \n",
       "32    3      0.195679  ago                  \n",
       "32    5      0.802286  ago                  \n",
       "46    1      0.101664  america              \n",
       "46    2      0.063258  america              \n",
       "46    4      0.067776  america              \n",
       "46    5      0.765868  america              \n",
       "47    5      0.992649  america great        \n",
       "48    5      0.987862  america great again  \n",
       "51    5      0.969061  america safe         \n",
       "52    2      0.981563  americafirst         \n",
       "53    2      0.972626  americafirst https   \n",
       "54    2      0.972626  americafirst https co\n",
       "57    1      0.418951  americans            \n",
       "57    3      0.122463  americans            \n",
       "57    4      0.457624  americans            \n",
       "58    2      0.236122  amp                  \n",
       "58    3      0.542518  amp                  \n",
       "58    4      0.101195  amp                  \n",
       "58    5      0.118061  amp                  \n",
       "82    3      0.982735  beat                 \n",
       "84    4      0.074684  become               \n",
       "84    5      0.921103  become               \n",
       "...  ..           ...     ...               \n",
       "960   4      0.155159  win                  \n",
       "960   5      0.147770  win                  \n",
       "965   1      0.116592  woman                \n",
       "965   4      0.246138  woman                \n",
       "965   5      0.634778  woman                \n",
       "966   1      0.239870  women                \n",
       "966   2      0.109032  women                \n",
       "966   4      0.654191  women                \n",
       "972   1      0.087798  work                 \n",
       "972   3      0.097553  work                 \n",
       "972   4      0.682871  work                 \n",
       "972   5      0.126819  work                 \n",
       "980   1      0.526109  would                \n",
       "980   3      0.407735  would                \n",
       "980   4      0.061379  would                \n",
       "981   2      0.088852  wow                  \n",
       "981   3      0.903325  wow                  \n",
       "986   1      0.170439  years                \n",
       "986   3      0.153395  years                \n",
       "986   4      0.272703  years                \n",
       "986   5      0.400532  years                \n",
       "987   5      0.968809  years ago            \n",
       "989   3      0.965981  yesterday            \n",
       "991   2      0.985256  york                 \n",
       "992   1      0.079956  you                  \n",
       "992   2      0.791288  you                  \n",
       "992   4      0.077199  you                  \n",
       "992   5      0.049628  you                  \n",
       "993   2      0.962420  you https            \n",
       "994   2      0.962420  you https co         \n",
       "\n",
       "[391 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[3, 4, 1, 5, 2])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyLDAvis.sklearn\n",
    " \n",
    "pyLDAvis.enable_notebook()\n",
    "% time pyLDAvis.sklearn.prepare(lda_model, dtm, vectorizer, mds=\"tsne\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}