Part1D_Finalizing_Data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "israeli-expression",
   "metadata": {},
   "source": [
    "# Project Overview\n",
    "In this notebook and subsequent notebooks I will attempt to predict mechanically meaningful labels associated with some fan-generated roleplaying game (RPG) content.\n",
    "\n",
    "I collected post and comment data from Reddit in multiple runs using a command-line tool I wrote for this purpose (see: [PRAW_FILES-CoDiaLS](https://github.com/nkuehnle/praw-codials))\n",
    "\n",
    "The actual text I will be attempting to predict is hosted on one of two websites: [GM Binder](https://gmbinder.com/) or [Homebrewery](https://homebrewery.naturalcrit.com/), which are tools/content hosting providers for fan-made RPG content, primarily Dungeons and Dragons 5th Edition (DnD5e).\n",
    "\n",
    "The labels make use of the user-assigned \"flair\" from the subreddit [/r/UnearthedArcana](https://www.reddit.com/r/UnearthedArcana/). UnearthedArcana describes itself as a source of \"homebrew\" (fan-generated) content and its community rules ensure that every post has a meaningful label (\"flair\") associated with it. These labels describe the \"type\" of game content that has been created, be it a character \"class\" (think achetypal high-fantasy characters knights, wizards, archers, etc) or a \"race\" (elves, dwarves, humans, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b84dc5c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The jupyter_black extension is already loaded. To reload it, use:\n",
      "  %reload_ext jupyter_black\n"
     ]
    }
   ],
   "source": [
    "%load_ext jupyter_black\n",
    "# Utility Imports\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "# Imports for data processing/handling\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle as pkl\n",
    "\n",
    "# NLP-specific processing\n",
    "from nltk.tokenize import word_tokenize\n",
    "from langdetect import detect\n",
    "\n",
    "# Custom modules\n",
    "from src.preprocessing import (\n",
    "    fix_dt,\n",
    "    data_io,\n",
    "    get_section_df,\n",
    "    sections_df_to_docs,\n",
    "    EmbeddingAwareTokenizer,\n",
    ")\n",
    "from src.eda_utils import get_char_counts, count_stopwords, calc_token_sizes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1def0239",
   "metadata": {},
   "source": [
    "# Bringing it all together\n",
    "\n",
    "At this point we've:\n",
    "1. Collected links containing fan-generated Dungeons and Dragons game content posted by Reddit users to [/r/UnearthedArcana](https://reddit.com/r/unearthedarcana) or [/r/DnDHomebrew](https://reddit.com/r/dndhomebrew) using my [PRAW_FILES-CoDiaLS](https://github.com/nkuehnle/praw-codials) tool\n",
    "2. Filered out or semi-manually reviewed links to get the largest set of unique, accurately \"flaired\" (labeled) links\n",
    "3. Scraped this content from [one](https://homebrewery.naturalcrit.com) of [two](https://gmbinder.com) websites that host this content (as literal string markdown text with some CSS elements)\n",
    "4. Cleaned as much irrelevant information from the texts (i.e. styling elements and so forth)\n",
    "\n",
    "### In this notebook I'll:\n",
    "1. Ensure column datatypes are properly set\n",
    "2. Filter once more for a few potential problems that might have been missed\n",
    "3. Load the text as either sections or full documents into two possible dataframes\n",
    "4. Get word and sentence tokens at both the section and document level (this takes a while)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "independent-phrase",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a constant (CWD) to use\n",
    "CWD = Path(os.getcwd())\n",
    "DATA = CWD / \"data\"\n",
    "PRAW_FILES = DATA / \"praw_data\"\n",
    "TEXT_FILES = DATA / \"text_files\"\n",
    "CLEAN_TEXT_FILES = DATA / \"clean_text_files\"\n",
    "CORPUS_FILES = DATA / \"corpus_files\"\n",
    "CORPUS_FILES.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "159df875",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 3250 entries, 0 to 2335\n",
      "Data columns (total 18 columns):\n",
      " #   Column                   Non-Null Count  Dtype  \n",
      "---  ------                   --------------  -----  \n",
      " 0   UID                      3250 non-null   int64  \n",
      " 1   link                     3250 non-null   object \n",
      " 2   submission_author        3237 non-null   object \n",
      " 3   submission_id            3250 non-null   object \n",
      " 4   submission_title         3250 non-null   object \n",
      " 5   subreddit                3250 non-null   object \n",
      " 6   submission_flair         3235 non-null   object \n",
      " 7   submission_score         3250 non-null   float64\n",
      " 8   submission_upvote_ratio  3247 non-null   float64\n",
      " 9   submission_date          2009 non-null   object \n",
      " 10  comment_author           2628 non-null   object \n",
      " 11  comment_id               2628 non-null   object \n",
      " 12  comment_score            2628 non-null   float64\n",
      " 13  comment_body             2628 non-null   object \n",
      " 14  comment_date             1867 non-null   object \n",
      " 15  src_url                  3250 non-null   object \n",
      " 16  manually_reviewed        1425 non-null   object \n",
      " 17  related_link             1472 non-null   object \n",
      "dtypes: float64(3), int64(1), object(14)\n",
      "memory usage: 482.4+ KB\n"
     ]
    }
   ],
   "source": [
    "metadata_path = CORPUS_FILES / \"Metadata.pkl\"\n",
    "metadata: pd.DataFrame = pd.read_pickle(metadata_path)\n",
    "metadata = metadata.sort_values(by=\"UID\")\n",
    "metadata.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7a3d121a",
   "metadata": {},
   "source": [
    "# Update manually reviewed /r/DnDHomeBrew content\n",
    "At the previous notebook I generated a list of labels from /r/DnDHomebrew (which has optional and/or less meaningful flair) to review.\n",
    "\n",
    "Here I'll incorporate any new information I've gathered from manually reviewing these texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7e75a88f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "368 /r/DnDHomebrew entries updated.\n"
     ]
    }
   ],
   "source": [
    "label_review_path = PRAW_FILES / \"LabelsToReview.csv\"\n",
    "# Read in reviewed labels\n",
    "reviewed_labels = pd.read_csv(label_review_path)\n",
    "reviewed_labels[\"manually_reviewed\"] = reviewed_labels[\"manually_reviewed\"].fillna(\n",
    "    False\n",
    ")\n",
    "manually_reviewed = reviewed_labels[\"manually_reviewed\"]\n",
    "\n",
    "reviewed_labels = reviewed_labels[manually_reviewed].copy()\n",
    "reviewed_labels[\"submission_flair\"] = reviewed_labels[\"corrected_flair\"]\n",
    "\n",
    "# Update values\n",
    "to_update = metadata[\"UID\"].isin(reviewed_labels[\"UID\"])\n",
    "mapper = reviewed_labels.set_index(\"UID\")[\"corrected_flair\"].to_dict()\n",
    "metadata.loc[to_update, \"submission_flair\"] = metadata.loc[to_update, \"UID\"].map(mapper)\n",
    "\n",
    "print(f\"{to_update.sum()} /r/DnDHomebrew entries updated.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "05e37815",
   "metadata": {},
   "source": [
    "### Set column datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8ea53a51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dates\n",
    "metadata[\"submission_date\"] = metadata[\"submission_date\"].apply(lambda x: fix_dt(x))\n",
    "metadata[\"comment_date\"] = metadata[\"comment_date\"].apply(lambda x: fix_dt(x))\n",
    "# Boolean\n",
    "metadata[\"manually_reviewed\"] = metadata[\"manually_reviewed\"].fillna(False)\n",
    "metadata[\"manually_reviewed\"] = metadata[\"manually_reviewed\"].astype(\"boolean\")\n",
    "# All primary submission links are related to the submission...\n",
    "metadata.loc[metadata[\"comment_id\"].isna(), \"related_link\"] = True\n",
    "metadata[\"related_link\"] = metadata[\"related_link\"].astype(\"boolean\")\n",
    "# Numbers\n",
    "metadata[\"comment_score\"] = metadata[\"comment_score\"].astype(\"Int32\")\n",
    "metadata[\"submission_score\"] = metadata[\"submission_score\"].astype(\"Int32\")\n",
    "metadata[\"submission_upvote_ratio\"] = metadata[\"submission_upvote_ratio\"].astype(\n",
    "    \"float32\"\n",
    ")\n",
    "metadata[\"UID\"] = metadata[\"UID\"].astype(\"Int16\")\n",
    "# Categorical\n",
    "metadata[\"subreddit\"] = metadata[\"subreddit\"].astype(\"category\")\n",
    "metadata[\"submission_flair\"] = metadata[\"submission_flair\"].astype(\"category\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ebb60c99",
   "metadata": {},
   "source": [
    "### Remove any remaining duplicates (now that DnDHomebrew content has flair)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d3143d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "dups = metadata.sort_values(\n",
    "    by=[\"manually_reviewed\", \"subreddit\", \"submission_date\", \"related_link\"],\n",
    "    axis=0,\n",
    "    ascending=[False, False, True, False],\n",
    ")\n",
    "dups = dups[dups.duplicated(subset=[\"src_url\", \"submission_flair\"], keep=\"first\")]\n",
    "metadata = metadata[~metadata[\"UID\"].isin(dups[\"UID\"])]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "37fea00f",
   "metadata": {},
   "source": [
    "Confirming no src_urls have been duplicated with differing flair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0c55cf3a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [UID, link, submission_author, submission_id, submission_title, subreddit, submission_flair, submission_score, submission_upvote_ratio, submission_date, comment_author, comment_id, comment_score, comment_body, comment_date, src_url, manually_reviewed, related_link]\n",
       "Index: []"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata[metadata[\"src_url\"].duplicated(keep=False)].sort_values(\"src_url\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b9a3a7ab",
   "metadata": {},
   "source": [
    "### Remove any invalid/missing content\n",
    "A few bad links have slipped through at some point. These will be dropped below."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3dd71441",
   "metadata": {},
   "source": [
    "#### Manual Review/missing flair\n",
    "After manual review, some links still had no relevant flair, seen and dropped below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7fb433c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata[metadata[\"submission_flair\"].isna()][\"link\"].values\n",
    "metadata = metadata[~metadata[\"submission_flair\"].isna()]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6b55b458",
   "metadata": {},
   "source": [
    "#### Missing/invalid content\n",
    "Likely collected in a previous pass or made evident after cleaning the markdown\n",
    "In a few cases, non-English texts are present. Due to the small number, we won't attempt training/translating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "86cd3ec1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dropped 24 texts based on absence of content:\n",
      "https://homebrewery.naturalcrit.com/share/1K6GHAH_Z1kNOiBXOMxNz6dKHfzaiOdg3YpbrFdctgy5p\n",
      "https://homebrewery.naturalcrit.com/share/1SNcTl5ohA2p3HYZtVqWLRS5kKJzlW7aEd2K9_HkberKm\n",
      "https://homebrewery.naturalcrit.com/share/1Zf0IQKmtx9miXcTDMlrG61uoXyjJKyTjudXLpyvDcjTU\n",
      "https://homebrewery.naturalcrit.com/share/1yARx8g8ufNapmrp-k72GVqmVupjOXlp5S1S3GYQpdozr\n",
      "https://homebrewery.naturalcrit.com/share/1PItn1Plb2qGd2zoFCtOVBY9ZRUSsDPZqyA48eDbqeVrI\n",
      "https://homebrewery.naturalcrit.com/share/1vm0x8Y63Rm3SIiWg6iEUT8bbcfIUuq7yxjih8vPgn1w1\n",
      "https://homebrewery.naturalcrit.com/share/16jtuooS8rYVhTFruUBM0kARAEqp8MKay2kM2QhL3Zf1v\n",
      "https://homebrewery.naturalcrit.com/share/126hk_D1cqhUnU-AcXeRNhm3PuVOVEiqDZ-J_GthMND0Y\n",
      "https://gmbinder.com/share/-MKHCarZv-OYPmgoduh9\n",
      "https://gmbinder.com/share/-MEUxzSRW9kf5wYfT0Oj\n",
      "https://gmbinder.com/share/-M_BKYtvcFV1OIBjaYP-\n",
      "https://gmbinder.com/share/-M_pNobz8LY49mB9eRiB\n",
      "https://gmbinder.com/share/-MdCpGu3Ga5xQt-uHW4h\n",
      "https://gmbinder.com/share/-MaOp5n_rJ34mUVIsrwD\n",
      "https://gmbinder.com/share/-MasFCl8pZobbu4M5cFr\n",
      "https://gmbinder.com/share/-MVC1WS7u7OEvbA69HHt\n",
      "https://gmbinder.com/share/-MVfl4G6lTqvl7Rnj8i5\n",
      "https://gmbinder.com/share/-N9MFWsfVsqNIvmGtsIf\n",
      "https://gmbinder.com/share/-M_21dYDnYBcNOiz7lho\n",
      "https://homebrewery.naturalcrit.com/share/W5Rp1BmAhs1Q\n",
      "https://homebrewery.naturalcrit.com/share/A40C1JoLZ70v\n",
      "https://gmbinder.com/share/-Mt8S5fHlKk68lKTIyJ2\n",
      "https://homebrewery.naturalcrit.com/share/myPa0Lo49-OP\n",
      "https://homebrewery.naturalcrit.com/share/BJXc7uX6dH\n",
      "5 non-english texts removed:\n",
      "https://homebrewery.naturalcrit.com/share/1KGpYTLPAAbdvtqQ_SWOfGmdloWxL4b78ljTk-mL8hVVt\n",
      "https://gmbinder.com/share/-MYPz7gxn3cOLZEI0wrj\n",
      "https://homebrewery.naturalcrit.com/share/Fg2Gu_WwgA_Q\n",
      "https://homebrewery.naturalcrit.com/share/M9DwzWnmwCfN\n",
      "https://homebrewery.naturalcrit.com/share/Ph_5tpP7gIPO\n",
      "3213 texts remain\n"
     ]
    }
   ],
   "source": [
    "metadata[\"raw_markdown\"] = metadata[\"UID\"].apply(\n",
    "    lambda x: data_io.read_src_txt_from_file(x, TEXT_FILES)\n",
    ")\n",
    "metadata[\"clean_markdown\"] = metadata[\"UID\"].apply(\n",
    "    lambda x: data_io.read_src_txt_from_file(x, CLEAN_TEXT_FILES)\n",
    ")\n",
    "\n",
    "to_drop = [\n",
    "    # Drop GMBinder links that are dead.\n",
    "    metadata[\"raw_markdown\"].str.contains(\"NoSuchKey\"),\n",
    "    # Drop Homebrwery links that are dead\n",
    "    metadata[\"raw_markdown\"].str.contains(\"This content has been moved\"),\n",
    "    # Some are not found\n",
    "    metadata[\"raw_markdown\"].str.contains(\"Error: File not found\"),\n",
    "    # There are a few examples of nonsensical text, which is generally short, thus I'm doing a check for anything less than 150 characters\n",
    "    metadata[\"raw_markdown\"].str.len() <= 200,\n",
    "    metadata[\"clean_markdown\"].str.len() <= 200,\n",
    "    # Drop anything totally empty.\n",
    "    metadata[\"raw_markdown\"] == \"\",\n",
    "    metadata[\"clean_markdown\"] == \"\",\n",
    "]\n",
    "\n",
    "drop_mask = None\n",
    "for mask in to_drop:\n",
    "    if isinstance(drop_mask, pd.Series) or isinstance(drop_mask, np.ndarray):\n",
    "        drop_mask = drop_mask | mask\n",
    "    else:\n",
    "        drop_mask = mask\n",
    "print(f\"Dropped {sum(drop_mask)} texts based on absence of content:\")\n",
    "print(*metadata[drop_mask][\"link\"], sep=\"\\n\")\n",
    "dropped = metadata[drop_mask].copy()\n",
    "metadata = metadata[~drop_mask]\n",
    "\n",
    "# Check language\n",
    "metadata[\"lang\"] = metadata[\"clean_markdown\"].map(detect)\n",
    "not_english = metadata[\"lang\"] != \"en\"\n",
    "print(f\"{not_english.sum()} non-english texts removed:\")\n",
    "print(*metadata[not_english][\"link\"], sep=\"\\n\")\n",
    "dropped = pd.concat([dropped, metadata[not_english]])\n",
    "metadata = metadata[~not_english]\n",
    "\n",
    "print(f\"{len(metadata)} texts remain\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "933f73fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>...</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "      <th>raw_markdown</th>\n",
       "      <th>clean_markdown</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [UID, link, submission_author, submission_id, submission_title, subreddit, submission_flair, submission_score, submission_upvote_ratio, submission_date, comment_author, comment_id, comment_score, comment_body, comment_date, src_url, manually_reviewed, related_link, raw_markdown, clean_markdown, lang]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 21 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropped[dropped[\"lang\"] == \"en\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "62ad3f33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>...</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "      <th>raw_markdown</th>\n",
       "      <th>clean_markdown</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>94</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1K6G...</td>\n",
       "      <td>Reinaldnaufal</td>\n",
       "      <td>lnxzo1</td>\n",
       "      <td>Barbarian Subclass - Path of the Siege | For t...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>286</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2021-02-20 03:22:27</td>\n",
       "      <td>...</td>\n",
       "      <td>goat7nn</td>\n",
       "      <td>1</td>\n",
       "      <td>Hey! Appreciate the input.\\n\\nI agree with the...</td>\n",
       "      <td>2021-02-22 03:07:42</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1K6...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1K6GHAH_Z1kNO...</td>\n",
       "      <td>\"stack\":\"Error: File not found: 1 K 6 GHAH_Z 1...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>239</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1SNc...</td>\n",
       "      <td>SkirtWearingSlutBoi</td>\n",
       "      <td>ln0sme</td>\n",
       "      <td>Egg's Enhanced Shields - Brand New Array of Ea...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Item</td>\n",
       "      <td>148</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2021-02-18 23:39:54</td>\n",
       "      <td>...</td>\n",
       "      <td>gny4xrm</td>\n",
       "      <td>7</td>\n",
       "      <td>[Homebrewery Link.](https://homebrewery.natura...</td>\n",
       "      <td>2021-02-18 23:50:25</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1SN...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1SNcTl5ohA2p3...</td>\n",
       "      <td>\"stack\":\"Error: File not found: 1 SNcTl 5 ohA ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>982</th>\n",
       "      <td>999</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1Zf0...</td>\n",
       "      <td>matsozetex11</td>\n",
       "      <td>nu6zwl</td>\n",
       "      <td>Darker Firearms - A take on firearm mechanics</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Item</td>\n",
       "      <td>57</td>\n",
       "      <td>0.92</td>\n",
       "      <td>2021-06-07 07:54:36</td>\n",
       "      <td>...</td>\n",
       "      <td>h0vxww8</td>\n",
       "      <td>3</td>\n",
       "      <td>This is my first homebrew posted publicly on t...</td>\n",
       "      <td>2021-06-07 07:56:59</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1Zf...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1Zf0IQKmtx9mi...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1888</th>\n",
       "      <td>2428</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1yAR...</td>\n",
       "      <td>maxnis</td>\n",
       "      <td>o95p8g</td>\n",
       "      <td>Ecrenians, Undead elves for a homebrew setting...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Race</td>\n",
       "      <td>6</td>\n",
       "      <td>0.87</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1yA...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1yARx8g8ufNap...</td>\n",
       "      <td>\"stack\":\"Error: File not found: 1 yARx 8 g 8 u...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1900</th>\n",
       "      <td>2440</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1PIt...</td>\n",
       "      <td>MBluna9</td>\n",
       "      <td>og6ocy</td>\n",
       "      <td>Summon Ooze and Summon Plant, the summoning sp...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>5</td>\n",
       "      <td>0.74</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1PI...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1PItn1Plb2qGd...</td>\n",
       "      <td>\"stack\":\"Error: File not found: 1 PItn 1 Plb 2...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>2517</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1vm0...</td>\n",
       "      <td>MrTzaangor</td>\n",
       "      <td>s28fr2</td>\n",
       "      <td>Magic From Beyond - All art credits in document</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1vm...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 1vm0x8Y63Rm3S...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1986</th>\n",
       "      <td>2529</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/16jt...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>sij0vw</td>\n",
       "      <td>College of Folk Bard subclass v2</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>3</td>\n",
       "      <td>0.80</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/16j...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 16jtuooS8rYVh...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2543</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/126h...</td>\n",
       "      <td>MrTzaangor</td>\n",
       "      <td>s35nzq</td>\n",
       "      <td>Races of Tairth - All art credits in document</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Race</td>\n",
       "      <td>10</td>\n",
       "      <td>1.00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/126...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>{\"stack\":\"Error: File not found: 126hk_D1cqhUn...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2223</th>\n",
       "      <td>2805</td>\n",
       "      <td>https://gmbinder.com/share/-MKHCarZv-OYPmgoduh9</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>lpphds</td>\n",
       "      <td>Marksman v3: A Rogue subclass for Flintlock sh...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>41</td>\n",
       "      <td>0.91</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gocdlcc</td>\n",
       "      <td>3</td>\n",
       "      <td>The colonial wars of North America were fought...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MKHCarZv-OYPmgoduh9</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Marksman (Rogue Subclass)\\n\\n\\nby Moonstrou...</td>\n",
       "      <td>## Marksman (Rogue Subclass)\\n\\n##### Flagbear...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2226</th>\n",
       "      <td>2809</td>\n",
       "      <td>https://gmbinder.com/share/-MEUxzSRW9kf5wYfT0Oj</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>i8jtxd</td>\n",
       "      <td>Muskets, Dueling Pistols, and other 18th-Centu...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Item</td>\n",
       "      <td>1304</td>\n",
       "      <td>0.99</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>g18tgvz</td>\n",
       "      <td>25</td>\n",
       "      <td>From the Seven Years War to the American Revol...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MEUxzSRW9kf5wYfT0Oj</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Flintlock Firearms (Equipment)\\n\\n\\nby Moon...</td>\n",
       "      <td>## Flintlock Firearms (Equipment)\\n\\n##### Fla...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2227</th>\n",
       "      <td>2810</td>\n",
       "      <td>https://gmbinder.com/share/-M_BKYtvcFV1OIBjaYP-</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>n7qkbg</td>\n",
       "      <td>Courtly Intrigues: Spells to Discredit and Dec...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>37</td>\n",
       "      <td>0.93</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gxe3flm</td>\n",
       "      <td>1</td>\n",
       "      <td>In some respects the 18th century was a transi...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-M_BKYtvcFV1OIBjaYP-</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Courtly Intrigues (Spells)\\n\\n\\nby Moonstro...</td>\n",
       "      <td>## Courtly Intrigues (Spells)\\n\\n##### Flagbea...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2231</th>\n",
       "      <td>2817</td>\n",
       "      <td>https://gmbinder.com/share/-M_pNobz8LY49mB9eRiB</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>ndq2o2</td>\n",
       "      <td>Field Engineering: Spells to counter Enemy Art...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>31</td>\n",
       "      <td>0.91</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gybw85g</td>\n",
       "      <td>1</td>\n",
       "      <td>In the long 18th Century, few weapons were as ...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-M_pNobz8LY49mB9eRiB</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Field Engineering (Spells)\\n\\n\\nby Moonstro...</td>\n",
       "      <td>## Field Engineering (Spells)\\n\\n##### Flagbea...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2232</th>\n",
       "      <td>2819</td>\n",
       "      <td>https://gmbinder.com/share/-MdCpGu3Ga5xQt-uHW4h</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>o8xvho</td>\n",
       "      <td>Colonial Backgrounds: character options for 18...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Background</td>\n",
       "      <td>58</td>\n",
       "      <td>0.95</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>h37p9d8</td>\n",
       "      <td>3</td>\n",
       "      <td>These new backgrounds are designed to suppleme...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MdCpGu3Ga5xQt-uHW4h</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Colonial Backgrounds\\n\\n\\nby Moonstrous\\n\\n...</td>\n",
       "      <td>## Colonial Backgrounds\\n\\n##### Flagbearer Games</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2234</th>\n",
       "      <td>2822</td>\n",
       "      <td>https://gmbinder.com/share/-MaOp5n_rJ34mUVIsrwD</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>njacgm</td>\n",
       "      <td>Rapid Deployment: Infantry Spells to Storm the...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>31</td>\n",
       "      <td>0.95</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gz66nwt</td>\n",
       "      <td>1</td>\n",
       "      <td>A [Song](https://www.loc.gov/resource/amss.as1...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MaOp5n_rJ34mUVIsrwD</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Rapid Deployment (Spells)\\n\\n\\nby Moonstrou...</td>\n",
       "      <td>## Rapid Deployment (Spells)\\n\\n##### Flagbear...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2235</th>\n",
       "      <td>2823</td>\n",
       "      <td>https://gmbinder.com/share/-MasFCl8pZobbu4M5cFr</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>nsvbu8</td>\n",
       "      <td>The Turncoat: From poisons to ciphers, pick di...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>52</td>\n",
       "      <td>0.92</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>h0okpii</td>\n",
       "      <td>3</td>\n",
       "      <td>&gt;Treason of the blackest dye was yesterday dis...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MasFCl8pZobbu4M5cFr</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Turncoat (Fighter Subclass)\\n\\n\\nby Moonstr...</td>\n",
       "      <td>## Turncoat (Fighter Subclass)\\n\\n##### Flagbe...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2239</th>\n",
       "      <td>2827</td>\n",
       "      <td>https://gmbinder.com/share/-MVC1WS7u7OEvbA69HHt</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>m0gm2p</td>\n",
       "      <td>Portage and Logjam: Spells to Traverse (and Tr...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>21</td>\n",
       "      <td>0.90</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gq7n320</td>\n",
       "      <td>1</td>\n",
       "      <td>*In Colonial America, a portage was a place of...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MVC1WS7u7OEvbA69HHt</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Portage and Logjam (Spells)\\n\\n\\nby Moonstr...</td>\n",
       "      <td>## Portage and Logjam (Spells)\\n\\n##### Flagbe...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2246</th>\n",
       "      <td>2835</td>\n",
       "      <td>https://gmbinder.com/share/-MVfl4G6lTqvl7Rnj8i5</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>m468sy</td>\n",
       "      <td>Revolutionary Partisans: Minuteman and Highlan...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Monster</td>\n",
       "      <td>233</td>\n",
       "      <td>0.98</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>gqsoqq0</td>\n",
       "      <td>7</td>\n",
       "      <td>Among the fiercest fighters in colonial militi...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MVfl4G6lTqvl7Rnj8i5</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Highlander and Minutemen (Enemies)\\n\\n\\nby ...</td>\n",
       "      <td>## Highlander and Minutemen (Enemies)\\n\\n#####...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>218</th>\n",
       "      <td>3206</td>\n",
       "      <td>https://gmbinder.com/share/-N9MFWsfVsqNIvmGtsIf</td>\n",
       "      <td>moonstrous</td>\n",
       "      <td>wneulg</td>\n",
       "      <td>Philadelphia, 1777: Adventure Map for a Revolu...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>World</td>\n",
       "      <td>26</td>\n",
       "      <td>0.86</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>ik4l9wz</td>\n",
       "      <td>5</td>\n",
       "      <td>&gt;*While the Continentals and Redcoats clash ne...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-N9MFWsfVsqNIvmGtsIf</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Philadelphia, 1777 (Map)\\n\\n\\nby Moonstrous...</td>\n",
       "      <td>## Philadelphia, 17 (Map)\\n\\n##### Flagbearer ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2329</th>\n",
       "      <td>4518</td>\n",
       "      <td>https://gmbinder.com/share/-M_21dYDnYBcNOiz7lho</td>\n",
       "      <td>Effusion-</td>\n",
       "      <td>s3ms3m</td>\n",
       "      <td>Pointless Sorcery: an alternative to sorcerer ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>3</td>\n",
       "      <td>0.67</td>\n",
       "      <td>2022-01-14 07:46:58</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-M_21dYDnYBcNOiz7lho</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>`NoSuchKey`The specified key does not exist.-M...</td>\n",
       "      <td>NoSuchKeyThe specified key does not exist. met...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2330</th>\n",
       "      <td>4519</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/W5Rp...</td>\n",
       "      <td>CapCece</td>\n",
       "      <td>m40dva</td>\n",
       "      <td>Are you an Inventor? A bold Experimentor? Beco...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>18</td>\n",
       "      <td>0.95</td>\n",
       "      <td>2021-03-13 06:00:07</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/W5R...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2331</th>\n",
       "      <td>4520</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/A40C...</td>\n",
       "      <td>Homebrewer_L_Rennick</td>\n",
       "      <td>lqztvq</td>\n",
       "      <td>Dodge Ratings - A Replacement for Armor Classes</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Mechanic</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2021-02-24 01:44:57</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/A40...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2332</th>\n",
       "      <td>4521</td>\n",
       "      <td>https://gmbinder.com/share/-Mt8S5fHlKk68lKTIyJ2</td>\n",
       "      <td>Metal-Wolf-Enrif</td>\n",
       "      <td>s308en</td>\n",
       "      <td>MWE's Hacks for DnD - WIP</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2022-01-13 14:07:15</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-Mt8S5fHlKk68lKTIyJ2</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>`NoSuchKey`The specified key does not exist.-M...</td>\n",
       "      <td>NoSuchKeyThe specified key does not exist. -Mt...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2333</th>\n",
       "      <td>4522</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/myPa...</td>\n",
       "      <td>PropitiousRaider</td>\n",
       "      <td>t55gxj</td>\n",
       "      <td>Conduit Bloodline Sorcerer Subclass for D&amp;D 5e...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>8</td>\n",
       "      <td>0.80</td>\n",
       "      <td>2022-03-02 17:35:59</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/myP...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2334</th>\n",
       "      <td>4523</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/BJXc...</td>\n",
       "      <td>himothebard</td>\n",
       "      <td>dg56yp</td>\n",
       "      <td>Circle of The Hydrohomies- A druidic circle ba...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>3</td>\n",
       "      <td>0.58</td>\n",
       "      <td>2019-10-10 21:53:23</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/BJX...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>Brew not found in Homebrewery database or Goog...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>318</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1KGp...</td>\n",
       "      <td>DoktorKarp</td>\n",
       "      <td>st73kp</td>\n",
       "      <td>The Entropist (V 2.0) | A frontline Artificier...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>186</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2022-02-15 16:38:38</td>\n",
       "      <td>...</td>\n",
       "      <td>hx1zlnd</td>\n",
       "      <td>3</td>\n",
       "      <td>Thanks for reading! As always, feedback is muc...</td>\n",
       "      <td>2022-02-15 16:54:02</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1KG...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>```metadata\\ntitle: Demon Slayer - The Wisteri...</td>\n",
       "      <td># Demon Slayer\\n# The Wisteria War\\n\\n# Razze\\...</td>\n",
       "      <td>it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>794</th>\n",
       "      <td>807</td>\n",
       "      <td>https://gmbinder.com/share/-MYPz7gxn3cOLZEI0wrj</td>\n",
       "      <td>McKirby98</td>\n",
       "      <td>n7ljl2</td>\n",
       "      <td>Way of the Element Bender - Final Version. A (...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>1863</td>\n",
       "      <td>0.99</td>\n",
       "      <td>2021-05-08 09:29:13</td>\n",
       "      <td>...</td>\n",
       "      <td>gxdgwz0</td>\n",
       "      <td>41</td>\n",
       "      <td>Hello again! Following the suggestions on my p...</td>\n",
       "      <td>2021-05-08 10:18:57</td>\n",
       "      <td>https://www.gmbinder.com/share/-MYPz7gxn3cOLZE...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>&lt;style&gt;\\n/* background image*/\\n .phb{ backgro...</td>\n",
       "      <td># Tradición Monástica\\n## Camino del Maestro E...</td>\n",
       "      <td>es</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>2996</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/Fg2G...</td>\n",
       "      <td>The9thHuman</td>\n",
       "      <td>wid557</td>\n",
       "      <td>Ciao, sono di nuovo io, ho creato una nuova cl...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Missing</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2022-08-07 11:02:46</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/Fg2...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td># Incantatore della Lama\\nIncantatori della La...</td>\n",
       "      <td># Incantatore della Lama\\nIncantatori della La...</td>\n",
       "      <td>it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>595</th>\n",
       "      <td>4092</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/M9Dw...</td>\n",
       "      <td>PropitiousRaider</td>\n",
       "      <td>yzczkt</td>\n",
       "      <td>Order of the Vampire V1.5 + Bonus Griffon Drak...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>109</td>\n",
       "      <td>0.99</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>ix8644b</td>\n",
       "      <td>3</td>\n",
       "      <td>Dude, i really liked the material, so i took t...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/M9D...</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>```metadata\\ntitle: Ordem do Vampiro\\ndescript...</td>\n",
       "      <td>:\\n# Ordem dos\\n# Vampiros\\n\\n\\nUma subclasse ...</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>4093</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/Ph_5...</td>\n",
       "      <td>PropitiousRaider</td>\n",
       "      <td>yzczkt</td>\n",
       "      <td>Order of the Vampire V1.5 + Bonus Griffon Drak...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>109</td>\n",
       "      <td>0.99</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>ix8644b</td>\n",
       "      <td>3</td>\n",
       "      <td>Dude, i really liked the material, so i took t...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/Ph_...</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>```metadata\\ntitle: Guardião dos Céus\\ndescrip...</td>\n",
       "      <td># Guardiao dos Ceus\\n\\n## Subclasse de Patrulh...</td>\n",
       "      <td>pt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>29 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       UID                                               link  \\\n",
       "91      94  https://homebrewery.naturalcrit.com/share/1K6G...   \n",
       "234    239  https://homebrewery.naturalcrit.com/share/1SNc...   \n",
       "982    999  https://homebrewery.naturalcrit.com/share/1Zf0...   \n",
       "1888  2428  https://homebrewery.naturalcrit.com/share/1yAR...   \n",
       "1900  2440  https://homebrewery.naturalcrit.com/share/1PIt...   \n",
       "1974  2517  https://homebrewery.naturalcrit.com/share/1vm0...   \n",
       "1986  2529  https://homebrewery.naturalcrit.com/share/16jt...   \n",
       "1999  2543  https://homebrewery.naturalcrit.com/share/126h...   \n",
       "2223  2805    https://gmbinder.com/share/-MKHCarZv-OYPmgoduh9   \n",
       "2226  2809    https://gmbinder.com/share/-MEUxzSRW9kf5wYfT0Oj   \n",
       "2227  2810    https://gmbinder.com/share/-M_BKYtvcFV1OIBjaYP-   \n",
       "2231  2817    https://gmbinder.com/share/-M_pNobz8LY49mB9eRiB   \n",
       "2232  2819    https://gmbinder.com/share/-MdCpGu3Ga5xQt-uHW4h   \n",
       "2234  2822    https://gmbinder.com/share/-MaOp5n_rJ34mUVIsrwD   \n",
       "2235  2823    https://gmbinder.com/share/-MasFCl8pZobbu4M5cFr   \n",
       "2239  2827    https://gmbinder.com/share/-MVC1WS7u7OEvbA69HHt   \n",
       "2246  2835    https://gmbinder.com/share/-MVfl4G6lTqvl7Rnj8i5   \n",
       "218   3206    https://gmbinder.com/share/-N9MFWsfVsqNIvmGtsIf   \n",
       "2329  4518    https://gmbinder.com/share/-M_21dYDnYBcNOiz7lho   \n",
       "2330  4519  https://homebrewery.naturalcrit.com/share/W5Rp...   \n",
       "2331  4520  https://homebrewery.naturalcrit.com/share/A40C...   \n",
       "2332  4521    https://gmbinder.com/share/-Mt8S5fHlKk68lKTIyJ2   \n",
       "2333  4522  https://homebrewery.naturalcrit.com/share/myPa...   \n",
       "2334  4523  https://homebrewery.naturalcrit.com/share/BJXc...   \n",
       "311    318  https://homebrewery.naturalcrit.com/share/1KGp...   \n",
       "794    807    https://gmbinder.com/share/-MYPz7gxn3cOLZEI0wrj   \n",
       "36    2996  https://homebrewery.naturalcrit.com/share/Fg2G...   \n",
       "595   4092  https://homebrewery.naturalcrit.com/share/M9Dw...   \n",
       "596   4093  https://homebrewery.naturalcrit.com/share/Ph_5...   \n",
       "\n",
       "         submission_author submission_id  \\\n",
       "91           Reinaldnaufal        lnxzo1   \n",
       "234    SkirtWearingSlutBoi        ln0sme   \n",
       "982           matsozetex11        nu6zwl   \n",
       "1888                maxnis        o95p8g   \n",
       "1900               MBluna9        og6ocy   \n",
       "1974            MrTzaangor        s28fr2   \n",
       "1986                   NaN        sij0vw   \n",
       "1999            MrTzaangor        s35nzq   \n",
       "2223            moonstrous        lpphds   \n",
       "2226            moonstrous        i8jtxd   \n",
       "2227            moonstrous        n7qkbg   \n",
       "2231            moonstrous        ndq2o2   \n",
       "2232            moonstrous        o8xvho   \n",
       "2234            moonstrous        njacgm   \n",
       "2235            moonstrous        nsvbu8   \n",
       "2239            moonstrous        m0gm2p   \n",
       "2246            moonstrous        m468sy   \n",
       "218             moonstrous        wneulg   \n",
       "2329             Effusion-        s3ms3m   \n",
       "2330               CapCece        m40dva   \n",
       "2331  Homebrewer_L_Rennick        lqztvq   \n",
       "2332      Metal-Wolf-Enrif        s308en   \n",
       "2333      PropitiousRaider        t55gxj   \n",
       "2334           himothebard        dg56yp   \n",
       "311             DoktorKarp        st73kp   \n",
       "794              McKirby98        n7ljl2   \n",
       "36             The9thHuman        wid557   \n",
       "595       PropitiousRaider        yzczkt   \n",
       "596       PropitiousRaider        yzczkt   \n",
       "\n",
       "                                       submission_title        subreddit  \\\n",
       "91    Barbarian Subclass - Path of the Siege | For t...  UnearthedArcana   \n",
       "234   Egg's Enhanced Shields - Brand New Array of Ea...  UnearthedArcana   \n",
       "982       Darker Firearms - A take on firearm mechanics  UnearthedArcana   \n",
       "1888  Ecrenians, Undead elves for a homebrew setting...  UnearthedArcana   \n",
       "1900  Summon Ooze and Summon Plant, the summoning sp...  UnearthedArcana   \n",
       "1974    Magic From Beyond - All art credits in document  UnearthedArcana   \n",
       "1986                   College of Folk Bard subclass v2  UnearthedArcana   \n",
       "1999      Races of Tairth - All art credits in document  UnearthedArcana   \n",
       "2223  Marksman v3: A Rogue subclass for Flintlock sh...  UnearthedArcana   \n",
       "2226  Muskets, Dueling Pistols, and other 18th-Centu...  UnearthedArcana   \n",
       "2227  Courtly Intrigues: Spells to Discredit and Dec...  UnearthedArcana   \n",
       "2231  Field Engineering: Spells to counter Enemy Art...  UnearthedArcana   \n",
       "2232  Colonial Backgrounds: character options for 18...  UnearthedArcana   \n",
       "2234  Rapid Deployment: Infantry Spells to Storm the...  UnearthedArcana   \n",
       "2235  The Turncoat: From poisons to ciphers, pick di...  UnearthedArcana   \n",
       "2239  Portage and Logjam: Spells to Traverse (and Tr...  UnearthedArcana   \n",
       "2246  Revolutionary Partisans: Minuteman and Highlan...  UnearthedArcana   \n",
       "218   Philadelphia, 1777: Adventure Map for a Revolu...  UnearthedArcana   \n",
       "2329  Pointless Sorcery: an alternative to sorcerer ...  UnearthedArcana   \n",
       "2330  Are you an Inventor? A bold Experimentor? Beco...  UnearthedArcana   \n",
       "2331    Dodge Ratings - A Replacement for Armor Classes  UnearthedArcana   \n",
       "2332                          MWE's Hacks for DnD - WIP  UnearthedArcana   \n",
       "2333  Conduit Bloodline Sorcerer Subclass for D&D 5e...  UnearthedArcana   \n",
       "2334  Circle of The Hydrohomies- A druidic circle ba...  UnearthedArcana   \n",
       "311   The Entropist (V 2.0) | A frontline Artificier...  UnearthedArcana   \n",
       "794   Way of the Element Bender - Final Version. A (...  UnearthedArcana   \n",
       "36    Ciao, sono di nuovo io, ho creato una nuova cl...      DnDHomebrew   \n",
       "595   Order of the Vampire V1.5 + Bonus Griffon Drak...  UnearthedArcana   \n",
       "596   Order of the Vampire V1.5 + Bonus Griffon Drak...  UnearthedArcana   \n",
       "\n",
       "     submission_flair  submission_score  submission_upvote_ratio  \\\n",
       "91           Subclass               286                     0.97   \n",
       "234              Item               148                     0.97   \n",
       "982              Item                57                     0.92   \n",
       "1888             Race                 6                     0.87   \n",
       "1900            Spell                 5                     0.74   \n",
       "1974            Spell                 3                     1.00   \n",
       "1986         Subclass                 3                     0.80   \n",
       "1999             Race                10                     1.00   \n",
       "2223            Class                41                     0.91   \n",
       "2226             Item              1304                     0.99   \n",
       "2227            Spell                37                     0.93   \n",
       "2231            Spell                31                     0.91   \n",
       "2232       Background                58                     0.95   \n",
       "2234            Spell                31                     0.95   \n",
       "2235         Subclass                52                     0.92   \n",
       "2239            Spell                21                     0.90   \n",
       "2246          Monster               233                     0.98   \n",
       "218             World                26                     0.86   \n",
       "2329         Subclass                 3                     0.67   \n",
       "2330         Subclass                18                     0.95   \n",
       "2331         Mechanic                 1                     1.00   \n",
       "2332       Compendium                 3                     0.72   \n",
       "2333         Subclass                 8                     0.80   \n",
       "2334         Subclass                 3                     0.58   \n",
       "311          Subclass               186                     0.97   \n",
       "794          Subclass              1863                     0.99   \n",
       "36            Missing                 3                     0.72   \n",
       "595          Subclass               109                     0.99   \n",
       "596          Subclass               109                     0.99   \n",
       "\n",
       "         submission_date  ... comment_id comment_score  \\\n",
       "91   2021-02-20 03:22:27  ...    goat7nn             1   \n",
       "234  2021-02-18 23:39:54  ...    gny4xrm             7   \n",
       "982  2021-06-07 07:54:36  ...    h0vxww8             3   \n",
       "1888                 NaT  ...        NaN          <NA>   \n",
       "1900                 NaT  ...        NaN          <NA>   \n",
       "1974                 NaT  ...        NaN          <NA>   \n",
       "1986                 NaT  ...        NaN          <NA>   \n",
       "1999                 NaT  ...        NaN          <NA>   \n",
       "2223                 NaT  ...    gocdlcc             3   \n",
       "2226                 NaT  ...    g18tgvz            25   \n",
       "2227                 NaT  ...    gxe3flm             1   \n",
       "2231                 NaT  ...    gybw85g             1   \n",
       "2232                 NaT  ...    h37p9d8             3   \n",
       "2234                 NaT  ...    gz66nwt             1   \n",
       "2235                 NaT  ...    h0okpii             3   \n",
       "2239                 NaT  ...    gq7n320             1   \n",
       "2246                 NaT  ...    gqsoqq0             7   \n",
       "218                  NaT  ...    ik4l9wz             5   \n",
       "2329 2022-01-14 07:46:58  ...        NaN          <NA>   \n",
       "2330 2021-03-13 06:00:07  ...        NaN          <NA>   \n",
       "2331 2021-02-24 01:44:57  ...        NaN          <NA>   \n",
       "2332 2022-01-13 14:07:15  ...        NaN          <NA>   \n",
       "2333 2022-03-02 17:35:59  ...        NaN          <NA>   \n",
       "2334 2019-10-10 21:53:23  ...        NaN          <NA>   \n",
       "311  2022-02-15 16:38:38  ...    hx1zlnd             3   \n",
       "794  2021-05-08 09:29:13  ...    gxdgwz0            41   \n",
       "36   2022-08-07 11:02:46  ...        NaN          <NA>   \n",
       "595                  NaT  ...    ix8644b             3   \n",
       "596                  NaT  ...    ix8644b             3   \n",
       "\n",
       "                                           comment_body        comment_date  \\\n",
       "91    Hey! Appreciate the input.\\n\\nI agree with the... 2021-02-22 03:07:42   \n",
       "234   [Homebrewery Link.](https://homebrewery.natura... 2021-02-18 23:50:25   \n",
       "982   This is my first homebrew posted publicly on t... 2021-06-07 07:56:59   \n",
       "1888                                                NaN                 NaT   \n",
       "1900                                                NaN                 NaT   \n",
       "1974                                                NaN                 NaT   \n",
       "1986                                                NaN                 NaT   \n",
       "1999                                                NaN                 NaT   \n",
       "2223  The colonial wars of North America were fought...                 NaT   \n",
       "2226  From the Seven Years War to the American Revol...                 NaT   \n",
       "2227  In some respects the 18th century was a transi...                 NaT   \n",
       "2231  In the long 18th Century, few weapons were as ...                 NaT   \n",
       "2232  These new backgrounds are designed to suppleme...                 NaT   \n",
       "2234  A [Song](https://www.loc.gov/resource/amss.as1...                 NaT   \n",
       "2235  >Treason of the blackest dye was yesterday dis...                 NaT   \n",
       "2239  *In Colonial America, a portage was a place of...                 NaT   \n",
       "2246  Among the fiercest fighters in colonial militi...                 NaT   \n",
       "218   >*While the Continentals and Redcoats clash ne...                 NaT   \n",
       "2329                                                NaN                 NaT   \n",
       "2330                                                NaN                 NaT   \n",
       "2331                                                NaN                 NaT   \n",
       "2332                                                NaN                 NaT   \n",
       "2333                                                NaN                 NaT   \n",
       "2334                                                NaN                 NaT   \n",
       "311   Thanks for reading! As always, feedback is muc... 2022-02-15 16:54:02   \n",
       "794   Hello again! Following the suggestions on my p... 2021-05-08 10:18:57   \n",
       "36                                                  NaN                 NaT   \n",
       "595   Dude, i really liked the material, so i took t...                 NaT   \n",
       "596   Dude, i really liked the material, so i took t...                 NaT   \n",
       "\n",
       "                                                src_url manually_reviewed  \\\n",
       "91    https://homebrewery.naturalcrit.com/source/1K6...             False   \n",
       "234   https://homebrewery.naturalcrit.com/source/1SN...             False   \n",
       "982   https://homebrewery.naturalcrit.com/source/1Zf...             False   \n",
       "1888  https://homebrewery.naturalcrit.com/source/1yA...             False   \n",
       "1900  https://homebrewery.naturalcrit.com/source/1PI...             False   \n",
       "1974  https://homebrewery.naturalcrit.com/source/1vm...             False   \n",
       "1986  https://homebrewery.naturalcrit.com/source/16j...             False   \n",
       "1999  https://homebrewery.naturalcrit.com/source/126...             False   \n",
       "2223    https://gmbinder.com/share/-MKHCarZv-OYPmgoduh9             False   \n",
       "2226    https://gmbinder.com/share/-MEUxzSRW9kf5wYfT0Oj             False   \n",
       "2227    https://gmbinder.com/share/-M_BKYtvcFV1OIBjaYP-             False   \n",
       "2231    https://gmbinder.com/share/-M_pNobz8LY49mB9eRiB             False   \n",
       "2232    https://gmbinder.com/share/-MdCpGu3Ga5xQt-uHW4h             False   \n",
       "2234    https://gmbinder.com/share/-MaOp5n_rJ34mUVIsrwD             False   \n",
       "2235    https://gmbinder.com/share/-MasFCl8pZobbu4M5cFr             False   \n",
       "2239    https://gmbinder.com/share/-MVC1WS7u7OEvbA69HHt             False   \n",
       "2246    https://gmbinder.com/share/-MVfl4G6lTqvl7Rnj8i5             False   \n",
       "218     https://gmbinder.com/share/-N9MFWsfVsqNIvmGtsIf             False   \n",
       "2329    https://gmbinder.com/share/-M_21dYDnYBcNOiz7lho             False   \n",
       "2330  https://homebrewery.naturalcrit.com/source/W5R...             False   \n",
       "2331  https://homebrewery.naturalcrit.com/source/A40...             False   \n",
       "2332    https://gmbinder.com/share/-Mt8S5fHlKk68lKTIyJ2             False   \n",
       "2333  https://homebrewery.naturalcrit.com/source/myP...             False   \n",
       "2334  https://homebrewery.naturalcrit.com/source/BJX...             False   \n",
       "311   https://homebrewery.naturalcrit.com/source/1KG...             False   \n",
       "794   https://www.gmbinder.com/share/-MYPz7gxn3cOLZE...              True   \n",
       "36    https://homebrewery.naturalcrit.com/source/Fg2...             False   \n",
       "595   https://homebrewery.naturalcrit.com/source/M9D...              True   \n",
       "596   https://homebrewery.naturalcrit.com/source/Ph_...              True   \n",
       "\n",
       "      related_link                                       raw_markdown  \\\n",
       "91            <NA>  {\"stack\":\"Error: File not found: 1K6GHAH_Z1kNO...   \n",
       "234           <NA>  {\"stack\":\"Error: File not found: 1SNcTl5ohA2p3...   \n",
       "982           <NA>  {\"stack\":\"Error: File not found: 1Zf0IQKmtx9mi...   \n",
       "1888          True  {\"stack\":\"Error: File not found: 1yARx8g8ufNap...   \n",
       "1900          True  {\"stack\":\"Error: File not found: 1PItn1Plb2qGd...   \n",
       "1974          True  {\"stack\":\"Error: File not found: 1vm0x8Y63Rm3S...   \n",
       "1986          True  {\"stack\":\"Error: File not found: 16jtuooS8rYVh...   \n",
       "1999          True  {\"stack\":\"Error: File not found: 126hk_D1cqhUn...   \n",
       "2223          <NA>  ## Marksman (Rogue Subclass)\\n\\n\\nby Moonstrou...   \n",
       "2226          <NA>  ## Flintlock Firearms (Equipment)\\n\\n\\nby Moon...   \n",
       "2227          <NA>  ## Courtly Intrigues (Spells)\\n\\n\\nby Moonstro...   \n",
       "2231          <NA>  ## Field Engineering (Spells)\\n\\n\\nby Moonstro...   \n",
       "2232          <NA>  ## Colonial Backgrounds\\n\\n\\nby Moonstrous\\n\\n...   \n",
       "2234          <NA>  ## Rapid Deployment (Spells)\\n\\n\\nby Moonstrou...   \n",
       "2235          <NA>  ## Turncoat (Fighter Subclass)\\n\\n\\nby Moonstr...   \n",
       "2239          <NA>  ## Portage and Logjam (Spells)\\n\\n\\nby Moonstr...   \n",
       "2246          <NA>  ## Highlander and Minutemen (Enemies)\\n\\n\\nby ...   \n",
       "218           <NA>  ## Philadelphia, 1777 (Map)\\n\\n\\nby Moonstrous...   \n",
       "2329          True  `NoSuchKey`The specified key does not exist.-M...   \n",
       "2330          True  Brew not found in Homebrewery database or Goog...   \n",
       "2331          True  Brew not found in Homebrewery database or Goog...   \n",
       "2332          True  `NoSuchKey`The specified key does not exist.-M...   \n",
       "2333          True  Brew not found in Homebrewery database or Goog...   \n",
       "2334          True  Brew not found in Homebrewery database or Goog...   \n",
       "311           <NA>  ```metadata\\ntitle: Demon Slayer - The Wisteri...   \n",
       "794           True  <style>\\n/* background image*/\\n .phb{ backgro...   \n",
       "36            True  # Incantatore della Lama\\nIncantatori della La...   \n",
       "595          False  ```metadata\\ntitle: Ordem do Vampiro\\ndescript...   \n",
       "596          False  ```metadata\\ntitle: Guardião dos Céus\\ndescrip...   \n",
       "\n",
       "                                         clean_markdown lang  \n",
       "91    \"stack\":\"Error: File not found: 1 K 6 GHAH_Z 1...  NaN  \n",
       "234   \"stack\":\"Error: File not found: 1 SNcTl 5 ohA ...  NaN  \n",
       "982                                                      NaN  \n",
       "1888  \"stack\":\"Error: File not found: 1 yARx 8 g 8 u...  NaN  \n",
       "1900  \"stack\":\"Error: File not found: 1 PItn 1 Plb 2...  NaN  \n",
       "1974                                                     NaN  \n",
       "1986                                                     NaN  \n",
       "1999                                                     NaN  \n",
       "2223  ## Marksman (Rogue Subclass)\\n\\n##### Flagbear...  NaN  \n",
       "2226  ## Flintlock Firearms (Equipment)\\n\\n##### Fla...  NaN  \n",
       "2227  ## Courtly Intrigues (Spells)\\n\\n##### Flagbea...  NaN  \n",
       "2231  ## Field Engineering (Spells)\\n\\n##### Flagbea...  NaN  \n",
       "2232  ## Colonial Backgrounds\\n\\n##### Flagbearer Games  NaN  \n",
       "2234  ## Rapid Deployment (Spells)\\n\\n##### Flagbear...  NaN  \n",
       "2235  ## Turncoat (Fighter Subclass)\\n\\n##### Flagbe...  NaN  \n",
       "2239  ## Portage and Logjam (Spells)\\n\\n##### Flagbe...  NaN  \n",
       "2246  ## Highlander and Minutemen (Enemies)\\n\\n#####...  NaN  \n",
       "218   ## Philadelphia, 17 (Map)\\n\\n##### Flagbearer ...  NaN  \n",
       "2329  NoSuchKeyThe specified key does not exist. met...  NaN  \n",
       "2330  Brew not found in Homebrewery database or Goog...  NaN  \n",
       "2331  Brew not found in Homebrewery database or Goog...  NaN  \n",
       "2332  NoSuchKeyThe specified key does not exist. -Mt...  NaN  \n",
       "2333  Brew not found in Homebrewery database or Goog...  NaN  \n",
       "2334  Brew not found in Homebrewery database or Goog...  NaN  \n",
       "311   # Demon Slayer\\n# The Wisteria War\\n\\n# Razze\\...   it  \n",
       "794   # Tradición Monástica\\n## Camino del Maestro E...   es  \n",
       "36    # Incantatore della Lama\\nIncantatori della La...   it  \n",
       "595   :\\n# Ordem dos\\n# Vampiros\\n\\n\\nUma subclasse ...   pt  \n",
       "596   # Guardiao dos Ceus\\n\\n## Subclasse de Patrulh...   pt  \n",
       "\n",
       "[29 rows x 21 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropped[dropped[\"lang\"] != \"en\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "348a7726",
   "metadata": {},
   "source": [
    "### Remove rare flair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "341e50b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "submission_flair\n",
       "Adventure        3\n",
       "Background       7\n",
       "Class          478\n",
       "Compendium     152\n",
       "Feat           122\n",
       "Item           166\n",
       "Mechanic       104\n",
       "Missing          0\n",
       "Monster        562\n",
       "Official         1\n",
       "Other            2\n",
       "Race           229\n",
       "Resource         5\n",
       "Spell          194\n",
       "Subclass      1187\n",
       "World            1\n",
       "Name: UID, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.groupby(by=[\"submission_flair\"])[\"UID\"].agg(\"count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6d0977dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>...</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "      <th>raw_markdown</th>\n",
       "      <th>clean_markdown</th>\n",
       "      <th>lang</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>337</th>\n",
       "      <td>344</td>\n",
       "      <td>https://gmbinder.com/share/-MkL3KLR7-tbnxdErq_R</td>\n",
       "      <td>AutoModerator</td>\n",
       "      <td>smtkbu</td>\n",
       "      <td>The Arcana Forge! For all your drafts, ideas, ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Official</td>\n",
       "      <td>16</td>\n",
       "      <td>0.91</td>\n",
       "      <td>2022-02-07 16:00:13</td>\n",
       "      <td>...</td>\n",
       "      <td>hwyqky9</td>\n",
       "      <td>2</td>\n",
       "      <td>Hey, /r/unearthedarcana! I decided to create a...</td>\n",
       "      <td>2022-02-14 22:41:15</td>\n",
       "      <td>https://www.gmbinder.com/share/-MkL3KLR7-tbnxd...</td>\n",
       "      <td>True</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;style&gt;\\n .phb#p1:after { display:none; }\\n \\n...</td>\n",
       "      <td>Mundane Guide to Guilds\\n\\nA supplement for Ca...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1018</th>\n",
       "      <td>1035</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1583...</td>\n",
       "      <td>Kaiburr_Kath-Hound</td>\n",
       "      <td>sfjdft</td>\n",
       "      <td>Tasha's Cauldron of Everything Style/Template ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Resource</td>\n",
       "      <td>50</td>\n",
       "      <td>0.96</td>\n",
       "      <td>2022-01-29 14:37:25</td>\n",
       "      <td>...</td>\n",
       "      <td>huq1kjv</td>\n",
       "      <td>4</td>\n",
       "      <td>Edit: The title should be **Xanathar’s Guide t...</td>\n",
       "      <td>2022-01-29 14:37:32</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/158...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>```metadata\\ntitle: Xanthar's Style\\ndescripti...</td>\n",
       "      <td># Xanathar's Guide to Anything\\n## . But not E...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1626</th>\n",
       "      <td>2072</td>\n",
       "      <td>https://gmbinder.com/share/-MEP1q7iP_x3LcQVvg_k</td>\n",
       "      <td>KibblesTasty</td>\n",
       "      <td>jn7t0w</td>\n",
       "      <td>Abducted! A very strange one-shot adventure ab...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Adventure</td>\n",
       "      <td>1046</td>\n",
       "      <td>0.99</td>\n",
       "      <td>2020-11-03 10:08:41</td>\n",
       "      <td>...</td>\n",
       "      <td>gazph80</td>\n",
       "      <td>24</td>\n",
       "      <td>[**GMBinder**](https://www.gmbinder.com/share/...</td>\n",
       "      <td>2020-11-03 10:08:55</td>\n",
       "      <td>https://www.gmbinder.com/share/-MEP1q7iP_x3LcQ...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>## Summary\\n\\nThe following is an adventure in...</td>\n",
       "      <td>## Summary\\n\\nThe following is an adventure in...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1708</th>\n",
       "      <td>2229</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1jQ1...</td>\n",
       "      <td>Kaiburr_Kath-Hound</td>\n",
       "      <td>srwhue</td>\n",
       "      <td>Dyslexia-Friendly Styling Code - Make your doc...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Resource</td>\n",
       "      <td>98</td>\n",
       "      <td>0.96</td>\n",
       "      <td>2022-02-13 23:47:18</td>\n",
       "      <td>...</td>\n",
       "      <td>hwud36p</td>\n",
       "      <td>2</td>\n",
       "      <td>Hey all, here's a project I've been wanting to...</td>\n",
       "      <td>2022-02-13 23:47:22</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1jQ...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>```metadata\\ntitle: Xanthar's Style - OpenDysl...</td>\n",
       "      <td># Xanathar's Guide to Anything\\n## . But not E...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1710</th>\n",
       "      <td>2231</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1h8J...</td>\n",
       "      <td>Kaiburr_Kath-Hound</td>\n",
       "      <td>srwhue</td>\n",
       "      <td>Dyslexia-Friendly Styling Code - Make your doc...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Resource</td>\n",
       "      <td>98</td>\n",
       "      <td>0.96</td>\n",
       "      <td>2022-02-13 23:47:18</td>\n",
       "      <td>...</td>\n",
       "      <td>hwud36p</td>\n",
       "      <td>2</td>\n",
       "      <td>Hey all, here's a project I've been wanting to...</td>\n",
       "      <td>2022-02-13 23:47:22</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1h8...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>```metadata\\ntitle: Tasha's Style - OpenDyslex...</td>\n",
       "      <td># Tasha's Cauldron of Every Single Thing\\n## T...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1716</th>\n",
       "      <td>2245</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/15OH...</td>\n",
       "      <td>Kaiburr_Kath-Hound</td>\n",
       "      <td>srwhue</td>\n",
       "      <td>Dyslexia-Friendly Styling Code - Make your doc...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Resource</td>\n",
       "      <td>98</td>\n",
       "      <td>0.96</td>\n",
       "      <td>2022-02-13 23:47:18</td>\n",
       "      <td>...</td>\n",
       "      <td>hwud36p</td>\n",
       "      <td>2</td>\n",
       "      <td>Hey all, here's a project I've been wanting to...</td>\n",
       "      <td>2022-02-13 23:47:22</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/15O...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>```metadata\\ntitle: PHB Style - OpenDyslexic\\n...</td>\n",
       "      <td># Player's Hand Book\\n\\n## Definitely not a \"H...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1717</th>\n",
       "      <td>2246</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1eK9...</td>\n",
       "      <td>Kaiburr_Kath-Hound</td>\n",
       "      <td>srwhue</td>\n",
       "      <td>Dyslexia-Friendly Styling Code - Make your doc...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Resource</td>\n",
       "      <td>98</td>\n",
       "      <td>0.96</td>\n",
       "      <td>2022-02-13 23:47:18</td>\n",
       "      <td>...</td>\n",
       "      <td>hwud36p</td>\n",
       "      <td>2</td>\n",
       "      <td>Hey all, here's a project I've been wanting to...</td>\n",
       "      <td>2022-02-13 23:47:22</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1eK...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>```metadata\\ntitle: Dyslexia-Friendly Style\\nd...</td>\n",
       "      <td># The Homebrewery\\n\\nWelcome traveler from an ...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1723</th>\n",
       "      <td>2253</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/bjmG...</td>\n",
       "      <td>dr3w_be4r</td>\n",
       "      <td>sx2xug</td>\n",
       "      <td>Harry Potter/Hogwarts Inspired Adventure! Evok...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Adventure</td>\n",
       "      <td>237</td>\n",
       "      <td>0.95</td>\n",
       "      <td>2022-02-20 14:54:50</td>\n",
       "      <td>...</td>\n",
       "      <td>hynvdze</td>\n",
       "      <td>1</td>\n",
       "      <td>Thank you! Yes https://homebrewery.naturalcrit...</td>\n",
       "      <td>2022-02-27 16:54:17</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/bjm...</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>```metadata\\ntitle: Evoking Extra Credit\\ndesc...</td>\n",
       "      <td>Credit: Wizards of the Coast\\n\\n# Evoking Extr...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1779</th>\n",
       "      <td>2312</td>\n",
       "      <td>https://gmbinder.com/share/-MVsRDHxhmCt-JRE_5GL</td>\n",
       "      <td>Finalplayer14</td>\n",
       "      <td>m5xiol</td>\n",
       "      <td>XGTE Class Traits Expanded + Artificer</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Other</td>\n",
       "      <td>34</td>\n",
       "      <td>0.87</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://www.gmbinder.com/share/-MVsRDHxhmCt-JR...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>/* Background */\\n .phb{ background-image: url...</td>\n",
       "      <td>Class Background Traits Expanded\\n\\nArt: dunge...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1880</th>\n",
       "      <td>2420</td>\n",
       "      <td>https://gmbinder.com/share/-Mcp-N9HQ6ZAOVvNpC0j</td>\n",
       "      <td>DevanT77</td>\n",
       "      <td>o8zasm</td>\n",
       "      <td>Class &amp; Subclass Variants. A reworking of ever...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Other</td>\n",
       "      <td>21</td>\n",
       "      <td>0.87</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://www.gmbinder.com/share/-Mcp-N9HQ6ZAOVv...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>/*ToC Styling*/\\n .toc a {color: inherit !impo...</td>\n",
       "      <td># Variants and House Rules\\n\\n### Artificer\\n#...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2175</th>\n",
       "      <td>2747</td>\n",
       "      <td>https://gmbinder.com/share/-MvkxgD1MbUVjq7MiiV-</td>\n",
       "      <td>PipFizzlebang</td>\n",
       "      <td>t0bivd</td>\n",
       "      <td>Frost Bitten : The Nest of the Remorhaz</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Adventure</td>\n",
       "      <td>11</td>\n",
       "      <td>0.84</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://gmbinder.com/share/-MvkxgD1MbUVjq7MiiV-</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>## The Remorhaz Lair\\n\\n\\nby PipFizzlebang\\n\\n...</td>\n",
       "      <td>## The Remorhaz Lair\\n\\nThe Nest the Remorhaz\\...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2222</th>\n",
       "      <td>2798</td>\n",
       "      <td>https://gmbinder.com/share/-Mv9wF89pU5Ltvzt0mSW</td>\n",
       "      <td>Spock_42</td>\n",
       "      <td>snfzhv</td>\n",
       "      <td>Planning a West Marches campaign set in the \"V...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>World</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>https://www.gmbinder.com/share/-Mv9wF89pU5Ltvz...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>.phb{ width: 210mm; height: 296.8mm; }\\n\\n/* B...</td>\n",
       "      <td>Valley of the Lost and Found\\n\\nA realm hidden...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       UID                                               link  \\\n",
       "337    344    https://gmbinder.com/share/-MkL3KLR7-tbnxdErq_R   \n",
       "1018  1035  https://homebrewery.naturalcrit.com/share/1583...   \n",
       "1626  2072    https://gmbinder.com/share/-MEP1q7iP_x3LcQVvg_k   \n",
       "1708  2229  https://homebrewery.naturalcrit.com/share/1jQ1...   \n",
       "1710  2231  https://homebrewery.naturalcrit.com/share/1h8J...   \n",
       "1716  2245  https://homebrewery.naturalcrit.com/share/15OH...   \n",
       "1717  2246  https://homebrewery.naturalcrit.com/share/1eK9...   \n",
       "1723  2253  https://homebrewery.naturalcrit.com/share/bjmG...   \n",
       "1779  2312    https://gmbinder.com/share/-MVsRDHxhmCt-JRE_5GL   \n",
       "1880  2420    https://gmbinder.com/share/-Mcp-N9HQ6ZAOVvNpC0j   \n",
       "2175  2747    https://gmbinder.com/share/-MvkxgD1MbUVjq7MiiV-   \n",
       "2222  2798    https://gmbinder.com/share/-Mv9wF89pU5Ltvzt0mSW   \n",
       "\n",
       "       submission_author submission_id  \\\n",
       "337        AutoModerator        smtkbu   \n",
       "1018  Kaiburr_Kath-Hound        sfjdft   \n",
       "1626        KibblesTasty        jn7t0w   \n",
       "1708  Kaiburr_Kath-Hound        srwhue   \n",
       "1710  Kaiburr_Kath-Hound        srwhue   \n",
       "1716  Kaiburr_Kath-Hound        srwhue   \n",
       "1717  Kaiburr_Kath-Hound        srwhue   \n",
       "1723           dr3w_be4r        sx2xug   \n",
       "1779       Finalplayer14        m5xiol   \n",
       "1880            DevanT77        o8zasm   \n",
       "2175       PipFizzlebang        t0bivd   \n",
       "2222            Spock_42        snfzhv   \n",
       "\n",
       "                                       submission_title        subreddit  \\\n",
       "337   The Arcana Forge! For all your drafts, ideas, ...  UnearthedArcana   \n",
       "1018  Tasha's Cauldron of Everything Style/Template ...  UnearthedArcana   \n",
       "1626  Abducted! A very strange one-shot adventure ab...      DnDHomebrew   \n",
       "1708  Dyslexia-Friendly Styling Code - Make your doc...      DnDHomebrew   \n",
       "1710  Dyslexia-Friendly Styling Code - Make your doc...      DnDHomebrew   \n",
       "1716  Dyslexia-Friendly Styling Code - Make your doc...      DnDHomebrew   \n",
       "1717  Dyslexia-Friendly Styling Code - Make your doc...      DnDHomebrew   \n",
       "1723  Harry Potter/Hogwarts Inspired Adventure! Evok...      DnDHomebrew   \n",
       "1779             XGTE Class Traits Expanded + Artificer  UnearthedArcana   \n",
       "1880  Class & Subclass Variants. A reworking of ever...  UnearthedArcana   \n",
       "2175            Frost Bitten : The Nest of the Remorhaz  UnearthedArcana   \n",
       "2222  Planning a West Marches campaign set in the \"V...  UnearthedArcana   \n",
       "\n",
       "     submission_flair  submission_score  submission_upvote_ratio  \\\n",
       "337          Official                16                     0.91   \n",
       "1018         Resource                50                     0.96   \n",
       "1626        Adventure              1046                     0.99   \n",
       "1708         Resource                98                     0.96   \n",
       "1710         Resource                98                     0.96   \n",
       "1716         Resource                98                     0.96   \n",
       "1717         Resource                98                     0.96   \n",
       "1723        Adventure               237                     0.95   \n",
       "1779            Other                34                     0.87   \n",
       "1880            Other                21                     0.87   \n",
       "2175        Adventure                11                     0.84   \n",
       "2222            World                 3                     1.00   \n",
       "\n",
       "         submission_date  ... comment_id comment_score  \\\n",
       "337  2022-02-07 16:00:13  ...    hwyqky9             2   \n",
       "1018 2022-01-29 14:37:25  ...    huq1kjv             4   \n",
       "1626 2020-11-03 10:08:41  ...    gazph80            24   \n",
       "1708 2022-02-13 23:47:18  ...    hwud36p             2   \n",
       "1710 2022-02-13 23:47:18  ...    hwud36p             2   \n",
       "1716 2022-02-13 23:47:18  ...    hwud36p             2   \n",
       "1717 2022-02-13 23:47:18  ...    hwud36p             2   \n",
       "1723 2022-02-20 14:54:50  ...    hynvdze             1   \n",
       "1779                 NaT  ...        NaN          <NA>   \n",
       "1880                 NaT  ...        NaN          <NA>   \n",
       "2175                 NaT  ...        NaN          <NA>   \n",
       "2222                 NaT  ...        NaN          <NA>   \n",
       "\n",
       "                                           comment_body        comment_date  \\\n",
       "337   Hey, /r/unearthedarcana! I decided to create a... 2022-02-14 22:41:15   \n",
       "1018  Edit: The title should be **Xanathar’s Guide t... 2022-01-29 14:37:32   \n",
       "1626  [**GMBinder**](https://www.gmbinder.com/share/... 2020-11-03 10:08:55   \n",
       "1708  Hey all, here's a project I've been wanting to... 2022-02-13 23:47:22   \n",
       "1710  Hey all, here's a project I've been wanting to... 2022-02-13 23:47:22   \n",
       "1716  Hey all, here's a project I've been wanting to... 2022-02-13 23:47:22   \n",
       "1717  Hey all, here's a project I've been wanting to... 2022-02-13 23:47:22   \n",
       "1723  Thank you! Yes https://homebrewery.naturalcrit... 2022-02-27 16:54:17   \n",
       "1779                                                NaN                 NaT   \n",
       "1880                                                NaN                 NaT   \n",
       "2175                                                NaN                 NaT   \n",
       "2222                                                NaN                 NaT   \n",
       "\n",
       "                                                src_url manually_reviewed  \\\n",
       "337   https://www.gmbinder.com/share/-MkL3KLR7-tbnxd...              True   \n",
       "1018  https://homebrewery.naturalcrit.com/source/158...             False   \n",
       "1626  https://www.gmbinder.com/share/-MEP1q7iP_x3LcQ...             False   \n",
       "1708  https://homebrewery.naturalcrit.com/source/1jQ...              True   \n",
       "1710  https://homebrewery.naturalcrit.com/source/1h8...              True   \n",
       "1716  https://homebrewery.naturalcrit.com/source/15O...              True   \n",
       "1717  https://homebrewery.naturalcrit.com/source/1eK...              True   \n",
       "1723  https://homebrewery.naturalcrit.com/source/bjm...             False   \n",
       "1779  https://www.gmbinder.com/share/-MVsRDHxhmCt-JR...             False   \n",
       "1880  https://www.gmbinder.com/share/-Mcp-N9HQ6ZAOVv...             False   \n",
       "2175    https://gmbinder.com/share/-MvkxgD1MbUVjq7MiiV-             False   \n",
       "2222  https://www.gmbinder.com/share/-Mv9wF89pU5Ltvz...             False   \n",
       "\n",
       "      related_link                                       raw_markdown  \\\n",
       "337           <NA>  <style>\\n .phb#p1:after { display:none; }\\n \\n...   \n",
       "1018          <NA>  ```metadata\\ntitle: Xanthar's Style\\ndescripti...   \n",
       "1626          <NA>  ## Summary\\n\\nThe following is an adventure in...   \n",
       "1708          True  ```metadata\\ntitle: Xanthar's Style - OpenDysl...   \n",
       "1710          True  ```metadata\\ntitle: Tasha's Style - OpenDyslex...   \n",
       "1716          True  ```metadata\\ntitle: PHB Style - OpenDyslexic\\n...   \n",
       "1717          True  ```metadata\\ntitle: Dyslexia-Friendly Style\\nd...   \n",
       "1723          <NA>  ```metadata\\ntitle: Evoking Extra Credit\\ndesc...   \n",
       "1779          True  /* Background */\\n .phb{ background-image: url...   \n",
       "1880          True  /*ToC Styling*/\\n .toc a {color: inherit !impo...   \n",
       "2175          True  ## The Remorhaz Lair\\n\\n\\nby PipFizzlebang\\n\\n...   \n",
       "2222          True  .phb{ width: 210mm; height: 296.8mm; }\\n\\n/* B...   \n",
       "\n",
       "                                         clean_markdown lang  \n",
       "337   Mundane Guide to Guilds\\n\\nA supplement for Ca...   en  \n",
       "1018  # Xanathar's Guide to Anything\\n## . But not E...   en  \n",
       "1626  ## Summary\\n\\nThe following is an adventure in...   en  \n",
       "1708  # Xanathar's Guide to Anything\\n## . But not E...   en  \n",
       "1710  # Tasha's Cauldron of Every Single Thing\\n## T...   en  \n",
       "1716  # Player's Hand Book\\n\\n## Definitely not a \"H...   en  \n",
       "1717  # The Homebrewery\\n\\nWelcome traveler from an ...   en  \n",
       "1723  Credit: Wizards of the Coast\\n\\n# Evoking Extr...   en  \n",
       "1779  Class Background Traits Expanded\\n\\nArt: dunge...   en  \n",
       "1880  # Variants and House Rules\\n\\n### Artificer\\n#...   en  \n",
       "2175  ## The Remorhaz Lair\\n\\nThe Nest the Remorhaz\\...   en  \n",
       "2222  Valley of the Lost and Found\\n\\nA realm hidden...   en  \n",
       "\n",
       "[12 rows x 21 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.groupby(by=[\"submission_flair\"])[\"UID\"].agg(\"count\")\n",
    "MAJOR_LABELS = [\n",
    "    \"Class\",\n",
    "    \"Item\",\n",
    "    \"Monster\",\n",
    "    \"Race\",\n",
    "    \"Spell\",\n",
    "    \"Subclass\",\n",
    "    \"Feat\",\n",
    "    \"Compendium\",\n",
    "    \"Mechanic\",\n",
    "    \"Background\",\n",
    "]\n",
    "metadata[~metadata[\"submission_flair\"].isin(MAJOR_LABELS)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "054d17d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = metadata[metadata[\"submission_flair\"].isin(MAJOR_LABELS)].copy()\n",
    "metadata[\"submission_flair\"] = metadata[\n",
    "    \"submission_flair\"\n",
    "].cat.remove_unused_categories()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "33e037db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 3201 entries, 0 to 2336\n",
      "Data columns (total 21 columns):\n",
      " #   Column                   Non-Null Count  Dtype         \n",
      "---  ------                   --------------  -----         \n",
      " 0   UID                      3201 non-null   Int16         \n",
      " 1   link                     3201 non-null   object        \n",
      " 2   submission_author        3189 non-null   object        \n",
      " 3   submission_id            3201 non-null   object        \n",
      " 4   submission_title         3201 non-null   object        \n",
      " 5   subreddit                3201 non-null   category      \n",
      " 6   submission_flair         3201 non-null   category      \n",
      " 7   submission_score         3201 non-null   Int32         \n",
      " 8   submission_upvote_ratio  3198 non-null   float32       \n",
      " 9   submission_date          1983 non-null   datetime64[ns]\n",
      " 10  comment_author           2597 non-null   object        \n",
      " 11  comment_id               2597 non-null   object        \n",
      " 12  comment_score            2597 non-null   Int32         \n",
      " 13  comment_body             2597 non-null   object        \n",
      " 14  comment_date             1849 non-null   datetime64[ns]\n",
      " 15  src_url                  3201 non-null   object        \n",
      " 16  manually_reviewed        3201 non-null   boolean       \n",
      " 17  related_link             1445 non-null   boolean       \n",
      " 18  raw_markdown             3201 non-null   object        \n",
      " 19  clean_markdown           3201 non-null   object        \n",
      " 20  lang                     3201 non-null   object        \n",
      "dtypes: Int16(1), Int32(2), boolean(2), category(2), datetime64[ns](2), float32(1), object(11)\n",
      "memory usage: 422.2+ KB\n"
     ]
    }
   ],
   "source": [
    "metadata = metadata.sort_values(\"UID\")\n",
    "metadata.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c35997c2",
   "metadata": {},
   "source": [
    "# Create Corpus Dataframes & Tokenize\n",
    "### Calculation of QC/Language Metrics\n",
    "Metrics are calculated at the sentence/word token and full string levels on:\n",
    "1. the raw markdown\n",
    "2. the cleaned markdown\n",
    "3. the section-level text\n",
    "Metrics for markdown are calculated first and use simple tokenization to get some basic information for EDA/QC purposes.\n",
    "Final/real-word implementation will not bother with this except as part of continual learning/QC\n",
    "\n",
    "**Speed:** This takes ~1-2 minutes for each phase.\n",
    "\n",
    "### Get section-level text\n",
    "This involves using the NLTK markdown corpus reader to split the text up into sections. The plan is for section-level data to mostly be used for cleaning out things like credit sections and/or unsupervised learning to get better segmented labels for supervised learning cases (i.e. multi-class sequence to sequence models)\n",
    "\n",
    "**Speed:** akes approx. ~4 minutes (~1.5 min to initialize NLTK markdown corpus reader, ~2.5 min to get section-level data--most of that is labeling the sections as credit/non-credit)\n",
    "\n",
    "### Custom word tokenization\n",
    "\n",
    "My tokenizer here will attempt to resolve a number of ambiguities that exist in the form of spelling variations in a smart way that maximizes the number of words taken from a pre-existing set of word embeddings. In this case, we'll use the GLoVe (Wikipedia+Gigaword) embeddings taken from Gensim. **For now we only need the list of vocabulary terms.** In the event that a word is unknown and rare within the text corpus, it will use the part-of-speech tag from NLTK as an approximation of the meaning.\n",
    "\n",
    "My custom tokenizer is slower than standard word tokenization largely due to its effort to retain as many tokens in a meaningful way as possible. The **entire** text that is being tagged is passed into the PoS tagger if an unknown term is encountered since performance is generally best within the full context of the surrounding tokens.\n",
    "\n",
    "**Note**: Section tokenization is *particularly* slow. It's ~10x faster to tokenize by document, but we lose information that defines sections if we tokenize first. A better future implementation would be to figure out where the start and end of each section by looking for the indices of key tokens surrounding each token/text. Since I only need to do this once and going from section tokens -> document tokens is trivial, I implemented it like this for now.\n",
    "\n",
    "**Speed:** Fitting the tokenizer takes ~1-2 minutes and mostly cross references uncommon or hyphen/other delimited characters with \"known words\" that are either highly common in the corpus or are present in the word embeddings' vocabulary.\n",
    "This tokenization is a bit overboard for bag-of-words models, but I would like to implement some sequence-to-sequence stuff later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "33340016",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating markdown-level QC metrics\n",
      "\tWord tokens not determined yet, defaulting to NLTK word_tokenize\n",
      "Getting sections-level dataframe.\n",
      "\tPreparing markdown corpus reader\n",
      "\tParsing sections\n",
      "Fitting tokenizer\n",
      "Tokenizing 123307 sections...\n",
      "Calculating section-level QC metrics\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 123307 entries, 0 to 123306\n",
      "Data columns (total 46 columns):\n",
      " #   Column                   Non-Null Count   Dtype         \n",
      "---  ------                   --------------   -----         \n",
      " 0   UID                      123307 non-null  Int16         \n",
      " 1   link                     123307 non-null  object        \n",
      " 2   submission_author        122992 non-null  object        \n",
      " 3   submission_id            123307 non-null  object        \n",
      " 4   submission_title         123307 non-null  object        \n",
      " 5   subreddit                123307 non-null  category      \n",
      " 6   submission_flair         123307 non-null  category      \n",
      " 7   submission_score         123307 non-null  Int32         \n",
      " 8   submission_upvote_ratio  123151 non-null  float32       \n",
      " 9   submission_date          76278 non-null   datetime64[ns]\n",
      " 10  comment_author           93625 non-null   object        \n",
      " 11  comment_id               93625 non-null   object        \n",
      " 12  comment_score            93625 non-null   Int32         \n",
      " 13  comment_body             93625 non-null   object        \n",
      " 14  comment_date             67614 non-null   datetime64[ns]\n",
      " 15  src_url                  123307 non-null  object        \n",
      " 16  manually_reviewed        123307 non-null  boolean       \n",
      " 17  related_link             59826 non-null   boolean       \n",
      " 18  raw_markdown             123307 non-null  object        \n",
      " 19  clean_markdown           123307 non-null  object        \n",
      " 20  lang                     123307 non-null  object        \n",
      " 21  raw_md_char_count        123307 non-null  Int32         \n",
      " 22  raw_md_hastag_runs       123307 non-null  Int32         \n",
      " 23  raw_md_asterisk_runs     123307 non-null  Int32         \n",
      " 24  clean_md_char_count      123307 non-null  Int32         \n",
      " 25  clean_md_hastag_runs     123307 non-null  Int32         \n",
      " 26  clean_md_asterisk_runs   123307 non-null  Int32         \n",
      " 27  clean_md_stopword_count  123307 non-null  Int32         \n",
      " 28  clean_md_word_count      123307 non-null  Int32         \n",
      " 29  clean_md_avg_word_len    123307 non-null  float64       \n",
      " 30  clean_md_sent_count      123307 non-null  Int32         \n",
      " 31  clean_md_avg_sent_len    123307 non-null  float64       \n",
      " 32  header_level             123307 non-null  Int8          \n",
      " 33  section_number           123307 non-null  Int16         \n",
      " 34  section_heading          123307 non-null  object        \n",
      " 35  section_text             123307 non-null  object        \n",
      " 36  is_credit                123307 non-null  boolean       \n",
      " 37  word_tokens              123307 non-null  object        \n",
      " 38  section_char_count       123307 non-null  Int32         \n",
      " 39  section_hastag_runs      123307 non-null  Int32         \n",
      " 40  section_asterisk_runs    123307 non-null  Int32         \n",
      " 41  section_stopword_count   123307 non-null  Int32         \n",
      " 42  section_word_count       123307 non-null  Int32         \n",
      " 43  section_avg_word_len     123307 non-null  float64       \n",
      " 44  section_sent_count       123307 non-null  Int32         \n",
      " 45  section_avg_sent_len     123307 non-null  float64       \n",
      "dtypes: Int16(2), Int32(17), Int8(1), boolean(3), category(2), datetime64[ns](2), float32(1), float64(4), object(14)\n",
      "memory usage: 32.1+ MB\n"
     ]
    }
   ],
   "source": [
    "sec_df_path = DATA / \"section_corpus.pkl\"\n",
    "tokenizer_path = DATA / \"tokenizer.pkl\"\n",
    "\n",
    "if sec_df_path.is_file():\n",
    "    # No longer need metadata, free up memory and import section df\n",
    "    del metadata\n",
    "    section_df = pd.read_pickle(sec_df_path)\n",
    "    with open(tokenizer_path, \"rb\") as p:\n",
    "        tokenizer = pkl.load(p)\n",
    "else:\n",
    "    print(\"Calculating markdown-level QC metrics\")\n",
    "    # Calculate basic markdown QC/lang metrics\n",
    "    get_char_counts(metadata, pref=\"raw_md\", text_col=\"raw_markdown\")\n",
    "    get_char_counts(metadata, pref=\"clean_md\", text_col=\"clean_markdown\")\n",
    "    count_stopwords(\n",
    "        metadata,\n",
    "        pref=\"clean_md\",\n",
    "        text_col=\"clean_markdown\",\n",
    "        tokenizer=word_tokenize,\n",
    "        text_to_lower=True,\n",
    "    )\n",
    "    calc_token_sizes(metadata, pref=\"clean_md\", text_col=\"clean_markdown\")\n",
    "\n",
    "    # Get sections\n",
    "    print(\"Getting sections-level dataframe.\")\n",
    "    section_df = get_section_df(metadata, CLEAN_TEXT_FILES)  # See function/module for details\n",
    "    del metadata\n",
    "\n",
    "    # Get tokens\n",
    "    if tokenizer_path.is_file():\n",
    "        print(\"Loading tokenizer\")\n",
    "        with open(tokenizer_path, \"rb\") as p:\n",
    "            tokenizer = pkl.load(p)\n",
    "    else:\n",
    "        print(\"Fitting tokenizer\")\n",
    "        import gensim.downloader as api\n",
    "\n",
    "        # Load WV for embedding vocab\n",
    "        vectors = api.load(\"glove-wiki-gigaword-100\")\n",
    "        vector_vocab = [k for k in vectors.key_to_index.keys()]\n",
    "        del vectors\n",
    "        # Create and fit tokenizer to corpus\n",
    "        tokenizer = EmbeddingAwareTokenizer(\n",
    "            embedding_vocab=vector_vocab,\n",
    "            min_counts_common_token=5,\n",
    "            unknown_token=\"\",  # If no unknown token is provided, will use NLTK-predicted POS tag\n",
    "            min_subtoken_size=4,\n",
    "            max_subtokens=3,\n",
    "        )\n",
    "        corpus = \"\\n\".join(section_df[\"section_text\"])\n",
    "        tokenizer.fit(corpus)\n",
    "        with open(tokenizer_path, \"wb\") as p:\n",
    "            pkl.dump(tokenizer, p, pkl.HIGHEST_PROTOCOL)\n",
    "\n",
    "    print(f\"Tokenizing {len(section_df)} sections...\")\n",
    "    section_df[\"word_tokens\"] = section_df[\"section_text\"].map(tokenizer.tokenize)\n",
    "    section_df.to_pickle(sec_df_path)\n",
    "\n",
    "    # Calculate section QC/lang metrics\n",
    "    print(\"Calculating section-level QC metrics\")\n",
    "    get_char_counts(section_df, pref=\"section\", text_col=\"section_text\")\n",
    "    count_stopwords(\n",
    "        section_df,\n",
    "        pref=\"section\",\n",
    "        token_col=\"word_tokens\",\n",
    "        tokenizer=tokenizer.tokenize,\n",
    "    )\n",
    "    calc_token_sizes(section_df, pref=\"section\", word_token_col=\"word_tokens\")\n",
    "\n",
    "    # Save data\n",
    "    section_df.to_pickle(sec_df_path)\n",
    "\n",
    "section_df.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fed763e8",
   "metadata": {},
   "source": [
    "### Convert to documents\n",
    "No need to tokenize here since we are just merging tokens in the correct order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "750c15bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting document-level dataframe from section-level dataframe\n",
      "Calculating document-level QC metrics\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3201 entries, 0 to 3200\n",
      "Data columns (total 54 columns):\n",
      " #   Column                     Non-Null Count  Dtype         \n",
      "---  ------                     --------------  -----         \n",
      " 0   UID                        3201 non-null   int16         \n",
      " 1   clean_text                 3201 non-null   object        \n",
      " 2   num_sections               3201 non-null   int64         \n",
      " 3   credit_text                3201 non-null   object        \n",
      " 4   num_credit_sections        3201 non-null   int64         \n",
      " 5   clean_word_tokens          3201 non-null   object        \n",
      " 6   credit_word_tokens         3201 non-null   object        \n",
      " 7   link                       3201 non-null   object        \n",
      " 8   submission_author          3189 non-null   object        \n",
      " 9   submission_id              3201 non-null   object        \n",
      " 10  submission_title           3201 non-null   object        \n",
      " 11  subreddit                  3201 non-null   category      \n",
      " 12  submission_flair           3201 non-null   category      \n",
      " 13  submission_score           3201 non-null   Int32         \n",
      " 14  submission_upvote_ratio    3198 non-null   float32       \n",
      " 15  submission_date            1983 non-null   datetime64[ns]\n",
      " 16  comment_author             2597 non-null   object        \n",
      " 17  comment_id                 2597 non-null   object        \n",
      " 18  comment_score              2597 non-null   Int32         \n",
      " 19  comment_body               2597 non-null   object        \n",
      " 20  comment_date               1849 non-null   datetime64[ns]\n",
      " 21  src_url                    3201 non-null   object        \n",
      " 22  manually_reviewed          3201 non-null   boolean       \n",
      " 23  related_link               1445 non-null   boolean       \n",
      " 24  raw_markdown               3201 non-null   object        \n",
      " 25  clean_markdown             3201 non-null   object        \n",
      " 26  lang                       3201 non-null   object        \n",
      " 27  raw_md_char_count          3201 non-null   Int32         \n",
      " 28  raw_md_hastag_runs         3201 non-null   Int32         \n",
      " 29  raw_md_asterisk_runs       3201 non-null   Int32         \n",
      " 30  clean_md_char_count        3201 non-null   Int32         \n",
      " 31  clean_md_hastag_runs       3201 non-null   Int32         \n",
      " 32  clean_md_asterisk_runs     3201 non-null   Int32         \n",
      " 33  clean_md_stopword_count    3201 non-null   Int32         \n",
      " 34  clean_md_word_count        3201 non-null   Int32         \n",
      " 35  clean_md_avg_word_len      3201 non-null   float64       \n",
      " 36  clean_md_sent_count        3201 non-null   Int32         \n",
      " 37  clean_md_avg_sent_len      3201 non-null   float64       \n",
      " 38  doc_main_char_count        3201 non-null   Int32         \n",
      " 39  doc_main_hastag_runs       3201 non-null   Int32         \n",
      " 40  doc_main_asterisk_runs     3201 non-null   Int32         \n",
      " 41  doc_credit_char_count      3201 non-null   Int32         \n",
      " 42  doc_credit_hastag_runs     3201 non-null   Int32         \n",
      " 43  doc_credit_asterisk_runs   3201 non-null   Int32         \n",
      " 44  doc_main_stopword_count    3201 non-null   Int32         \n",
      " 45  doc_credit_stopword_count  3201 non-null   Int32         \n",
      " 46  doc_main_word_count        3201 non-null   Int32         \n",
      " 47  doc_main_avg_word_len      3201 non-null   float64       \n",
      " 48  doc_main_sent_count        3201 non-null   Int32         \n",
      " 49  doc_main_avg_sent_len      3201 non-null   float64       \n",
      " 50  doc_credit_word_count      3201 non-null   Int32         \n",
      " 51  doc_credit_avg_word_len    3201 non-null   float64       \n",
      " 52  doc_credit_sent_count      3201 non-null   Int32         \n",
      " 53  doc_credit_avg_sent_len    3201 non-null   float64       \n",
      "dtypes: Int32(23), boolean(2), category(2), datetime64[ns](2), float32(1), float64(6), int16(1), int64(2), object(15)\n",
      "memory usage: 1022.8+ KB\n"
     ]
    }
   ],
   "source": [
    "doc_df_path = DATA / \"document_corpus.pkl\"\n",
    "\n",
    "if doc_df_path.is_file():\n",
    "    # del section_df\n",
    "    doc_df = pd.read_pickle(doc_df_path)\n",
    "else:\n",
    "    print(\"Getting document-level dataframe from section-level dataframe\")\n",
    "    doc_df = sections_df_to_docs(section_df)  # See function/module for details\n",
    "    del section_df\n",
    "    # Calculate document QC/lang metrics\n",
    "    print(\"Calculating document-level QC metrics\")\n",
    "    get_char_counts(doc_df, pref=\"doc_main\", text_col=\"clean_text\")\n",
    "    get_char_counts(doc_df, pref=\"doc_credit\", text_col=\"credit_text\")\n",
    "    count_stopwords(\n",
    "        doc_df,\n",
    "        pref=\"doc_main\",\n",
    "        token_col=\"clean_word_tokens\",\n",
    "        tokenizer=tokenizer.tokenize,\n",
    "    )\n",
    "    count_stopwords(\n",
    "        doc_df,\n",
    "        pref=\"doc_credit\",\n",
    "        token_col=\"credit_word_tokens\",\n",
    "        tokenizer=tokenizer.tokenize,\n",
    "    )\n",
    "    del tokenizer\n",
    "    calc_token_sizes(doc_df, pref=\"doc_main\", word_token_col=\"clean_word_tokens\")\n",
    "    calc_token_sizes(doc_df, pref=\"doc_credit\", word_token_col=\"credit_word_tokens\")\n",
    "\n",
    "    doc_df.to_pickle(DATA / \"document_corpus.pkl\")\n",
    "\n",
    "doc_df.info()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py310",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}