Part1B_Processing_Comments.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "israeli-expression",
   "metadata": {},
   "source": [
    "# Project Overview\n",
    "In this notebook and subsequent notebooks I will attempt to predict mechanically meaningful labels associated with some fan-generated roleplaying game (RPG) content.\n",
    "\n",
    "I collected post and comment data from Reddit in multiple runs using a command-line tool I wrote for this purpose (see: [PRAW-CoDiaLS](https://github.com/nkuehnle/praw-codials))\n",
    "\n",
    "The actual text I will be attempting to predict is hosted on one of two websites: [GM Binder](https://gmbinder.com/) or [Homebrewery](https://homebrewery.naturalcrit.com/), which are tools/content hosting providers for fan-made RPG content, primarily Dungeons and Dragons 5th Edition (DnD5e).\n",
    "\n",
    "The labels make use of the user-assigned \"flair\" from the subreddit [/r/UnearthedArcana](https://www.reddit.com/r/UnearthedArcana/). UnearthedArcana describes itself as a source of \"homebrew\" (fan-generated) content and its community rules ensure that every post has a meaningful label (\"flair\") associated with it. These labels describe the \"type\" of game content that has been created, be it a character \"class\" (think achetypal high-fantasy characters knights, wizards, archers, etc) or a \"race\" (elves, dwarves, humans, etc).\n",
    "\n",
    "At the start of this document we have only the data about the Reddit submission including the submission flair link to the service described above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f0df1d02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility Imports\n",
    "import os\n",
    "from pathlib import Path\n",
    "# Imports for data processing/handling\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "# Plotting\n",
    "import matplotlib as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3776ac50",
   "metadata": {},
   "source": [
    "# This Notebook\n",
    "\n",
    "In this notebook, I'll use the output of manual review of the multi-link comments to further filter and clean the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "independent-phrase",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a constant (CWD) to use\n",
    "CWD = Path(os.getcwd())\n",
    "DATA = CWD / 'data'\n",
    "PRAW = DATA / 'praw_data'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5dbb6ab2",
   "metadata": {},
   "source": [
    "# Data Imports\n",
    "There are now two files I'll be working with:\n",
    "1. CommentsFiltered.csv\n",
    "2. CommentsDropped.csv\n",
    "2. MultiLinkCmts.csv\n",
    "\n",
    "If you are not familiar with Reddit, when users post it can take two forms, \"Submissions\" and \"comments.\"\n",
    "\n",
    "Submissions are associated with a specific community or \"subreddit\" and users can comment on Submissions to discuss them or provide additional commentary.\n",
    "Submissions can come in multiple forms: link, text, image, etc. The submissions here are only link submissions. Comments are searched for all other types of submissions as users will frequently post an image version of their content and then link to the text copy in the comments on these subreddits. The vast majority of content on /r/UnearthedArcana comes in this second form (an image submission with the link in the comments).\n",
    "\n",
    "In any case, PRAW-CoDiaLS is written so that they both share the same overall structure, although certain fields are only used for comments (and blank otherwise).\n",
    "\n",
    "### Shared Fields\n",
    "\n",
    "| column name | description |\n",
    "| ----- | ----- |\n",
    "| link | Raw text URL |\n",
    "| submission_author | Reddit user who posted the comment |\n",
    "| submission_id | A unique ID used for locating the submission on Reddit |\n",
    "| submission_title | The title of the submission on Reddit |\n",
    "| subreddit | The name of the Subreddit the content was found on (in this case, either UnearthedArcana or DnDHomebrew) |\n",
    "| submission_flair | the figure number as indicated in the manuscript |\n",
    "| submission_score | The approximate number of net positive votes received by the submission on Reddit, the actual values are obfuscated by Reddit to avoid manipulation |\n",
    "| submission_upvote_ratio | The ratio of positive to negative votes |\n",
    "| submission_date | Date on which the submission was originally made |\n",
    "| comment_author | Reddit user who posted the comment |\n",
    "| comment_id | A unique ID used for locating the comment on Reddit |\n",
    "| comment_score | The approximate number of net positive votes received by the comment on Reddit, the actual values are obfuscated by Reddit to avoid manipulation |\n",
    "| comment_body | Text of the comment containing markdown elements |\n",
    "| comment_date | Date on which the comment was originally posted |\n",
    "\n",
    "### Other Fields\n",
    "\n",
    "| column name | description |\n",
    "| ----- | ----- |\n",
    "| processed | Boolean indicator for whether link has been manually reviewed (MultiLinkCmts.csv only) |\n",
    "| related_link | Boolean indicator for whether link/text was related to the labeled post (MultiLinkCmts.csv only) |\n",
    "| discard_reason | Label indicating why the link was discarded (CommentsDiscarded.csv) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "19363ea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cmts_df = pd.read_csv(PRAW/'CommentsFiltered.csv', index_col='idx').sort_index(axis=0)\n",
    "cmts_df['manually_reviewed'] = False\n",
    "discarded_cmts = pd.read_csv(PRAW/'CommentsDropped.csv', index_col='idx').sort_index(axis=0)\n",
    "multilink_cmts = pd.read_csv(PRAW/'MultiLinkCmts.csv', index_col='idx').sort_index(axis=0)\n",
    "mismatched_author_cmts = pd.read_csv(PRAW/'AuthorIssues.csv', index_col='idx').sort_index(axis=0)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "89d19b63",
   "metadata": {},
   "source": [
    "# Mismatched Authorship Comments"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42e02dd2",
   "metadata": {},
   "source": [
    "#### Missing/Invalid Content\n",
    "As before, this includes any links that have either been removed/deleted or otherwise do not really provide useful examples of English 5th edition content (foreign language, styling guides, thought-pieces, etc)\n",
    "\n",
    "#### Duplication/Redundancy\n",
    "Other instances of duplication exist within the dataset. These will be handled similarly to how duplicate submissions were handled in the previous notebook (i.e. prioritizing submissions to /r/UnearthedArcana and earlier postings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "09a19fb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "mismatched_author_cmts['flair_correct'] = mismatched_author_cmts['submission_flair'] == mismatched_author_cmts['corrected_flair']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cf922516",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dropped 8 links with invalid/missing content.\n"
     ]
    }
   ],
   "source": [
    "null_rows = mismatched_author_cmts['corrected_flair'].isna() & mismatched_author_cmts['manually_reviewed']\n",
    "null_df = mismatched_author_cmts[null_rows].copy()\n",
    "null_df['discard_reason'] = \"Invalid/Missing Content Type\"\n",
    "mismatched_author_cmts = mismatched_author_cmts[~null_rows]\n",
    "print(f\"Dropped {len(null_df)} links with invalid/missing content.\")\n",
    "discarded_cmts = pd.concat([discarded_cmts, null_df[~null_df.index.isin(discarded_cmts.index)]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6bcc04e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "66.27% of remaining comments had incorrect flair\n"
     ]
    }
   ],
   "source": [
    "err_rate = ((~mismatched_author_cmts['flair_correct']).sum())/len(mismatched_author_cmts)*100\n",
    "print(f\"{err_rate:.2f}% of remaining comments had incorrect flair\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ba80150e",
   "metadata": {},
   "source": [
    "#### Dropping remaining duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c3192615",
   "metadata": {},
   "outputs": [],
   "source": [
    "mismatched_author_cmts = mismatched_author_cmts.sort_values(by=[\"submission_date\",\"comment_date\"])\n",
    "duplicated = mismatched_author_cmts.duplicated(subset=['src_url', 'corrected_flair'], keep='first')\n",
    "dup_df = mismatched_author_cmts[duplicated].copy()\n",
    "dup_df['discard_reason'] = \"Duplicate Unambiguous Flair\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, dup_df[~dup_df.index.isin(discarded_cmts)]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ac206019",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All duplicated links have consistent flair\n"
     ]
    }
   ],
   "source": [
    "mismatches = mismatched_author_cmts.duplicated(subset=[\"src_url\"]).sum() - mismatched_author_cmts.duplicated(subset=[\"src_url\",\"corrected_flair\"]).sum()\n",
    "if mismatches != 0:\n",
    "    print(f\"There are {mismatches} links with inconsistent flair assignments\")\n",
    "    all_dups = mismatched_author_cmts.duplicated(subset=[\"src_url\"], keep=False).index\n",
    "    dups_after_correction = mismatched_author_cmts.duplicated(subset=[\"src_url\",\"corrected_flair\"], keep=False).index\n",
    "    mismatched = [dup for dup in all_dups if dup not in dups_after_correction]\n",
    "    print(mismatched_author_cmts[mismatched_author_cmts.index.isin(mismatched)])\n",
    "else:\n",
    "    print(\"All duplicated links have consistent flair\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9ee4bc30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31 duplicate links with un-ambiguous submission flair dropped\n"
     ]
    }
   ],
   "source": [
    "mismatched_author_cmts = mismatched_author_cmts[~duplicated]\n",
    "mismatched_author_cmts.sort_index()\n",
    "print(f\"{duplicated.sum()} duplicate links with un-ambiguous submission flair dropped\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8fadcc03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64.82% of remaining comments had incorrect flair\n"
     ]
    }
   ],
   "source": [
    "err_rate = ((~mismatched_author_cmts['flair_correct']).sum())/len(mismatched_author_cmts)*100\n",
    "print(f\"{err_rate:.2f}% of remaining comments had incorrect flair\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f0d2e720",
   "metadata": {},
   "source": [
    "### Fixing/re-merging content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "922095bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 links with missing/invalid content types removed from primary comment dataset.\n"
     ]
    }
   ],
   "source": [
    "null_links = null_df['link'].unique()\n",
    "null_rows = cmts_df['link'].isin(null_links)\n",
    "null_df = cmts_df[null_rows].copy()\n",
    "null_df['submission_flair'] = np.nan\n",
    "null_df['discard_reason'] = \"Invalid/Missing Content Type\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, null_df[~null_df.index.isin(discarded_cmts)]])\n",
    "cmts_df = cmts_df[~null_rows]\n",
    "print(f\"{null_rows.sum()} links with missing/invalid content types removed from primary comment dataset.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "83155ce0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26 entries updated\n"
     ]
    }
   ],
   "source": [
    "url_mapper = mismatched_author_cmts.set_index('link')['corrected_flair'].to_dict()\n",
    "\n",
    "flair_to_correct = cmts_df['link'].isin(url_mapper.keys())\n",
    "mismatched_author_cmts[mismatched_author_cmts['link'].isin(cmts_df['link'])]\n",
    "flair_to_correct = flair_to_correct & cmts_df['link'].isin(mismatched_author_cmts['link'])\n",
    "\n",
    "cmts_df.loc[flair_to_correct, 'submission_flair'] = cmts_df[flair_to_correct]['link'].map(url_mapper).values\n",
    "cmts_df.loc[flair_to_correct, 'manually_reviewed'] = True\n",
    "\n",
    "print(f\"{flair_to_correct.sum()} entries updated\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "aed3aec0",
   "metadata": {},
   "outputs": [],
   "source": [
    "mismatched_author_cmts['submission_flair'] = mismatched_author_cmts['corrected_flair']\n",
    "mismatched_author_cmts = mismatched_author_cmts.drop(columns=['flair_correct', 'corrected_flair', 'src_url'])\n",
    "mismatched_author_cmts = mismatched_author_cmts[~mismatched_author_cmts.index.isin(cmts_df.index)]\n",
    "cmts_df = pd.concat([cmts_df, mismatched_author_cmts[~mismatched_author_cmts.index.isin(cmts_df.index)]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8ea417ab",
   "metadata": {},
   "source": [
    "# Multi-Link Comment Content\n",
    "Next, I'll re-incorporate the manually-reviewed multi-link comment content.\n",
    "\n",
    "I'll perform some limited processing to prune bad content and/or duplicates before I merge them back in.\n",
    "\n",
    "There will be a few angles I'll use for filtering:\n",
    "\n",
    "#### Missing/Invalid Content\n",
    "As before, this includes any links that have either been removed/deleted or otherwise do not really provide useful examples of English 5th edition content (foreign language, styling guides, thought-pieces, etc)\n",
    "\n",
    "#### Content Relation to Parent Submission\n",
    "Discovered links were flagged as either related or unrelated to the main (parent) submission the comment was made under.\n",
    "\n",
    "In many cases, these unrelated posts still provided useful content, so they will be kept. However, I'll preferentially keep the ones that were related to the parent submission.\n",
    "\n",
    "#### Duplication/Redundancy\n",
    "Other instances of duplication exist within the dataset. These will be handled similarly to how duplicate submissions were handled in the previous notebook (i.e. prioritizing submissions to /r/UnearthedArcana and earlier postings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ab2e211f",
   "metadata": {},
   "outputs": [],
   "source": [
    "multilink_cmts['flair_correct'] = multilink_cmts['submission_flair'] == multilink_cmts['corrected_flair']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "18363017",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='related_link', ylabel='flair_correct'>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGxCAYAAACeKZf2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAnEElEQVR4nO3df1TUdb7H8dcAAi4KiiKhIWJmYvgTbqVeNdvExY671e7qrm1kaidSQ2T1rqybCVaUvxa1QLum6M1tyWx/VGhxU1t/3M4GWVeT3BQNf6AItuCvQGDuH17n7AQYDAMzfHw+zplzmM98vzPvqYM+/X7nh8VqtVoFAABgCA9XDwAAAOBMxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAo3i5egBXqK2t1enTp9WxY0dZLBZXjwMAABrBarXqwoUL6t69uzw8Gj4+c1PGzenTpxUaGurqMQAAgANOnDihW2+9tcHbb8q46dixo6Rr/3H8/f1dPA0AAGiMiooKhYaG2v4eb8hNGTfXT0X5+/sTNwAAtDHf95ISXlAMAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMclN+KzgAoO2bPXu2zp07J0kKCgrSypUrXTwR3AVxAwBok86dO6ezZ8+6egy4IU5LAQAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwiperBwCAtqYodYCrR4Ck6n92keT5/z+f5v+LG+i58ICrR5DEkRsAAGAY4gYAABiFuAEAAEYhbgAAgFGIGwAAYBTiBgAAGIW4AQAARiFuAACAUYgbAABgFOIGAAAYxS3iJiMjQ+Hh4fL19VVUVJR27959w+03b96sQYMG6Qc/+IFCQkL0+OOPq6ysrJWmBQAA7szlcZOdna3ExEQtWLBA+/fv18iRIxUbG6uioqJ6t9+zZ4/i4uI0bdo0ffHFF9qyZYs++eQTTZ8+vZUnBwAA7sjlcbNixQpNmzZN06dPV0REhNLT0xUaGqrMzMx6t//444/Vq1cvJSQkKDw8XP/+7/+uJ598Unl5ea08OQAAcEcujZuqqirl5+crJibGbj0mJkb79u2rd5/hw4fr5MmTysnJkdVq1dmzZ/XWW2/pgQceaI2RAQCAm3Np3JSWlqqmpkbBwcF268HBwTpz5ky9+wwfPlybN2/WpEmT5O3trVtuuUWdOnXS6tWrG3ycyspKVVRU2F0AAICZXH5aSpIsFovddavVWmftukOHDikhIUELFy5Ufn6+tm/frmPHjik+Pr7B+09LS1NAQIDtEhoa6tT5AQCtL9CnRl3+/xLoU+PqceBGvFz54F27dpWnp2edozQlJSV1juZcl5aWphEjRmjevHmSpIEDB8rPz08jR47Uc889p5CQkDr7JCcnKykpyXa9oqKCwAGANu63Q/7p6hHgplx65Mbb21tRUVHKzc21W8/NzdXw4cPr3efy5cvy8LAf29PTU9K1Iz718fHxkb+/v90FAACYyeWnpZKSkrRu3TqtX79eBQUFmjNnjoqKimynmZKTkxUXF2fbfsKECXr77beVmZmpwsJC7d27VwkJCbrrrrvUvXt3Vz0NAADgJlx6WkqSJk2apLKyMqWmpqq4uFiRkZHKyclRWFiYJKm4uNjuM2+mTJmiCxcu6OWXX9avf/1rderUSffdd59eeuklVz0FAADgRizWhs7lGKyiokIBAQEqLy/nFBWAJitKHeDqEQC31HPhgRa9/8b+/e3y01IAAADORNwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAo7hF3GRkZCg8PFy+vr6KiorS7t27b7h9ZWWlFixYoLCwMPn4+Oi2227T+vXrW2laAADgzrxcPUB2drYSExOVkZGhESNGaO3atYqNjdWhQ4fUs2fPeveZOHGizp49q9dee019+vRRSUmJqqurW3lyAADgjixWq9XqygHuvvtuDR06VJmZmba1iIgIPfjgg0pLS6uz/fbt2/WLX/xChYWFCgwMdOgxKyoqFBAQoPLycvn7+zs8O4CbU1HqAFePALilngsPtOj9N/bvb5eelqqqqlJ+fr5iYmLs1mNiYrRv37569/nrX/+q6OhoLVmyRD169FDfvn01d+5cXblypcHHqaysVEVFhd0FAACYyaWnpUpLS1VTU6Pg4GC79eDgYJ05c6befQoLC7Vnzx75+vrqT3/6k0pLSzVjxgydP3++wdfdpKWlKSUlxenzAwAA9+MWLyi2WCx2161Wa52162pra2WxWLR582bdddddGj9+vFasWKGsrKwGj94kJyervLzcdjlx4oTTnwMAAHAPLj1y07VrV3l6etY5SlNSUlLnaM51ISEh6tGjhwICAmxrERERslqtOnnypG6//fY6+/j4+MjHx8e5wwMAALfk0iM33t7eioqKUm5urt16bm6uhg8fXu8+I0aM0OnTp3Xx4kXb2j/+8Q95eHjo1ltvbdF5AQCA+3P5aamkpCStW7dO69evV0FBgebMmaOioiLFx8dLunZKKS4uzrb95MmT1aVLFz3++OM6dOiQ/va3v2nevHmaOnWq2rdv76qnAQAA3ITLP+dm0qRJKisrU2pqqoqLixUZGamcnByFhYVJkoqLi1VUVGTbvkOHDsrNzdXTTz+t6OhodenSRRMnTtRzzz3nqqcAAADciMs/58YV+JwbAM3B59wA9eNzbgAAAFoAcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwikNxk5qaqsuXL9dZv3LlilJTU5s9FAAAgKMcipuUlBS7b+W+7vLly0pJSWn2UAAAAI5yKG6sVqssFkud9c8//1yBgYHNHgoAAMBRTfpW8M6dO8tischisahv3752gVNTU6OLFy8qPj7e6UMCAAA0VpPiJj09XVarVVOnTlVKSooCAgJst3l7e6tXr14aNmyY04cEAABorCbFzWOPPSZJCg8P14gRI+Tl1aTdAQAAWpxDr7m5dOmSPvzwwzrr77//vrZt29bsoQAAABzlUNzMnz9fNTU1ddatVqvmz5/f7KEAAAAc5VDcfPXVV+rfv3+d9X79+unIkSPNHgoAAMBRDsVNQECACgsL66wfOXJEfn5+zR4KAADAUQ7FzY9//GMlJibq6NGjtrUjR47o17/+tX784x87bTgAAICmcihuli5dKj8/P/Xr10/h4eEKDw9XRESEunTpomXLljl7RgAAgEZz6L3cAQEB2rdvn3Jzc/X555+rffv2GjhwoEaNGuXs+QAAAJrE4Q+qsVgsiomJ0ahRo+Tj41Pv1zEAAAC0NodOS9XW1mrx4sXq0aOHOnTooGPHjkmSnnnmGb322mtOHRAAAKApHIqb5557TllZWVqyZIm8vb1t6wMGDNC6deucNhwAAEBTORQ3mzZt0quvvqpHHnlEnp6etvWBAwfqyy+/dNpwAAAATeVQ3Jw6dUp9+vSps15bW6urV682eygAAABHORQ3d955p3bv3l1nfcuWLRoyZEizhwIAAHCUQ++WevbZZ/Xoo4/q1KlTqq2t1dtvv63Dhw9r06ZNevfdd509IwAAQKM5dORmwoQJys7OVk5OjiwWixYuXKiCggK98847Gjt2rLNnBAAAaLQmH7mprq7W888/r6lTp+qjjz5qiZkAAAAc1uQjN15eXlq6dKlqampaYh4AAIBmcei01P33369du3Y5eRQAAIDmc+gFxbGxsUpOTtbBgwcVFRUlPz8/u9v5ZnAAAOAqDsXNU089JUlasWJFndssFgunrAAAgMs4FDe1tbXOngMAAMApmvyam+rqanl5eengwYMtMQ8AAECzOPRuqbCwME49AQAAt+TQu6V+97vfKTk5WefPn3f2PAAAAM3i0GtuVq1apSNHjqh79+4KCwur826pTz/91CnDAQAANJVDcfPggw86eQwAAADncPiLMwEAANyRQ3FzXX5+vgoKCmSxWNS/f38NGTLEWXMBAAA4xKG4KSkp0S9+8Qvt2rVLnTp1ktVqVXl5ucaMGaM//vGPCgoKcvacAAAAjeLQu6WefvppVVRU6IsvvtD58+f1zTff6ODBg6qoqFBCQoKzZwQAAGg0h47cbN++Xf/93/+tiIgI21r//v31yiuvKCYmxmnDAQAANJVDR25qa2vVrl27Ouvt2rXjqxkAAIBLORQ39913n2bPnq3Tp0/b1k6dOqU5c+bohz/8odOGAwAAaCqH4ubll1/WhQsX1KtXL912223q06ePwsPDdeHCBa1evdrZMwIAADSaQ6+5CQ0N1aeffqrc3Fx9+eWXslqt6t+/v+6//35nzwcAANAkzfqcm7Fjx2rs2LHOmgUAAKDZHDotlZCQoFWrVtVZf/nll5WYmNjcmQAAABzmUNxs3bpVI0aMqLM+fPhwvfXWW80eCgAAwFEOxU1ZWZkCAgLqrPv7+6u0tLTZQwEAADjKobjp06ePtm/fXmd927Zt6t27d7OHAgAAcJRDLyhOSkrSrFmzdO7cOd13332SpA8//FDLly9Xenq6M+cDAABoEofiZurUqaqsrNTzzz+vxYsXS5J69eqlzMxMxcXFOXVAAACApnDotJQkPfXUUzp58qTOnj2riooKFRYW1gmbvXv3qrKystlDAgAANJbDcXNdUFCQOnToUO9tsbGxOnXqVHMfAgAAoNGaHTc3YrVaW/LuAQAA6mjRuAEAAGhtxA0AADAKcQMAAIzSonFjsVha8u4BAADqaHLcWK1Wff3117py5UqjtgUAAGhNDsXN7bffrpMnT37vthcuXODrGAAAQKtqctx4eHjo9ttvV1lZWUvMAwAA0CwOveZmyZIlmjdvng4ePOjseQAAAJrFoe+W+tWvfqXLly9r0KBB8vb2Vvv27e1uP3/+vFOGAwAAaCqH4sbZ3/ydkZGhpUuXqri4WHfeeafS09M1cuTI791v7969Gj16tCIjI/XZZ585dSYAANA2ORQ3jz32mNMGyM7OVmJiojIyMjRixAitXbtWsbGxOnTokHr27NngfuXl5YqLi9MPf/hDnT171mnzAACAtq3Rr7mpqKiw+/lGl6ZYsWKFpk2bpunTpysiIkLp6ekKDQ1VZmbmDfd78sknNXnyZA0bNqxJjwcAAMzW6Ljp3LmzSkpKJEmdOnVS586d61yurzdWVVWV8vPzFRMTY7ceExOjffv2Nbjfhg0bdPToUT377LONepzKyspmBRgAAGg7Gn1aaseOHQoMDJQk7dy50ykPXlpaqpqaGgUHB9utBwcH68yZM/Xu89VXX2n+/PnavXu3vLwaN35aWppSUlKaPS8AAHB/jY6b0aNH1/uzM3z3axqsVmu9X91QU1OjyZMnKyUlRX379m30/ScnJyspKcl2vaKiQqGhoY4PDAAA3JZDLyi+7vLlyyoqKlJVVZXd+sCBAxu1f9euXeXp6VnnKE1JSUmdoznStU88zsvL0/79+zVr1ixJUm1traxWq7y8vPTBBx/ovvvuq7Ofj4+PfHx8Gvu0AABAG+ZQ3Jw7d06PP/64tm3bVu/tNTU1jbofb29vRUVFKTc3Vw899JBtPTc3Vz/5yU/qbO/v768DBw7YrWVkZGjHjh166623FB4e3oRngZvB7Nmzde7cOUlSUFCQVq5c6eKJAAAtzaG4SUxM1DfffKOPP/5YY8aM0Z/+9CedPXtWzz33nJYvX96k+0pKStKjjz6q6OhoDRs2TK+++qqKiooUHx8v6doppVOnTmnTpk3y8PBQZGSk3f7dunWTr69vnXVAuhbifFQAANxcHIqbHTt26C9/+Yv+7d/+TR4eHgoLC9PYsWPl7++vtLQ0PfDAA42+r0mTJqmsrEypqakqLi5WZGSkcnJyFBYWJkkqLi5WUVGRI2MCAICbkEPfLXXp0iV169ZNkhQYGGg77D9gwAB9+umnTb6/GTNm6Pjx46qsrFR+fr5GjRpluy0rK0u7du1qcN9Fixbx6cQAAMDGobi54447dPjwYUnS4MGDtXbtWp06dUpr1qxRSEiIUwcEAABoCodfc1NcXCxJevbZZzVu3Dht3rxZ3t7eysrKcuZ8AAAATeJQ3DzyyCO2n4cMGaLjx4/ryy+/VM+ePdW1a1enDQcAANBUzfqcm+t+8IMfaOjQoc64KwAAgGZpdNz86yf8fp8VK1Y4NAwAAEBzNTpuNmzYoMjISHl5eclischqtda7XX1fmwAAANBaGh035eXl2rp1q7p166bevXvrk08+UZcuXVpyNgAAgCZr9FvBO3furGPHjkmSjh8/rtra2hYbCgAAwFGNPnLz05/+VKNGjVL37t1lsVgUHR0tT0/PerctLCx02oAAAABN0ei4efXVV/Xwww/ryJEjSkhI0BNPPKGOHTu25GwAAABN1qS3gv/oRz+SJOXn52v27NnEDQAAcDsOfc7Nhg0bnD0HAACAUzj03VIAAADuirgBAABGccrXL6CuqHmbXD0CJPl/c9FW8MXfXOT/ixvIXxrn6hEAGI4jNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAo3i5egCgJdW286v3ZwCAuYgbGO3iHbGuHgEA0Mo4LQUAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADCKW8RNRkaGwsPD5evrq6ioKO3evbvBbd9++22NHTtWQUFB8vf317Bhw/T++++34rQAAMCduTxusrOzlZiYqAULFmj//v0aOXKkYmNjVVRUVO/2f/vb3zR27Fjl5OQoPz9fY8aM0YQJE7R///5WnhwAALgji9VqtbpygLvvvltDhw5VZmambS0iIkIPPvig0tLSGnUfd955pyZNmqSFCxc2avuKigoFBASovLxc/v7+Ds39faLmbWqR+wXauvylca4eodmKUge4egTALfVceKBF77+xf3+79MhNVVWV8vPzFRMTY7ceExOjffv2Neo+amtrdeHCBQUGBja4TWVlpSoqKuwuAADATC6Nm9LSUtXU1Cg4ONhuPTg4WGfOnGnUfSxfvlyXLl3SxIkTG9wmLS1NAQEBtktoaGiz5gYAAO7L5a+5kSSLxWJ33Wq11lmrzxtvvKFFixYpOztb3bp1a3C75ORklZeX2y4nTpxo9swAAMA9ebnywbt27SpPT886R2lKSkrqHM35ruzsbE2bNk1btmzR/ffff8NtfXx85OPj0+x5AQCA+3PpkRtvb29FRUUpNzfXbj03N1fDhw9vcL833nhDU6ZM0R/+8Ac98MADLT0mAABoQ1x65EaSkpKS9Oijjyo6OlrDhg3Tq6++qqKiIsXHx0u6dkrp1KlT2rTp2ruP3njjDcXFxWnlypW65557bEd92rdvr4CAAJc9DwAA4B5cHjeTJk1SWVmZUlNTVVxcrMjISOXk5CgsLEySVFxcbPeZN2vXrlV1dbVmzpypmTNn2tYfe+wxZWVltfb4AADAzbg8biRpxowZmjFjRr23fTdYdu3a1fIDAQCANsst3i0FAADgLMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwCjEDQAAMApxAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQMAAIxC3AAAAKMQNwAAwChuETcZGRkKDw+Xr6+voqKitHv37htu/9FHHykqKkq+vr7q3bu31qxZ00qTAgAAd+fyuMnOzlZiYqIWLFig/fv3a+TIkYqNjVVRUVG92x87dkzjx4/XyJEjtX//fv32t79VQkKCtm7d2sqTAwAAd+TyuFmxYoWmTZum6dOnKyIiQunp6QoNDVVmZma9269Zs0Y9e/ZUenq6IiIiNH36dE2dOlXLli1r5ckBAIA7cmncVFVVKT8/XzExMXbrMTEx2rdvX737/M///E+d7ceNG6e8vDxdvXq1xWYFAABtg5crH7y0tFQ1NTUKDg62Ww8ODtaZM2fq3efMmTP1bl9dXa3S0lKFhITU2aeyslKVlZW26+Xl5ZKkioqK5j6FBtVUXmmx+wbaspb8vWstF76tcfUIgFtq6d/v6/dvtVpvuJ1L4+Y6i8Vid91qtdZZ+77t61u/Li0tTSkpKXXWQ0NDmzoqgGYKWB3v6hEAtJS0gFZ5mAsXLiggoOHHcmncdO3aVZ6ennWO0pSUlNQ5OnPdLbfcUu/2Xl5e6tKlS737JCcnKykpyXa9trZW58+fV5cuXW4YUTBDRUWFQkNDdeLECfn7+7t6HABOxO/3zcVqterChQvq3r37Dbdzadx4e3srKipKubm5euihh2zrubm5+slPflLvPsOGDdM777xjt/bBBx8oOjpa7dq1q3cfHx8f+fj42K116tSpecOjzfH39+cPP8BQ/H7fPG50xOY6l79bKikpSevWrdP69etVUFCgOXPmqKioSPHx1w5dJycnKy4uzrZ9fHy8vv76ayUlJamgoEDr16/Xa6+9prlz57rqKQAAADfi8tfcTJo0SWVlZUpNTVVxcbEiIyOVk5OjsLAwSVJxcbHdZ96Eh4crJydHc+bM0SuvvKLu3btr1apV+ulPf+qqpwAAANyIxfp9LzkG2rjKykqlpaUpOTm5zulJAG0bv9+oD3EDAACM4vLX3AAAADgTcQMAAIxC3MBYWVlZvOUfAG5CxA3c3pQpU2SxWOpcjhw54urRADhJfb/j/3qZMmWKq0dEG+Lyt4IDjfGjH/1IGzZssFsLCgpy0TQAnK24uNj2c3Z2thYuXKjDhw/b1tq3b2+3/dWrVxv84FaAIzdoE3x8fHTLLbfYXVauXKkBAwbIz89PoaGhmjFjhi5evNjgfXz++ecaM2aMOnbsKH9/f0VFRSkvL892+759+zRq1Ci1b99eoaGhSkhI0KVLl1rj6QE3vX/93Q4ICJDFYrFd//bbb9WpUye9+eabuvfee+Xr66vXX39dixYt0uDBg+3uJz09Xb169bJb27BhgyIiIuTr66t+/fopIyOj9Z4YXIK4QZvl4eGhVatW6eDBg9q4caN27Nih//iP/2hw+0ceeUS33nqrPvnkE+Xn52v+/Pm2f/kdOHBA48aN08MPP6z//d//VXZ2tvbs2aNZs2a11tMB8D1+85vfKCEhQQUFBRo3blyj9vnP//xPLViwQM8//7wKCgr0wgsv6JlnntHGjRtbeFq4Eqel0Ca8++676tChg+16bGystmzZYrseHh6uxYsX66mnnmrwX2VFRUWaN2+e+vXrJ0m6/fbbbbctXbpUkydPVmJiou22VatWafTo0crMzJSvr28LPCsATZGYmKiHH364SfssXrxYy5cvt+0XHh6uQ4cOae3atXrsscdaYky4AeIGbcKYMWOUmZlpu+7n56edO3fqhRde0KFDh1RRUaHq6mp9++23unTpkvz8/OrcR1JSkqZPn67/+q//0v3336+f//znuu222yRJ+fn5OnLkiDZv3mzb3mq1qra2VseOHVNERETLP0kANxQdHd2k7c+dO6cTJ05o2rRpeuKJJ2zr1dXVjfryRbRdxA3aBD8/P/Xp08d2/euvv9b48eMVHx+vxYsXKzAwUHv27NG0adN09erVeu9j0aJFmjx5st577z1t27ZNzz77rP74xz/qoYceUm1trZ588kklJCTU2a9nz54t9rwANN53/9Hi4eGh737I/r/+/tfW1kq6dmrq7rvvttvO09OzhaaEOyBu0Cbl5eWpurpay5cvl4fHtZeOvfnmm9+7X9++fdW3b1/NmTNHv/zlL7VhwwY99NBDGjp0qL744gu7gALg3oKCgnTmzBlZrVZZLBZJ0meffWa7PTg4WD169FBhYaEeeeQRF00JVyBu0Cbddtttqq6u1urVqzVhwgTt3btXa9asaXD7K1euaN68efrZz36m8PBwnTx5Up988ont2+R/85vf6J577tHMmTP1xBNPyM/PTwUFBcrNzdXq1atb62kBaIJ7771X586d05IlS/Szn/1M27dv17Zt2+Tv72/bZtGiRUpISJC/v79iY2NVWVmpvLw8ffPNN0pKSnLh9GhJvFsKbdLgwYO1YsUKvfTSS4qMjNTmzZuVlpbW4Paenp4qKytTXFyc+vbtq4kTJyo2NlYpKSmSpIEDB+qjjz7SV199pZEjR2rIkCF65plnFBIS0lpPCUATRUREKCMjQ6+88ooGDRqkv//975o7d67dNtOnT9e6deuUlZWlAQMGaPTo0crKylJ4eLiLpkZr4FvBAQCAUThyAwAAjELcAAAAoxA3AADAKMQNAAAwCnEDAACMQtwAAACjEDcAAMAoxA0AADAKcQOgVdx7771KTEx0yWPv2rVLFotF//znP79326ysLHXq1Ml2fdGiRRo8eHCTHq9Xr15KT09v0j4AnIe4AeCWmhIkLWnu3Ln68MMPXToDgKbhizMBNFtVVZW8vb1dPUaL6NChgzp06ODqMQA0AUduADTZvffeq1mzZikpKUldu3bV2LFjdejQIY0fP14dOnRQcHCwHn30UZWWljZ4H6+//rqio6PVsWNH3XLLLZo8ebJKSkokScePH9eYMWMkSZ07d5bFYtGUKVMkSVarVUuWLFHv3r3Vvn17DRo0SG+99Zbdfefk5Khv375q3769xowZo+PHjzv8XL97WmrKlCl68MEHtWzZMoWEhKhLly6aOXOmrl692uB9bNiwQQEBAcrNzXV4DgCNR9wAcMjGjRvl5eWlvXv36sUXX9To0aM1ePBg5eXlafv27Tp79qwmTpzY4P5VVVVavHixPv/8c/35z3/WsWPHbAETGhqqrVu3SpIOHz6s4uJirVy5UpL0u9/9Ths2bFBmZqa++OILzZkzR7/61a/00UcfSZJOnDihhx9+WOPHj9dnn32m6dOna/78+U597jt37tTRo0e1c+dObdy4UVlZWcrKyqp322XLlmnu3Ll6//33NXbsWKfOAaB+nJYC4JA+ffpoyZIlkqSFCxdq6NCheuGFF2y3r1+/XqGhofrHP/6hvn371tl/6tSptp979+6tVatW6a677tLFixfVoUMHBQYGSpK6detme4HvpUuXtGLFCu3YsUPDhg2z7btnzx6tXbtWo0ePVmZmpnr37q3f//73slgsuuOOO3TgwAG99NJLTnvunTt31ssvvyxPT0/169dPDzzwgD788EM98cQTdtslJydr48aN2rVrlwYMGOC0xwdwY8QNAIdER0fbfs7Pz9fOnTvrfW3K0aNH642b/fv3a9GiRfrss890/vx51dbWSpKKiorUv3//eh/z0KFD+vbbb+scAamqqtKQIUMkSQUFBbrnnntksVhst18PIWe588475enpabseEhKiAwcO2G2zfPlyXbp0SXl5eerdu7dTHx/AjRE3ABzi5+dn+7m2tlYTJkyo9+hISEhInbVLly4pJiZGMTExev311xUUFKSioiKNGzdOVVVVDT7m9QB677331KNHD7vbfHx8JF17TU5La9eund11i8Vim+26kSNH6r333tObb77p9NNiAG6MuAHQbEOHDtXWrVvVq1cveXl9/x8rX375pUpLS/Xiiy8qNDRUkpSXl2e3zfV3X9XU1NjW+vfvLx8fHxUVFWn06NH13nf//v315z//2W7t448/bsrTcYq77rpLTz/9tMaNGydPT0/Nmzev1WcAbla8oBhAs82cOVPnz5/XL3/5S/39739XYWGhPvjgA02dOtUuTq7r2bOnvL29tXr1ahUWFuqvf/2rFi9ebLdNWFiYLBaL3n33XZ07d04XL15Ux44dNXfuXM2ZM0cbN27U0aNHtX//fr3yyivauHGjJCk+Pl5Hjx5VUlKSDh8+rD/84Q8Nvti3pQ0bNkzbtm1Tamqqfv/737tkBuBmRNwAaLbu3btr7969qqmp0bhx4xQZGanZs2crICBAHh51/5gJCgpSVlaWtmzZov79++vFF1/UsmXL7Lbp0aOHUlJSNH/+fAUHB2vWrFmSpMWLF2vhwoVKS0tTRESExo0bp3feeUfh4eGSroXT1q1b9c4772jQoEFas2aN3QudW9uIESP03nvv6ZlnntGqVatcNgdwM7FYW+MENQAAQCvhyA0AADAKcQPgphIbG2v7SoXvXlx5+gqA83BaCsBN5dSpU7py5Uq9twUGBto+PBBA20XcAAAAo3BaCgAAGIW4AQAARiFuAACAUYgbAABgFOIGAAAYhbgBAABGIW4AAIBRiBsAAGCU/wMSz0pEvbaIkAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(multilink_cmts, x=\"related_link\", y=\"flair_correct\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "315b80da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "related_link\n",
       "False    0.347273\n",
       "True     0.839378\n",
       "Name: flair_correct, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multilink_cmts.groupby('related_link')['flair_correct'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2d630d9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='flair_correct', ylabel='related_link'>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGzCAYAAADT4Tb9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAshklEQVR4nO3dfVRVdd7//9cR5UAYkHdEhUiahqKlUCYtNafCwbnRbIqVpVlScVkpMtbouMYbcqSrjNCuQM0bNJuJuuyaqynUWGWl2Z1E11SYaVmH7ChCCWgFCfv3h9/Ob06AHQ4HzvHT87HWXov92Z/P3u9ti3z52Xc2y7IsAQAAGKKLvwsAAADwJcINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADBKV38XIEn5+fl6+OGH5XQ6NWTIEOXl5Wn06NEt9p0+fbo2btzYrH3w4MH66KOPPDpeU1OTvvrqK5199tmy2Wztqh0AAHQOy7JUV1en8847T126nGZ+xvKzp59+2urWrZv1xBNPWOXl5dbs2bOtsLAw64svvmix/7Fjxyyn0+laKioqrB49eliLFi3y+JgVFRWWJBYWFhYWFpYzcKmoqDjt3/M2y/LvhzNHjhypESNGqKCgwNUWHx+vSZMmKScn52fH/+Mf/9DkyZN18OBBxcbGenTMmpoaRUZGqqKiQuHh4V7XDgAAOk9tba1iYmJ07NgxRUREtNrPr5elGhoaVFpaqnnz5rm1p6SkaPfu3R7tY926dbrmmmtOG2zq6+tVX1/vWq+rq5MkhYeHE24AADjD/NwtJX69obiqqkqNjY2Kiopya4+KitLhw4d/drzT6dTWrVuVnp5+2n45OTmKiIhwLTExMe2qGwAABK6AeFrqpwnMsiyPbvQtLCxUZGSkJk2adNp+8+fPV01NjWupqKhoT7kAACCA+fWyVK9evRQUFNRslqaysrLZbM5PWZal9evXa+rUqQoODj5tX7vdLrvd3u56AQBA4PPrzE1wcLASExNVUlLi1l5SUqLk5OTTjn3ttdd04MABzZgxoyNLBAAAZxi/v+cmKytLU6dOVVJSkkaNGqU1a9bI4XAoIyND0qlLSocOHdKmTZvcxq1bt04jR45UQkKCP8oGAAAByu/hJi0tTdXV1crOzpbT6VRCQoKKi4tdTz85nU45HA63MTU1NdqyZYtWrFjhj5IBAEAA8/t7bvyhtrZWERERqqmp4VFwAADOEJ7+/R0QT0sBAAD4CuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBR/P6eGwAAvDF79mwdPXpUktS7d2/efQYXwg0A4Ix09OhRHTlyxN9lIABxWQoAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAoXf1dAACcaRzZQ/1dAiSdPNZTUtD/+/kr/rsEgL4LP/B3CZKYuQEAAIYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAowREuMnPz1dcXJxCQkKUmJionTt3nrZ/fX29FixYoNjYWNntdvXv31/r16/vpGoBAEAg8/u3pYqKipSZman8/HxdeeWVWr16tVJTU1VeXq6+ffu2OObGG2/UkSNHtG7dOg0YMECVlZU6efJkJ1cOAAACkd/DTW5urmbMmKH09HRJUl5enrZv366CggLl5OQ0679t2za99tpr+uyzz9SjRw9JUr9+/TqzZAAAEMD8elmqoaFBpaWlSklJcWtPSUnR7t27Wxzz/PPPKykpSQ899JDOP/98DRw4UHPnztV3333X6nHq6+tVW1vrtgAAADP5deamqqpKjY2NioqKcmuPiorS4cOHWxzz2WefadeuXQoJCdH//M//qKqqSjNnztTXX3/d6n03OTk5WrJkic/rBwAAgScgbii22Wxu65ZlNWv7UVNTk2w2m5566ildfvnlmjBhgnJzc1VYWNjq7M38+fNVU1PjWioqKnx+DgAAIDD4deamV69eCgoKajZLU1lZ2Ww250fR0dE6//zzFRER4WqLj4+XZVn68ssvddFFFzUbY7fbZbfbfVs8AAAISH6duQkODlZiYqJKSkrc2ktKSpScnNzimCuvvFJfffWVjh8/7mr75JNP1KVLF11wwQUdWi8AAAh8fr8slZWVpbVr12r9+vXau3ev5syZI4fDoYyMDEmnLilNmzbN1X/KlCnq2bOnbrvtNpWXl+v111/Xfffdp9tvv12hoaH+Og0AABAg/P4oeFpamqqrq5WdnS2n06mEhAQVFxcrNjZWkuR0OuVwOFz9u3fvrpKSEt17771KSkpSz549deONN2rp0qX+OgUAABBAbJZlWf4uorPV1tYqIiJCNTU1Cg8P93c5AM4wjuyh/i4Bkua+1VPV9UGSpJ72Ri2/otrPFaHvwg86dP+e/v3t95kbAAC80cPe2OLPAOEGAHBG+vPwY/4uAQHK7zcUAwAA+BLhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGCUgAg3+fn5iouLU0hIiBITE7Vz585W+7766quy2WzNlo8//rgTKwYAAIHK7+GmqKhImZmZWrBggcrKyjR69GilpqbK4XCcdty+ffvkdDpdy0UXXdRJFQMAgEDm93CTm5urGTNmKD09XfHx8crLy1NMTIwKCgpOO65Pnz4699xzXUtQUFAnVQwAAAKZX8NNQ0ODSktLlZKS4taekpKi3bt3n3bs8OHDFR0drauvvlo7duw4bd/6+nrV1ta6LQAAwEx+DTdVVVVqbGxUVFSUW3tUVJQOHz7c4pjo6GitWbNGW7Zs0XPPPadBgwbp6quv1uuvv97qcXJychQREeFaYmJifHoeAAAgcHT1dwGSZLPZ3NYty2rW9qNBgwZp0KBBrvVRo0apoqJCy5cv15gxY1ocM3/+fGVlZbnWa2trCTgAABjKrzM3vXr1UlBQULNZmsrKymazOadzxRVXaP/+/a1ut9vtCg8Pd1sAAICZ/BpugoODlZiYqJKSErf2kpISJScne7yfsrIyRUdH+7o8AABwBvL7ZamsrCxNnTpVSUlJGjVqlNasWSOHw6GMjAxJpy4pHTp0SJs2bZIk5eXlqV+/fhoyZIgaGhq0efNmbdmyRVu2bPHnaQAAgADh93CTlpam6upqZWdny+l0KiEhQcXFxYqNjZUkOZ1Ot3feNDQ0aO7cuTp06JBCQ0M1ZMgQvfjii5owYYK/TgEAAAQQm2VZlr+L6Gy1tbWKiIhQTU0N998AaDNH9lB/lwAEpL4LP+jQ/Xv697ffX+IHAADgS4QbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAoXoWbl19+udVt//Vf/+V1MQAAAO3lVbi5/vrr9e677zZrz8vL05///Od2FwUAAOAtr8LNo48+qgkTJqi8vNzVtnz5ci1atEgvvviiz4oDAABoq67eDLrttttUXV2tlJQU7dq1S0VFRVq2bJm2bt2q5ORkX9cIAADgMa/CjSTNnTtX1dXVSkpKUmNjo1566SWNHDnSl7UBAAC0mcfhZuXKlc3aoqOjddZZZ2nMmDF6++239fbbb0uSZs2a5bsKAQAA2sBmWZblSce4uDjPdmiz6bPPPmtXUR2ttrZWERERqqmpUXh4uL/LAXCGcWQP9XcJQEDqu/CDDt2/p39/ezxzc/DgQZ8UBgAA0JEC4iV++fn5iouLU0hIiBITE7Vz506Pxr3xxhvq2rWrLr300o4tEAAAnDG8uqG4sbFRhYWFevnll1VZWammpia37a+88orH+yoqKlJmZqby8/N15ZVXavXq1UpNTVV5ebn69u3b6riamhpNmzZNV199tY4cOeLNaQAAAAN5NXMze/ZszZ49W42NjUpISNAll1zitrRFbm6uZsyYofT0dMXHxysvL08xMTEqKCg47bi77rpLU6ZM0ahRo7w5BQAAYCivZm6efvppPfPMM5owYUK7Dt7Q0KDS0lLNmzfPrT0lJUW7d+9uddyGDRv06aefavPmzVq6dOnPHqe+vl719fWu9draWu+LBgAAAc2rmZvg4GANGDCg3QevqqpSY2OjoqKi3NqjoqJ0+PDhFsfs379f8+bN01NPPaWuXT3LZjk5OYqIiHAtMTEx7a4dAAAEJq/CzR//+EetWLFCHj5F/rNsNpvbumVZzdqkU/f6TJkyRUuWLNHAgQM93v/8+fNVU1PjWioqKtpdMwAACExeXZbatWuXduzYoa1bt2rIkCHq1q2b2/bnnnvOo/306tVLQUFBzWZpKisrm83mSFJdXZ327NmjsrIy3XPPPZKkpqYmWZalrl276qWXXtKvfvWrZuPsdrvsdrunpwcAAM5gXoWbyMhIXXfdde0+eHBwsBITE1VSUuK2v5KSEk2cOLFZ//DwcH3wgfsLgvLz8/XKK6/ov//7vz1+0SAAADCXV+Fmw4YNPisgKytLU6dOVVJSkkaNGqU1a9bI4XAoIyND0qlLSocOHdKmTZvUpUsXJSQkuI3v06ePQkJCmrUDAIBfJq8/nOkraWlpqq6uVnZ2tpxOpxISElRcXKzY2FhJktPplMPh8HOVAADgTOHxt6VGjBihl19+Weecc46GDx/e4g2/P3rvvfd8VmBH4NtSANqDb0sBLTvjvi01ceJE1025kyZNaneBAAAAHcHjmRuTMHMDoD2YuQFaFigzNwHx4UwAAABf8fiy1DnnnHPa+2z+3ddff+11QQAAAO3hcbjJy8vrwDIAAAB8w+Nwc+utt7Z55w8++KAyMjIUGRnZ5rEAAADe6NB7bpYtW8YlKgAA0Kk6NNz8Ah/EAgAAfsbTUgAAwCiEGwAAYBTCDQAAMArhBgAAGKVDw83o0aMVGhrakYcAAABw4/F7bmpraz3e6Y/feyguLm57RQAAAO3gcbiJjIz0+PMLjY2NXhcEAADQHh6Hmx07drh+/vzzzzVv3jxNnz5do0aNkiS9+eab2rhxo3JycnxfJQAAgIc8Djdjx451/Zydna3c3FzddNNNrrbf//73Gjp0qNasWePVpxoAAAB8wasbit98800lJSU1a09KStI777zT7qIAAAC85VW4iYmJ0apVq5q1r169WjExMe0uCgAAwFseX5b6d48++qiuv/56bd++XVdccYUk6a233tKnn36qLVu2+LRAAACAtvBq5mbChAn65JNP9Pvf/15ff/21qqurNXHiRH3yySeaMGGCr2sEAADwmFczN9KpS1PLli3zZS0AAADt5vUbinfu3KlbbrlFycnJOnTokCTpySef1K5du3xWHAAAQFt5FW62bNmi8ePHKzQ0VO+9957q6+slSXV1dczmAAAAv/Iq3CxdulSrVq3SE088oW7durnak5OT9d577/msOAAAgLbyKtzs27dPY8aMadYeHh6uY8eOtbcmAAAAr3kVbqKjo3XgwIFm7bt27dKFF17Y7qIAAAC85VW4ueuuuzR79my9/fbbstls+uqrr/TUU09p7ty5mjlzpq9rBAAA8JhXj4Lff//9qqmp0bhx4/T9999rzJgxstvtmjt3ru655x5f1wgAAOAxr99z89e//lULFixQeXm5mpqaNHjwYHXv3t2XtQEAALSZV5elbr/9dtXV1emss85SUlKSLr/8cnXv3l0nTpzQ7bff7usaAQAAPOZVuNm4caO+++67Zu3fffedNm3a1O6iAAAAvNWmy1K1tbWyLEuWZamurk4hISGubY2NjSouLlafPn18XiQAAICn2hRuIiMjZbPZZLPZNHDgwGbbbTablixZ4rPiAAAA2qpN4WbHjh2yLEu/+tWvtGXLFvXo0cO1LTg4WLGxsTrvvPN8XiQAAICn2hRuxo4dK0k6ePCgYmJi1KWL19/dBAAA6BBePQoeGxsrSfr222/lcDjU0NDgtn3YsGHtrwwAAMALXoWbo0eP6rbbbtPWrVtb3N7Y2NiuogAAALzl1XWlzMxMffPNN3rrrbcUGhqqbdu2aePGjbrooov0/PPP+7pGAAAAj3k1c/PKK6/of//3f3XZZZepS5cuio2N1bXXXqvw8HDl5OToN7/5ja/rBAAA8IhXMzcnTpxwvc+mR48eOnr0qCRp6NCheu+993xXHQAAQBt5FW4GDRqkffv2SZIuvfRSrV69WocOHdKqVasUHR3t0wIBAADawut7bpxOpyRp0aJF2rZtm/r27auVK1dq2bJlbd5ffn6+4uLiFBISosTERO3cubPVvrt27dKVV16pnj17KjQ0VBdffLEeffRRb04DAAAYyKt7bm6++WbXz8OHD9fnn3+ujz/+WH379lWvXr3atK+ioiJlZmYqPz9fV155pVavXq3U1FSVl5erb9++zfqHhYXpnnvu0bBhwxQWFqZdu3bprrvuUlhYmO68805vTgcAABjEZlmW5c8CRo4cqREjRqigoMDVFh8fr0mTJiknJ8ejfUyePFlhYWF68sknW9xeX1+v+vp613ptba1iYmJUU1Oj8PDw9p0AgF8cR/ZQf5cABKS+Cz/o0P3X1tYqIiLiZ//+9njmJisry+OD5+bmetSvoaFBpaWlmjdvnlt7SkqKdu/e7dE+ysrKtHv3bi1durTVPjk5OXzzCgCAXwiPw01ZWZlH/Ww2m8cHr6qqUmNjo6Kiotzao6KidPjw4dOOveCCC3T06FGdPHlSixcvVnp6eqt958+f7xbOfpy5AQAA5vE43OzYsaPDivhpILIs62dD0s6dO3X8+HG99dZbmjdvngYMGKCbbrqpxb52u112u91n9QIAgMDl1Q3FPzpw4IA+/fRTjRkzRqGhoR6Fkn/Xq1cvBQUFNZulqaysbDab81NxcXGSTr1b58iRI1q8eHGr4QYAAPxyePUoeHV1ta6++moNHDhQEyZMcD0Wnp6erj/+8Y8e7yc4OFiJiYkqKSlxay8pKVFycrLH+7Esy+2GYQAA8MvlVbiZM2eOunXrJofDobPOOsvVnpaWpm3btrVpX1lZWVq7dq3Wr1+vvXv3as6cOXI4HMrIyJB06n6ZadOmufo//vjj+uc//6n9+/dr//792rBhg5YvX65bbrnFm1MBAACG8eqy1EsvvaTt27frggsucGu/6KKL9MUXX7RpX2lpaaqurlZ2dracTqcSEhJUXFys2NhYSZLT6ZTD4XD1b2pq0vz583Xw4EF17dpV/fv314MPPqi77rrLm1MBAACG8SrcnDhxwm3G5kdVVVVe3bg7c+ZMzZw5s8VthYWFbuv33nuv7r333jYfAwAA/DJ4dVlqzJgx2rRpk2vdZrOpqalJDz/8sMaNG+ez4gAAANrKq5mb5cuXa+zYsdqzZ48aGhp0//3366OPPtLXX3+tN954w9c1AgAAeKzNMzc//PCDZs6cqeeff16XX365rr32Wp04cUKTJ09WWVmZ+vfv3xF1AgAAeKTNMzfdunXThx9+qJ49e/JJAwAAEHC8uudm2rRpWrduna9rAQAAaDev7rlpaGjQ2rVrVVJSoqSkJIWFhblt9/TDmQAAAL7mVbj58MMPNWLECEnSJ5984ratLZ9fAAAA8DWvwk1HfkQTAACgPby65wYAACBQEW4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMERLjJz89XXFycQkJClJiYqJ07d7ba97nnntO1116r3r17Kzw8XKNGjdL27ds7sVoAABDI/B5uioqKlJmZqQULFqisrEyjR49WamqqHA5Hi/1ff/11XXvttSouLlZpaanGjRun3/3udyorK+vkygEAQCCyWZZl+bOAkSNHasSIESooKHC1xcfHa9KkScrJyfFoH0OGDFFaWpoWLlzoUf/a2lpFRESopqZG4eHhXtUN4JfLkT3U3yUAAanvwg86dP+e/v3t15mbhoYGlZaWKiUlxa09JSVFu3fv9mgfTU1NqqurU48ePVrtU19fr9raWrcFAACYya/hpqqqSo2NjYqKinJrj4qK0uHDhz3axyOPPKITJ07oxhtvbLVPTk6OIiIiXEtMTEy76gYAAIHL7/fcSJLNZnNbtyyrWVtL/v73v2vx4sUqKipSnz59Wu03f/581dTUuJaKiop21wwAAAJTV38evFevXgoKCmo2S1NZWdlsNuenioqKNGPGDD377LO65pprTtvXbrfLbre3u14AABD4/DpzExwcrMTERJWUlLi1l5SUKDk5udVxf//73zV9+nT97W9/029+85uOLhMAAJxB/DpzI0lZWVmaOnWqkpKSNGrUKK1Zs0YOh0MZGRmSTl1SOnTokDZt2iTpVLCZNm2aVqxYoSuuuMI16xMaGqqIiAi/nQcAAAgMfg83aWlpqq6uVnZ2tpxOpxISElRcXKzY2FhJktPpdHvnzerVq3Xy5Endfffduvvuu13tt956qwoLCzu7fAAAEGD8/p4bf+A9NwDag/fcAC0LlPfc+H3mBuhIs2fP1tGjRyVJvXv31ooVK/xcEQCgoxFuYLSjR4/qyJEj/i4DANCJAuI9NwAAAL5CuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABG6ervAkyVeN8mf5cASeHfHHcleOc3x/nvEgBKH57m7xIAGI6ZGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADBKV38XAHSkpm5hLf4MADAX4QZGOz4o1d8lAAA6WUBclsrPz1dcXJxCQkKUmJionTt3ttrX6XRqypQpGjRokLp06aLMzMzOKxQAAAQ8v4eboqIiZWZmasGCBSorK9Po0aOVmpoqh8PRYv/6+nr17t1bCxYs0CWXXNLJ1QIAgEDn93CTm5urGTNmKD09XfHx8crLy1NMTIwKCgpa7N+vXz+tWLFC06ZNU0RERCdXCwAAAp1fw01DQ4NKS0uVkpLi1p6SkqLdu3f77Dj19fWqra11WwAAgJn8Gm6qqqrU2NioqKgot/aoqCgdPnzYZ8fJyclRRESEa4mJifHZvgEAQGDx+2UpSbLZbG7rlmU1a2uP+fPnq6amxrVUVFT4bN8AACCw+PVR8F69eikoKKjZLE1lZWWz2Zz2sNvtstvtPtsfAAAIXH6duQkODlZiYqJKSkrc2ktKSpScnOynqgAAwJnM7y/xy8rK0tSpU5WUlKRRo0ZpzZo1cjgcysjIkHTqktKhQ4e0adMm15j3339fknT8+HEdPXpU77//voKDgzV48GB/nAIAAAggfg83aWlpqq6uVnZ2tpxOpxISElRcXKzY2FhJp17a99N33gwfPtz1c2lpqf72t78pNjZWn3/+eWeWDgAAApDfw40kzZw5UzNnzmxxW2FhYbM2y7I6uCIAAHCmCoinpQAAAHyFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMEhDhJj8/X3FxcQoJCVFiYqJ27tx52v6vvfaaEhMTFRISogsvvFCrVq3qpEoBAECg83u4KSoqUmZmphYsWKCysjKNHj1aqampcjgcLfY/ePCgJkyYoNGjR6usrEx//vOfNWvWLG3ZsqWTKwcAAIHI7+EmNzdXM2bMUHp6uuLj45WXl6eYmBgVFBS02H/VqlXq27ev8vLyFB8fr/T0dN1+++1avnx5J1cOAAACUVd/HryhoUGlpaWaN2+eW3tKSop2797d4pg333xTKSkpbm3jx4/XunXr9MMPP6hbt27NxtTX16u+vt61XlNTI0mqra1t7ym0qrH+uw7bN3Am68jfu85S932jv0sAAlJH/37/uH/Lsk7bz6/hpqqqSo2NjYqKinJrj4qK0uHDh1scc/jw4Rb7nzx5UlVVVYqOjm42JicnR0uWLGnWHhMT047qAXgj4rEMf5cAoKPkRHTKYerq6hQR0fqx/BpufmSz2dzWLctq1vZz/Vtq/9H8+fOVlZXlWm9qatLXX3+tnj17nvY4MENtba1iYmJUUVGh8PBwf5cDwIf4/f5lsSxLdXV1Ou+8807bz6/hplevXgoKCmo2S1NZWdlsduZH5557bov9u3btqp49e7Y4xm63y263u7VFRkZ6XzjOSOHh4fzPDzAUv9+/HKebsfmRX28oDg4OVmJiokpKStzaS0pKlJyc3OKYUaNGNev/0ksvKSkpqcX7bQAAwC+L35+WysrK0tq1a7V+/Xrt3btXc+bMkcPhUEbGqevy8+fP17Rp01z9MzIy9MUXXygrK0t79+7V+vXrtW7dOs2dO9dfpwAAAAKI3++5SUtLU3V1tbKzs+V0OpWQkKDi4mLFxsZKkpxOp9s7b+Li4lRcXKw5c+bo8ccf13nnnaeVK1fq+uuv99cpIMDZ7XYtWrSo2aVJAGc+fr/REpv1c89TAQAAnEH8flkKAADAlwg3AADAKIQbAABgFMINjFVYWMj7jADgF4hwg4A3ffp02Wy2ZsuBAwf8XRoAH2npd/zfl+nTp/u7RJxB/P4oOOCJX//619qwYYNbW+/evf1UDQBfczqdrp+Lioq0cOFC7du3z9UWGhrq1r+1DyUDEjM3OEPY7Xade+65bsuKFSs0dOhQhYWFKSYmRjNnztTx48db3cf//d//ady4cTr77LMVHh6uxMRE7dmzx7V99+7dGjNmjEJDQxUTE6NZs2bpxIkTnXF6wC/ev/9uR0REyGazuda///57RUZG6plnntFVV12lkJAQbd68WYsXL9all17qtp+8vDz169fPrW3Dhg2Kj49XSEiILr74YuXn53feicEvCDc4Y3Xp0kUrV67Uhx9+qI0bN+qVV17R/fff32r/m2++WRdccIHeffddlZaWat68ea5/+X3wwQcaP368Jk+erH/9618qKirSrl27dM8993TW6QD4GX/60580a9Ys7d27V+PHj/dozBNPPKEFCxbor3/9q/bu3atly5bpL3/5izZu3NjB1cKfuCyFM8ILL7yg7t27u9ZTU1P17LPPutbj4uL0wAMP6D/+4z9a/VeZw+HQfffdp4svvliSdNFFF7m2Pfzww5oyZYoyMzNd21auXKmxY8eqoKBAISEhHXBWANoiMzNTkydPbtOYBx54QI888ohrXFxcnMrLy7V69WrdeuutHVEmAgDhBmeEcePGqaCgwLUeFhamHTt2aNmyZSovL1dtba1Onjyp77//XidOnFBYWFizfWRlZSk9PV1PPvmkrrnmGt1www3q37+/JKm0tFQHDhzQU0895epvWZaampp08OBBxcfHd/xJAjitpKSkNvU/evSoKioqNGPGDN1xxx2u9pMnT3r0ZWmcuQg3OCOEhYVpwIABrvUvvvhCEyZMUEZGhh544AH16NFDu3bt0owZM/TDDz+0uI/FixdrypQpevHFF7V161YtWrRITz/9tK677jo1NTXprrvu0qxZs5qN69u3b4edFwDP/fQfLV26dNFPvyD077//TU1Nkk5dmho5cqRbv6CgoA6qEoGAcIMz0p49e3Ty5Ek98sgj6tLl1K1jzzzzzM+OGzhwoAYOHKg5c+bopptu0oYNG3TddddpxIgR+uijj9wCFIDA1rt3bx0+fFiWZclms0mS3n//fdf2qKgonX/++frss8908803+6lK+APhBmek/v376+TJk3rsscf0u9/9Tm+88YZWrVrVav/vvvtO9913n/7whz8oLi5OX375pd59913X1+T/9Kc/6YorrtDdd9+tO+64Q2FhYdq7d69KSkr02GOPddZpAWiDq666SkePHtVDDz2kP/zhD9q2bZu2bt2q8PBwV5/Fixdr1qxZCg8PV2pqqurr67Vnzx598803ysrK8mP16Eg8LYUz0qWXXqrc3Fz953/+pxISEvTUU08pJyen1f5BQUGqrq7WtGnTNHDgQN14441KTU3VkiVLJEnDhg3Ta6+9pv3792v06NEaPny4/vKXvyg6OrqzTglAG8XHxys/P1+PP/64LrnkEr3zzjuaO3euW5/09HStXbtWhYWFGjp0qMaOHavCwkLFxcX5qWp0Bpv10wuWAAAAZzBmbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuALSJZVm688471aNHD9lsNkVGRiozM9Pj8a+++qpsNpuOHTvWYTUC+GUj3ABok23btqmwsFAvvPCCnE6nEhIS2jQ+OTlZTqdTERERHVRhYCksLFRkZKS/ywB+UfhwJoA2+fTTTxUdHa3k5GRJUteubfvfSHBwsM4999xWtzc2Nspms7m+9t4ZfvjhB3Xr1s3jdgCBjZkbAB6bPn267r33XjkcDtlsNvXr169Zn82bNyspKUlnn322zj33XE2ZMkWVlZWu7T+9LPXjzMYLL7ygwYMHy26364svvvjZWtavX68hQ4bIbrcrOjpa99xzj2ubw+HQxIkT1b17d4WHh+vGG2/UkSNHXNsXL16sSy+9VOvXr9eFF14ou90uy7Jks9m0atUqTZw4UWFhYVq6dKkk6Z///KcSExMVEhKiCy+8UEuWLNHJkydd+zt27JjuvPNORUVFKSQkRAkJCXrhhRf06quv6rbbblNNTY1sNptsNpsWL17cxj91AG3FzA0Aj61YsUL9+/fXmjVr9O677yooKEg33HCDW5+GhgY98MADGjRokCorKzVnzhxNnz5dxcXFre7322+/VU5OjtauXauePXuqT58+p62joKBAWVlZevDBB5Wamqqamhq98cYbkk7dEzRp0iSFhYXptdde08mTJzVz5kylpaXp1Vdfde3jwIEDeuaZZ7RlyxYFBQW52hctWqScnBw9+uijCgoK0vbt23XLLbdo5cqVGj16tD799FPdeeedrr5NTU1KTU1VXV2dNm/erP79+6u8vFxBQUFKTk5WXl6eFi5cqH379kmSunfv3qY/cwBesACgDR599FErNjbWtT527Fhr9uzZrfZ/5513LElWXV2dZVmWtWPHDkuS9c0331iWZVkbNmywJFnvv/++xzWcd9551oIFC1rc9tJLL1lBQUGWw+FwtX300UeWJOudd96xLMuyFi1aZHXr1s2qrKx0GyvJyszMdGsbPXq0tWzZMre2J5980oqOjrYsy7K2b99udenSxdq3b1+L9WzYsMGKiIjw+NwAtB+XpQD4VFlZmSZOnKjY2FidffbZuuqqqySdulTUmuDgYA0bNsyj/VdWVuqrr77S1Vdf3eL2vXv3KiYmRjExMa62wYMHKzIyUnv37nW1xcbGqnfv3s3GJyUlua2XlpYqOztb3bt3dy133HGHnE6nvv32W73//vu64IILNHDgQI/qB9DxuCwFwGdOnDihlJQUpaSkaPPmzerdu7ccDofGjx+vhoaGVseFhobKZrN5dIzQ0NDTbrf+370zP9ceFhbW4viftjc1NWnJkiWaPHlys74hISE/Ww+Azke4AeAzH3/8saqqqvTggw+6Zk727Nnj02OcffbZ6tevn15++WWNGzeu2fbBgwfL4XCooqLCVUN5eblqamoUHx/f5uONGDFC+/bt04ABA1rcPmzYMH355Zf65JNPWpy9CQ4OVmNjY5uPC8B7hBsAPtO3b18FBwfrscceU0ZGhj788EM98MADPj/O4sWLlZGRoT59+rhu5n3jjTd077336pprrtGwYcN08803Ky8vz3VD8dixY5tdcvLEwoUL9dvf/lYxMTG64YYb1KVLF/3rX//SBx98oKVLl2rs2LEaM2aMrr/+euXm5mrAgAH6+OOPZbPZ9Otf/1r9+vXT8ePH9fLLL+uSSy7RWWedpbPOOsvnfyYA/n/ccwPAZ3r37q3CwkI9++yzGjx4sB588EEtX77c58e59dZblZeXp/z8fA0ZMkS//e1vtX//fkmSzWbTP/7xD51zzjkaM2aMrrnmGl144YUqKiry6ljjx4/XCy+8oJKSEl122WW64oorlJubq9jYWFefLVu26LLLLtNNN92kwYMH6/7773fN1iQnJysjI0NpaWnq3bu3Hnroofb/AQA4LZtlWZa/iwAAAPAVZm4AAIBRCDcAAs6/P3b902Xnzp3+Lg9AgOOyFICAc+DAgVa3nX/++Tx+DeC0CDcAAMAoXJYCAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABjl/wMf+IaPmp3tBwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(multilink_cmts, x=\"flair_correct\", y=\"related_link\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7bca3cd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "flair_correct\n",
       "False    0.147268\n",
       "True     0.629126\n",
       "Name: related_link, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multilink_cmts.groupby('flair_correct')['related_link'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f33ae91f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44.98% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post\n"
     ]
    }
   ],
   "source": [
    "reviewed_cmts = multilink_cmts[multilink_cmts['manually_reviewed']]\n",
    "err_rate = ((~reviewed_cmts['flair_correct']).sum()/len(reviewed_cmts))*100\n",
    "print(f\"{err_rate:.2f}% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9eac18c4",
   "metadata": {},
   "source": [
    "#### Invalid/Missing Content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "401ac9d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removed 2 rows with missing or invalid content.\n"
     ]
    }
   ],
   "source": [
    "null_rows = multilink_cmts['corrected_flair'].isna()\n",
    "null_df = multilink_cmts[null_rows].copy()\n",
    "null_df['discard_reason'] = \"Invalid/Missing Content Type\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, null_df[~null_df.index.isin(discarded_cmts.index)]])\n",
    "multilink_cmts = multilink_cmts[~null_rows]\n",
    "print(f\"Removed {null_rows.sum()} rows with missing or invalid content.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "57c29f49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After filtering, 44.86% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\n"
     ]
    }
   ],
   "source": [
    "reviewed_cmts = multilink_cmts[multilink_cmts['manually_reviewed']]\n",
    "err_rate = (~reviewed_cmts['flair_correct']).sum()/len(reviewed_cmts)*100\n",
    "print(f\"After filtering, {err_rate:.2f}% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dfe16c38",
   "metadata": {},
   "source": [
    "#### Duplicate Links with Mixed Relationships to the Titular Post\n",
    "\n",
    "In a few particular cases, a link was duplicated within the dataset and included both posts related to and unrelated to the main submission/flair in question.\n",
    "\n",
    "I made to sure keep the related instances in these cases. I did not bother to fix the flair given that these will be discarded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "58acc7a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['https://homebrewery.naturalcrit.com/source/1-M_mgIo6auMreJ7TIIXkIfisYSUnkO1RYoo6GRrq8ZRr',\n",
       "       'https://homebrewery.naturalcrit.com/source/1-RG_M7OJzm2oMw0cMMXyyzAq3zD1OkJ-Ge_07X_o7ZFE',\n",
       "       'https://homebrewery.naturalcrit.com/source/1-u6fWltBA4ULSqyO0gzZck9zVI344gFiYwhDrq2dcXSl',\n",
       "       'https://homebrewery.naturalcrit.com/source/10HyHBXn_Ol0tae9-t5vpZzIGWKPe5l8W6fCTK77Sd8DD',\n",
       "       'https://homebrewery.naturalcrit.com/source/10WTDFO5SRDmh0x9l8mj-aT_aF0Dce-Kpk_hn68Pftw5j',\n",
       "       'https://homebrewery.naturalcrit.com/source/10_9qbCu_jDdHfG72oZmLv86sX1F_RKp8g9zTdkduosDo',\n",
       "       'https://homebrewery.naturalcrit.com/source/11D123fkj2EWNYBvaXuJKGuqydSzJUYBcowQ5pXPpqxKb',\n",
       "       'https://homebrewery.naturalcrit.com/source/11s4ieAqZiqStfRT3fxFHrHOSLKaUzaifsojvlVDoKawB',\n",
       "       'https://homebrewery.naturalcrit.com/source/13WCQFSjOhpiJJtx6Oq_oZ-aRw_72O6tz8rPrcDxTSiRb',\n",
       "       'https://homebrewery.naturalcrit.com/source/13at3aA6Vd-x6A2mxErg-f8GM0VYm4GeqtN96wsMmubR5',\n",
       "       ...\n",
       "       'https://homebrewery.naturalcrit.com/source/1yjOSStdpYoH7qBuCcjYiHsSHTF01JkYaccEG_4pSv-6O',\n",
       "       'https://homebrewery.naturalcrit.com/source/1ys1ZUyxW9nqoLO9hHUYKuQ_aZ2ID52IUdPfV5PNL86ct',\n",
       "       'https://homebrewery.naturalcrit.com/source/1yuf-qJIezBp6amETuOSghf_TIx07ESI2dh5QrfxJEh_m',\n",
       "       'https://homebrewery.naturalcrit.com/source/1zD40AwGgmlYk3GbiqURXeSPif5t_CaQol8Wf_0PvHm9T',\n",
       "       'https://homebrewery.naturalcrit.com/source/1zF0cq4tQScBk5i3fVqgb_Jh4UUiKZ2S4FBdvdFLSAnX9',\n",
       "       'https://homebrewery.naturalcrit.com/source/1zPl3bj5ILT6s4QB9Klq-l36NCKfDPa0tLSseDEXb84pu',\n",
       "       'https://www.gmbinder.com/share/-MSfA82gv8V69JAoqFVq',\n",
       "       'https://www.gmbinder.com/share/-MW4c30CbGMWLRNgJxgb',\n",
       "       'https://www.gmbinder.com/share/-MtR2w88UA77uX5gCoB2/-MtVAaUoGw_yIam0T0fz/source',\n",
       "       'https://www.gmbinder.com/share/-MvHSJ6CCGPVTHBx0GGk/-MwsghDAelJRrPB2nzDG/source'],\n",
       "      dtype='object', name='src_url', length=142)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "link_mixture = multilink_cmts.groupby('src_url')['related_link'].mean()\n",
    "multilink_cmts['related_link'] = multilink_cmts['related_link'].astype(bool)\n",
    "mixed_links = link_mixture[(link_mixture>0) & (link_mixture<1)].index\n",
    "mixed_links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6f75e872",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>corrected_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "      <th>src_url</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>flair_correct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3930</th>\n",
       "      <td>https://gmbinder.com/share/-MSfA82gv8V69JAoqFVq</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>ynxm9m</td>\n",
       "      <td>laserllama's Alternate Ranger Class (Update!) ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>601</td>\n",
       "      <td>0.98</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>ivdcakk</td>\n",
       "      <td>2</td>\n",
       "      <td>Glad you enjoy the execution!\\n\\nAt some point...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://www.gmbinder.com/share/-MSfA82gv8V69JA...</td>\n",
       "      <td>Class</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1286</th>\n",
       "      <td>https://gmbinder.com/share/-MSfA82gv8V69JAoqFVq</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>lyfven</td>\n",
       "      <td>The Alternate Fighter (v1.2.0 Update!) - Becom...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>475</td>\n",
       "      <td>0.98</td>\n",
       "      <td>3/5/2021 16:40</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>gpsfg1d</td>\n",
       "      <td>19</td>\n",
       "      <td>Hello everyone! Here is the third (public) upd...</td>\n",
       "      <td>3/5/2021 16:41</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>https://www.gmbinder.com/share/-MSfA82gv8V69JA...</td>\n",
       "      <td>Class</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3570</th>\n",
       "      <td>https://gmbinder.com/share/-MW4c30CbGMWLRNgJxgb</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>yvb5yt</td>\n",
       "      <td>laserllama's Alternate Ranger: Expanded - A Mu...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>268</td>\n",
       "      <td>0.98</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>iwdb9t0</td>\n",
       "      <td>7</td>\n",
       "      <td>Hello everyone - today I’m sharing a compendiu...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>https://www.gmbinder.com/share/-MW4c30CbGMWLRN...</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1223</th>\n",
       "      <td>https://gmbinder.com/share/-MW4c30CbGMWLRNgJxgb</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>m54olt</td>\n",
       "      <td>Ranger Archetypes (New &amp; Updated) - Track your...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>623</td>\n",
       "      <td>0.99</td>\n",
       "      <td>3/14/2021 21:26</td>\n",
       "      <td>LaserLlama</td>\n",
       "      <td>gvg0bt0</td>\n",
       "      <td>1</td>\n",
       "      <td>No worries, I get them mixed up sometimes too!...</td>\n",
       "      <td>4/22/2021 13:57</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://www.gmbinder.com/share/-MW4c30CbGMWLRN...</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1914</th>\n",
       "      <td>https://gmbinder.com/share/-MtR2w88UA77uX5gCoB2</td>\n",
       "      <td>StarSailGames</td>\n",
       "      <td>t2drfi</td>\n",
       "      <td>Circle of Geometry: Summon Fractal Creatures a...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>50</td>\n",
       "      <td>0.93</td>\n",
       "      <td>2/27/2022 2:32</td>\n",
       "      <td>StarSailGames</td>\n",
       "      <td>hylf7pg</td>\n",
       "      <td>3</td>\n",
       "      <td>Hi everyone! Today I'm sharing a quirky druid ...</td>\n",
       "      <td>2/27/2022 2:33</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://www.gmbinder.com/share/-MtR2w88UA77uX5...</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2888</th>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1zD4...</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>wngw7y</td>\n",
       "      <td>Give me a D&amp;D monster and I'll homebrew you a ...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Monster</td>\n",
       "      <td>149</td>\n",
       "      <td>0.95</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>ik72rt0</td>\n",
       "      <td>1</td>\n",
       "      <td>* [Manticores](https://homebrewery.naturalcrit...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1zD...</td>\n",
       "      <td>5e</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2423</th>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1zF0...</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>whtmbi</td>\n",
       "      <td>Give me a D&amp;D monster and I'll homebrew you a ...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Monster</td>\n",
       "      <td>185</td>\n",
       "      <td>0.95</td>\n",
       "      <td>1659807006</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>ij818f8</td>\n",
       "      <td>3</td>\n",
       "      <td>* [Silver Dragons](https://homebrewery.natural...</td>\n",
       "      <td>1659814781</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1zF...</td>\n",
       "      <td>Monster</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3680</th>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1zF0...</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>z092x8</td>\n",
       "      <td>Give me a D&amp;D monster and I'll homebrew you a ...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Monster</td>\n",
       "      <td>26</td>\n",
       "      <td>0.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>ix5tk1y</td>\n",
       "      <td>2</td>\n",
       "      <td>Chromatics\\n\\n* [Black Dragons](https://homebr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1zF...</td>\n",
       "      <td>5e</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2644</th>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1zPl...</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>whtmbi</td>\n",
       "      <td>Give me a D&amp;D monster and I'll homebrew you a ...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Monster</td>\n",
       "      <td>185</td>\n",
       "      <td>0.95</td>\n",
       "      <td>1659807006</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>ij9p553</td>\n",
       "      <td>1</td>\n",
       "      <td>Oh cool, here's a little CoS care package:\\n\\n...</td>\n",
       "      <td>1659843328</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1zP...</td>\n",
       "      <td>Monster</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3699</th>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1zPl...</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>z092x8</td>\n",
       "      <td>Give me a D&amp;D monster and I'll homebrew you a ...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Monster</td>\n",
       "      <td>26</td>\n",
       "      <td>0.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Oh_Hi_Mark_</td>\n",
       "      <td>ix5rwzx</td>\n",
       "      <td>2</td>\n",
       "      <td>I'm in the process of redoing these, but here'...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1zP...</td>\n",
       "      <td>5e</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>284 rows × 19 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   link submission_author  \\\n",
       "idx                                                                         \n",
       "3930    https://gmbinder.com/share/-MSfA82gv8V69JAoqFVq        LaserLlama   \n",
       "1286    https://gmbinder.com/share/-MSfA82gv8V69JAoqFVq        LaserLlama   \n",
       "3570    https://gmbinder.com/share/-MW4c30CbGMWLRNgJxgb        LaserLlama   \n",
       "1223    https://gmbinder.com/share/-MW4c30CbGMWLRNgJxgb        LaserLlama   \n",
       "1914    https://gmbinder.com/share/-MtR2w88UA77uX5gCoB2     StarSailGames   \n",
       "...                                                 ...               ...   \n",
       "2888  https://homebrewery.naturalcrit.com/share/1zD4...       Oh_Hi_Mark_   \n",
       "2423  https://homebrewery.naturalcrit.com/share/1zF0...       Oh_Hi_Mark_   \n",
       "3680  https://homebrewery.naturalcrit.com/share/1zF0...       Oh_Hi_Mark_   \n",
       "2644  https://homebrewery.naturalcrit.com/share/1zPl...       Oh_Hi_Mark_   \n",
       "3699  https://homebrewery.naturalcrit.com/share/1zPl...       Oh_Hi_Mark_   \n",
       "\n",
       "     submission_id                                   submission_title  \\\n",
       "idx                                                                     \n",
       "3930        ynxm9m  laserllama's Alternate Ranger Class (Update!) ...   \n",
       "1286        lyfven  The Alternate Fighter (v1.2.0 Update!) - Becom...   \n",
       "3570        yvb5yt  laserllama's Alternate Ranger: Expanded - A Mu...   \n",
       "1223        m54olt  Ranger Archetypes (New & Updated) - Track your...   \n",
       "1914        t2drfi  Circle of Geometry: Summon Fractal Creatures a...   \n",
       "...            ...                                                ...   \n",
       "2888        wngw7y  Give me a D&D monster and I'll homebrew you a ...   \n",
       "2423        whtmbi  Give me a D&D monster and I'll homebrew you a ...   \n",
       "3680        z092x8  Give me a D&D monster and I'll homebrew you a ...   \n",
       "2644        whtmbi  Give me a D&D monster and I'll homebrew you a ...   \n",
       "3699        z092x8  Give me a D&D monster and I'll homebrew you a ...   \n",
       "\n",
       "            subreddit corrected_flair  submission_score  \\\n",
       "idx                                                       \n",
       "3930  UnearthedArcana           Class               601   \n",
       "1286  UnearthedArcana           Class               475   \n",
       "3570  UnearthedArcana      Compendium               268   \n",
       "1223  UnearthedArcana      Compendium               623   \n",
       "1914  UnearthedArcana        Subclass                50   \n",
       "...               ...             ...               ...   \n",
       "2888      DnDHomebrew         Monster               149   \n",
       "2423      DnDHomebrew         Monster               185   \n",
       "3680      DnDHomebrew         Monster                26   \n",
       "2644      DnDHomebrew         Monster               185   \n",
       "3699      DnDHomebrew         Monster                26   \n",
       "\n",
       "      submission_upvote_ratio  submission_date comment_author comment_id  \\\n",
       "idx                                                                        \n",
       "3930                     0.98              NaN     LaserLlama    ivdcakk   \n",
       "1286                     0.98   3/5/2021 16:40     LaserLlama    gpsfg1d   \n",
       "3570                     0.98              NaN     LaserLlama    iwdb9t0   \n",
       "1223                     0.99  3/14/2021 21:26     LaserLlama    gvg0bt0   \n",
       "1914                     0.93   2/27/2022 2:32  StarSailGames    hylf7pg   \n",
       "...                       ...              ...            ...        ...   \n",
       "2888                     0.95              NaN    Oh_Hi_Mark_    ik72rt0   \n",
       "2423                     0.95       1659807006    Oh_Hi_Mark_    ij818f8   \n",
       "3680                     0.93              NaN    Oh_Hi_Mark_    ix5tk1y   \n",
       "2644                     0.95       1659807006    Oh_Hi_Mark_    ij9p553   \n",
       "3699                     0.93              NaN    Oh_Hi_Mark_    ix5rwzx   \n",
       "\n",
       "      comment_score                                       comment_body  \\\n",
       "idx                                                                      \n",
       "3930              2  Glad you enjoy the execution!\\n\\nAt some point...   \n",
       "1286             19  Hello everyone! Here is the third (public) upd...   \n",
       "3570              7  Hello everyone - today I’m sharing a compendiu...   \n",
       "1223              1  No worries, I get them mixed up sometimes too!...   \n",
       "1914              3  Hi everyone! Today I'm sharing a quirky druid ...   \n",
       "...             ...                                                ...   \n",
       "2888              1  * [Manticores](https://homebrewery.naturalcrit...   \n",
       "2423              3  * [Silver Dragons](https://homebrewery.natural...   \n",
       "3680              2  Chromatics\\n\\n* [Black Dragons](https://homebr...   \n",
       "2644              1  Oh cool, here's a little CoS care package:\\n\\n...   \n",
       "3699              2  I'm in the process of redoing these, but here'...   \n",
       "\n",
       "         comment_date  manually_reviewed  related_link  \\\n",
       "idx                                                      \n",
       "3930              NaN               True         False   \n",
       "1286   3/5/2021 16:41               True          True   \n",
       "3570              NaN               True          True   \n",
       "1223  4/22/2021 13:57               True         False   \n",
       "1914   2/27/2022 2:33               True         False   \n",
       "...               ...                ...           ...   \n",
       "2888              NaN               True         False   \n",
       "2423       1659814781               True          True   \n",
       "3680              NaN               True         False   \n",
       "2644       1659843328               True          True   \n",
       "3699              NaN               True         False   \n",
       "\n",
       "                                                src_url submission_flair  \\\n",
       "idx                                                                        \n",
       "3930  https://www.gmbinder.com/share/-MSfA82gv8V69JA...            Class   \n",
       "1286  https://www.gmbinder.com/share/-MSfA82gv8V69JA...            Class   \n",
       "3570  https://www.gmbinder.com/share/-MW4c30CbGMWLRN...       Compendium   \n",
       "1223  https://www.gmbinder.com/share/-MW4c30CbGMWLRN...         Subclass   \n",
       "1914  https://www.gmbinder.com/share/-MtR2w88UA77uX5...         Subclass   \n",
       "...                                                 ...              ...   \n",
       "2888  https://homebrewery.naturalcrit.com/source/1zD...               5e   \n",
       "2423  https://homebrewery.naturalcrit.com/source/1zF...          Monster   \n",
       "3680  https://homebrewery.naturalcrit.com/source/1zF...               5e   \n",
       "2644  https://homebrewery.naturalcrit.com/source/1zP...          Monster   \n",
       "3699  https://homebrewery.naturalcrit.com/source/1zP...               5e   \n",
       "\n",
       "      flair_correct  \n",
       "idx                  \n",
       "3930           True  \n",
       "1286           True  \n",
       "3570           True  \n",
       "1223          False  \n",
       "1914           True  \n",
       "...             ...  \n",
       "2888          False  \n",
       "2423           True  \n",
       "3680          False  \n",
       "2644           True  \n",
       "3699          False  \n",
       "\n",
       "[284 rows x 19 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multilink_cmts[multilink_cmts['src_url'].isin(mixed_links)].sort_values(by=\"link\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2ea6a258",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "142 duplicates dropped in favor of a comment associated with a relevant parent post.\n"
     ]
    }
   ],
   "source": [
    "inds_to_drop = (multilink_cmts['src_url'].isin(mixed_links)) & (~multilink_cmts['related_link'])\n",
    "unrelated_dups = multilink_cmts[inds_to_drop].copy()\n",
    "unrelated_dups['discard_reason'] = \"Unrelated duplicate link\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, unrelated_dups[~unrelated_dups.index.isin(discarded_cmts.index)]])\n",
    "multilink_cmts = multilink_cmts[~inds_to_drop]\n",
    "print(f\"{inds_to_drop.sum()} duplicates dropped in favor of a comment associated with a relevant parent post.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "162b4e42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After filtering, 35.23% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\n"
     ]
    }
   ],
   "source": [
    "reviewed_cmts = multilink_cmts[multilink_cmts['manually_reviewed']]\n",
    "err_rate = (~reviewed_cmts['flair_correct']).sum()/len(reviewed_cmts)*100\n",
    "print(f\"After filtering, {err_rate:.2f}% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "26113333",
   "metadata": {},
   "source": [
    "#### Dropping remaining duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9cf4ed90",
   "metadata": {},
   "outputs": [],
   "source": [
    "multilink_cmts = multilink_cmts.sort_values(by=[\"submission_date\",\"comment_date\"])\n",
    "duplicated = multilink_cmts.duplicated(subset=['src_url', 'corrected_flair'], keep='first')\n",
    "dup_df = multilink_cmts[duplicated].copy()\n",
    "dup_df['discard_reason'] = \"Duplicate Unambiguous Flair\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, dup_df[~dup_df.index.isin(discarded_cmts.index)]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "72fb40d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All duplicated links have consistent flair\n"
     ]
    }
   ],
   "source": [
    "mismatches = multilink_cmts.duplicated(subset=[\"src_url\"]).sum() - multilink_cmts.duplicated(subset=[\"src_url\",\"corrected_flair\"]).sum()\n",
    "if mismatches != 0:\n",
    "    print(f\"There are {mismatches} links with inconsistent flair assignments\")\n",
    "    all_dups = multilink_cmts.duplicated(subset=[\"src_url\"], keep=False).index\n",
    "    dups_after_correction = multilink_cmts.duplicated(subset=[\"src_url\",\"corrected_flair\"], keep=False).index\n",
    "    mismatched = [dup for dup in all_dups if dup not in dups_after_correction]\n",
    "    print(multilink_cmts[multilink_cmts.index.isin(mismatched)])\n",
    "else:\n",
    "    print(\"All duplicated links have consistent flair\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "cc686115",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15 duplicate links with un-ambiguous submission flair dropped\n"
     ]
    }
   ],
   "source": [
    "multilink_cmts = multilink_cmts[~duplicated]\n",
    "multilink_cmts.sort_index()\n",
    "print(f\"{duplicated.sum()} duplicate links with un-ambiguous submission flair dropped\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "80df8441",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After filtering, 35.01% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\n"
     ]
    }
   ],
   "source": [
    "reviewed_cmts = multilink_cmts[multilink_cmts['manually_reviewed']]\n",
    "err_rate = (~reviewed_cmts['flair_correct']).sum()/len(reviewed_cmts)*100\n",
    "print(f\"After filtering, {err_rate:.2f}% of manually reviewed links from multi-link comments differed from the submission flair associated with their parent post.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6e1c3b75",
   "metadata": {},
   "source": [
    "### Fixing/re-merging content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bc0678cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 links with missing/invalid content types removed from primary comment dataset.\n"
     ]
    }
   ],
   "source": [
    "null_links = null_df['link'].unique()\n",
    "null_rows = cmts_df['link'].isin(null_links)\n",
    "null_df = cmts_df[null_rows].copy()\n",
    "null_df['submission_flair'] = np.nan\n",
    "null_df['discard_reason'] = \"Invalid/Missing Content Type\"\n",
    "discarded_cmts = pd.concat([discarded_cmts, null_df[~null_df.index.isin(discarded_cmts.index)]])\n",
    "cmts_df = cmts_df[~null_rows]\n",
    "print(f\"{null_rows.sum()} links with missing/invalid content types removed from primary comment dataset.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c10b3ee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "url_mapper = multilink_cmts.set_index('link')['corrected_flair'].to_dict()\n",
    "flair_to_correct = cmts_df['link'].isin(url_mapper.keys())\n",
    "flair_to_correct = flair_to_correct & cmts_df['link'].isin(multilink_cmts['link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c4e35c95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 entries updated\n"
     ]
    }
   ],
   "source": [
    "cmts_df.loc[flair_to_correct, 'submission_flair'] = cmts_df[flair_to_correct]['link'].map(url_mapper).values\n",
    "cmts_df.loc[flair_to_correct, 'manually_reviewed'] = True\n",
    "print(f\"{flair_to_correct.sum()} entries updated\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e43a49cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "multilink_cmts['submission_flair'] = multilink_cmts['corrected_flair']\n",
    "multilink_cmts = multilink_cmts.drop(columns=['flair_correct', 'corrected_flair', 'src_url'])\n",
    "cmts_df = pd.concat([cmts_df,multilink_cmts[~multilink_cmts.index.isin(cmts_df.index)]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e078ff72",
   "metadata": {},
   "source": [
    "### Part 2 -- Duplications: Content updates/versioning\n",
    "\n",
    "Now that we've re-incorporated all of the content we've revieed so far, we'll further filter down the dataset.\n",
    "\n",
    "Some content gets re-posted occasionally with updates. These often aren't materially different and they can skew our perception of how the resulting model performs in the real world. i.e. if we use an earlier verion of a document for training then attempt to predict a slightly modified/updated version during testing, we'll over-estimate performance\n",
    "\n",
    "The earlier processing by date will have removed some instances of this, but only in select cases. Here I'll keep the oldest instance of each link for which the flair is the same. This should fix most cases of versioning, while also fixing a few other issues surrounding re-posted content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "579997f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "91 duplicate links with un-ambiguous submission flair dropped\n"
     ]
    }
   ],
   "source": [
    "cmt_dups = cmts_df.sort_values(by=[\"manually_reviewed\",'subreddit','submission_date','related_link'], axis=0, ascending=[False,False,True,False])\n",
    "cmt_dups = cmt_dups[cmt_dups.duplicated(subset=['link','submission_flair'], keep='first')]\n",
    "cmt_dups['discard_reason'] = 'Duplicate Unambiguous Flair'\n",
    "discarded_cmts = pd.concat([discarded_cmts, cmt_dups])\n",
    "print(f\"{len(cmt_dups)} duplicate links with un-ambiguous submission flair dropped\")\n",
    "cmts_df = cmts_df[~cmts_df.index.isin(cmt_dups.index)]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "83fcae42",
   "metadata": {},
   "source": [
    "### Part 3 -- Remaining duplicates\n",
    "\n",
    "I'll manually sort out any remaining issues in the code blocks below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "0c20e1aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2189</th>\n",
       "      <td>https://gmbinder.com/share/-Mt-8R9tQYhuLrhMR5-L</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>sp43pv</td>\n",
       "      <td>Conjure insect - A druid cantrip</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>783</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1644491044.0</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>hwcq3c5</td>\n",
       "      <td>66</td>\n",
       "      <td>I don't know about you, but I love semi-useles...</td>\n",
       "      <td>1644491171.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2899</th>\n",
       "      <td>https://gmbinder.com/share/-Mt-8R9tQYhuLrhMR5-L</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>wo8xq3</td>\n",
       "      <td>Circle of Vermin - final-ish version</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>241</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1660490698.0</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>ik9gy7p</td>\n",
       "      <td>6</td>\n",
       "      <td>Here is the final (more or less) version of my...</td>\n",
       "      <td>1660490783.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 link     submission_author  \\\n",
       "idx                                                                           \n",
       "2189  https://gmbinder.com/share/-Mt-8R9tQYhuLrhMR5-L  Sensitive_Coyote_865   \n",
       "2899  https://gmbinder.com/share/-Mt-8R9tQYhuLrhMR5-L  Sensitive_Coyote_865   \n",
       "\n",
       "     submission_id                      submission_title        subreddit  \\\n",
       "idx                                                                         \n",
       "2189        sp43pv      Conjure insect - A druid cantrip  UnearthedArcana   \n",
       "2899        wo8xq3  Circle of Vermin - final-ish version  UnearthedArcana   \n",
       "\n",
       "     submission_flair  submission_score  submission_upvote_ratio  \\\n",
       "idx                                                                \n",
       "2189            Spell               783                     0.98   \n",
       "2899         Subclass               241                     0.99   \n",
       "\n",
       "     submission_date        comment_author comment_id  comment_score  \\\n",
       "idx                                                                    \n",
       "2189    1644491044.0  Sensitive_Coyote_865    hwcq3c5             66   \n",
       "2899    1660490698.0  Sensitive_Coyote_865    ik9gy7p              6   \n",
       "\n",
       "                                           comment_body  comment_date  \\\n",
       "idx                                                                     \n",
       "2189  I don't know about you, but I love semi-useles...  1644491171.0   \n",
       "2899  Here is the final (more or less) version of my...  1660490783.0   \n",
       "\n",
       "      manually_reviewed related_link  \n",
       "idx                                   \n",
       "2189              False          NaN  \n",
       "2899              False          NaN  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "remaining_dups = cmts_df.duplicated(subset='link', keep=False)\n",
    "cmts_df[remaining_dups].sort_values(by=['link','manually_reviewed'], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f578b805",
   "metadata": {},
   "outputs": [],
   "source": [
    "to_remove = cmts_df.duplicated(subset='link', keep='last')\n",
    "# This appears to have been updated since the original posting\n",
    "manual_removal = cmts_df[to_remove].copy()\n",
    "manual_removal ['discard_reason'] = 'Manually removed mislabeled'\n",
    "cmts_df = cmts_df[~to_remove]\n",
    "discarded_cmts = pd.concat([discarded_cmts, manual_removal])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "fc6cb5cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UA crosspost                    302\n",
       "Simple duplicate                235\n",
       "Has UA primary submission       215\n",
       "Newer multi-comment             181\n",
       "Unrelated duplicate link        142\n",
       "Duplicate Unambiguous Flair     137\n",
       "Missing path                     33\n",
       "Invalid/Missing Content Type     18\n",
       "Manually removed mislabeled       1\n",
       "Name: discard_reason, dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discarded_cmts['discard_reason'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "2ab5eb0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "discarded_cmts.to_csv(PRAW/'FinalCommentsDropped.csv', index_label='idx')\n",
    "cmts_df.to_csv(PRAW/'FinalCommentsFiltered.csv', index_label='idx')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "c739df7216996af44517d475e62739a71a77ba3d210c35292fab841551565cb7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}