DiversiPy Documentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Documentation of DiversiPy\n",
    "- This notebook explains the logic of DiversiPy and documents it.\n",
    "- DiversiPy was developed by Jin Woo Won for COMS W3101 Programming Languages (Python) class project on June 11, 2018.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nApplication that finds redundant vocabulary and suggests some synonyms\\n'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Application that finds redundant vocabulary and suggests some synonyms\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please copy and paste your input text here: These excellant intentions were strengthed when he enterd the Father Superior's diniing-room, though, stricttly speakin, it was not a dining-room, for the Father Superior had only two rooms alltogether; they were, however, much larger and more comfortable than Father Zossima's. But tehre was was no great luxury about the furnishng of these rooms eithar. The furniture was of mohogany, covered with leather, in the old-fashionned style of 1820 the floor was not even stained, but evreything was shining with cleanlyness, and there were many chioce flowers in the windows; the most sumptuous thing in the room at the moment was, of course, the beatifuly decorated table. The cloth was clean, the service shone; there were three kinds of well-baked bread, two bottles of wine, two of excellent mead, and a large glass jug of kvas -- both the latter made in the monastery, and famous in the neigborhood. There was no vodka. Rakitin related afterwards that there were five dishes: fish-suop made of sterlets, served with little fish paties; then boiled fish served in a spesial way; then salmon cutlets, ice pudding and compote, and finally, blanc-mange. Rakitin found out about all these good things, for he could not resist peeping into the kitchen, where he already had a footing. He had a footting everywhere, and got informaiton about everything. He was of an uneasy and envious temper. He was well aware of his own considerable abilities, and nervously exaggerated them in his self-conceit. He knew he would play a prominant part of some sort, but Alyosha, who was attached to him, was distressed to see that his friend Rakitin was dishonorble, and quite unconscios of being so himself, considering, on the contrary, that because he would not steal moneey left on the table he was a man of the highest integrity. Neither Alyosha nor anyone else could have infleunced him in that.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"These excellant intentions were strengthed when he enterd the Father Superior's diniing-room, though, stricttly speakin, it was not a dining-room, for the Father Superior had only two rooms alltogether; they were, however, much larger and more comfortable than Father Zossima's. But tehre was was no great luxury about the furnishng of these rooms eithar. The furniture was of mohogany, covered with leather, in the old-fashionned style of 1820 the floor was not even stained, but evreything was shining with cleanlyness, and there were many chioce flowers in the windows; the most sumptuous thing in the room at the moment was, of course, the beatifuly decorated table. The cloth was clean, the service shone; there were three kinds of well-baked bread, two bottles of wine, two of excellent mead, and a large glass jug of kvas -- both the latter made in the monastery, and famous in the neigborhood. There was no vodka. Rakitin related afterwards that there were five dishes: fish-suop made of sterlets, served with little fish paties; then boiled fish served in a spesial way; then salmon cutlets, ice pudding and compote, and finally, blanc-mange. Rakitin found out about all these good things, for he could not resist peeping into the kitchen, where he already had a footing. He had a footting everywhere, and got informaiton about everything. He was of an uneasy and envious temper. He was well aware of his own considerable abilities, and nervously exaggerated them in his self-conceit. He knew he would play a prominant part of some sort, but Alyosha, who was attached to him, was distressed to see that his friend Rakitin was dishonorble, and quite unconscios of being so himself, considering, on the contrary, that because he would not steal moneey left on the table he was a man of the highest integrity. Neither Alyosha nor anyone else could have infleunced him in that.\""
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Take the text input\n",
    "while True:\n",
    "    text = input(\"Please copy and paste your input text here: \")\n",
    "    try:\n",
    "        text = str(text)\n",
    "        break\n",
    "    except Exception:\n",
    "        print(\"Please input a string\")\n",
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Process the text input\n",
    "- Using NLTK, tokenizes and tags the input text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('These', 'DT'),\n",
       " ('excellant', 'JJ'),\n",
       " ('intentions', 'NNS'),\n",
       " ('were', 'VBD'),\n",
       " ('strengthed', 'VBN'),\n",
       " ('when', 'WRB'),\n",
       " ('he', 'PRP'),\n",
       " ('enterd', 'VBZ'),\n",
       " ('the', 'DT'),\n",
       " ('Father', 'NNP'),\n",
       " ('Superior', 'NNP'),\n",
       " (\"'s\", 'POS'),\n",
       " ('diniing-room', 'NN'),\n",
       " (',', ','),\n",
       " ('though', 'IN'),\n",
       " (',', ','),\n",
       " ('stricttly', 'RB'),\n",
       " ('speakin', 'FW'),\n",
       " (',', ','),\n",
       " ('it', 'PRP'),\n",
       " ('was', 'VBD'),\n",
       " ('not', 'RB'),\n",
       " ('a', 'DT'),\n",
       " ('dining-room', 'NN'),\n",
       " (',', ','),\n",
       " ('for', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('Father', 'NNP'),\n",
       " ('Superior', 'NNP'),\n",
       " ('had', 'VBD'),\n",
       " ('only', 'RB'),\n",
       " ('two', 'CD'),\n",
       " ('rooms', 'NNS'),\n",
       " ('alltogether', 'RB'),\n",
       " (';', ':'),\n",
       " ('they', 'PRP'),\n",
       " ('were', 'VBD'),\n",
       " (',', ','),\n",
       " ('however', 'RB'),\n",
       " (',', ','),\n",
       " ('much', 'JJ'),\n",
       " ('larger', 'JJR'),\n",
       " ('and', 'CC'),\n",
       " ('more', 'RBR'),\n",
       " ('comfortable', 'JJ'),\n",
       " ('than', 'IN'),\n",
       " ('Father', 'NNP'),\n",
       " ('Zossima', 'NNP'),\n",
       " (\"'s\", 'POS'),\n",
       " ('.', '.'),\n",
       " ('But', 'CC'),\n",
       " ('tehre', 'NN'),\n",
       " ('was', 'VBD'),\n",
       " ('was', 'VBD'),\n",
       " ('no', 'DT'),\n",
       " ('great', 'JJ'),\n",
       " ('luxury', 'NN'),\n",
       " ('about', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('furnishng', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('these', 'DT'),\n",
       " ('rooms', 'NNS'),\n",
       " ('eithar', 'VBP'),\n",
       " ('.', '.'),\n",
       " ('The', 'DT'),\n",
       " ('furniture', 'NN'),\n",
       " ('was', 'VBD'),\n",
       " ('of', 'IN'),\n",
       " ('mohogany', 'NN'),\n",
       " (',', ','),\n",
       " ('covered', 'VBN'),\n",
       " ('with', 'IN'),\n",
       " ('leather', 'NN'),\n",
       " (',', ','),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('old-fashionned', 'JJ'),\n",
       " ('style', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('1820', 'CD'),\n",
       " ('the', 'DT'),\n",
       " ('floor', 'NN'),\n",
       " ('was', 'VBD'),\n",
       " ('not', 'RB'),\n",
       " ('even', 'RB'),\n",
       " ('stained', 'VBN'),\n",
       " (',', ','),\n",
       " ('but', 'CC'),\n",
       " ('evreything', 'VBG'),\n",
       " ('was', 'VBD'),\n",
       " ('shining', 'VBG'),\n",
       " ('with', 'IN'),\n",
       " ('cleanlyness', 'NN'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('there', 'EX'),\n",
       " ('were', 'VBD'),\n",
       " ('many', 'JJ'),\n",
       " ('chioce', 'JJ'),\n",
       " ('flowers', 'NNS'),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('windows', 'NNS'),\n",
       " (';', ':'),\n",
       " ('the', 'DT'),\n",
       " ('most', 'RBS'),\n",
       " ('sumptuous', 'JJ'),\n",
       " ('thing', 'NN'),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('room', 'NN'),\n",
       " ('at', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('moment', 'NN'),\n",
       " ('was', 'VBD'),\n",
       " (',', ','),\n",
       " ('of', 'IN'),\n",
       " ('course', 'NN'),\n",
       " (',', ','),\n",
       " ('the', 'DT'),\n",
       " ('beatifuly', 'NN'),\n",
       " ('decorated', 'VBD'),\n",
       " ('table', 'JJ'),\n",
       " ('.', '.'),\n",
       " ('The', 'DT'),\n",
       " ('cloth', 'NN'),\n",
       " ('was', 'VBD'),\n",
       " ('clean', 'JJ'),\n",
       " (',', ','),\n",
       " ('the', 'DT'),\n",
       " ('service', 'NN'),\n",
       " ('shone', 'NN'),\n",
       " (';', ':'),\n",
       " ('there', 'EX'),\n",
       " ('were', 'VBD'),\n",
       " ('three', 'CD'),\n",
       " ('kinds', 'NNS'),\n",
       " ('of', 'IN'),\n",
       " ('well-baked', 'JJ'),\n",
       " ('bread', 'NN'),\n",
       " (',', ','),\n",
       " ('two', 'CD'),\n",
       " ('bottles', 'NNS'),\n",
       " ('of', 'IN'),\n",
       " ('wine', 'NN'),\n",
       " (',', ','),\n",
       " ('two', 'CD'),\n",
       " ('of', 'IN'),\n",
       " ('excellent', 'JJ'),\n",
       " ('mead', 'NN'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('a', 'DT'),\n",
       " ('large', 'JJ'),\n",
       " ('glass', 'NN'),\n",
       " ('jug', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('kvas', 'NN'),\n",
       " ('--', ':'),\n",
       " ('both', 'CC'),\n",
       " ('the', 'DT'),\n",
       " ('latter', 'NN'),\n",
       " ('made', 'VBN'),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('monastery', 'NN'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('famous', 'JJ'),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('neigborhood', 'NN'),\n",
       " ('.', '.'),\n",
       " ('There', 'EX'),\n",
       " ('was', 'VBD'),\n",
       " ('no', 'DT'),\n",
       " ('vodka', 'NN'),\n",
       " ('.', '.'),\n",
       " ('Rakitin', 'NNP'),\n",
       " ('related', 'JJ'),\n",
       " ('afterwards', 'NNS'),\n",
       " ('that', 'IN'),\n",
       " ('there', 'EX'),\n",
       " ('were', 'VBD'),\n",
       " ('five', 'CD'),\n",
       " ('dishes', 'NNS'),\n",
       " (':', ':'),\n",
       " ('fish-suop', 'JJ'),\n",
       " ('made', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('sterlets', 'NNS'),\n",
       " (',', ','),\n",
       " ('served', 'VBD'),\n",
       " ('with', 'IN'),\n",
       " ('little', 'JJ'),\n",
       " ('fish', 'JJ'),\n",
       " ('paties', 'NNS'),\n",
       " (';', ':'),\n",
       " ('then', 'RB'),\n",
       " ('boiled', 'VBD'),\n",
       " ('fish', 'JJ'),\n",
       " ('served', 'VBN'),\n",
       " ('in', 'IN'),\n",
       " ('a', 'DT'),\n",
       " ('spesial', 'JJ'),\n",
       " ('way', 'NN'),\n",
       " (';', ':'),\n",
       " ('then', 'RB'),\n",
       " ('salmon', 'VBZ'),\n",
       " ('cutlets', 'NNS'),\n",
       " (',', ','),\n",
       " ('ice', 'NN'),\n",
       " ('pudding', 'NN'),\n",
       " ('and', 'CC'),\n",
       " ('compote', 'NN'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('finally', 'RB'),\n",
       " (',', ','),\n",
       " ('blanc-mange', 'NN'),\n",
       " ('.', '.'),\n",
       " ('Rakitin', 'NNP'),\n",
       " ('found', 'VBD'),\n",
       " ('out', 'RP'),\n",
       " ('about', 'IN'),\n",
       " ('all', 'PDT'),\n",
       " ('these', 'DT'),\n",
       " ('good', 'JJ'),\n",
       " ('things', 'NNS'),\n",
       " (',', ','),\n",
       " ('for', 'IN'),\n",
       " ('he', 'PRP'),\n",
       " ('could', 'MD'),\n",
       " ('not', 'RB'),\n",
       " ('resist', 'VB'),\n",
       " ('peeping', 'VBG'),\n",
       " ('into', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('kitchen', 'NN'),\n",
       " (',', ','),\n",
       " ('where', 'WRB'),\n",
       " ('he', 'PRP'),\n",
       " ('already', 'RB'),\n",
       " ('had', 'VBD'),\n",
       " ('a', 'DT'),\n",
       " ('footing', 'NN'),\n",
       " ('.', '.'),\n",
       " ('He', 'PRP'),\n",
       " ('had', 'VBD'),\n",
       " ('a', 'DT'),\n",
       " ('footting', 'NN'),\n",
       " ('everywhere', 'RB'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('got', 'VBD'),\n",
       " ('informaiton', 'NNS'),\n",
       " ('about', 'IN'),\n",
       " ('everything', 'NN'),\n",
       " ('.', '.'),\n",
       " ('He', 'PRP'),\n",
       " ('was', 'VBD'),\n",
       " ('of', 'IN'),\n",
       " ('an', 'DT'),\n",
       " ('uneasy', 'JJ'),\n",
       " ('and', 'CC'),\n",
       " ('envious', 'JJ'),\n",
       " ('temper', 'NN'),\n",
       " ('.', '.'),\n",
       " ('He', 'PRP'),\n",
       " ('was', 'VBD'),\n",
       " ('well', 'RB'),\n",
       " ('aware', 'JJ'),\n",
       " ('of', 'IN'),\n",
       " ('his', 'PRP$'),\n",
       " ('own', 'JJ'),\n",
       " ('considerable', 'JJ'),\n",
       " ('abilities', 'NNS'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('nervously', 'RB'),\n",
       " ('exaggerated', 'VBD'),\n",
       " ('them', 'PRP'),\n",
       " ('in', 'IN'),\n",
       " ('his', 'PRP$'),\n",
       " ('self-conceit', 'NN'),\n",
       " ('.', '.'),\n",
       " ('He', 'PRP'),\n",
       " ('knew', 'VBD'),\n",
       " ('he', 'PRP'),\n",
       " ('would', 'MD'),\n",
       " ('play', 'VB'),\n",
       " ('a', 'DT'),\n",
       " ('prominant', 'JJ'),\n",
       " ('part', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('some', 'DT'),\n",
       " ('sort', 'NN'),\n",
       " (',', ','),\n",
       " ('but', 'CC'),\n",
       " ('Alyosha', 'NNP'),\n",
       " (',', ','),\n",
       " ('who', 'WP'),\n",
       " ('was', 'VBD'),\n",
       " ('attached', 'VBN'),\n",
       " ('to', 'TO'),\n",
       " ('him', 'PRP'),\n",
       " (',', ','),\n",
       " ('was', 'VBD'),\n",
       " ('distressed', 'VBN'),\n",
       " ('to', 'TO'),\n",
       " ('see', 'VB'),\n",
       " ('that', 'IN'),\n",
       " ('his', 'PRP$'),\n",
       " ('friend', 'NN'),\n",
       " ('Rakitin', 'NNP'),\n",
       " ('was', 'VBD'),\n",
       " ('dishonorble', 'JJ'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('quite', 'RB'),\n",
       " ('unconscios', 'NNS'),\n",
       " ('of', 'IN'),\n",
       " ('being', 'VBG'),\n",
       " ('so', 'RB'),\n",
       " ('himself', 'PRP'),\n",
       " (',', ','),\n",
       " ('considering', 'VBG'),\n",
       " (',', ','),\n",
       " ('on', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('contrary', 'JJ'),\n",
       " (',', ','),\n",
       " ('that', 'IN'),\n",
       " ('because', 'IN'),\n",
       " ('he', 'PRP'),\n",
       " ('would', 'MD'),\n",
       " ('not', 'RB'),\n",
       " ('steal', 'VB'),\n",
       " ('moneey', 'NN'),\n",
       " ('left', 'VBN'),\n",
       " ('on', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('table', 'NN'),\n",
       " ('he', 'PRP'),\n",
       " ('was', 'VBD'),\n",
       " ('a', 'DT'),\n",
       " ('man', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('highest', 'JJS'),\n",
       " ('integrity', 'NN'),\n",
       " ('.', '.'),\n",
       " ('Neither', 'CC'),\n",
       " ('Alyosha', 'NNP'),\n",
       " ('nor', 'CC'),\n",
       " ('anyone', 'NN'),\n",
       " ('else', 'RB'),\n",
       " ('could', 'MD'),\n",
       " ('have', 'VB'),\n",
       " ('infleunced', 'VBN'),\n",
       " ('him', 'PRP'),\n",
       " ('in', 'IN'),\n",
       " ('that', 'DT'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Process the text input\n",
    "import nltk\n",
    "\n",
    "# Take only nouns, verbs, adjectives, and adverbs (Using speech tagging)\n",
    "tokenized_text = nltk.word_tokenize(text)\n",
    "tagged_text = nltk.pos_tag(tokenized_text)\n",
    "\n",
    "tagged_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Process Words\n",
    "- Using NLTK's lemmatizing feature, turn the words into their basic forms (e.g. \"is\" -> \"be\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Process words\n",
    "\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "def lemmatize_list(input_list, flag='n'):\n",
    "    '''\n",
    "    Produces a set consisting of lemmatized words from a list of possibly\n",
    "    conjugated/inflected words\n",
    "    \n",
    "    flag: should be 'v' for verb, 'n' for noun, and 'a' for others\n",
    "    '''\n",
    "    result = set()\n",
    "    \n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "    for word in input_list:\n",
    "        result.add(lemmatizer.lemmatize(word, flag))\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filter using tags\n",
    "- Take out only nouns, verbs, adjectives, and adverbs using tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'adjective': ['excellant',\n",
       "  'much',\n",
       "  'large',\n",
       "  'comfortable',\n",
       "  'great',\n",
       "  'old-fashionned',\n",
       "  'many',\n",
       "  'chioce',\n",
       "  'sumptuous',\n",
       "  'table',\n",
       "  'clean',\n",
       "  'well-baked',\n",
       "  'excellent',\n",
       "  'large',\n",
       "  'famous',\n",
       "  'related',\n",
       "  'fish-suop',\n",
       "  'little',\n",
       "  'fish',\n",
       "  'fish',\n",
       "  'spesial',\n",
       "  'good',\n",
       "  'uneasy',\n",
       "  'envious',\n",
       "  'aware',\n",
       "  'own',\n",
       "  'considerable',\n",
       "  'prominant',\n",
       "  'dishonorble',\n",
       "  'contrary',\n",
       "  'high'],\n",
       " 'adverb': ['stricttly',\n",
       "  'not',\n",
       "  'only',\n",
       "  'alltogether',\n",
       "  'however',\n",
       "  'more',\n",
       "  'not',\n",
       "  'even',\n",
       "  'most',\n",
       "  'then',\n",
       "  'then',\n",
       "  'finally',\n",
       "  'not',\n",
       "  'already',\n",
       "  'everywhere',\n",
       "  'well',\n",
       "  'nervously',\n",
       "  'quite',\n",
       "  'so',\n",
       "  'not',\n",
       "  'else'],\n",
       " 'noun': ['intention',\n",
       "  'diniing-room',\n",
       "  'dining-room',\n",
       "  'room',\n",
       "  'tehre',\n",
       "  'luxury',\n",
       "  'furnishng',\n",
       "  'room',\n",
       "  'furniture',\n",
       "  'mohogany',\n",
       "  'leather',\n",
       "  'style',\n",
       "  'floor',\n",
       "  'cleanlyness',\n",
       "  'flower',\n",
       "  'window',\n",
       "  'thing',\n",
       "  'room',\n",
       "  'moment',\n",
       "  'course',\n",
       "  'beatifuly',\n",
       "  'cloth',\n",
       "  'service',\n",
       "  'shone',\n",
       "  'kind',\n",
       "  'bread',\n",
       "  'bottle',\n",
       "  'wine',\n",
       "  'mead',\n",
       "  'glass',\n",
       "  'jug',\n",
       "  'kvas',\n",
       "  'latter',\n",
       "  'monastery',\n",
       "  'neigborhood',\n",
       "  'vodka',\n",
       "  'afterwards',\n",
       "  'dish',\n",
       "  'made',\n",
       "  'sterlets',\n",
       "  'paties',\n",
       "  'way',\n",
       "  'cutlet',\n",
       "  'ice',\n",
       "  'pudding',\n",
       "  'compote',\n",
       "  'blanc-mange',\n",
       "  'thing',\n",
       "  'kitchen',\n",
       "  'footing',\n",
       "  'footting',\n",
       "  'informaiton',\n",
       "  'everything',\n",
       "  'temper',\n",
       "  'ability',\n",
       "  'self-conceit',\n",
       "  'part',\n",
       "  'sort',\n",
       "  'friend',\n",
       "  'unconscios',\n",
       "  'moneey',\n",
       "  'table',\n",
       "  'man',\n",
       "  'integrity',\n",
       "  'anyone'],\n",
       " 'verb': ['be',\n",
       "  'strengthed',\n",
       "  'enterd',\n",
       "  'be',\n",
       "  'have',\n",
       "  'be',\n",
       "  'be',\n",
       "  'be',\n",
       "  'eithar',\n",
       "  'be',\n",
       "  'cover',\n",
       "  'be',\n",
       "  'stain',\n",
       "  'evreything',\n",
       "  'be',\n",
       "  'shin',\n",
       "  'be',\n",
       "  'be',\n",
       "  'decorate',\n",
       "  'be',\n",
       "  'be',\n",
       "  'make',\n",
       "  'be',\n",
       "  'be',\n",
       "  'serve',\n",
       "  'boil',\n",
       "  'serve',\n",
       "  'salmon',\n",
       "  'find',\n",
       "  'resist',\n",
       "  'peep',\n",
       "  'have',\n",
       "  'have',\n",
       "  'get',\n",
       "  'be',\n",
       "  'be',\n",
       "  'exaggerate',\n",
       "  'know',\n",
       "  'play',\n",
       "  'be',\n",
       "  'attach',\n",
       "  'be',\n",
       "  'distress',\n",
       "  'see',\n",
       "  'be',\n",
       "  'be',\n",
       "  'consider',\n",
       "  'steal',\n",
       "  'leave',\n",
       "  'be',\n",
       "  'have',\n",
       "  'infleunced']}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter via tags\n",
    "\n",
    "def nvaa(tagged_text):\n",
    "    '''\n",
    "    Takes in tagged text, returns a dictionary of with four\n",
    "    keys noun, verb, adjective, and adverb, and discards\n",
    "    other parts of speech.\n",
    "    \n",
    "    Also lemmatizes each words in the lists.\n",
    "    '''\n",
    "    result = dict()\n",
    "    result['noun'] = []\n",
    "    result['verb'] = []\n",
    "    result['adjective'] = []\n",
    "    result['adverb'] = []\n",
    "    \n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    \n",
    "    for unit in tagged_text:\n",
    "        noun = ['NN', 'NNS']\n",
    "        verb = ['VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ']\n",
    "        adj = ['JJ', 'JJR', 'JJS']\n",
    "        adv = ['RB', 'RBR', 'RBS']\n",
    "        tag = unit[1]\n",
    "        if tag in noun:\n",
    "            n = lemmatizer.lemmatize(unit[0], 'n')\n",
    "            result['noun'].append(n)\n",
    "        if tag in verb:\n",
    "            v = lemmatizer.lemmatize(unit[0], 'v')\n",
    "            result['verb'].append(v)\n",
    "        if tag in adj:\n",
    "            a = lemmatizer.lemmatize(unit[0], 'a')\n",
    "            result['adjective'].append(a)\n",
    "        if tag in adv:\n",
    "            a = lemmatizer.lemmatize(unit[0], 'a')\n",
    "            result['adverb'].append(unit[0])\n",
    "\n",
    "    return result\n",
    "\n",
    "four_pos = nvaa(tagged_text)\n",
    "four_pos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find redundant Vocabulary\n",
    "- Find frequencies of each word and sorts it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'ability': 1,\n",
       "         'afterwards': 1,\n",
       "         'anyone': 1,\n",
       "         'beatifuly': 1,\n",
       "         'blanc-mange': 1,\n",
       "         'bottle': 1,\n",
       "         'bread': 1,\n",
       "         'cleanlyness': 1,\n",
       "         'cloth': 1,\n",
       "         'compote': 1,\n",
       "         'course': 1,\n",
       "         'cutlet': 1,\n",
       "         'diniing-room': 1,\n",
       "         'dining-room': 1,\n",
       "         'dish': 1,\n",
       "         'everything': 1,\n",
       "         'floor': 1,\n",
       "         'flower': 1,\n",
       "         'footing': 1,\n",
       "         'footting': 1,\n",
       "         'friend': 1,\n",
       "         'furnishng': 1,\n",
       "         'furniture': 1,\n",
       "         'glass': 1,\n",
       "         'ice': 1,\n",
       "         'informaiton': 1,\n",
       "         'integrity': 1,\n",
       "         'intention': 1,\n",
       "         'jug': 1,\n",
       "         'kind': 1,\n",
       "         'kitchen': 1,\n",
       "         'kvas': 1,\n",
       "         'latter': 1,\n",
       "         'leather': 1,\n",
       "         'luxury': 1,\n",
       "         'made': 1,\n",
       "         'man': 1,\n",
       "         'mead': 1,\n",
       "         'mohogany': 1,\n",
       "         'moment': 1,\n",
       "         'monastery': 1,\n",
       "         'moneey': 1,\n",
       "         'neigborhood': 1,\n",
       "         'part': 1,\n",
       "         'paties': 1,\n",
       "         'pudding': 1,\n",
       "         'room': 3,\n",
       "         'self-conceit': 1,\n",
       "         'service': 1,\n",
       "         'shone': 1,\n",
       "         'sort': 1,\n",
       "         'sterlets': 1,\n",
       "         'style': 1,\n",
       "         'table': 1,\n",
       "         'tehre': 1,\n",
       "         'temper': 1,\n",
       "         'thing': 2,\n",
       "         'unconscios': 1,\n",
       "         'vodka': 1,\n",
       "         'way': 1,\n",
       "         'window': 1,\n",
       "         'wine': 1})"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find Redundant vocabulary\n",
    "from collections import Counter\n",
    "\n",
    "def count_words(input_list):\n",
    "    return Counter(input_list)\n",
    "\n",
    "noun_count = count_words(four_pos['noun'])\n",
    "noun_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('room', 3),\n",
       "             ('thing', 2),\n",
       "             ('intention', 1),\n",
       "             ('diniing-room', 1),\n",
       "             ('dining-room', 1),\n",
       "             ('tehre', 1),\n",
       "             ('luxury', 1),\n",
       "             ('furnishng', 1),\n",
       "             ('furniture', 1),\n",
       "             ('mohogany', 1),\n",
       "             ('leather', 1),\n",
       "             ('style', 1),\n",
       "             ('floor', 1),\n",
       "             ('cleanlyness', 1),\n",
       "             ('flower', 1),\n",
       "             ('window', 1),\n",
       "             ('moment', 1),\n",
       "             ('course', 1),\n",
       "             ('beatifuly', 1),\n",
       "             ('cloth', 1),\n",
       "             ('service', 1),\n",
       "             ('shone', 1),\n",
       "             ('kind', 1),\n",
       "             ('bread', 1),\n",
       "             ('bottle', 1),\n",
       "             ('wine', 1),\n",
       "             ('mead', 1),\n",
       "             ('glass', 1),\n",
       "             ('jug', 1),\n",
       "             ('kvas', 1),\n",
       "             ('latter', 1),\n",
       "             ('monastery', 1),\n",
       "             ('neigborhood', 1),\n",
       "             ('vodka', 1),\n",
       "             ('afterwards', 1),\n",
       "             ('dish', 1),\n",
       "             ('made', 1),\n",
       "             ('sterlets', 1),\n",
       "             ('paties', 1),\n",
       "             ('way', 1),\n",
       "             ('cutlet', 1),\n",
       "             ('ice', 1),\n",
       "             ('pudding', 1),\n",
       "             ('compote', 1),\n",
       "             ('blanc-mange', 1),\n",
       "             ('kitchen', 1),\n",
       "             ('footing', 1),\n",
       "             ('footting', 1),\n",
       "             ('informaiton', 1),\n",
       "             ('everything', 1),\n",
       "             ('temper', 1),\n",
       "             ('ability', 1),\n",
       "             ('self-conceit', 1),\n",
       "             ('part', 1),\n",
       "             ('sort', 1),\n",
       "             ('friend', 1),\n",
       "             ('unconscios', 1),\n",
       "             ('moneey', 1),\n",
       "             ('table', 1),\n",
       "             ('man', 1),\n",
       "             ('integrity', 1),\n",
       "             ('anyone', 1)])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create OrderedDict sorted by number of occurrences\n",
    "from collections import OrderedDict\n",
    "\n",
    "def sorted_counter(input_counter):\n",
    "    return OrderedDict(sorted(input_counter.items(), reverse=True, key=lambda x: x[1]))\n",
    "    \n",
    "sorted_noun_count = sorted_counter(noun_count)\n",
    "sorted_noun_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take the most frequently used words of each category\n",
    "\n",
    "sorted_freq_nouns = sorted_counter(count_words(four_pos['noun']))\n",
    "sorted_freq_verbs = sorted_counter(count_words(four_pos['verb']))\n",
    "sorted_freq_adjs = sorted_counter(count_words(four_pos['adjective']))\n",
    "sorted_freq_advs = sorted_counter(count_words(four_pos['adverb']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Suggest synonyms\n",
    "- Using NLTK's corpus, suggest synonyms within the context of its part of speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('freedom', {'exemption', 'freedom'})"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Suggest some synonyms\n",
    "from nltk.corpus import wordnet\n",
    "\n",
    "def suggest_syn(word):\n",
    "    synonyms = set()\n",
    "    for syn in wordnet.synsets(word):\n",
    "        for l in syn.lemmas():\n",
    "            synonyms.add(l.name())\n",
    "    return word, synonyms\n",
    "\n",
    "suggest_syn('freedom')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('room', {'elbow_room', 'room', 'way', 'board'})\n",
      "('thing', {'thing', 'affair', 'matter'})\n",
      "('intention', {'purpose', 'intent', 'aim', 'intention', 'design'})\n",
      "('diniing-room', set())\n",
      "('dining-room', {'dining_room', 'dining-room'})\n",
      "('tehre', set())\n",
      "('luxury', {'lavishness', 'sumptuousness', 'luxury', 'opulence', 'luxuriousness', 'sumptuosity'})\n",
      "('furnishng', set())\n",
      "('furniture', {'article_of_furniture', 'furniture', 'piece_of_furniture'})\n",
      "('mohogany', set())\n",
      "('leather', {'leather'})\n",
      "('style', {'title', 'stylus', 'manner', 'panache', 'style', 'vogue', 'trend', 'elan', 'flair', 'fashion', 'mode', 'expressive_style', 'way', 'dash'})\n"
     ]
    }
   ],
   "source": [
    "for i, (word, n) in enumerate(sorted_freq_nouns.items()):\n",
    "    print(suggest_syn(word))\n",
    "    if i > 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('room', {'board', 'elbow_room', 'room', 'way'}),\n",
       " ('thing', {'affair', 'matter', 'thing'}),\n",
       " ('intention', {'aim', 'design', 'intent', 'intention', 'purpose'}),\n",
       " ('diniing-room', set()),\n",
       " ('dining-room', {'dining-room', 'dining_room'}),\n",
       " ('tehre', set()),\n",
       " ('luxury',\n",
       "  {'lavishness',\n",
       "   'luxuriousness',\n",
       "   'luxury',\n",
       "   'opulence',\n",
       "   'sumptuosity',\n",
       "   'sumptuousness'}),\n",
       " ('furnishng', set()),\n",
       " ('furniture', {'article_of_furniture', 'furniture', 'piece_of_furniture'}),\n",
       " ('mohogany', set()),\n",
       " ('leather', {'leather'})]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def suggest_syn_list(sorted_freq_words, num=10):\n",
    "    result = []\n",
    "    for i, (word, n) in enumerate(sorted_freq_words.items()):\n",
    "        result.append(suggest_syn(word))\n",
    "        if i > num:\n",
    "            break\n",
    "    return result\n",
    "suggest_syn_list(sorted_freq_nouns, 9)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}