notebooks/Python Classification and Feature Extraction for Timbl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Classification Feature Extraction for Timbl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2017 by [Damir Cavar](http://cavar.me/damir/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**License:** [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) ([CA BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tutorial related to the discussion of feature extraction for classification and clustering in the textbook [Machine Learning: The Art and Science of Algorithms that Make Sense of Data](https://www.cs.bris.ac.uk/~flach/mlbook/) by [Peter Flach](https://www.cs.bris.ac.uk/~flach/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial was developed as part of my course material for the course Machine Learning for Computational Linguistics in the [Computational Linguistics Program](http://cl.indiana.edu/) of the [Department of Linguistics](http://www.indiana.edu/~lingdept/) at [Indiana University](https://www.indiana.edu/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text1 = \"\"\"The city will pay for it by taxes on properties selling for more\n",
    "than $5 million.\n",
    "\n",
    "The real estate transfer tax, as it's called, was increased last year for both\n",
    "residential and commercial properties. The hike was approved by voters in\n",
    "November.\n",
    "Powered by SmartAsset.com\n",
    "SmartAsset.com\n",
    "\n",
    "The tax starts at 2.25% and goes up to 3% for properties worth at least $25\n",
    "million. It's expected to bring in an average of $45 million a year, according\n",
    "to the city controller. But the money goes into the city's general fund and is\n",
    "also expected to be used for affordable housing and senior support services.\n",
    "\n",
    "The free tuition plan is expected to impact about 28,000 residents who currently\n",
    "take classes at City College of San Francisco and encourage more people to sign\n",
    "up. Chancellor Susan Lamb said the school has the capacity for 85,000 students.\n",
    "\n",
    "It's difficult to predict how many more people will enroll, and how much the\n",
    "free-tuition plan will end up costing. San Francisco has committed $5.4 million\n",
    "a year for the next two years, and then will have to reassess. That includes a\n",
    "one-time $500,000 stipend to City College to help handle an influx of students.\n",
    "\n",
    "Related: Why New York's 'tuition-free colleges' still cost $14K\n",
    "\n",
    "San Francisco's tuition-free plan is more progressive than others round the\n",
    "country. First, everyone is eligible as long as they have resided in San Francisco\n",
    "for at least one year.\n",
    "\n",
    "It covers the $46 cost per credit no matter how rich you are, \"even to the\n",
    "children of the founders of Facebook,\" said city lawmaker Jane Kim.\n",
    "\n",
    "You don't have to be enrolled full-time or be a recent high school graduate.\n",
    "This means that people who are seeking job retraining or want to take a few\n",
    "foreign language courses won't have to pay for the cost of the credits.\n",
    "\n",
    "Related: Rhode Island governor wants to make college free, too\n",
    "\n",
    "Students will still be on the hook for the mandatory $17 per semester fee at\n",
    "City College and the cost of books, so college won't necessarily be free.\n",
    "\n",
    "What also sets apart San Francisco's plan is that it offers the poorest students\n",
    "additional money to help pay for these other expenses. An individual has to earn\n",
    "less than $17,000 a year to qualify for the aid, or less than $37,000 for a\n",
    "family of four. Eligible full-time students will get $500 a year and part-time\n",
    "students will get $200 a year.\n",
    "\n",
    "\"We have the fastest growing income gap than any city across the nation,\" Kim said\n",
    "on Monday at a press conference.\n",
    "\n",
    "\"Making city college free is going to provide greater opportunities for more San\n",
    "Franciscans to enter into the middle class and more San Franciscans to stay in the\n",
    "middle class if they currently are,\" she said.\n",
    "\n",
    "The push for free tuition is gaining support across the country. Tennessee started\n",
    "offering free community college to residents in 2015, and will expand the program\n",
    "this year to include adults returning to school. Lawmakers in New York are\n",
    "discussing a program that would make four-year and two-year public colleges\n",
    "tuition-free for residents who earn less than $125,000 a year. And Rhode Island's\n",
    "governor is pushing for two free years at public colleges for recent high school\n",
    "graduates.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokens1 = word_tokenize(text1.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'.': 26, 'the': 24, 'to': 24, 'for': 18, ',': 14, '$': 13, 'and': 12, 'a': 12, 'year': 9, 'of': 8, 'is': 8, \"'s\": 8, 'will': 8, 'San': 7, 'free': 7, 'at': 7, 'The': 6, 'than': 6, 'in': 6, 'city': 6, 'more': 6, 'have': 5, 'be': 5, 'students': 5, 'Francisco': 5, 'cost': 4, 'said': 4, 'plan': 4, 'school': 4, 'college': 4, 'million': 4, 'are': 4, 'It': 3, 'on': 3, 'or': 3, 'up': 3, \"n't\": 3, 'colleges': 3, 'residents': 3, 'less': 3, 'people': 3, 'by': 3, 'pay': 3, 'that': 3, 'how': 3, 'City': 3, 'expected': 3, 'has': 3, 'properties': 3, 'who': 3, 'it': 3, 'College': 3, '``': 3, \"''\": 3, 'as': 3, 'across': 2, 'middle': 2, 'earn': 2, 'goes': 2, 'they': 2, 'Related': 2, 'tuition-free': 2, 'full-time': 2, '%': 2, 'support': 2, 'an': 2, 'years': 2, 'take': 2, 'York': 2, 'was': 2, 'public': 2, ':': 2, 'class': 2, 'Rhode': 2, 'Kim': 2, 'money': 2, 'two': 2, 'least': 2, 'also': 2, 'New': 2, 'program': 2, 'currently': 2, 'per': 2, 'governor': 2, 'Island': 2, 'help': 2, 'into': 2, 'tax': 2, 'get': 2, 'country': 2, 'SmartAsset.com': 2, 'tuition': 2, 'wo': 2, 'Franciscans': 2, 'make': 2, 'high': 2, 'recent': 2, 'still': 2, 'progressive': 1, 'wants': 1, 'last': 1, 'fund': 1, 'We': 1, 'she': 1, '5': 1, 'called': 1, 'affordable': 1, 'Powered': 1, 'transfer': 1, 'foreign': 1, 'enter': 1, '125,000': 1, 'community': 1, 'pushing': 1, 'approved': 1, 'gap': 1, 'going': 1, 'even': 1, 'income': 1, '17': 1, 'nation': 1, 'include': 1, 'gaining': 1, 'apart': 1, 'part-time': 1, 'capacity': 1, 'matter': 1, 'used': 1, '28,000': 1, 'individual': 1, 'sign': 1, '500,000': 1, 'fastest': 1, '46': 1, 'resided': 1, 'classes': 1, 'commercial': 1, 'First': 1, 'Chancellor': 1, 'worth': 1, 'starts': 1, 'round': 1, 'influx': 1, 'credits': 1, 'You': 1, 'too': 1, 'voters': 1, 'estate': 1, 'What': 1, 'Making': 1, 'handle': 1, \"'tuition-free\": 1, '37,000': 1, 'hike': 1, 'seeking': 1, 'stay': 1, 'Monday': 1, 'push': 1, '45': 1, 'four': 1, 'additional': 1, 'qualify': 1, 'books': 1, 'Susan': 1, 'necessarily': 1, 'much': 1, 'provide': 1, 'opportunities': 1, 'offering': 1, 'real': 1, 'do': 1, 'lawmaker': 1, 'then': 1, '85,000': 1, '14K': 1, 'Lawmakers': 1, 'semester': 1, 'eligible': 1, 'Eligible': 1, 'reassess': 1, 'both': 1, 'aid': 1, 'adults': 1, 'four-year': 1, 'An': 1, '17,000': 1, 'this': 1, 'founders': 1, 'retraining': 1, '2.25': 1, 'any': 1, 'This': 1, 'November': 1, 'housing': 1, 'And': 1, 'senior': 1, 'other': 1, 'one-time': 1, 'conference': 1, 'if': 1, 'mandatory': 1, 'courses': 1, 'long': 1, 'free-tuition': 1, 'graduate': 1, 'would': 1, 'controller': 1, 'poorest': 1, '5.4': 1, 'so': 1, \"'\": 1, 'according': 1, 'children': 1, 'increased': 1, 'next': 1, 'discussing': 1, 'many': 1, 'Jane': 1, 'fee': 1, 'family': 1, 'returning': 1, 'growing': 1, 'want': 1, 'services': 1, 'impact': 1, 'no': 1, 'press': 1, 'expenses': 1, 'That': 1, 'But': 1, 'hook': 1, '2015': 1, 'graduates': 1, 'enrolled': 1, '3': 1, 'everyone': 1, 'language': 1, 'includes': 1, 'average': 1, 'offers': 1, 'greater': 1, 'others': 1, 'one': 1, 'you': 1, 'rich': 1, 'means': 1, 'job': 1, 'encourage': 1, 'these': 1, 'expand': 1, 'started': 1, 'Lamb': 1, 'taxes': 1, 'costing': 1, 'bring': 1, 'committed': 1, 'stipend': 1, 'about': 1, 'Students': 1, 'general': 1, 'difficult': 1, 'two-year': 1, 'Why': 1, 'Tennessee': 1, 'enroll': 1, 'selling': 1, 'end': 1, 'predict': 1, '25': 1, 'covers': 1, '200': 1, 'residential': 1, '500': 1, 'Facebook': 1, 'few': 1, 'credit': 1, 'sets': 1})\n"
     ]
    }
   ],
   "source": [
    "fp = Counter(tokens1)\n",
    "\n",
    "print(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('across', 2, 6), ('progressive', 1, 11), ('have', 5, 4), ('middle', 2, 6), ('wants', 1, 5), ('earn', 2, 4), ('last', 1, 4), ('It', 3, 2), ('for', 18, 3), ('fund', 1, 4), ('of', 8, 2), ('on', 3, 2), ('We', 1, 2), ('she', 1, 3), ('goes', 2, 4), ('they', 2, 4), ('5', 1, 1), ('called', 1, 6), ('cost', 4, 4), ('affordable', 1, 10), ('Powered', 1, 7), ('.', 26, 1), ('or', 3, 2), ('transfer', 1, 8), ('foreign', 1, 7), ('up', 3, 2), ('enter', 1, 5), ('said', 4, 4), ('Related', 2, 7), (\"n't\", 3, 3), ('125,000', 1, 7), ('colleges', 3, 8), ('residents', 3, 9), ('and', 12, 3), ('tuition-free', 2, 12), ('full-time', 2, 9), ('community', 1, 9), ('pushing', 1, 7), ('approved', 1, 8), ('%', 2, 1), ('support', 2, 7), ('gap', 1, 3), ('going', 1, 5), ('even', 1, 4), ('income', 1, 6), ('an', 2, 2), ('years', 2, 5), ('17', 1, 2), ('nation', 1, 6), ('include', 1, 7), ('gaining', 1, 7), ('apart', 1, 5), ('part-time', 1, 9), ('capacity', 1, 8), ('less', 3, 4), ('matter', 1, 6), ('people', 3, 6), ('used', 1, 4), ('28,000', 1, 6), ('individual', 1, 10), ('sign', 1, 4), ('500,000', 1, 7), ('fastest', 1, 7), ('46', 1, 2), ('by', 3, 2), ('resided', 1, 7), ('classes', 1, 7), ('commercial', 1, 10), ('First', 1, 5), ('Chancellor', 1, 10), ('take', 2, 4), ('worth', 1, 5), ('starts', 1, 6), ('York', 2, 4), ('round', 1, 5), ('influx', 1, 6), ('credits', 1, 7), ('was', 2, 3), ('You', 1, 3), ('too', 1, 3), ('voters', 1, 6), ('estate', 1, 6), ('What', 1, 4), ('public', 2, 6), ('Making', 1, 6), ('handle', 1, 6), (\"'tuition-free\", 1, 13), ('pay', 3, 3), ('The', 6, 3), (':', 2, 1), ('class', 2, 5), ('that', 3, 4), ('how', 3, 3), ('the', 24, 3), ('37,000', 1, 6), ('hike', 1, 4), ('Rhode', 2, 5), ('City', 3, 4), ('seeking', 1, 7), ('stay', 1, 4), ('Monday', 1, 6), ('push', 1, 4), ('45', 1, 2), ('four', 1, 4), ('additional', 1, 10), ('Kim', 2, 3), ('money', 2, 5), ('than', 6, 4), ('qualify', 1, 7), ('two', 2, 3), ('expected', 3, 8), ('books', 1, 5), ('Susan', 1, 5), ('necessarily', 1, 11), ('much', 1, 4), ('provide', 1, 7), ('opportunities', 1, 13), ('offering', 1, 8), ('has', 3, 3), ('real', 1, 4), ('properties', 3, 10), ('do', 1, 2), ('who', 3, 3), ('lawmaker', 1, 8), ('in', 6, 2), ('then', 1, 4), ('85,000', 1, 6), ('14K', 1, 3), ('Lawmakers', 1, 9), ('semester', 1, 8), ('is', 8, 2), ('eligible', 1, 8), ('Eligible', 1, 8), ('reassess', 1, 8), ('both', 1, 4), ('to', 24, 2), ('aid', 1, 3), ('adults', 1, 6), ('four-year', 1, 9), ('An', 1, 2), ('17,000', 1, 6), ('this', 1, 4), ('San', 7, 3), ('least', 2, 5), ('founders', 1, 8), ('retraining', 1, 10), ('2.25', 1, 4), ('any', 1, 3), ('also', 2, 4), ('plan', 4, 4), ('This', 1, 4), ('be', 5, 2), ('November', 1, 8), ('housing', 1, 7), ('And', 1, 3), ('city', 6, 4), ('it', 3, 2), ('free', 7, 4), ('senior', 1, 6), ('other', 1, 5), (\"'s\", 8, 2), ('one-time', 1, 8), ('College', 3, 7), ('conference', 1, 10), ('if', 1, 2), ('New', 2, 3), ('``', 3, 2), ('mandatory', 1, 9), ('courses', 1, 7), ('long', 1, 4), ('free-tuition', 1, 12), ('program', 2, 7), ('graduate', 1, 8), ('would', 1, 5), ('controller', 1, 10), ('currently', 2, 9), ('poorest', 1, 7), ('per', 2, 3), ('5.4', 1, 3), ('so', 1, 2), (\"'\", 1, 1), ('according', 1, 9), ('children', 1, 8), ('increased', 1, 9), ('school', 4, 6), ('governor', 2, 8), ('Island', 2, 6), ('next', 1, 4), ('help', 2, 4), ('discussing', 1, 10), ('many', 1, 4), ('$', 13, 1), ('Jane', 1, 4), ('fee', 1, 3), ('college', 4, 7), ('a', 12, 1), ('family', 1, 6), ('returning', 1, 9), ('into', 2, 4), ('growing', 1, 7), ('want', 1, 4), ('services', 1, 8), ('impact', 1, 6), ('no', 1, 2), (\"''\", 3, 2), ('press', 1, 5), ('expenses', 1, 8), ('That', 1, 4), ('more', 6, 4), ('But', 1, 3), ('hook', 1, 4), ('2015', 1, 4), ('tax', 2, 3), (',', 14, 1), ('graduates', 1, 9), ('enrolled', 1, 8), ('3', 1, 1), ('get', 2, 3), ('everyone', 1, 8), ('country', 2, 7), ('language', 1, 8), ('includes', 1, 8), ('average', 1, 7), ('offers', 1, 6), ('greater', 1, 7), ('others', 1, 6), ('one', 1, 3), ('million', 4, 7), ('you', 1, 3), ('rich', 1, 4), ('means', 1, 5), ('job', 1, 3), ('SmartAsset.com', 2, 14), ('encourage', 1, 9), ('these', 1, 5), ('at', 7, 2), ('expand', 1, 6), ('as', 3, 2), ('started', 1, 7), ('Lamb', 1, 4), ('students', 5, 8), ('taxes', 1, 5), ('tuition', 2, 7), ('costing', 1, 7), ('will', 8, 4), ('bring', 1, 5), ('committed', 1, 9), ('stipend', 1, 7), ('about', 1, 5), ('Students', 1, 8), ('wo', 2, 2), ('general', 1, 7), ('difficult', 1, 9), ('two-year', 1, 8), ('Why', 1, 3), ('Tennessee', 1, 9), ('enroll', 1, 6), ('selling', 1, 7), ('Franciscans', 2, 11), ('make', 2, 4), ('high', 2, 4), ('end', 1, 3), ('predict', 1, 7), ('recent', 2, 6), ('25', 1, 2), ('covers', 1, 6), ('200', 1, 3), ('residential', 1, 11), ('are', 4, 3), ('year', 9, 4), ('Francisco', 5, 9), ('still', 2, 5), ('500', 1, 3), ('Facebook', 1, 8), ('few', 1, 3), ('credit', 1, 6), ('sets', 1, 4)]\n"
     ]
    }
   ],
   "source": [
    "model = [ (i, fp[i], len(i)) for i in fp ]\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\t6\tacross\n",
      "1\t11\tprogressive\n",
      "5\t4\thave\n",
      "2\t6\tmiddle\n",
      "1\t5\twants\n",
      "2\t4\tearn\n",
      "1\t4\tlast\n",
      "3\t2\tIt\n",
      "18\t3\tfor\n",
      "1\t4\tfund\n",
      "8\t2\tof\n",
      "3\t2\ton\n",
      "1\t2\tWe\n",
      "1\t3\tshe\n",
      "2\t4\tgoes\n",
      "2\t4\tthey\n",
      "1\t1\t5\n",
      "1\t6\tcalled\n",
      "4\t4\tcost\n",
      "1\t10\taffordable\n",
      "1\t7\tPowered\n",
      "26\t1\t.\n",
      "3\t2\tor\n",
      "1\t8\ttransfer\n",
      "1\t7\tforeign\n",
      "3\t2\tup\n",
      "1\t5\tenter\n",
      "4\t4\tsaid\n",
      "2\t7\tRelated\n",
      "3\t3\tn't\n",
      "1\t7\t125,000\n",
      "3\t8\tcolleges\n",
      "3\t9\tresidents\n",
      "12\t3\tand\n",
      "2\t12\ttuition-free\n",
      "2\t9\tfull-time\n",
      "1\t9\tcommunity\n",
      "1\t7\tpushing\n",
      "1\t8\tapproved\n",
      "2\t1\t%\n",
      "2\t7\tsupport\n",
      "1\t3\tgap\n",
      "1\t5\tgoing\n",
      "1\t4\teven\n",
      "1\t6\tincome\n",
      "2\t2\tan\n",
      "2\t5\tyears\n",
      "1\t2\t17\n",
      "1\t6\tnation\n",
      "1\t7\tinclude\n",
      "1\t7\tgaining\n",
      "1\t5\tapart\n",
      "1\t9\tpart-time\n",
      "1\t8\tcapacity\n",
      "3\t4\tless\n",
      "1\t6\tmatter\n",
      "3\t6\tpeople\n",
      "1\t4\tused\n",
      "1\t6\t28,000\n",
      "1\t10\tindividual\n",
      "1\t4\tsign\n",
      "1\t7\t500,000\n",
      "1\t7\tfastest\n",
      "1\t2\t46\n",
      "3\t2\tby\n",
      "1\t7\tresided\n",
      "1\t7\tclasses\n",
      "1\t10\tcommercial\n",
      "1\t5\tFirst\n",
      "1\t10\tChancellor\n",
      "2\t4\ttake\n",
      "1\t5\tworth\n",
      "1\t6\tstarts\n",
      "2\t4\tYork\n",
      "1\t5\tround\n",
      "1\t6\tinflux\n",
      "1\t7\tcredits\n",
      "2\t3\twas\n",
      "1\t3\tYou\n",
      "1\t3\ttoo\n",
      "1\t6\tvoters\n",
      "1\t6\testate\n",
      "1\t4\tWhat\n",
      "2\t6\tpublic\n",
      "1\t6\tMaking\n",
      "1\t6\thandle\n",
      "1\t13\t'tuition-free\n",
      "3\t3\tpay\n",
      "6\t3\tThe\n",
      "2\t1\t:\n",
      "2\t5\tclass\n",
      "3\t4\tthat\n",
      "3\t3\thow\n",
      "24\t3\tthe\n",
      "1\t6\t37,000\n",
      "1\t4\thike\n",
      "2\t5\tRhode\n",
      "3\t4\tCity\n",
      "1\t7\tseeking\n",
      "1\t4\tstay\n",
      "1\t6\tMonday\n",
      "1\t4\tpush\n",
      "1\t2\t45\n",
      "1\t4\tfour\n",
      "1\t10\tadditional\n",
      "2\t3\tKim\n",
      "2\t5\tmoney\n",
      "6\t4\tthan\n",
      "1\t7\tqualify\n",
      "2\t3\ttwo\n",
      "3\t8\texpected\n",
      "1\t5\tbooks\n",
      "1\t5\tSusan\n",
      "1\t11\tnecessarily\n",
      "1\t4\tmuch\n",
      "1\t7\tprovide\n",
      "1\t13\topportunities\n",
      "1\t8\toffering\n",
      "3\t3\thas\n",
      "1\t4\treal\n",
      "3\t10\tproperties\n",
      "1\t2\tdo\n",
      "3\t3\twho\n",
      "1\t8\tlawmaker\n",
      "6\t2\tin\n",
      "1\t4\tthen\n",
      "1\t6\t85,000\n",
      "1\t3\t14K\n",
      "1\t9\tLawmakers\n",
      "1\t8\tsemester\n",
      "8\t2\tis\n",
      "1\t8\teligible\n",
      "1\t8\tEligible\n",
      "1\t8\treassess\n",
      "1\t4\tboth\n",
      "24\t2\tto\n",
      "1\t3\taid\n",
      "1\t6\tadults\n",
      "1\t9\tfour-year\n",
      "1\t2\tAn\n",
      "1\t6\t17,000\n",
      "1\t4\tthis\n",
      "7\t3\tSan\n",
      "2\t5\tleast\n",
      "1\t8\tfounders\n",
      "1\t10\tretraining\n",
      "1\t4\t2.25\n",
      "1\t3\tany\n",
      "2\t4\talso\n",
      "4\t4\tplan\n",
      "1\t4\tThis\n",
      "5\t2\tbe\n",
      "1\t8\tNovember\n",
      "1\t7\thousing\n",
      "1\t3\tAnd\n",
      "6\t4\tcity\n",
      "3\t2\tit\n",
      "7\t4\tfree\n",
      "1\t6\tsenior\n",
      "1\t5\tother\n",
      "8\t2\t's\n",
      "1\t8\tone-time\n",
      "3\t7\tCollege\n",
      "1\t10\tconference\n",
      "1\t2\tif\n",
      "2\t3\tNew\n",
      "3\t2\t``\n",
      "1\t9\tmandatory\n",
      "1\t7\tcourses\n",
      "1\t4\tlong\n",
      "1\t12\tfree-tuition\n",
      "2\t7\tprogram\n",
      "1\t8\tgraduate\n",
      "1\t5\twould\n",
      "1\t10\tcontroller\n",
      "2\t9\tcurrently\n",
      "1\t7\tpoorest\n",
      "2\t3\tper\n",
      "1\t3\t5.4\n",
      "1\t2\tso\n",
      "1\t1\t'\n",
      "1\t9\taccording\n",
      "1\t8\tchildren\n",
      "1\t9\tincreased\n",
      "4\t6\tschool\n",
      "2\t8\tgovernor\n",
      "2\t6\tIsland\n",
      "1\t4\tnext\n",
      "2\t4\thelp\n",
      "1\t10\tdiscussing\n",
      "1\t4\tmany\n",
      "13\t1\t$\n",
      "1\t4\tJane\n",
      "1\t3\tfee\n",
      "4\t7\tcollege\n",
      "12\t1\ta\n",
      "1\t6\tfamily\n",
      "1\t9\treturning\n",
      "2\t4\tinto\n",
      "1\t7\tgrowing\n",
      "1\t4\twant\n",
      "1\t8\tservices\n",
      "1\t6\timpact\n",
      "1\t2\tno\n",
      "3\t2\t''\n",
      "1\t5\tpress\n",
      "1\t8\texpenses\n",
      "1\t4\tThat\n",
      "6\t4\tmore\n",
      "1\t3\tBut\n",
      "1\t4\thook\n",
      "1\t4\t2015\n",
      "2\t3\ttax\n",
      "14\t1\t,\n",
      "1\t9\tgraduates\n",
      "1\t8\tenrolled\n",
      "1\t1\t3\n",
      "2\t3\tget\n",
      "1\t8\teveryone\n",
      "2\t7\tcountry\n",
      "1\t8\tlanguage\n",
      "1\t8\tincludes\n",
      "1\t7\taverage\n",
      "1\t6\toffers\n",
      "1\t7\tgreater\n",
      "1\t6\tothers\n",
      "1\t3\tone\n",
      "4\t7\tmillion\n",
      "1\t3\tyou\n",
      "1\t4\trich\n",
      "1\t5\tmeans\n",
      "1\t3\tjob\n",
      "2\t14\tSmartAsset.com\n",
      "1\t9\tencourage\n",
      "1\t5\tthese\n",
      "7\t2\tat\n",
      "1\t6\texpand\n",
      "3\t2\tas\n",
      "1\t7\tstarted\n",
      "1\t4\tLamb\n",
      "5\t8\tstudents\n",
      "1\t5\ttaxes\n",
      "2\t7\ttuition\n",
      "1\t7\tcosting\n",
      "8\t4\twill\n",
      "1\t5\tbring\n",
      "1\t9\tcommitted\n",
      "1\t7\tstipend\n",
      "1\t5\tabout\n",
      "1\t8\tStudents\n",
      "2\t2\two\n",
      "1\t7\tgeneral\n",
      "1\t9\tdifficult\n",
      "1\t8\ttwo-year\n",
      "1\t3\tWhy\n",
      "1\t9\tTennessee\n",
      "1\t6\tenroll\n",
      "1\t7\tselling\n",
      "2\t11\tFranciscans\n",
      "2\t4\tmake\n",
      "2\t4\thigh\n",
      "1\t3\tend\n",
      "1\t7\tpredict\n",
      "2\t6\trecent\n",
      "1\t2\t25\n",
      "1\t6\tcovers\n",
      "1\t3\t200\n",
      "1\t11\tresidential\n",
      "4\t3\tare\n",
      "9\t4\tyear\n",
      "5\t9\tFrancisco\n",
      "2\t5\tstill\n",
      "1\t3\t500\n",
      "1\t8\tFacebook\n",
      "1\t3\tfew\n",
      "1\t6\tcredit\n",
      "1\t4\tsets\n"
     ]
    }
   ],
   "source": [
    "for x in model:\n",
    "    print( \"\\t\".join( (str(x[1]), str(x[2]), x[0]) ) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopw = stopwords.words(\"english\")\n",
    "stopw.append(\"us\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\t6\tacross\t0\n",
      "1\t11\tprogressive\t0\n",
      "5\t4\thave\t1\n",
      "2\t6\tmiddle\t0\n",
      "1\t5\twants\t0\n",
      "2\t4\tearn\t0\n",
      "1\t4\tlast\t0\n",
      "3\t2\tIt\t0\n",
      "18\t3\tfor\t1\n",
      "1\t4\tfund\t0\n",
      "8\t2\tof\t1\n",
      "3\t2\ton\t1\n",
      "1\t2\tWe\t0\n",
      "1\t3\tshe\t1\n",
      "2\t4\tgoes\t0\n",
      "2\t4\tthey\t1\n",
      "1\t1\t5\t0\n",
      "1\t6\tcalled\t0\n",
      "4\t4\tcost\t0\n",
      "1\t10\taffordable\t0\n",
      "1\t7\tPowered\t0\n",
      "26\t1\t.\t0\n",
      "3\t2\tor\t1\n",
      "1\t8\ttransfer\t0\n",
      "1\t7\tforeign\t0\n",
      "3\t2\tup\t1\n",
      "1\t5\tenter\t0\n",
      "4\t4\tsaid\t0\n",
      "2\t7\tRelated\t0\n",
      "3\t3\tn't\t0\n",
      "1\t7\t125,000\t0\n",
      "3\t8\tcolleges\t0\n",
      "3\t9\tresidents\t0\n",
      "12\t3\tand\t1\n",
      "2\t12\ttuition-free\t0\n",
      "2\t9\tfull-time\t0\n",
      "1\t9\tcommunity\t0\n",
      "1\t7\tpushing\t0\n",
      "1\t8\tapproved\t0\n",
      "2\t1\t%\t0\n",
      "2\t7\tsupport\t0\n",
      "1\t3\tgap\t0\n",
      "1\t5\tgoing\t0\n",
      "1\t4\teven\t0\n",
      "1\t6\tincome\t0\n",
      "2\t2\tan\t1\n",
      "2\t5\tyears\t0\n",
      "1\t2\t17\t0\n",
      "1\t6\tnation\t0\n",
      "1\t7\tinclude\t0\n",
      "1\t7\tgaining\t0\n",
      "1\t5\tapart\t0\n",
      "1\t9\tpart-time\t0\n",
      "1\t8\tcapacity\t0\n",
      "3\t4\tless\t0\n",
      "1\t6\tmatter\t0\n",
      "3\t6\tpeople\t0\n",
      "1\t4\tused\t0\n",
      "1\t6\t28,000\t0\n",
      "1\t10\tindividual\t0\n",
      "1\t4\tsign\t0\n",
      "1\t7\t500,000\t0\n",
      "1\t7\tfastest\t0\n",
      "1\t2\t46\t0\n",
      "3\t2\tby\t1\n",
      "1\t7\tresided\t0\n",
      "1\t7\tclasses\t0\n",
      "1\t10\tcommercial\t0\n",
      "1\t5\tFirst\t0\n",
      "1\t10\tChancellor\t0\n",
      "2\t4\ttake\t0\n",
      "1\t5\tworth\t0\n",
      "1\t6\tstarts\t0\n",
      "2\t4\tYork\t0\n",
      "1\t5\tround\t0\n",
      "1\t6\tinflux\t0\n",
      "1\t7\tcredits\t0\n",
      "2\t3\twas\t1\n",
      "1\t3\tYou\t0\n",
      "1\t3\ttoo\t1\n",
      "1\t6\tvoters\t0\n",
      "1\t6\testate\t0\n",
      "1\t4\tWhat\t0\n",
      "2\t6\tpublic\t0\n",
      "1\t6\tMaking\t0\n",
      "1\t6\thandle\t0\n",
      "1\t13\t'tuition-free\t0\n",
      "3\t3\tpay\t0\n",
      "6\t3\tThe\t0\n",
      "2\t1\t:\t0\n",
      "2\t5\tclass\t0\n",
      "3\t4\tthat\t1\n",
      "3\t3\thow\t1\n",
      "24\t3\tthe\t1\n",
      "1\t6\t37,000\t0\n",
      "1\t4\thike\t0\n",
      "2\t5\tRhode\t0\n",
      "3\t4\tCity\t0\n",
      "1\t7\tseeking\t0\n",
      "1\t4\tstay\t0\n",
      "1\t6\tMonday\t0\n",
      "1\t4\tpush\t0\n",
      "1\t2\t45\t0\n",
      "1\t4\tfour\t0\n",
      "1\t10\tadditional\t0\n",
      "2\t3\tKim\t0\n",
      "2\t5\tmoney\t0\n",
      "6\t4\tthan\t1\n",
      "1\t7\tqualify\t0\n",
      "2\t3\ttwo\t0\n",
      "3\t8\texpected\t0\n",
      "1\t5\tbooks\t0\n",
      "1\t5\tSusan\t0\n",
      "1\t11\tnecessarily\t0\n",
      "1\t4\tmuch\t0\n",
      "1\t7\tprovide\t0\n",
      "1\t13\topportunities\t0\n",
      "1\t8\toffering\t0\n",
      "3\t3\thas\t1\n",
      "1\t4\treal\t0\n",
      "3\t10\tproperties\t0\n",
      "1\t2\tdo\t1\n",
      "3\t3\twho\t1\n",
      "1\t8\tlawmaker\t0\n",
      "6\t2\tin\t1\n",
      "1\t4\tthen\t1\n",
      "1\t6\t85,000\t0\n",
      "1\t3\t14K\t0\n",
      "1\t9\tLawmakers\t0\n",
      "1\t8\tsemester\t0\n",
      "8\t2\tis\t1\n",
      "1\t8\teligible\t0\n",
      "1\t8\tEligible\t0\n",
      "1\t8\treassess\t0\n",
      "1\t4\tboth\t1\n",
      "24\t2\tto\t1\n",
      "1\t3\taid\t0\n",
      "1\t6\tadults\t0\n",
      "1\t9\tfour-year\t0\n",
      "1\t2\tAn\t0\n",
      "1\t6\t17,000\t0\n",
      "1\t4\tthis\t1\n",
      "7\t3\tSan\t0\n",
      "2\t5\tleast\t0\n",
      "1\t8\tfounders\t0\n",
      "1\t10\tretraining\t0\n",
      "1\t4\t2.25\t0\n",
      "1\t3\tany\t1\n",
      "2\t4\talso\t0\n",
      "4\t4\tplan\t0\n",
      "1\t4\tThis\t0\n",
      "5\t2\tbe\t1\n",
      "1\t8\tNovember\t0\n",
      "1\t7\thousing\t0\n",
      "1\t3\tAnd\t0\n",
      "6\t4\tcity\t0\n",
      "3\t2\tit\t1\n",
      "7\t4\tfree\t0\n",
      "1\t6\tsenior\t0\n",
      "1\t5\tother\t1\n",
      "8\t2\t's\t0\n",
      "1\t8\tone-time\t0\n",
      "3\t7\tCollege\t0\n",
      "1\t10\tconference\t0\n",
      "1\t2\tif\t1\n",
      "2\t3\tNew\t0\n",
      "3\t2\t``\t0\n",
      "1\t9\tmandatory\t0\n",
      "1\t7\tcourses\t0\n",
      "1\t4\tlong\t0\n",
      "1\t12\tfree-tuition\t0\n",
      "2\t7\tprogram\t0\n",
      "1\t8\tgraduate\t0\n",
      "1\t5\twould\t0\n",
      "1\t10\tcontroller\t0\n",
      "2\t9\tcurrently\t0\n",
      "1\t7\tpoorest\t0\n",
      "2\t3\tper\t0\n",
      "1\t3\t5.4\t0\n",
      "1\t2\tso\t1\n",
      "1\t1\t'\t0\n",
      "1\t9\taccording\t0\n",
      "1\t8\tchildren\t0\n",
      "1\t9\tincreased\t0\n",
      "4\t6\tschool\t0\n",
      "2\t8\tgovernor\t0\n",
      "2\t6\tIsland\t0\n",
      "1\t4\tnext\t0\n",
      "2\t4\thelp\t0\n",
      "1\t10\tdiscussing\t0\n",
      "1\t4\tmany\t0\n",
      "13\t1\t$\t0\n",
      "1\t4\tJane\t0\n",
      "1\t3\tfee\t0\n",
      "4\t7\tcollege\t0\n",
      "12\t1\ta\t1\n",
      "1\t6\tfamily\t0\n",
      "1\t9\treturning\t0\n",
      "2\t4\tinto\t1\n",
      "1\t7\tgrowing\t0\n",
      "1\t4\twant\t0\n",
      "1\t8\tservices\t0\n",
      "1\t6\timpact\t0\n",
      "1\t2\tno\t1\n",
      "3\t2\t''\t0\n",
      "1\t5\tpress\t0\n",
      "1\t8\texpenses\t0\n",
      "1\t4\tThat\t0\n",
      "6\t4\tmore\t1\n",
      "1\t3\tBut\t0\n",
      "1\t4\thook\t0\n",
      "1\t4\t2015\t0\n",
      "2\t3\ttax\t0\n",
      "14\t1\t,\t0\n",
      "1\t9\tgraduates\t0\n",
      "1\t8\tenrolled\t0\n",
      "1\t1\t3\t0\n",
      "2\t3\tget\t0\n",
      "1\t8\teveryone\t0\n",
      "2\t7\tcountry\t0\n",
      "1\t8\tlanguage\t0\n",
      "1\t8\tincludes\t0\n",
      "1\t7\taverage\t0\n",
      "1\t6\toffers\t0\n",
      "1\t7\tgreater\t0\n",
      "1\t6\tothers\t0\n",
      "1\t3\tone\t0\n",
      "4\t7\tmillion\t0\n",
      "1\t3\tyou\t1\n",
      "1\t4\trich\t0\n",
      "1\t5\tmeans\t0\n",
      "1\t3\tjob\t0\n",
      "2\t14\tSmartAsset.com\t0\n",
      "1\t9\tencourage\t0\n",
      "1\t5\tthese\t1\n",
      "7\t2\tat\t1\n",
      "1\t6\texpand\t0\n",
      "3\t2\tas\t1\n",
      "1\t7\tstarted\t0\n",
      "1\t4\tLamb\t0\n",
      "5\t8\tstudents\t0\n",
      "1\t5\ttaxes\t0\n",
      "2\t7\ttuition\t0\n",
      "1\t7\tcosting\t0\n",
      "8\t4\twill\t1\n",
      "1\t5\tbring\t0\n",
      "1\t9\tcommitted\t0\n",
      "1\t7\tstipend\t0\n",
      "1\t5\tabout\t1\n",
      "1\t8\tStudents\t0\n",
      "2\t2\two\t0\n",
      "1\t7\tgeneral\t0\n",
      "1\t9\tdifficult\t0\n",
      "1\t8\ttwo-year\t0\n",
      "1\t3\tWhy\t0\n",
      "1\t9\tTennessee\t0\n",
      "1\t6\tenroll\t0\n",
      "1\t7\tselling\t0\n",
      "2\t11\tFranciscans\t0\n",
      "2\t4\tmake\t0\n",
      "2\t4\thigh\t0\n",
      "1\t3\tend\t0\n",
      "1\t7\tpredict\t0\n",
      "2\t6\trecent\t0\n",
      "1\t2\t25\t0\n",
      "1\t6\tcovers\t0\n",
      "1\t3\t200\t0\n",
      "1\t11\tresidential\t0\n",
      "4\t3\tare\t1\n",
      "9\t4\tyear\t0\n",
      "5\t9\tFrancisco\t0\n",
      "2\t5\tstill\t0\n",
      "1\t3\t500\t0\n",
      "1\t8\tFacebook\t0\n",
      "1\t3\tfew\t1\n",
      "1\t6\tcredit\t0\n",
      "1\t4\tsets\t0\n"
     ]
    }
   ],
   "source": [
    "def isStopword(word):\n",
    "    if word in stopw:\n",
    "        return(1)\n",
    "    return(0)\n",
    "\n",
    "for x in model:\n",
    "    print( \"\\t\".join( (str(x[1]), str(x[2]), x[0], str(isStopword(x[0]))) ) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import pos_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokens1 = word_tokenize(text1).lower())\n",
    "posTokens = pos_tag(tokens1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['C', 'D', 'R', '.', ':', 'W', ',', 'N', 'P', 'J', 'T', \"'\", 'V', 'M', '`', 'I', '$']\n"
     ]
    }
   ],
   "source": [
    "tags = list( set( [ x[1][0] for x in posTokens ] ) )\n",
    "\n",
    "print(tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "leftOfToken = defaultdict(Counter)\n",
    "rightOfToken = defaultdict(Counter)\n",
    "\n",
    "for i in range(len(posTokens)):\n",
    "    tag = posTokens[i][1][0]\n",
    "    token = (posTokens[i][0], tag)\n",
    "    if i > 0:\n",
    "        ltag = posTokens[i - 1][1][0]\n",
    "        leftOfToken[token][ltag] += 1\n",
    "    if i < len(posTokens) - 1:\n",
    "        rtag = posTokens[i + 1][1][0]\n",
    "        rightOfToken[token][rtag] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 growing V\n",
      "1 7 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 3 0 0 1 0 0 0 2 0 0 0 0 2 0 year N\n",
      "1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 cost N\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 everyone N\n",
      "0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 class N\n",
      "0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Why W\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 books N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 : :\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 But C\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 retraining V\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 general J\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 hike N\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 many J\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 nation N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 credit N\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 so I\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 do V\n",
      "1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 Rhode N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 approved V\n",
      "0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 It P\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 capacity N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 difficult J\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 community N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 include V\n",
      "2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 % N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 returning V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 impact V\n",
      "1 0 0 0 0 0 0 0 0 3 0 0 0 0 0 1 0 0 0 0 2 0 0 0 0 0 1 0 0 0 2 0 0 0 students N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 This D\n",
      "0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 Related V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 C\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 enter V\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 Tennessee N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 affordable J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 increased V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 started V\n",
      "0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 help V\n",
      "1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 2 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 1 0 more J\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 You P\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 job N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 Lawmakers N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 First N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 2 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 City N\n",
      "0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 tax N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 this D\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 two C\n",
      "0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 she P\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 influx N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 14K C\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Jane N\n",
      "2 0 0 0 0 0 0 0 0 1 2 0 7 0 0 12 0 0 0 0 0 0 0 0 17 0 6 0 0 0 0 0 0 1 the D\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 pushing V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 sign V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 Lamb N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 Powered V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 make V\n",
      "0 1 0 0 0 0 0 0 0 3 0 0 4 0 0 5 0 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 $ $\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 family N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 stay V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 least J\n",
      "6 0 0 0 0 0 0 0 0 0 0 0 4 0 0 2 0 0 0 0 0 0 0 0 9 0 3 0 0 0 0 0 0 0 a D\n",
      "1 0 0 0 0 0 0 6 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 1 0 0 4 0 0 1 0 is V\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 sets V\n",
      "1 0 2 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 2 0 0 1 0 be V\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 An D\n",
      "0 0 2 0 0 0 0 9 0 1 0 0 12 0 0 0 0 1 2 0 0 0 0 0 3 0 0 0 0 18 0 0 0 0 to T\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 free-tuition N\n",
      "0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 currently R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 credits N\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 costing V\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 progressive J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 37,000 C\n",
      "0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 middle J\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 that W\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 enroll V\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 real J\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 estate N\n",
      "0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 as I\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5.4 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 offering V\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 1 0 0 0 0 0 0 0 0 2 0 0 0 0 properties N\n",
      "0 0 2 0 0 0 0 0 2 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 3 0 1 0 0 0 0 have V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 graduate N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 one C\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 graduates N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 no D\n",
      "0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 program N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 public J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 adults N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 2 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 n't R\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Chancellor N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Susan N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 opportunities N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 you P\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 expected V\n",
      "0 4 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 3 1 0 0 0 0 1 0 1 0 city N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 courses N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 resided V\n",
      "0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 take V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 500,000 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 gaining V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 November N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 What W\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 fastest J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 even R\n",
      "1 0 0 1 0 0 0 0 0 3 0 0 0 0 0 2 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 San N\n",
      "4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 million C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Making V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 four-year J\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 necessarily R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 aid N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 85,000 C\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 housing N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 2 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 on I\n",
      "0 0 0 0 0 0 0 1 0 5 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 4 than I\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 both D\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 taxes N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 round V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 reassess V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 expand V\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 matter N\n",
      "1 0 0 0 0 0 0 9 0 1 0 0 7 0 0 0 0 2 7 0 0 0 0 0 2 1 5 0 0 0 0 0 1 0 for I\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 28,000 C\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 transfer N\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 by I\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 individual N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2.25 C\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 offers V\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 then R\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 predict V\n",
      "3 0 1 0 0 0 0 18 0 1 0 0 3 0 0 0 0 2 7 0 0 0 1 0 5 4 1 0 0 3 0 2 0 0 . .\n",
      "0 0 0 2 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 `` `\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 commercial J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 committed V\n",
      "0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 tuition N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 conference N\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 York N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 press N\n",
      "1 0 0 0 0 0 0 8 0 1 0 0 4 0 0 0 0 4 0 1 0 0 0 0 1 0 0 0 3 2 0 1 2 0 , ,\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 includes V\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 two-year J\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 bring V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 controller N\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 encourage V\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 full-time J\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 support N\n",
      "0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 who W\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 Franciscans N\n",
      "0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 2 0 0 0 0 's V\n",
      "0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 2 0 0 0 0 are V\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 next J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 used V\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 long R\n",
      "0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 0 0 0 1 0 0 1 0 Francisco N\n",
      "0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 people N\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 much J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 We P\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 too R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 one-time J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 classes N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 into I\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 foreign J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 apart J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 called V\n",
      "0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 earn V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 qualify V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 SmartAsset.com N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 language N\n",
      "0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 College N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 200 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 about I\n",
      "0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 2 0 0 0 0 or C\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 want V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 services N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 And C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 that I\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 few J\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 means V\n",
      "1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 how W\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 part-time J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 any D\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 stipend N\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 covers V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 selling V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 worth V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 discussing V\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 other J\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 push N\n",
      "0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 3 0 0 0 0 0 0 0 's P\n",
      "0 0 0 0 0 0 0 4 0 0 0 0 2 0 0 1 0 1 1 0 0 0 0 0 2 0 3 0 0 0 0 0 0 0 at I\n",
      "0 0 0 0 0 0 3 7 0 2 0 0 0 0 0 0 0 0 1 1 0 0 1 0 0 0 5 0 0 3 1 0 0 0 and C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25 C\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 senior J\n",
      "1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 residents N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 residential J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 125,000 C\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 Eligible J\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 Island N\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 less J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 enrolled V\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 founders N\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 Students N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 end V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 an D\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 hook N\n",
      "1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 goes V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eligible J\n",
      "1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 years N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 children N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 fund N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 last J\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 lawmaker N\n",
      "0 1 0 0 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 1 0 0 0 0 school N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 greater J\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 also R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 recent J\n",
      "0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 has V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 if I\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 they P\n",
      "0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 money N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 income N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 17 C\n",
      "0 0 0 0 0 0 0 2 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 1 0 0 0 plan N\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 would M\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 rich J\n",
      "0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 '' '\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17,000 C\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 expenses N\n",
      "0 0 0 0 0 0 0 3 0 0 0 0 3 0 0 0 0 1 2 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 in I\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 2 1 0 0 1 0 0 0 college N\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 wo M\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 going V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 others N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 wants V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 gap N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 additional J\n",
      "0 0 0 0 0 0 0 2 1 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 said V\n",
      "0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 1 2 0 0 0 0 0 4 0 0 0 0 0 0 0 0 1 of I\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 fee N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 46 C\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 New N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 semester N\n",
      "1 1 0 0 0 0 0 2 0 0 0 0 2 0 0 1 0 0 0 0 1 0 0 1 4 0 0 0 0 1 0 0 0 0 free J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 up R\n",
      "0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 governor N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 500 C\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 provide V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ' P\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 Facebook N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 pay V\n",
      "0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 tuition-free J\n",
      "0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 high J\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 still R\n",
      "1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 per I\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 across I\n",
      "0 0 0 4 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 2 0 0 0 0 0 0 0 The D\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 according V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 1 0 it P\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 'tuition-free J\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 That D\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 as R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 poorest J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 Monday N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 these D\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 starts V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 get V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 more R\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 average N\n",
      "1 0 1 0 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 7 0 0 0 0 will M\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 four C\n",
      "0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 was V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 2015 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 seeking V\n",
      "0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 Kim N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 handle V\n",
      "0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 country N\n",
      "0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 colleges N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 mandatory J\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 cost V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 3 C\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 voters N\n"
     ]
    }
   ],
   "source": [
    "for token in leftOfToken.keys():\n",
    "    leftVector = []\n",
    "    rightVector = []\n",
    "    for tag in tags:\n",
    "        leftVector.append(leftOfToken[token][tag])\n",
    "        rightVector.append(rightOfToken[token][tag])\n",
    "    print(\" \".join([ str(x) for x in leftVector ]), \" \".join([ str(x) for x in rightVector ]), token[0], token[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 caused V\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 unstable J\n",
      "0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 flight N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 Hobart N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 clothes N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 presidential J\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 spoke V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 behaved V\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 Texas N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 believe V\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 passengers N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Airlines N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 her P\n",
      "0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 pilot N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 in I\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 plane N\n",
      "0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 CNN N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Charlie N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 she P\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 said V\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 took V\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 about I\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 way N\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 San N\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 whom W\n",
      "0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 divorce N\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 United N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 social J\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 street N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 Sunday N\n",
      "0 0 0 0 0 0 0 5 0 2 0 0 0 0 0 0 0 0 1 0 0 0 1 0 2 0 0 0 0 2 0 0 1 0 , ,\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 Francisco-bound N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 over I\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 on I\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 out I\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 Austin N\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 media N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 them P\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 mentally R\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 not R\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 other J\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 that W\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 's P\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 and C\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 The D\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 concerns N\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 delayed V\n",
      "0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 naming J\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 2 0 0 0 0 0 0 0 0 3 0 1 0 0 0 0 0 0 0 the D\n",
      "0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 spokesman N\n",
      "0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 Passengers N\n",
      "0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 express V\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 election N\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 . .\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 among I\n",
      "0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 intercom N\n",
      "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 began V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 Saturday N\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 after I\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 a D\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 of I\n",
      "0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 is V\n",
      "0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 speaking V\n",
      "0 0 0 0 0 0 0 2 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 2 0 0 0 0 to T\n",
      "0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 was V\n",
      "0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 issues N\n",
      "0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 boarded V\n"
     ]
    }
   ],
   "source": [
    "text2 = \"\"\"A flight out of Austin, Texas, was delayed after\n",
    "a pilot behaved in a way that caused passengers to believe\n",
    "she was mentally unstable, a United Airlines spokesman said Sunday.\n",
    "The pilot, whom CNN is not naming, boarded the plane in street\n",
    "clothes and began speaking to passengers over the intercom,\n",
    "spokesman Charlie Hobart said.\n",
    "Passengers on Saturday's San Francisco-bound flight took to\n",
    "social media to express concerns after the pilot spoke to\n",
    "them about her divorce and the presidential election, among\n",
    "other issues. \"\"\"\n",
    "\n",
    "tokens2 = word_tokenize(text2).lower())\n",
    "\n",
    "posTokens2 = pos_tag(tokens2)\n",
    "\n",
    "leftOfToken2 = defaultdict(Counter)\n",
    "rightOfToken2 = defaultdict(Counter)\n",
    "\n",
    "for i in range(len(posTokens2)):\n",
    "    token = (posTokens2[i][0], posTokens2[i][1][0])\n",
    "    if i > 0:\n",
    "        ltag = posTokens2[i - 1][1][0]\n",
    "        leftOfToken2[token][ltag] += 1\n",
    "    if i < len(posTokens2) - 1:\n",
    "        rtag = posTokens2[i + 1][1][0]\n",
    "        rightOfToken2[token][rtag] += 1\n",
    "\n",
    "for token in leftOfToken2.keys():\n",
    "    leftVector = []\n",
    "    rightVector = []\n",
    "    for tag in tags:\n",
    "        leftVector.append(leftOfToken2[token][tag])\n",
    "        rightVector.append(rightOfToken2[token][tag])\n",
    "    print(\" \".join([ str(x) for x in leftVector ]), \" \".join([ str(x) for x in rightVector ]), token[0], token[1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Timbl in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}