README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTRC-Features [![Build Status](https://travis-ci.org/htrc/htrc-feature-reader.svg?branch=master)](https://travis-ci.org/htrc/htrc-feature-reader) [![PyPI version](https://badge.fury.io/py/htrc-feature-reader.svg)](https://badge.fury.io/py/htrc-feature-reader) [![Anaconda-Server Badge](https://anaconda.org/htrc/htrc-feature-reader/badges/installer/conda.svg)](https://anaconda.org/htrc/htrc-feature-reader)\n",
    "=============\n",
    "\n",
    "Tools for working with the [HTRC Extracted Features dataset](https://sharc.hathitrust.org/features), a dataset of page-level text features extracted from 17 million digitized works.\n",
    "\n",
    "This library provides a `FeatureReader` for parsing files, which are handled as `Volume` objects with collections of `Page` objects. Volumes provide access to metadata (e.g. language), volume-wide feature information (e.g. token counts), and access to Pages. Pages allow you to easily parse page-level features, particularly token lists.\n",
    "\n",
    "This library makes heavy use of [Pandas](pandas.pydata.org), returning many data representations as DataFrames. This is the leading way of dealing with structured data in Python, so this library doesn't try to reinvent the wheel. Since refactoring around Pandas, the primary benefit of using the HTRC Feature Reader is performance: reading the json structures and parsing them is generally faster than custom code. You also get convenient access to common information, such as case-folded token counts or part-of-page specific character counts. Details of the public methods provided by this library can be found in the [HTRC Feature Reader docs](http://htrc.github.io/htrc-feature-reader/htrc_features/feature_reader.m.html).\n",
    "\n",
    "**Table of Contents**: [Installation](#Installation) | [Usage](#Usage) | \n",
    "[Additional Notes](#Additional-Notes)\n",
    "\n",
    "**Links**: \n",
    "[HTRC Feature Reader Documentation](http://htrc.github.io/htrc-feature-reader/htrc_features/feature_reader.m.html) | [HTRC Extracted Features Dataset](https://sharc.hathitrust.org/features)\n",
    "\n",
    "**Citation**:\n",
    "Peter Organisciak and Boris Capitanu, \"Text Mining in Python through the HTRC Feature Reader,\" *Programming Historian*, (22 November 2016), http://programminghistorian.org/lessons/text-mining-with-extracted-features.\n",
    "\n",
    "\n",
    "## Installation\n",
    "\n",
    "To install,\n",
    "\n",
    "```bash\n",
    "    pip install htrc-feature-reader\n",
    "```\n",
    "\n",
    "That's it! This library is written for Python 3.0+. For Python beginners, you'll need [pip](https://pip.pypa.io/en/stable/installing/).\n",
    "\n",
    "Alternately, if you are using [Anaconda](https://www.continuum.io/downloads), you can install with\n",
    "\n",
    "```bash\n",
    "    conda install -c htrc htrc-feature-reader\n",
    "```\n",
    "\n",
    "The `conda` approach is recommended, because it makes sure that some of the hard-to-install dependencies are properly installed.\n",
    "\n",
    "Given the nature of data analysis, using iPython with Jupyter notebooks for preparing your scripts interactively is a recommended convenience. Most basically, it can be installed with `pip install ipython[notebook]` and run with `ipython notebook` from the command line, which starts a session that you can access through your browser. If this doesn't work, consult the [iPython documentation](http://ipython.readthedocs.org/).\n",
    "\n",
    "Optional: [installing the development version](#Installing-the-development-version). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "*Note: for new Python users, a more in-depth lesson is published by Programming Historian: [Text Mining in Python through the HTRC Feature Reader](http://programminghistorian.org/lessons/text-mining-with-extracted-features). That lesson is also the official citation associated the HTRC Feature Reader library.*\n",
    "\n",
    "### Reading feature files\n",
    "\n",
    "The easiest way to start using this library is to use the Volume interface, which takes a path to an Extracted Features file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href='http://hdl.handle.net/2027/hvd.32044093320364'>The Nautilus.</a></strong> by <em>Delaware Museum of Natural History.</em> (1904, 222 pages) - <code>hvd.32044093320364</code>"
      ],
      "text/plain": [
       "<htrc_features.feature_reader.Volume at 0x7f824834d1d0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from htrc_features import Volume\n",
    "vol = Volume('data/ef2-stubby/hvd/34926/hvd.32044093320364.json.bz2')\n",
    "vol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FeatureReader can also download files at read time, by reference to a HathiTrust volume id. For example, if I want [both of volumes of Pride and Prejudice](https://catalog.hathitrust.org/Record/100323335), I can see that the URLs are babel.hathitrust.org/cgi/pt?id=__hvd.32044013656053__ and babel.hathitrust.org/cgi/pt?id=__hvd.32044013656061__. In the FeatureReader, these can be called with the `ids=[]` argument, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pride and prejudice. v.1\n",
      "Pride and prejudice. v.2\n"
     ]
    }
   ],
   "source": [
    "for htid in [\"hvd.32044013656053\", \"hvd.32044013656061\"]:\n",
    "    vol = Volume(htid)\n",
    "    print(vol.title, vol.enumeration_chronology)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This downloads the file temporarily, using the HTRC's web-based download link (e.g. https://data.analytics.hathitrust.org/features/get?download-id={{URL}}). One good pairing with this feature is the [HTRC Python SDK](https://github.com/htrc/HTRC-PythonSDK)'s functionality for downloading collections. \n",
    "\n",
    "For example, I have a small collection of knitting-related books at https://babel.hathitrust.org/cgi/mb?a=listis&c=1174943610. To read the feature files for those books:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from htrc import workset\n",
    "volids = workset.load_hathitrust_collection('https://babel.hathitrust.org/cgi/mb?a=listis&c=1174943610')\n",
    "FeatureReader(ids=volids).first().title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that for large jobs, it is faster to download your dataset beforehand, using the `rsync` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Volume\n",
    "\n",
    "A [Volume](http://htrc.github.io/htrc-feature-reader/htrc_features/feature_reader.m.html#htrc_features.feature_reader.Volume) contains information about the current work and access to the pages of the work. All the metadata fields from the HTRC JSON file are accessible as properties of the volume object, including _title_, _language_, _imprint_, _oclc_, _pubDate_, and _genre_. The main identifier _id_ and _pageCount_ are also accessible, and you can find the URL for the Full View of the text in the HathiTrust Digital Library - if it exists - with `vol.handle_url`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Volume hvd.32044013656061 is a 306 page text from 1903 written in eng. You can doublecheck at http://hdl.handle.net/2027/hvd.32044013656061'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Volume {} is a {} page text from {} written in {}. You can doublecheck at {}\".format(vol.id, vol.page_count, \n",
    "                                                                                      vol.year, vol.language, \n",
    "                                                                                      vol.handle_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the *Extracted Features* dataset, so the features are easily accessible. To most popular is token counts, which are returned as a Pandas DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>token</th>\n",
       "      <th>pos</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <th>body</th>\n",
       "      <th>abode</th>\n",
       "      <th>NN</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <th>body</th>\n",
       "      <th>head</th>\n",
       "      <th>NN</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <th>body</th>\n",
       "      <th>for</th>\n",
       "      <th>IN</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <th>body</th>\n",
       "      <th>three</th>\n",
       "      <th>CD</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <th>body</th>\n",
       "      <th>would</th>\n",
       "      <th>MD</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <th>body</th>\n",
       "      <th>The</th>\n",
       "      <th>DT</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <th>body</th>\n",
       "      <th>any</th>\n",
       "      <th>DT</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <th>body</th>\n",
       "      <th>surprise</th>\n",
       "      <th>NN</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <th>body</th>\n",
       "      <th>make</th>\n",
       "      <th>VB</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <th>body</th>\n",
       "      <th>I</th>\n",
       "      <th>PRP</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           count\n",
       "page section token    pos       \n",
       "201  body    abode    NN       1\n",
       "117  body    head     NN       1\n",
       "126  body    for      IN       1\n",
       "210  body    three    CD       1\n",
       "224  body    would    MD       1\n",
       "89   body    The      DT       1\n",
       "283  body    any      DT       1\n",
       "63   body    surprise NN       1\n",
       "152  body    make     VB       1\n",
       "170  body    I        PRP      3"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = vol.tokenlist()\n",
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other extracted features are discussed below.\n",
    "\n",
    "The full included metadata can be seen with `vol.parser.meta`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['id', 'metadata_schema_version', 'enumeration_chronology', 'type_of_resource', 'title', 'date_created', 'pub_date', 'language', 'access_profile', 'isbn', 'issn', 'lccn', 'oclc', 'page_count', 'feature_schema_version', 'ht_bib_url', 'genre', 'handle_url', 'imprint', 'names', 'source_institution', 'classification', 'issuance', 'bibliographic_format', 'government_document', 'hathitrust_record_number', 'rights_attributes', 'pub_place', 'volume_identifier', 'source_institution_record_number', 'last_update_date'])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.parser.meta.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These fields are mapped to attributes in `Volume`, so `vol.oclc` will return the oclc field from that metadata. As a convenience, `Volume.year` returns the `pub_date` information and `Volume.author` returns the `contributor information`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('1903', ['Austen, Jane 1775-1817 '])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.year, vol.author"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the minimal metadata included with the extracted feature files is insufficient, you can fetch HT's metadata record from the Bib API with `vol.metadata`.\n",
    "Remember that this calls the HTRC servers for each volume, so can add considerable overhead. The result is a MARC file, returned as a [pymarc](https://github.com/edsu/pymarc) record object. For example, to get the publisher information from field `260`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Boston : Little, Brown, 1903.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.metadata['260'].value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_At large-scales, using `vol.metadata` is an impolite and inefficient amount of server pinging; there are better ways to query the API than one volume at a time. Read about the [HTRC Solr Proxy](https://wiki.htrc.illinois.edu/display/COM/Solr+Proxy+API+User+Guide)._\n",
    "\n",
    "Another source of bibliographic metadata is the HathiTrust Bib API. You can access this information through the URL returned with `vol.ht_bib_url`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://catalog.hathitrust.org/api/volumes/full/htid/hvd.32044013656061.json'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.ht_bib_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Volumes also have direct access to volume-wide info of features stored in pages. For example, you can get a list of words per page through [Volume.tokens_per_page()](http://htrc.github.io/htrc-feature-reader/htrc_features/feature_reader.m.html#htrc_features.feature_reader.Volume.tokens_per_page). We'll discuss these features [below](#Volume-stats-collecting), after looking first at Pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for the most part, the properties of the `Page` and `Volume` objects aligns with the names in the HTRC Extracted Features schema, except they are converted to follow [Python naming conventions](https://google.github.io/styleguide/pyguide.html?showone=Naming#Naming): converting the `CamelCase` of the schema to `lowercase_with_underscores`. E.g. `beginLineChars` from the HTRC data is accessible as `Page.begin_line_chars`.\n",
    "\n",
    "## The fun stuff: playing with token counts and character counts\n",
    "\n",
    "Token counts are returned by `Volume.tokenlist()` (or `Page.tokenlist()`. By default, part-of-speech tagged, case-sensitive counts are returned for the body.\n",
    "\n",
    "The token count information is returned as a DataFrame with a MultiIndex (page, section, token, and part of speech) and one column (count)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         count\n",
      "page section token  pos       \n",
      "1    body    Austen .        1\n",
      "             Pride  NNP      1\n",
      "             and    CC       1\n"
     ]
    }
   ],
   "source": [
    "print(vol.tokenlist()[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Page.tokenlist()` can be manipulated in various ways. You can case-fold, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>lowercase</th>\n",
       "      <th>pos</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>218</th>\n",
       "      <th>body</th>\n",
       "      <th>what</th>\n",
       "      <th>WP</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <th>body</th>\n",
       "      <th>pemberley</th>\n",
       "      <th>NNP</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <th>body</th>\n",
       "      <th>comes</th>\n",
       "      <th>VBZ</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <th>body</th>\n",
       "      <th>took</th>\n",
       "      <th>VBD</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <th>body</th>\n",
       "      <th>necessary</th>\n",
       "      <th>JJ</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            count\n",
       "page section lowercase pos       \n",
       "218  body    what      WP       1\n",
       "30   body    pemberley NNP      1\n",
       "213  body    comes     VBZ      2\n",
       "183  body    took      VBD      1\n",
       "51   body    necessary JJ       1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl = vol.tokenlist(case=False)\n",
    "tl.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, you can combine part of speech counts into a single integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <th>body</th>\n",
       "      <th>family</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <th>body</th>\n",
       "      <th>journey</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <th>body</th>\n",
       "      <th>Perhaps</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <th>body</th>\n",
       "      <th>at</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <th>body</th>\n",
       "      <th>so</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      count\n",
       "page section token         \n",
       "264  body    family       2\n",
       "47   body    journey      1\n",
       "98   body    Perhaps      1\n",
       "49   body    at           2\n",
       "227  body    so           1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl = vol.tokenlist(pos=False)\n",
    "tl.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Section arguments are valid here: 'header', 'body', 'footer', 'all', and 'group'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>lowercase</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">9</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">header</th>\n",
       "      <th>'s</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>and</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>austen</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jane</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prejudice</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        count\n",
       "page section lowercase       \n",
       "9    header  's             1\n",
       "             and            1\n",
       "             austen         1\n",
       "             jane           1\n",
       "             prejudice      1"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl = vol.tokenlist(section=\"header\", case=False, pos=False)\n",
    "tl.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also drop the section index altogether if you're content with the default 'body'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>lowercase</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <th>suppose</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <th>would</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                count\n",
       "page lowercase       \n",
       "247  suppose        1\n",
       "76   would          2"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.tokenlist(drop_section=True, case=False, pos=False).sample(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MultiIndex makes it easy to slice the results, and it is althogether more memory-efficient. For example, to return just the nouns (`NN`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>body</th>\n",
       "      <th>prejudiceJane</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>body</th>\n",
       "      <th>Volume</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>body</th>\n",
       "      <th>vol</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <th>body</th>\n",
       "      <th>./■</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            count\n",
       "page section token               \n",
       "1    body    prejudiceJane      1\n",
       "9    body    Volume             1\n",
       "10   body    vol                3\n",
       "12   body    ./■                1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl = vol.tokenlist()\n",
    "tl.xs('NN', level='pos').head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are new to Pandas DataFrames, you might find it easier to learn by converting the index to columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>page</th>\n",
       "      <th>section</th>\n",
       "      <th>token</th>\n",
       "      <th>pos</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>body</td>\n",
       "      <td>prejudiceJane</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>9</td>\n",
       "      <td>body</td>\n",
       "      <td>Volume</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>10</td>\n",
       "      <td>body</td>\n",
       "      <td>vol</td>\n",
       "      <td>NN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>12</td>\n",
       "      <td>body</td>\n",
       "      <td>./■</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>12</td>\n",
       "      <td>body</td>\n",
       "      <td>/</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43178</th>\n",
       "      <td>297</td>\n",
       "      <td>body</td>\n",
       "      <td>spite</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43187</th>\n",
       "      <td>297</td>\n",
       "      <td>body</td>\n",
       "      <td>uncle</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43191</th>\n",
       "      <td>297</td>\n",
       "      <td>body</td>\n",
       "      <td>warmest</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43195</th>\n",
       "      <td>297</td>\n",
       "      <td>body</td>\n",
       "      <td>wife</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43226</th>\n",
       "      <td>305</td>\n",
       "      <td>body</td>\n",
       "      <td>NON-RECEIPT</td>\n",
       "      <td>NN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7224 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       page section          token pos  count\n",
       "3         1    body  prejudiceJane  NN      1\n",
       "19        9    body         Volume  NN      1\n",
       "40       10    body            vol  NN      3\n",
       "51       12    body            ./■  NN      1\n",
       "53       12    body              /  NN      1\n",
       "...     ...     ...            ...  ..    ...\n",
       "43178   297    body          spite  NN      1\n",
       "43187   297    body          uncle  NN      1\n",
       "43191   297    body        warmest  NN      1\n",
       "43195   297    body           wife  NN      1\n",
       "43226   305    body    NON-RECEIPT  NN      1\n",
       "\n",
       "[7224 rows x 5 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simpler_tl = df.reset_index()\n",
    "simpler_tl[simpler_tl.pos == 'NN']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you prefer not to use Pandas, you can always convert the object, with methods like `to_dict` and `to_csv`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'page,section,token,pos,count\\n1,body,Austen,.,1\\n1,body,Pride,NNP,1\\n1,body,and,CC,1\\n'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl[:3].to_csv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get just the unique tokens, `Volume.tokens` provides them as a set. Here I select a specific page for brevity and a minimum count, but you can run the method without arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'\"', ',', '.', 'You', 'been', 'have', 'his', 'in', 'of', 'the', 'you'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.tokens(page_select=21, min_count=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to token lists, you can also access other section features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tokenCount</th>\n",
       "      <th>lineCount</th>\n",
       "      <th>emptyLineCount</th>\n",
       "      <th>capAlphaSeq</th>\n",
       "      <th>sentenceCount</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305</th>\n",
       "      <td>49</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>306 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      tokenCount  lineCount  emptyLineCount  capAlphaSeq  sentenceCount\n",
       "page                                                                   \n",
       "1              4          1               0            1              1\n",
       "2             15         10               4            2              1\n",
       "3              0          0               0            0              0\n",
       "4              0          0               0            0              0\n",
       "5              0          0               0            0              0\n",
       "...          ...        ...             ...          ...            ...\n",
       "302            0          0               0            0              0\n",
       "303            0          0               0            0              0\n",
       "304            0          0               0            0              0\n",
       "305           49         11               2            3              3\n",
       "306            2          3               1            1              1\n",
       "\n",
       "[306 rows x 5 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vol.section_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chunking\n",
    "\n",
    "If you're working in an instance where you hope to have comparably sized document units, you can use 'chunking' to roll pages into chunks that aim for a specific length. e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                              count\n",
      "chunk section token      pos       \n",
      "5     body    husbands   NNS      3\n",
      "2     body    frequently RB       3\n",
      "              domestic   JJ       3\n",
      "3     body    :          :       10\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chunk</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>10054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>10065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>12327</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       count\n",
       "chunk       \n",
       "1      12453\n",
       "2       9888\n",
       "3       9887\n",
       "4      10129\n",
       "5      10054\n",
       "6      10065\n",
       "7      12327"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_chunk = vol.tokenlist(chunk=True, chunk_target=10000)\n",
    "print(by_chunk.sample(4))\n",
    "# Count words per chunk\n",
    "by_chunk.groupby(level='chunk').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiprocessing\n",
    "\n",
    "For large jobs, you'll want to use multiprocessing or multithreading to speed up your process. This is left up to your preferred method, either within Python or by spawning multiple scripts from the command line. Here are two approaches that I like.\n",
    "\n",
    "#### Dask\n",
    "\n",
    "Dask offers easy multithreading (shared resources) and multiprocessing (separate processes) in Python, and is particularly convenient because it includes a subset of Pandas DataFrames.\n",
    "\n",
    "Here is a minimal example, that lazily loads token frequencies from a list of volume IDs, and counts them up by part of speech tag.\n",
    "\n",
    "```python\n",
    "import dask.dataframe as dd\n",
    "from dask import delayed\n",
    "\n",
    "def get_tokenlist(vol):\n",
    "    ''' Load a one volume feature reader, get that volume, and return its tokenlist '''\n",
    "    return FeatureReader(ids=[volid]).first().tokenlist()\n",
    "\n",
    "delayed_dfs = [delayed(get_tokenlist)(volid) for volid in volids]\n",
    "\n",
    "# Create a dask\n",
    "ddf = (dd.from_delayed(delayed_dfs)\n",
    "         .reset_index()\n",
    "         .groupby('pos')[['count']]\n",
    "         .sum()\n",
    "      )\n",
    "\n",
    "# Run processing\n",
    "ddf.compute()\n",
    "```\n",
    "\n",
    "Here is an example of 78 volumes being processed in 24 seconds with 31 threads:\n",
    "\n",
    "![Counting POS in 78 books about knitting](data/dask-progress.png)\n",
    "\n",
    "This example used multithreading. Due to the nature of Python, certain functions won't parallelize well. In our case, the part where the JSON is read from the file and converted to a DataFrame (the light green parts of the graphic) won't speed up because Python dicts lock the Global Interpreter Lock (GIL). However, because Pandas releases the GIL, nearly everything you do after parsing the JSON will be very quick.\n",
    "\n",
    "To better understand what happens when `ddf.compute()`, here is a graph for 4 volumes:\n",
    "\n",
    "![](data/dask-graph.png)\n",
    "\n",
    "\n",
    "#### GNU Parallel\n",
    "As an alternative to multiprocessing in Python, my preference is to have simpler Python scripts and to use GNU Parallel on the command line. To do this, you can set up your Python script to take variable length arguments of feature file paths, and to print to stdout.\n",
    "\n",
    "This psuedo-code shows how that you'd use parallel, where the number of parallel processes is 90% the number of cores, and 50 paths are sent to the script at a time (if you send too little at a time, the initialization time of the script can add up).\n",
    "\n",
    "```bash\n",
    "find feature-files/ -name '*json.bz2' | parallel --eta --jobs 90% -n 50 python your_script.py >output.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Notes\n",
    "\n",
    "### Installing the development version\n",
    "\n",
    "    git clone https://github.com/htrc/htrc-feature-reader.git\n",
    "    cd htrc-feature-reader\n",
    "    python setup.py install\n",
    "\n",
    "### Iterating through the JSON files\n",
    "\n",
    "If you need to do fast, highly customized processing without instantiating Volumes, FeatureReader has a convenient generator for getting the raw JSON as a Python dict: `fr.jsons()`. This simply does the file reading, optional decompression, and JSON parsing.\n",
    "\n",
    "### Downloading files within the library\n",
    "\n",
    "`utils` includes an Rsyncing utility, `download_file`. This requires Rsync to be installed on your system.\n",
    "\n",
    "**Usage:**\n",
    "\n",
    "Download one file to the current directory:\n",
    "    \n",
    "```\n",
    "utils.download_file(htids='nyp.33433042068894')\n",
    "```\n",
    "\n",
    "Download multiple files to the current directory:\n",
    "\n",
    "```\n",
    "ids = ['nyp.33433042068894', 'nyp.33433074943592', 'nyp.33433074943600']\n",
    "utils.download_file(htids=ids)\n",
    "```\n",
    "\n",
    "Download file to `/tmp`:\n",
    "```\n",
    "utils.download_file(htids='nyp.33433042068894', outdir='/tmp')\n",
    "```\n",
    "\n",
    "Download file to current directory, keeping pairtree directory structure,\n",
    "i.e. `./nyp/pairtree_root/33/43/30/42/06/88/94/33433042068894/nyp.33433042068894.json.bz2`:\n",
    "\n",
    "```\n",
    "utils.download_file(htids='nyp.33433042068894', keep_dirs=True)\n",
    "    ```\n",
    "\n",
    "### Getting the Rsync URL\n",
    "\n",
    "If you have a HathiTrust Volume ID and want to be able to download the features for a specific book, `hrtc_features.utils` contains an [id_to_rsync](http://htrc.github.io/htrc-feature-reader/htrc_features/utils.m.html#htrc_features.utils.id_to_rsync) function. This uses the [pairtree](http://pythonhosted.org/Pairtree/) library but has a fallback written with that library is not installed, since it isn't compatible with Python 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from htrc_features import utils\n",
    "utils.id_to_rsync('miun.adx6300.0001.001')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [ID to Rsync notebook](examples/ID_to_Rsync_Link.ipynb) for more information on this format and on Rsyncing lists of urls.\n",
    "\n",
    "There is also a command line utility installed with the HTRC Feature Reader:\n",
    "\n",
    "```bash\n",
    "$ htid2rsync miun.adx6300.0001.001\n",
    "miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced Features\n",
    "\n",
    "In the beta Extracted Features release, schema 2.0, a few features were separated out to an advanced files. However, *this designation is no longer present starting with schema 3.0*, meaning information like `beginLineChars`, `endLineChars`, and `capAlphaSeq` are always available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 0, 0, 0, 0, 0, 0, 4, 1]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is the longest sequence of capital letter on each page?\n",
    "vol.cap_alpha_seqs()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         count\n",
      "page section place char       \n",
      "2    body    end   -         1\n",
      "                   :         1\n",
      "                   I         1\n",
      "                   f         1\n",
      "                   t         1\n"
     ]
    }
   ],
   "source": [
    "end_line_chars = vol.end_line_chars()\n",
    "print(end_line_chars.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         count\n",
      "page section place char       \n",
      "45   body    end   !         1\n",
      "75   body    end   !         1\n",
      "77   body    end   !         1\n",
      "91   body    end   !         1\n",
      "92   body    end   !         1\n"
     ]
    }
   ],
   "source": [
    "# Find pages that have lines ending with \"!\"\n",
    "idx = pd.IndexSlice\n",
    "print(end_line_chars.loc[idx[:,:,:,'!'],].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing\n",
    "\n",
    "This library is meant to be compatible with Python 3.2+ and Python 2.7+. Tests are written for py.test and can be run with `setup.py test`, or directly with `python -m py.test -v`.\n",
    "\n",
    "If you find a bug, leave an issue on the issue tracker, or contact Peter Organisciak at `organisciak+htrc@gmail.com`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}