Skip to content

Latest commit

 

History

History
92 lines (78 loc) · 7.85 KB

README.md

File metadata and controls

92 lines (78 loc) · 7.85 KB
Raw

Description

A parser that reads HTMLs downloaded from Pacer.gov and breaks them up into JSON format.

Usage

To run the parser:

python parse_pacer.py [OPTIONS] INPATH OUTPATH

Arguments

  • INPATH: Relative path to the folder where HTMLs will be read, e.g. ../../data/pacer/ilnd/html.
  • OUTPATH: Relative path to the folder where JSONs will be written, e.g. ../../data/pacer/ilnd/json.

If you are using the parser in conjunction with SCALES's Pacer scraper, you will likely want your input and output directories to be the scraper-generated html and json folders within your chosen court directory, as outlined here.

Options

  • -c, --court TEXT (defaults to none) The standard abbreviation for the district court being parsed, e.g. ilnd. If not specified, and if using the directory structure mentioned above, the parser will inference the court abbreviation from the parent folder.
  • -d, --debug (flag) Turns off concurrency in the parser. Useful for ensuring that error traces are printed properly.
  • -f, --force-rerun (flag) Tells the parser to process HTMLs even when their corresponding JSONs already exist. Useful for obtaining fresh parses after scraping updates to existing dockets.
  • -nw, --n-workers INTEGER (defaults to 16) Number of concurrent workers to run simultaneously - i.e., no. of simultaneous parses running.

Shell scripts

Two shell scripts, parse_all.sh and parse_subset.sh, are provided for batch runs across multiple court directories. To run them:

sh parse_all.sh INPATH [OPTIONS]
sh parse_subset.sh INPATH -s STARTDIR -e ENDDIR [OPTIONS]

where INPATH is the relative path to a parent folder containing multiple court directories, STARTDIR and ENDDIR define the inclusive alphabetical range of court directories to parse (e.g. nyed through nywd), and OPTIONS are any command-line options you would like to pass through to parse_pacer.py (e.g. --debug, --force-rerun, --n-workers).

Note: each court directory in the batch must include an HTML folder for input and a JSON folder for output, as is true in the scraper-generated directory structure.

JSON Schema

The following fields are inferenced from the filepath:

  • case_id (string) - Pacer's case ID, which has the form O:YY-TY-##### (where O is a court office code, YY is a year, TY is the case type, and ##### is a numeric identifier associated with this case)
  • case_type (string) - usually 'cr' (criminal) or 'cv' (civil); other types are acceptable ('mc', 'bk'...), but they will result in an incomplete parse
  • download_court (string) - read from the command line if passed in with the -c option
  • ucid (string) - SCALES's case ID (stands for 'unique case id'), generated by prepending the court abbreviation to the Pacer case ID and used to ensure that cases with identical Pacer IDs from different districts can be distinguished from one another

The following fields are pulled from the header of the Pacer docket:

  • header_case_id (string) - similar to case_id, but pulled from the docket itself rather than the filepath
  • case_name (string)
  • filing_date (string)
  • terminating_date (string)
  • case_status (string) - 'open' if a terminating date is listed, else 'closed'
  • judge (string)
  • referred_judge (string) - only present when the case was referred to a second judge
  • nature_suit (string) - civil cases only
  • jury_demand (string) - civil cases only
  • cause (string) - civil cases only
  • jurisdiction (string) - civil cases only
  • monetary_demand (string) - civil cases only
  • lead_case_id (string) - only present when the case is part of multi-district litigation (MDL)
  • other_court (string) - only present when another case ID is provided by Pacer as 'Case in other court'; doesn't pick up all alternate case IDs (e.g. appeals court case numbers)
  • case_flags (list of strings) - only present when there are flags listed in the upper right corner of the Pacer docket
  • mdl_code (integer)

The following fields are pulled from the body of the Pacer docket:

  • plaintiffs, defendants, bankruptcy_parties, other_parties, misc_participants (dictionary) - each key is the name of a participant in the case, and each value is a dictionary with the following structure:
    • counsel (dictionary): - each key is the name of a lawyer representing this participant, and each value is a dictionary with the following structure:
      • office (string)
      • is_lead_attorney (boolean)
      • is_pro_hac_vice (boolean)
      • additional_info (dictionary) - keys vary according to the information in the docket ('Designation,' 'Bar Status,' etc.)
    • is_pro_se (boolean)
    • roles (list of strings) - 'Plaintiff,' 'Petitioner,' 'Movant,' etc.
  • pending_counts, terminated_counts (dictionary) - criminal cases only; each key is the name of a party who was charged with a criminal count, and each value is a list in which each element has the following dictionary structure:
    • counts (string)
    • disposition (string)
  • complaints (dictionary) - certain criminal cases only; each key is the name of a party who was charged with a criminal count, and each value is the statute(s) specified as the basis of the charges
  • docket_available (boolean)
  • docket (list of dictionaries) - contains one item per docket entry, structured as follows:
    • date_filed (string)
    • ind (string) - Pacer's numerical index for this entry (can be an empty string, as not all Pacer entries are numbered)
    • docket_text (string)
    • documents (dictionary) - each key is either a non-zero attachment number or '0' for the main document, and each value is a dictionary with the following structure:
      • url (string) - the Pacer URL for this document
      • span (dictionary) - the starting and ending indices (within docket_text) of the hyperlink to the document, formatted as a dictionary with keys start and end
    • edges (list of tuples) - each element is a three-value tuple (encoded in graph-edge format) representing a hyperlink between two docket entries, with the first value encoding the index of the source entry within docket, the second value encoding the index of the target entry, and the third value encoding the starting and ending indices of the hyperlink within docket_text (as specified in span above)

The following fields are not pulled directly from the Pacer docket, and are primarily meant for internal use:

  • mdl_id_source (string) - the origin of mdl_code (either 'lead_case_id' or 'flags')
  • is_mdl (boolean) - true if mdl_code is non-null or if the case has any MDL flags
  • is_multi (boolean) - true if is_mdl is true or if any of lead_case_id, member_case_key, or other_court is non-null
  • member_case_key (string) - a UCID-formatted version of lead_case_id (or a copy of ucid if this case is a lead case); used to write MDL-related data to an external file for improved performance
  • source (string) - used to distinguish between JSONs from this parser and similarly-formatted JSONs from other sources); if generated by this parser, will always be 'pacer'
  • download_url (string) - the URL from which this HTML was downloaded; only present if parsing an HTML from the SCALES scraper

The following fields are pulled from the 'Transaction Receipt' at the bottom of the Pacer docket:

  • billable_pages (integer)
  • cost (float)
  • download_timestamp (string)
  • n_docket_reports (integer) - the number of times the SCALES scraper has modified this docket (1 if there have never been updates, >1 if new docket entries have been added after the initial download)
  • pacer_case_id (integer) - the unique numerical ID that Pacer uses internally to identify this document (pulled from Pacer's XML responses to user queries; not visible on the docket sheet itself)