Using-TroveHarvester-to-get-newspaper-articles-in-bulk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Using the Trove Newspaper Harvester on the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<p>If you haven't used one of these notebooks before, they're basically web pages in which you can write, edit, and run live code. They're meant to encourage experimentation, so don't feel nervous. Just try running a few cells and see what happens!.</p>\n",
    "\n",
    "<p>\n",
    "    Some tips:\n",
    "    <ul>\n",
    "        <li>Code cells have boxes around them.</li>\n",
    "        <li>To run a code cell click on the cell and then hit <b>Shift+Enter</b>. The <b>Shift+Enter</b> combo will also move you to the next cell, so it's a quick way to work through the notebook.</li>\n",
    "        <li>While a cell is running a <b>*</b> appears in the square brackets next to the cell. Once the cell has finished running the asterisk will be replaced with a number.</li>\n",
    "        <li>In most cases you'll want to start from the top of notebook and work your way down running each cell in turn. Later cells might depend on the results of earlier ones.</li>\n",
    "        <li>To edit a code cell, just click on it and type stuff. Remember to run the cell once you've finished editing.</li>\n",
    "    </ul>\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The Trove Newspaper & Gazette Harvester is a command line tool and Python package that helps you download large quantities of digitised articles from [Trove](http://trove.nla.gov.au/)'s newspapers and gazettes. \n",
    "\n",
    "If you'd like to install and run the TroveHarvester on your local system see [the package documentation](https://wragge.github.io/trove-newspaper-harvester/).\n",
    "\n",
    "This notebook demonstrates the basic use of the command line tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Getting started\n",
    "\n",
    "Run the cell below to set some things up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "If you were running TroveHarvester on your local system, you could access the basic help information by entering this on the command line:\n",
    "\n",
    "``` bash\n",
    "troveharvester -h\n",
    "```\n",
    "\n",
    "In this notebook environment you need start with a `!` to run the command-line TroveHarvester script. Click on the cell below and hit `Shift+Enter` to view the TroveHarvester's basic options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: troveharvester [-h] {start,restart,report} ...\n",
      "\n",
      "positional arguments:\n",
      "  {start,restart,report}\n",
      "    start               start a new harvest\n",
      "    restart             restart an unfinished harvest\n",
      "    report              report on a harvest\n",
      "\n",
      "options:\n",
      "  -h, --help            show this help message and exit\n"
     ]
    }
   ],
   "source": [
    "!troveharvester -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Before we go any further you should make sure you have a Trove API key. For non-commercial projects, you just  fill out a simple form and your API key is generated instantly. Follow the instructions in the Trove Help to [obtain your own Trove API Key](https://trove.nla.gov.au/about/create-something/using-api).\n",
    "\n",
    "Once you've created a key, you can access it at any time on the 'For developers' tab of your Trove user profile.\n",
    "\n",
    "Copy your API key now, and paste it in the cell below, between the quotes. Then hit `Shift+Enter` to save your key as a variable called `api_key`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Insert your Trove API key\n",
    "API_KEY = \"YOUR API KEY\"\n",
    "\n",
    "# Use api key value from environment variables if it is available\n",
    "if os.getenv(\"TROVE_API_KEY\"):\n",
    "    API_KEY = os.getenv(\"TROVE_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What do you want to harvest?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The TroveHarvester translates queries from the Trove web interface into something that the API can understand. So all you need to do is construct your query using the web interface. Once you're happy with the results you're getting just copy the url.\n",
    "\n",
    "Once you've constructed your query and copied the url, paste it between the quotes in the cell below and hit `Shift+Enter` to save it as a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "query = \"https://trove.nla.gov.au/search/category/newspapers?keyword=wragge%201902&l-artType=newspapers&l-state=Queensland&l-title=840\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Running the harvest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default the harvester will save all the article metadata to a CSV formatted file called `results.csv`. If you'd like to save the full OCRd text of all the articles, just add the `--text` parameter. If you'd like copies of the articles as JPG images, add the `--image` option. You can also save PDFs of all the articles by adding the `--pdf` parameter, but be warned that this will slow down your harvest considerably and can consume large amounts of disk space. So use with care!\n",
    "\n",
    "Now we're ready to start the harvest! Just run the code in the cell below. You can delete the `--text` parameter if you're not interested in saving the full text of every article. You could also try adding `--image` to save articles as images (this will slow down the harvest)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!troveharvester start \"$query\" $API_KEY --text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll know the harvest is finished when the asterisk in the square brackets of the cell above turns into a number. \n",
    "\n",
    "If the harvest stops before it's finished, you can restart it by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!troveharvester restart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to check the details of a finished harvest, just run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!troveharvester report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Harvest results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "You can also view the results of the harvest in the `data` directory.\n",
    "\n",
    "See the GLAM Workbench for detailed information about [the files included in each harvest](https://glam-workbench.net/trove-harvester/#your-harvested-data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Download your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    If you're using this notebook through the MyBinder service (it'll say `mybinder` in the url) make sure you <b>download your data</b> once the harvest is finished as it will not be preserved!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once your harvest is complete, you probably want to download the results. The easiest way to do this is to zip up the results folder. Run the following cell to zip up the folder containing all the data from your most recent harvest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List all the harvest folders and sort by date\n",
    "harvests = sorted(\n",
    "    [d for d in os.listdir(\"data\") if os.path.isdir(os.path.join(\"data\", d))]\n",
    ")\n",
    "# Get the most recent\n",
    "timestamp = harvests[-1]\n",
    "# Zip up the folder\n",
    "shutil.make_archive(\n",
    "    os.path.join(\"data\", timestamp), \"zip\", os.path.join(\"data\", timestamp)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once your zip file has been created you can find it in the data directory. Or just run the cell below to create a handy download link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        f'<a href=\"data/{timestamp}.zip\" download=\"{timestamp}.zip\">data/{timestamp}.zip</a>'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Explore your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Have a look at the [Exploring your TroveHarvest data](2-Exploring-your-TroveHarvester-data.ipynb) for some ideas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherratt](https://timsherratt.au) for the [GLAM Workbench](https://glam-workbench.net/).  \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "rocrate": {
   "author": [
    {
     "name": "Sherratt, Tim",
     "orcid": "https://orcid.org/0000-0001-7956-4498"
    }
   ],
   "category": "harvesting",
   "description": "The Trove Newspaper & Gazette Harvester is a command line tool and Python package that helps you download large quantities of digitised articles from Trove's newspapers and gazettes. This notebook demonstrates the basic use of the command line tool.",
   "mainEntityOfPage": "https://glam-workbench.net/trove-harvester/using-troveharvester/",
   "name": "Using the Trove Newspaper Harvester on the command line",
   "position": 3
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}