Finding-unpublished-works-entering-public-domain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding unpublished works that might be entering the public domain on 1 January 2019\n",
    "\n",
    "Changes to Australian copyright legislation mean that many unpublished resources will be entering the public domain on 1 January 2019. This notebook attempts to harvest the details of some of these resources from Trove.\n",
    "\n",
    "As with most things involving copyright, there's no real way to be certain what will be entering the public domain. The main problem is that if there's no known author then the copyright period depends on if and when the work was 'made public'. Add to that general issues around the accuracy and completeness of the metadata and all I can really do is create a list of *some* of the things which could *potentially* be entering the public domain based on the available metadata.\n",
    "\n",
    "The basic methodology is:\n",
    "\n",
    "* Search in Trove's 'Diaries, letters, archives' zone for ['Unpublished' Australian materials](https://trove.nla.gov.au/collection/result?l-format=Unpublished&q=&l-australian=y)\n",
    "* For each record check to see if there are any listed creators.\n",
    "* If there are creators, look to see if they have a death date and if that date is before 1949\n",
    "* If all creators died before 1949 then save the item metadata\n",
    "* If there are no creators, look to see if the creation date of the item is before 1949, if so save the metadata\n",
    "\n",
    "If you just want the data, here's a [CSV file you can download](unpublished_works_entering_pd_20181006.csv). Look below for a preview.\n",
    "\n",
    "If you want to play with the data a bit, [here's another notebook](Exploring-unpublished-works-entering-public-domain.ipynb) with a few ideas.\n",
    "\n",
    "For more information on the changes see the NSLA guide to [Preparing for copyright term changes in 2019](https://www.nsla.org.au/sites/default/files/documents/nsla.copyright-preparing-changes-2019.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import requests_cache\n",
    "from IPython.display import HTML, display\n",
    "from requests.adapters import HTTPAdapter\n",
    "from requests.packages.urllib3.util.retry import Retry\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "s = requests_cache.CachedSession()\n",
    "retries = Retry(total=10, backoff_factor=0.2, status_forcelist=[500, 502, 503, 504])\n",
    "s.mount(\"http://\", HTTPAdapter(max_retries=retries))\n",
    "s.mount(\"https://\", HTTPAdapter(max_retries=retries))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Load variables from the .env file if it exists\n",
    "# Use %%capture to suppress messages\n",
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Insert your Trove API key\n",
    "API_KEY = \"YOUR API KEY\"\n",
    "\n",
    "if os.getenv(\"TROVE_API_KEY\"):\n",
    "    API_KEY = os.getenv(\"TROVE_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Harvest the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "api_url = \"http://api.trove.nla.gov.au/v2/result\"\n",
    "# Insert your Trove API key where indicated\n",
    "params = {\n",
    "    \"q\": \" \",\n",
    "    \"zone\": \"collection\",\n",
    "    \"encoding\": \"json\",\n",
    "    \"l-format\": \"Unpublished\",\n",
    "    \"l-australian\": \"y\",\n",
    "    \"include\": \"holdings\",\n",
    "    \"key\": API_KEY,\n",
    "    \"n\": \"100\",\n",
    "    \"bulkHarvest\": \"true\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# How many things are we processing?\n",
    "response = s.get(api_url, params=params)\n",
    "data = response.json()\n",
    "total = int(data[\"response\"][\"zone\"][0][\"records\"][\"total\"])\n",
    "print(total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def check_creators(creators):\n",
    "    \"\"\"\n",
    "    Make sure all creators have a death date before 1949.\n",
    "    \"\"\"\n",
    "    opening = False\n",
    "    count = 0\n",
    "    for creator in creators:\n",
    "        year = get_latest_year(creator)\n",
    "        if year and int(year) < 1949:\n",
    "            count += 1\n",
    "    if len(creators) == count:\n",
    "        opening = True\n",
    "    return opening\n",
    "\n",
    "\n",
    "def check_date(issued):\n",
    "    \"\"\"\n",
    "    Check if the latest issued date is before 1949.\n",
    "    \"\"\"\n",
    "    opening = False\n",
    "    year = get_latest_year(issued)\n",
    "    if year and int(year) < 1949:\n",
    "        opening = True\n",
    "    return opening\n",
    "\n",
    "\n",
    "def get_latest_year(value):\n",
    "    \"\"\"\n",
    "    Get a year from the end of a string.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        year = re.search(r\"\\b(\\d{4})$\", value).group(1)\n",
    "    except (AttributeError, TypeError):\n",
    "        year = None\n",
    "    return year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def harvest(max_records=None):\n",
    "    items = []\n",
    "    start = \"*\"\n",
    "\n",
    "    with tqdm(total=total) as pbar:\n",
    "        while start:\n",
    "            params[\"s\"] = start\n",
    "            response = s.get(api_url, params=params)\n",
    "            # print(response.url)\n",
    "            data = response.json()\n",
    "            for record in data[\"response\"][\"zone\"][0][\"records\"][\"work\"]:\n",
    "                opening = False\n",
    "                creators = record.get(\"contributor\")\n",
    "                issued = record.get(\"issued\")\n",
    "                if creators:\n",
    "                    opening = check_creators(creators)\n",
    "                elif issued:\n",
    "                    opening = check_date(str(issued))\n",
    "                if opening:\n",
    "                    try:\n",
    "                        creator = \" | \".join(creators)\n",
    "                    except TypeError:\n",
    "                        creator = creators\n",
    "                    try:\n",
    "                        nuc = record[\"holding\"][0][\"nuc\"]\n",
    "                    except KeyError:\n",
    "                        nuc = None\n",
    "                    item = {\n",
    "                        \"id\": record[\"id\"],\n",
    "                        \"title\": record[\"title\"],\n",
    "                        \"creator\": creator,\n",
    "                        \"date\": issued,\n",
    "                        \"trove_url\": record[\"troveUrl\"],\n",
    "                        \"nuc\": nuc,\n",
    "                    }\n",
    "                    items.append(item)\n",
    "            try:\n",
    "                start = data[\"response\"][\"zone\"][0][\"records\"][\"nextStart\"]\n",
    "            except KeyError:\n",
    "                start = None\n",
    "            pbar.update(100)\n",
    "            # Stop iteration once max number of records inspected (mainly for testing)\n",
    "            if max_records and pbar.n >= max_records:\n",
    "                break\n",
    "            if not response.from_cache:\n",
    "                time.sleep(0.2)\n",
    "    return items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "items = harvest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the results to a dataframe and have a look inside"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(items)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "# How many items are there?\n",
    "df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the results as a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "date_str = datetime.datetime.now().strftime(\"%Y%m%d\")\n",
    "csv_file = \"unpublished_works_entering_pd_{}.csv\".format(date_str)\n",
    "df.to_csv(csv_file, index=False)\n",
    "# Make a download link\n",
    "display(HTML('<a target=\"_blank\" href=\"{}\">Download CSV file</a>'.format(csv_file)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ignore -- this is just used for testing in development\n",
    "if os.getenv(\"GW_STATUS\") == \"dev\":\n",
    "    fa_test = harvest(200)\n",
    "    df_test = pd.DataFrame(fa_test)\n",
    "    assert not df_test.empty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherratt](https://timsherratt.org/) for the [GLAM Workbench](https://glam-workbench.net/). Support this project by becoming a [GitHub sponsor](https://github.com/sponsors/wragge)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}