diff --git a/README.md b/README.md index 28decfc2..dd7cacea 100644 --- a/README.md +++ b/README.md @@ -14,19 +14,22 @@ Book Usage Data Workflows provides Apache Airflow workflows for fetching, proces A telescope a type of workflow used to ingest data from different data sources, and to run workflows that process and output data to other places. Workflows are built on top of Apache Airflow's DAGs. -The workflows include: Directory of Open Access Books (DOAB), Google Analytics, Google Books, JSTOR, Oapen IRUS UK, -Oapen Metadata, Onix, UCL Discovery and an Onix Workflow for combining all of this data. +The workflows include: Google Analytics, Google Books, JSTOR, IRUS Fulcrum, IRUS OAPEN, +Onix, UCL Discovery and an Onix Workflow for combining all of this data. | Telescope Workflow | Description | | ------------- | ------------- | -| Directory of Open Access Books (DOAB) | DOAB is a community-driven discovery service that indexes and provides access to scholarly, peer-reviewed open access books and helps users to find trusted open access book publishers. All DOAB services are free of charge and all data is freely available. | +| Crossref Events | Crossref Event Data captures discussion on scholarly content and acts as a hub for the storage and distribution of this data. An event may be a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media. | +| Crossref Metadata | Crossref is a non-for-profit membership organisation working on making scholarly communications better. It is an official Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation. They provide metadata for every DOI that is registered with Crossref. | | Google Analytics | Google Analytics is a web-based service that allows groups to track usage of their web properties. It offers vistor counts, statistics, and other breakdowns such as country or origin for visitors. If publishers or partners already have Google Analytics already setup of their website, this usage data is able to be ingested | -| Google Books | The Google Books Partner program enables selling books through the Google Play store and offering a preview on Google books. As a publisher it is possible to download reports on Google Books data, currently there are 3 report types available (sales summary, sales transaction and traffic) of which we use the latter 2 |the latter 2 -| JSTOR | JSTOR provides publisher usage reports, the reports offer details about the use of journal or book content by institution, and country. Journal reports also include usage by issue and article. Usage is aligned with the COUNTER 5 standard of Item Requests (views + downloads) | -| Oapen IRUS UK | IRUS-UK provides OAPEN COUNTER standard access reports. Almost all books on OAPEN are provided as a whole book PDF file. The reports show access figures for each month as well as the location of the access. Since the location info includes an IP-address, the original data is handled only from within the OAPEN Google Cloud project. | -| Oapen Metadata | The OAPEN repository hosts more than 15,000 peer reviewed academic books. OAPEN is working with the Directory of Open Access Books (DOAB), which are used by library catalogues making their books more discoverable. All their metadata feeds are available under a CC0 1.0 license. All of the OA books hosted on OAPEN are registered to DOAB. There are around 1300 titles that are hosted on the OAPEN repository but are not registered to DOAB, as those titles are published under a license that does not allow reuse | +| Google Books | The Google Books Partner program enables selling books through the Google Play store and offering a preview on Google books. As a publisher it is possible to download reports on Google Books data, currently there are 3 report types available (sales summary, sales transaction and traffic) of which we use the latter 2 | +| IRUS Fulcrum | IRUS provides COUNTER standard access reports for books hosted on the Fulcrum platform. The reports show access figures for each month and the country of usage | +| IRUS OAPEN | IRUS provides COUNTER standard access reports for books hosted on the OAPEN platform. Almost all books on OAPEN are provided as a whole book PDF file. The reports show access figures for each month as well as the location of the access. Since the location info includes an IP-address, the original data is handled only from within the OAPEN Google Cloud project | +| JSTOR | JSTOR provides publisher usage reports, the reports offer details about the use of journal or book content by institution, and country. Journal reports also include usage by issue and article. Usage is aligned with the COUNTER 5 standard of Item Requests (views + downloads) | +| OAPEN Metadata | The OAPEN Library hosts more than 18,000 Open Access books. OAPEN enables libraries and aggregators to use the metadata of all available titles in the OAPEN Library, made available under a CC0 1.0 license. The metadata is available in different formats and the OAPEN metadata telescope harvests the data in XML format | | Onix | ONIX is a standard format that book publishers use to share information about the books that they have published. Publishers that have ONIX feeds are given credentials and access to their own upload folder on the Mellon SFTP server. The publisher uploads their ONIX feed to their upload folder on a weekly, fortnightly or monthly basis. The ONIX telescope downloads, transforms (with the ONIX parser Java command line tool) and then loads the ONIX data into BigQuery for further processing | -| UCL Discovery | UCL Discovery is UCL's open access repository, showcasing and providing access to the full texts of UCL research publications.The metadata for all eprints is obtained from their publicly available CSV file (https://discovery.ucl.ac.uk/cgi/search/advanced). | +| Thoth | Thoth is a free, open metadata service that publishers can choose to utilise as a solution for metadata storage. Thoth can provide metadata upon request in a number of formats. The Thoth telescope uses the Thoth Export API to download metadata in an ONIX format. | +| UCL Discovery | UCL Discovery is UCL's open access repository, showcasing and providing access to the full texts of UCL research publications.The metadata for all eprints is obtained from their publicly available CSV file (https://discovery.ucl.ac.uk/cgi/search/advanced) | ## Documentation diff --git a/docs/README.md b/docs/README.md index d7a60e4c..3be1ad19 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,5 +18,5 @@ 4) Activate virtual env: `conda activate ` 5) Add docs directory directory to Python path: `PYTHONPATH="${PYTHONPATH}:/oaebu-workflows/docs"` followed by `export PYTHONPATH`. Make sure to replace `pathtorepository` with the file path to the `oaebu-workflows` folder 6) Install documentation dependencies: `pip install -r requirements.txt` -7) Build documentation: `make html` -8) To view the documentation, open `_build/html/index.html`. +7) Build documentation: `make html` or `make latexpdf` +8) To view the documentation, open `_build/html/index.html` or `_build/latex/index.html`. diff --git a/docs/bad_project/help/faq.md b/docs/bad_project/help/faq.md deleted file mode 100644 index 4ef01e9d..00000000 --- a/docs/bad_project/help/faq.md +++ /dev/null @@ -1,98 +0,0 @@ -# Frequently Asked Questions - -This section of the help guide contains frequently asking questions about using the pilot project dashboards: -* [I’m having trouble seeing some visualisations](#im-having-trouble-accessing-the-dashboards-or-viewing-some-visualisations) -* [Some of the usage data is missing for a specific book title](#some-of-the-usage-data-is-missing-for-a-specific-book-title) -* [I can’t find a specific title](#i-cant-find-a-specific-title) -* [The title I am looking for is in some dashboards but not others](#the-title-i-am-looking-for-is-in-some-dashboards-but-not-others) -* [I can’t find a specific author in the ‘Select Author’ dropdown](#i-cant-find-a-specific-author-in-the-select-author-dropdown) -* [Why is there a relatively low number of ISBNs with usage data?](#why-is-there-a-relatively-low-number-of-isbns-with-usage-data) -* [I can only see data for one title/author/country/institution](#i-can-only-see-data-for-one-title-author-country-or-institution) -* [I’m trying to use ‘Options’ on a visualisation and getting an error](#im-trying-to-use-options-on-a-visualisation-and-getting-an-error) - -## I’m having trouble accessing the dashboards or viewing some visualisations - -If you are unable to access the dashboards, please check your web browser settings. if you have a setting in your web browser that 'blocks all cookies’, you will not be able to see the dashboards. To enable the cookies: -* Navigate to the ‘cookies in use’ section in your web browser and check for any blocked cookies. For example for the Chrome web browser, navigate to the 'padlock' icon next to the URL window, and click 'cookies' and then ‘Blocked’. If the cookies from 'es.io' or the link you use to access the dashboard are blocked, set them to 'allow'. - -Please use an up-to-date web browser. The dashboards should be compatible with the latest versions of Firefox, Chrome, Safari and Edge Chromium (see for specific versions of each browser). - -Please send any bug reports to: coki@curtin.edu.au - -## Some of the usage data is missing for a specific book title - -Some of the reasons why you may not be able to see usage data about a book title are: -* The book may not have been published as an open access eBook during that time period -* An ISBN may have been assigned, but the open access version may not yet have been released. -* Data sources have usage data for different time periods. For example, the University of Michigan Press has data sources for the following time periods: - * Usage data is from September 2020 for Google Books - * Usage data is from January 2018 onwards for JSTOR - * Usage data is from January 2018 to March 2020 for OAPEN (Counter 4), and from April 2020 for OAPEN (Counter 5) - * Event data is from May 2018 onwards for Crossref Events - * For publishers using Google Analytics data, data is only available for the previous 26 months from the activation of data ingestion [see the Google Analytic telescope documentation for more details](../../oaebu_workflows/telescopes/google_analytics). - -## I can’t find a specific title - -``` eval_rst -.. tip:: Try searching by author or ISBN if you can’t find a specific title -``` - -Data about open access eBooks comes from different sources, and sometimes there are inconsistencies in this data. You may not be able to find a specific title for some of the following reasons: -* A book has not yet been released, so the Title and ISBN might not be in the data sources yet. -* A book is released but very recently, so the Title and ISBN might not be in the most recent data source. -* Titles and ISBNs presented in the dashboards (aside from the manual data upload dashboards) are obtained from the publisher's ONIX feed. A book may have a slightly different title, for example with the word ‘The’ in a title from one data source, but not in the ONIX feed. In this case, try leaving out the word ‘The’ to see if the title you are looking for is returned. -* A book has a shorter title across different data sources. For example one data source has the title “Sounding Together”, while a different data source has the title “Sounding Together: Collaborative Perspectives on U.S. Music in the 21st Century” - -For publishers, the [Data Quality](../user_guides/data_qual_db.md) and [Overview](../user_guides/overview_db) dashboards may be helpful for identifying issues with data quality, and discrepancies between data from different sources. - -## The title I am looking for is in some dashboards but not others -The dashboards are built on usage data from different data sources. Sometimes a title is in one data source but not another, hence it may not appear in all Title/ISBN filters across all dashboards. - -## I can’t find a specific author in the ‘Select Author’ dropdown -If a book is not yet released, the Title and ISBN may be in the pilot project dashboards, but without the author details. - -## Why is there a relatively low number of ISBNs with usage data? -This can happen when only partial data is available for the current month, or for the last few months. Data is transferred from our data sources at different frequencies, with Crossref events data and ONIX feeds generally updated weekly, with usage data updated approximately monthly. Find out more in the user documentation about [Book Usage Data Workflows](../../oaebu_workflows/index). - -## I can only see data for one title, author, country or institution -There could be filters applied in the dashboard. First try clearing all filters applied in the filter dropdowns. For example, to remove all authors from the ‘Select Author(s)/Volume Editor(s)’ filter, click on the X in a circle icon. - -``` eval_rst -.. image:: ../images/faq1.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard is to use the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/faq2.png - :width: 500 -``` - -If filters are still active, reload the dashboard by clicking on its link in the navigation menu at the top right of any dashboard. - -``` eval_rst -.. image:: ../images/faq3.png - :width: 200 -``` - -## I’m trying to use ‘Options’ on a visualisation and getting an error -Some of the header text in the dashboards display the ‘Options’ symbol (three dots). However, these options are redundant because they are on a header rather than a visualisation itself. - -You may also see the following error ‘No requests logged. The element hasn't logged any requests (yet). This usually means that there was no need to fetch any data or that the element has not yet started fetching data.’ - -To see the options for a visualisation itself, click on the ‘Options’ symbol (three dots) beside the visualisation rather than beside the header. - -Click here: - -``` eval_rst -.. image:: ../images/faq4.png - :width: 500 -``` - -Don’t click here: - -``` eval_rst -.. image:: ../images/faq5.png - :width: 500 -``` diff --git a/docs/bad_project/help/glossary.md b/docs/bad_project/help/glossary.md index a2431994..fafe9cff 100644 --- a/docs/bad_project/help/glossary.md +++ b/docs/bad_project/help/glossary.md @@ -4,89 +4,85 @@ ``` eval_rst .. glossary:: + BAD + The Book Analytics Dashboard Project (2022-2025) - a term used to refer to the Mellon Foundation funded project that is focused on creating a sustainable open access Book focused analytics service [1]_ + COKI - Curtin Open Knowledge Initiative - a team of data scientists, software developers and researchers at Curtin University, Perth, Australia + Curtin Open Knowledge Initiative - a team of data scientists, software developers and researchers at Curtin University, Perth, Australia [2]_ Crossref - Crossref is a Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation, that makes metadata available for all DOIs registered with them [1]_ + Crossref is a Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation, that makes metadata available for all DOIs registered with them [3]_ dashboard A dashboard is an interactive, up-to-date page of visualisations that aggregate and summarise data from different sources data source A public or pilot project dashboard partner source of data about open access eBooks and their usage, such as views, downloads and online mentions - - DOAB - Directory of Open Access Books - - eISBN - An identifier for eBooks used by some publishers and platforms, specifically the manual EBSCO data source in the pilot project - - Elasticsearch - Elasticsearch is a search and analytics engine that enables fast searches for large sets of data [2]_ + + eBook + A book publication made available in electronic or digital form Google Books - Google Books provides paid and free (open access) eBooks [3]_ + Google Books provides paid and free (open access) eBooks [4]_ + + IRUS + IRUS – a service for capturing and processing institutional repository usage data, making it possible for institutional repositories and platforms to generate COUNTER compliant usage data [5]_ + + IRUS Fulcrum + Fulcrum is a “community-developed, open source platform for digital scholarship”. IRUS provides COUNTER standard usgae reports for eBooks hosted on the Fulcrum platform [6]_ + + IRUS OAPEN + IRUS provides COUNTER standard usage reports for eBooks hosted on the OAPEN library and platform [7]_ JSTOR - JSTOR is a digital library, which offers over 7000 open access eBooks [4]_ - - Kibana - Kibana is a free and open user interface to Elasticsearch.[5]_ Kibana is used in the 2020-2022 data dashboard pilot to analyse, search, interact with and visualize the Elasticsearch data + JSTOR is a digital library, which offers over 7000 open access eBooks [8]_ + + Looker Studio + A dashboarding solution provided by Google [9]_ OAeBU - Open Access eBook Usage (2020 - 2022) - a term used to refer to the Mellon Foundation funded pilot project Developing a Pilot Data Trust for Open Access Ebook Usage [6]_ + Open Access eBook Usage (2020 - 2022) - a term used to refer to the Mellon Foundation funded pilot project Developing a Pilot Data Trust for Open Access Ebook Usage [10]_ OAPEN - OAPEN is a not-for-profit organisation dedicated to open access, peer-reviewed books, operating three platforms: OAPEN Library; OAPEN Open Access Books Toolkit; and Directory of Open Access Books [7]_ - - OAPEN IRUS-UK - OAPEN IRUS-UK – a service for capturing and processing institutional repository usage data, making it possible for institutional repositories to generate COUNTER compliant usage data [8]_ + OAPEN is a not-for-profit organisation dedicated to open access, peer-reviewed books, operating three platforms: OAPEN Library; OAPEN Open Access Books Toolkit; and Directory of Open Access Books [11]_ open access - Open access (OA) is free access to information, and unrestricted use of electronic resources for all [9]_ + Open access (OA) is free access to information, and unrestricted use of electronic resources for all [12]_ ONIX - ONIX for Books (ONline Information eXchange) is a standard format that book publishers use to share information about the books that they have published [10]_ - - publisher - A scholarly eBook publisher, who participated in the pilot project to provide metadata for their titles, and metrics of their usage - - shard - A database shard is a way of storing data, so that load can be spread for large amounts of data [11]_ + ONIX for Books (ONline Information eXchange) is a standard format that book publishers use to share information about the books that they have published [13]_ SFTP SSH File Transfer Protocol telescope - A telescope is a data workflow that fetches and ingests data from a data source. Some telescopes run workflows that process and output data to other places. [12]_ Workflows are built on top of Apache Airflow's Directed Acyclic Graph (DAGs), where a DAG is “a collection of organized tasks that you want to schedule and run” [13]_ - - - -.. [1] https://www.crossref.org/community/ + A telescope is a data workflow that fetches and ingests data from a data source. Some telescopes run workflows that process and output data to other places. [14]_ Workflows are built on top of Apache Airflow's Directed Acyclic Graph (DAGs), where a DAG is “a collection of organized tasks that you want to schedule and run” [14]_ + -.. [2] https://www.elastic.co/elasticsearch/ +.. [1] https://openknowledge.community/projects/bad-project/ -.. [3] https://play.google.com/books/publish/ +.. [2] https://openknowledge.community/ -.. [4] https://about.jstor.org/librarians/books/open-access-books-jstor/ +.. [3] https://www.crossref.org/community/ -.. [5] https://www.elastic.co/kibana/ +.. [4] https://play.google.com/books/publish/ -.. [6] https://educopia.org/data_trust/ +.. [5] https://www.jisc.ac.uk/irus -.. [7] https://www.oapen.org/oapen/1891940-organisation +.. [6] https://www.fulcrum.org/ -.. [8] https://www.jisc.ac.uk/irus +.. [7] https://www.oapen.org/ -.. [9] https://en.unesco.org/open-access/what-open-access +.. [8] https://about.jstor.org/librarians/books/open-access-books-jstor/ -.. [10] https://bisg.org/general/custom.asp?page=ONIXforBooks +.. [9] https://cloud.google.com/looker-studio -.. [11] https://en.wikipedia.org/wiki/Shard_(database_architecture) +.. [10] https://educopia.org/data_trust/ -.. [12] https://github.com/The-Academic-Observatory/oaebu-workflows +.. [11] https://www.oapen.org/oapen/1891940-organisation -.. [13] https://cloud.google.com/composer/docs/run-apache-airflow-dag +.. [12] https://en.unesco.org/open-access/what-open-access +.. [13] https://bisg.org/general/custom.asp?page=ONIXforBooks +.. [14] https://cloud.google.com/composer/docs/run-apache-airflow-dag \ No newline at end of file diff --git a/docs/bad_project/help/index.rst b/docs/bad_project/help/index.rst index 8dfcf2cf..6538a770 100644 --- a/docs/bad_project/help/index.rst +++ b/docs/bad_project/help/index.rst @@ -1,5 +1,6 @@ .. toctree:: :maxdepth: 1 + glossary contact diff --git a/docs/bad_project/help/known_issues.md b/docs/bad_project/help/known_issues.md deleted file mode 100644 index b92b8acd..00000000 --- a/docs/bad_project/help/known_issues.md +++ /dev/null @@ -1,13 +0,0 @@ -# Known Issues - -This section contains details of any known issues, and any workarounds. If you find an issue that is not covered here or in the [FAQ](./faq.md), please get in touch with us at: coki@curtin.edu.au - -## Warning message ‘Terms list might be incomplete ...’ - -In the bespoke dashboard sections where data is manually uploaded (such as the University of Michigan Press’s Fulcrum, MUSE and EBSCO dashboards), a warning message may appear on the filters: ‘Terms list might be incomplete because the request is taking too long…’. This is an expected warning message for this dashboard, and these filters can still be used. - -## Error ‘No results found’ when select an individual year in a table -When a year filter is added by clicking the + sign in a table, the error ‘No results found’ may appear. The error message may then also appear in the dashboard filters ‘Filtering occurs on the … field, which doesn’t exist on any documents in the … index pattern’. - -This is a known issue. The dashboard can be reloaded by clicking on the relevant dashboard link in the navigation menu at the top right of each dashboard. To view data for a specific year, filter via any of the bar graphs rather than the tables. - diff --git a/docs/bad_project/images/authors_downloads1.png b/docs/bad_project/images/authors_downloads1.png deleted file mode 100644 index b32b530a..00000000 Binary files a/docs/bad_project/images/authors_downloads1.png and /dev/null differ diff --git a/docs/bad_project/images/authors_events1.png b/docs/bad_project/images/authors_events1.png deleted file mode 100644 index d76772be..00000000 Binary files a/docs/bad_project/images/authors_events1.png and /dev/null differ diff --git a/docs/bad_project/images/authors_filters1.png b/docs/bad_project/images/authors_filters1.png deleted file mode 100644 index afb96593..00000000 Binary files a/docs/bad_project/images/authors_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/authors_filters2.png b/docs/bad_project/images/authors_filters2.png deleted file mode 100644 index 5b07f1f6..00000000 Binary files a/docs/bad_project/images/authors_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/authors_views1.png b/docs/bad_project/images/authors_views1.png deleted file mode 100644 index 939f1285..00000000 Binary files a/docs/bad_project/images/authors_views1.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_downloads1.png b/docs/bad_project/images/bisac_downloads1.png deleted file mode 100644 index 0f1fff7e..00000000 Binary files a/docs/bad_project/images/bisac_downloads1.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_filters1.png b/docs/bad_project/images/bisac_filters1.png deleted file mode 100644 index dfc4bba9..00000000 Binary files a/docs/bad_project/images/bisac_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_filters2.png b/docs/bad_project/images/bisac_filters2.png deleted file mode 100644 index 6d3bf435..00000000 Binary files a/docs/bad_project/images/bisac_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_filters3.png b/docs/bad_project/images/bisac_filters3.png deleted file mode 100644 index 945a4a06..00000000 Binary files a/docs/bad_project/images/bisac_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_summary1.png b/docs/bad_project/images/bisac_summary1.png deleted file mode 100644 index 77f175eb..00000000 Binary files a/docs/bad_project/images/bisac_summary1.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_summary2.png b/docs/bad_project/images/bisac_summary2.png deleted file mode 100644 index 4ad87a14..00000000 Binary files a/docs/bad_project/images/bisac_summary2.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_summary3.png b/docs/bad_project/images/bisac_summary3.png deleted file mode 100644 index d5c62b06..00000000 Binary files a/docs/bad_project/images/bisac_summary3.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_summary4.png b/docs/bad_project/images/bisac_summary4.png deleted file mode 100644 index ad33a356..00000000 Binary files a/docs/bad_project/images/bisac_summary4.png and /dev/null differ diff --git a/docs/bad_project/images/bisac_summary5.png b/docs/bad_project/images/bisac_summary5.png deleted file mode 100644 index ab0df3be..00000000 Binary files a/docs/bad_project/images/bisac_summary5.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_downloads1.png b/docs/bad_project/images/book_isbns_downloads1.png deleted file mode 100644 index df7d4863..00000000 Binary files a/docs/bad_project/images/book_isbns_downloads1.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_downloads2.png b/docs/bad_project/images/book_isbns_downloads2.png deleted file mode 100644 index 815c2cec..00000000 Binary files a/docs/bad_project/images/book_isbns_downloads2.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_downloads3.png b/docs/bad_project/images/book_isbns_downloads3.png deleted file mode 100644 index 3bbddb8c..00000000 Binary files a/docs/bad_project/images/book_isbns_downloads3.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_downloads4.png b/docs/bad_project/images/book_isbns_downloads4.png deleted file mode 100644 index 294e882f..00000000 Binary files a/docs/bad_project/images/book_isbns_downloads4.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_filter1.png b/docs/bad_project/images/book_isbns_filter1.png deleted file mode 100644 index eebcd0c0..00000000 Binary files a/docs/bad_project/images/book_isbns_filter1.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_filter2.png b/docs/bad_project/images/book_isbns_filter2.png deleted file mode 100644 index ed61e31f..00000000 Binary files a/docs/bad_project/images/book_isbns_filter2.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_filter3.png b/docs/bad_project/images/book_isbns_filter3.png deleted file mode 100644 index 6d4ead49..00000000 Binary files a/docs/bad_project/images/book_isbns_filter3.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary1.png b/docs/bad_project/images/book_isbns_summary1.png deleted file mode 100644 index 5f7ae926..00000000 Binary files a/docs/bad_project/images/book_isbns_summary1.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary10.png b/docs/bad_project/images/book_isbns_summary10.png deleted file mode 100644 index d62abc3a..00000000 Binary files a/docs/bad_project/images/book_isbns_summary10.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary12.png b/docs/bad_project/images/book_isbns_summary12.png deleted file mode 100644 index 0bb8acbb..00000000 Binary files a/docs/bad_project/images/book_isbns_summary12.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary13.png b/docs/bad_project/images/book_isbns_summary13.png deleted file mode 100644 index 3bae9fda..00000000 Binary files a/docs/bad_project/images/book_isbns_summary13.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary14.png b/docs/bad_project/images/book_isbns_summary14.png deleted file mode 100644 index 0473af1d..00000000 Binary files a/docs/bad_project/images/book_isbns_summary14.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary15.png b/docs/bad_project/images/book_isbns_summary15.png deleted file mode 100644 index 1aa320cc..00000000 Binary files a/docs/bad_project/images/book_isbns_summary15.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary16.png b/docs/bad_project/images/book_isbns_summary16.png deleted file mode 100644 index 43afc6ad..00000000 Binary files a/docs/bad_project/images/book_isbns_summary16.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary17.png b/docs/bad_project/images/book_isbns_summary17.png deleted file mode 100644 index 26e5620a..00000000 Binary files a/docs/bad_project/images/book_isbns_summary17.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary18.png b/docs/bad_project/images/book_isbns_summary18.png deleted file mode 100644 index 8a04490c..00000000 Binary files a/docs/bad_project/images/book_isbns_summary18.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary19.png b/docs/bad_project/images/book_isbns_summary19.png deleted file mode 100644 index 743e8c78..00000000 Binary files a/docs/bad_project/images/book_isbns_summary19.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary2.png b/docs/bad_project/images/book_isbns_summary2.png deleted file mode 100644 index ec6b3339..00000000 Binary files a/docs/bad_project/images/book_isbns_summary2.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary20.png b/docs/bad_project/images/book_isbns_summary20.png deleted file mode 100644 index 98d962d4..00000000 Binary files a/docs/bad_project/images/book_isbns_summary20.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary21.png b/docs/bad_project/images/book_isbns_summary21.png deleted file mode 100644 index 6fdddbb1..00000000 Binary files a/docs/bad_project/images/book_isbns_summary21.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary22.png b/docs/bad_project/images/book_isbns_summary22.png deleted file mode 100644 index 148a3647..00000000 Binary files a/docs/bad_project/images/book_isbns_summary22.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary23.png b/docs/bad_project/images/book_isbns_summary23.png deleted file mode 100644 index cde8ba48..00000000 Binary files a/docs/bad_project/images/book_isbns_summary23.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary24.png b/docs/bad_project/images/book_isbns_summary24.png deleted file mode 100644 index d6ec4a53..00000000 Binary files a/docs/bad_project/images/book_isbns_summary24.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary3.png b/docs/bad_project/images/book_isbns_summary3.png deleted file mode 100644 index 02d2bbda..00000000 Binary files a/docs/bad_project/images/book_isbns_summary3.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary4.png b/docs/bad_project/images/book_isbns_summary4.png deleted file mode 100644 index 3bc1d17b..00000000 Binary files a/docs/bad_project/images/book_isbns_summary4.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary5.png b/docs/bad_project/images/book_isbns_summary5.png deleted file mode 100644 index 5173c768..00000000 Binary files a/docs/bad_project/images/book_isbns_summary5.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary6.png b/docs/bad_project/images/book_isbns_summary6.png deleted file mode 100644 index b6a1792a..00000000 Binary files a/docs/bad_project/images/book_isbns_summary6.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary7.png b/docs/bad_project/images/book_isbns_summary7.png deleted file mode 100644 index 5258a480..00000000 Binary files a/docs/bad_project/images/book_isbns_summary7.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary8.png b/docs/bad_project/images/book_isbns_summary8.png deleted file mode 100644 index 1dbf35e7..00000000 Binary files a/docs/bad_project/images/book_isbns_summary8.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_summary9.png b/docs/bad_project/images/book_isbns_summary9.png deleted file mode 100644 index 1c5ef83a..00000000 Binary files a/docs/bad_project/images/book_isbns_summary9.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_top_titles_1.png b/docs/bad_project/images/book_isbns_top_titles_1.png deleted file mode 100644 index 83c79ee2..00000000 Binary files a/docs/bad_project/images/book_isbns_top_titles_1.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_top_titles_2.png b/docs/bad_project/images/book_isbns_top_titles_2.png deleted file mode 100644 index ce0f9adf..00000000 Binary files a/docs/bad_project/images/book_isbns_top_titles_2.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_views1.png b/docs/bad_project/images/book_isbns_views1.png deleted file mode 100644 index 5917028a..00000000 Binary files a/docs/bad_project/images/book_isbns_views1.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_views2.png b/docs/bad_project/images/book_isbns_views2.png deleted file mode 100644 index 4856d7b9..00000000 Binary files a/docs/bad_project/images/book_isbns_views2.png and /dev/null differ diff --git a/docs/bad_project/images/book_isbns_views3.png b/docs/bad_project/images/book_isbns_views3.png deleted file mode 100644 index 13308ba2..00000000 Binary files a/docs/bad_project/images/book_isbns_views3.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads1.png b/docs/bad_project/images/countries_downloads1.png deleted file mode 100644 index 0ee6cf51..00000000 Binary files a/docs/bad_project/images/countries_downloads1.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads10.png b/docs/bad_project/images/countries_downloads10.png deleted file mode 100644 index 9f3570b1..00000000 Binary files a/docs/bad_project/images/countries_downloads10.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads11.png b/docs/bad_project/images/countries_downloads11.png deleted file mode 100644 index 05adba6f..00000000 Binary files a/docs/bad_project/images/countries_downloads11.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads12.png b/docs/bad_project/images/countries_downloads12.png deleted file mode 100644 index c279ff9d..00000000 Binary files a/docs/bad_project/images/countries_downloads12.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads13.png b/docs/bad_project/images/countries_downloads13.png deleted file mode 100644 index bb3ccfb3..00000000 Binary files a/docs/bad_project/images/countries_downloads13.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads14.png b/docs/bad_project/images/countries_downloads14.png deleted file mode 100644 index 5dd4082c..00000000 Binary files a/docs/bad_project/images/countries_downloads14.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads15.png b/docs/bad_project/images/countries_downloads15.png deleted file mode 100644 index db717c41..00000000 Binary files a/docs/bad_project/images/countries_downloads15.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads16.png b/docs/bad_project/images/countries_downloads16.png deleted file mode 100644 index 69362d8e..00000000 Binary files a/docs/bad_project/images/countries_downloads16.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads17.png b/docs/bad_project/images/countries_downloads17.png deleted file mode 100644 index 631cb14c..00000000 Binary files a/docs/bad_project/images/countries_downloads17.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads2.png b/docs/bad_project/images/countries_downloads2.png deleted file mode 100644 index 828b6ee8..00000000 Binary files a/docs/bad_project/images/countries_downloads2.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads3.png b/docs/bad_project/images/countries_downloads3.png deleted file mode 100644 index a3c012ce..00000000 Binary files a/docs/bad_project/images/countries_downloads3.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads4.png b/docs/bad_project/images/countries_downloads4.png deleted file mode 100644 index b391131c..00000000 Binary files a/docs/bad_project/images/countries_downloads4.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads5.png b/docs/bad_project/images/countries_downloads5.png deleted file mode 100644 index 44e263cd..00000000 Binary files a/docs/bad_project/images/countries_downloads5.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads6.png b/docs/bad_project/images/countries_downloads6.png deleted file mode 100644 index 751ac678..00000000 Binary files a/docs/bad_project/images/countries_downloads6.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads7.png b/docs/bad_project/images/countries_downloads7.png deleted file mode 100644 index c03eb981..00000000 Binary files a/docs/bad_project/images/countries_downloads7.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads8.png b/docs/bad_project/images/countries_downloads8.png deleted file mode 100644 index 2724a8f7..00000000 Binary files a/docs/bad_project/images/countries_downloads8.png and /dev/null differ diff --git a/docs/bad_project/images/countries_downloads9.png b/docs/bad_project/images/countries_downloads9.png deleted file mode 100644 index 3fa4b248..00000000 Binary files a/docs/bad_project/images/countries_downloads9.png and /dev/null differ diff --git a/docs/bad_project/images/countries_filters1.png b/docs/bad_project/images/countries_filters1.png deleted file mode 100644 index e3f14ba8..00000000 Binary files a/docs/bad_project/images/countries_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/countries_filters2.png b/docs/bad_project/images/countries_filters2.png deleted file mode 100644 index 0998257d..00000000 Binary files a/docs/bad_project/images/countries_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/countries_filters3.png b/docs/bad_project/images/countries_filters3.png deleted file mode 100644 index 6d4ead49..00000000 Binary files a/docs/bad_project/images/countries_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_filters1.png b/docs/bad_project/images/ebsco_filters1.png deleted file mode 100644 index 70d9fc2f..00000000 Binary files a/docs/bad_project/images/ebsco_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_filters2.png b/docs/bad_project/images/ebsco_filters2.png deleted file mode 100644 index b67eb979..00000000 Binary files a/docs/bad_project/images/ebsco_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_filters3.png b/docs/bad_project/images/ebsco_filters3.png deleted file mode 100644 index 856c5ad3..00000000 Binary files a/docs/bad_project/images/ebsco_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_subjects1.png b/docs/bad_project/images/ebsco_subjects1.png deleted file mode 100644 index 4040b522..00000000 Binary files a/docs/bad_project/images/ebsco_subjects1.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_subjects2.png b/docs/bad_project/images/ebsco_subjects2.png deleted file mode 100644 index 0730ea38..00000000 Binary files a/docs/bad_project/images/ebsco_subjects2.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_summary1.png b/docs/bad_project/images/ebsco_summary1.png deleted file mode 100644 index ddba4040..00000000 Binary files a/docs/bad_project/images/ebsco_summary1.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_summary2.png b/docs/bad_project/images/ebsco_summary2.png deleted file mode 100644 index c25b795a..00000000 Binary files a/docs/bad_project/images/ebsco_summary2.png and /dev/null differ diff --git a/docs/bad_project/images/ebsco_summary3.png b/docs/bad_project/images/ebsco_summary3.png deleted file mode 100644 index e4a71923..00000000 Binary files a/docs/bad_project/images/ebsco_summary3.png and /dev/null differ diff --git a/docs/bad_project/images/faq1.png b/docs/bad_project/images/faq1.png deleted file mode 100644 index bbf999fb..00000000 Binary files a/docs/bad_project/images/faq1.png and /dev/null differ diff --git a/docs/bad_project/images/faq2.png b/docs/bad_project/images/faq2.png deleted file mode 100644 index dc25ac42..00000000 Binary files a/docs/bad_project/images/faq2.png and /dev/null differ diff --git a/docs/bad_project/images/faq3.png b/docs/bad_project/images/faq3.png deleted file mode 100644 index 8efdd7f4..00000000 Binary files a/docs/bad_project/images/faq3.png and /dev/null differ diff --git a/docs/bad_project/images/faq4.png b/docs/bad_project/images/faq4.png deleted file mode 100644 index 8bea226a..00000000 Binary files a/docs/bad_project/images/faq4.png and /dev/null differ diff --git a/docs/bad_project/images/faq5.png b/docs/bad_project/images/faq5.png deleted file mode 100644 index f7591fd3..00000000 Binary files a/docs/bad_project/images/faq5.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_books1.png b/docs/bad_project/images/fulcrum_books1.png deleted file mode 100644 index c500ae9b..00000000 Binary files a/docs/bad_project/images/fulcrum_books1.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_books2.png b/docs/bad_project/images/fulcrum_books2.png deleted file mode 100644 index e910d1b8..00000000 Binary files a/docs/bad_project/images/fulcrum_books2.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_books3.png b/docs/bad_project/images/fulcrum_books3.png deleted file mode 100644 index 1d5a05d0..00000000 Binary files a/docs/bad_project/images/fulcrum_books3.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_books4.png b/docs/bad_project/images/fulcrum_books4.png deleted file mode 100644 index ceec146f..00000000 Binary files a/docs/bad_project/images/fulcrum_books4.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_books5.png b/docs/bad_project/images/fulcrum_books5.png deleted file mode 100644 index de2f9432..00000000 Binary files a/docs/bad_project/images/fulcrum_books5.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_institutions.png b/docs/bad_project/images/fulcrum_institutions.png deleted file mode 100644 index 2b42e6b6..00000000 Binary files a/docs/bad_project/images/fulcrum_institutions.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_publishers.png b/docs/bad_project/images/fulcrum_publishers.png deleted file mode 100644 index c401fe19..00000000 Binary files a/docs/bad_project/images/fulcrum_publishers.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_select_authors1.png b/docs/bad_project/images/fulcrum_select_authors1.png deleted file mode 100644 index dfc4bba9..00000000 Binary files a/docs/bad_project/images/fulcrum_select_authors1.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_select_authors2.png b/docs/bad_project/images/fulcrum_select_authors2.png deleted file mode 100644 index 6d3bf435..00000000 Binary files a/docs/bad_project/images/fulcrum_select_authors2.png and /dev/null differ diff --git a/docs/bad_project/images/fulcrum_select_authors3.png b/docs/bad_project/images/fulcrum_select_authors3.png deleted file mode 100644 index 9c9a6a09..00000000 Binary files a/docs/bad_project/images/fulcrum_select_authors3.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads1.png b/docs/bad_project/images/institutions_downloads1.png deleted file mode 100644 index 4a05c4fd..00000000 Binary files a/docs/bad_project/images/institutions_downloads1.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads10.png b/docs/bad_project/images/institutions_downloads10.png deleted file mode 100644 index 1fb09ffc..00000000 Binary files a/docs/bad_project/images/institutions_downloads10.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads11.png b/docs/bad_project/images/institutions_downloads11.png deleted file mode 100644 index 1e38c2cb..00000000 Binary files a/docs/bad_project/images/institutions_downloads11.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads12.png b/docs/bad_project/images/institutions_downloads12.png deleted file mode 100644 index e4c2e05a..00000000 Binary files a/docs/bad_project/images/institutions_downloads12.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads13.png b/docs/bad_project/images/institutions_downloads13.png deleted file mode 100644 index 0c86105a..00000000 Binary files a/docs/bad_project/images/institutions_downloads13.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads14.png b/docs/bad_project/images/institutions_downloads14.png deleted file mode 100644 index a048baa5..00000000 Binary files a/docs/bad_project/images/institutions_downloads14.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads15.png b/docs/bad_project/images/institutions_downloads15.png deleted file mode 100644 index 2cfc1e29..00000000 Binary files a/docs/bad_project/images/institutions_downloads15.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads16.png b/docs/bad_project/images/institutions_downloads16.png deleted file mode 100644 index 442454c7..00000000 Binary files a/docs/bad_project/images/institutions_downloads16.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads2.png b/docs/bad_project/images/institutions_downloads2.png deleted file mode 100644 index 42322cdc..00000000 Binary files a/docs/bad_project/images/institutions_downloads2.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads3.png b/docs/bad_project/images/institutions_downloads3.png deleted file mode 100644 index 0db1b3f4..00000000 Binary files a/docs/bad_project/images/institutions_downloads3.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads4.png b/docs/bad_project/images/institutions_downloads4.png deleted file mode 100644 index 4c056a28..00000000 Binary files a/docs/bad_project/images/institutions_downloads4.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads5.png b/docs/bad_project/images/institutions_downloads5.png deleted file mode 100644 index a5880ca5..00000000 Binary files a/docs/bad_project/images/institutions_downloads5.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads6.png b/docs/bad_project/images/institutions_downloads6.png deleted file mode 100644 index 88fbce68..00000000 Binary files a/docs/bad_project/images/institutions_downloads6.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads7.png b/docs/bad_project/images/institutions_downloads7.png deleted file mode 100644 index b51d7ba4..00000000 Binary files a/docs/bad_project/images/institutions_downloads7.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads8.png b/docs/bad_project/images/institutions_downloads8.png deleted file mode 100644 index 2c196220..00000000 Binary files a/docs/bad_project/images/institutions_downloads8.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_downloads9.png b/docs/bad_project/images/institutions_downloads9.png deleted file mode 100644 index 54d1a291..00000000 Binary files a/docs/bad_project/images/institutions_downloads9.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_filters1.png b/docs/bad_project/images/institutions_filters1.png deleted file mode 100644 index e3f14ba8..00000000 Binary files a/docs/bad_project/images/institutions_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_filters2.png b/docs/bad_project/images/institutions_filters2.png deleted file mode 100644 index 0998257d..00000000 Binary files a/docs/bad_project/images/institutions_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/institutions_filters3.png b/docs/bad_project/images/institutions_filters3.png deleted file mode 100644 index 6d4ead49..00000000 Binary files a/docs/bad_project/images/institutions_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events1.png b/docs/bad_project/images/mentions_events1.png deleted file mode 100644 index 21622549..00000000 Binary files a/docs/bad_project/images/mentions_events1.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events2.png b/docs/bad_project/images/mentions_events2.png deleted file mode 100644 index 54f8c0b7..00000000 Binary files a/docs/bad_project/images/mentions_events2.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events3.png b/docs/bad_project/images/mentions_events3.png deleted file mode 100644 index a38f318d..00000000 Binary files a/docs/bad_project/images/mentions_events3.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events4.png b/docs/bad_project/images/mentions_events4.png deleted file mode 100644 index 0a8db887..00000000 Binary files a/docs/bad_project/images/mentions_events4.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events5.png b/docs/bad_project/images/mentions_events5.png deleted file mode 100644 index 15a89ffe..00000000 Binary files a/docs/bad_project/images/mentions_events5.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events6.png b/docs/bad_project/images/mentions_events6.png deleted file mode 100644 index a201535e..00000000 Binary files a/docs/bad_project/images/mentions_events6.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events7.png b/docs/bad_project/images/mentions_events7.png deleted file mode 100644 index 1df9ccae..00000000 Binary files a/docs/bad_project/images/mentions_events7.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_events8.png b/docs/bad_project/images/mentions_events8.png deleted file mode 100644 index de881bcc..00000000 Binary files a/docs/bad_project/images/mentions_events8.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_filters1.png b/docs/bad_project/images/mentions_filters1.png deleted file mode 100644 index eebcd0c0..00000000 Binary files a/docs/bad_project/images/mentions_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_filters2.png b/docs/bad_project/images/mentions_filters2.png deleted file mode 100644 index ed61e31f..00000000 Binary files a/docs/bad_project/images/mentions_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/mentions_filters3.png b/docs/bad_project/images/mentions_filters3.png deleted file mode 100644 index 6d4ead49..00000000 Binary files a/docs/bad_project/images/mentions_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/muse_books1.png b/docs/bad_project/images/muse_books1.png deleted file mode 100644 index bfd3bab5..00000000 Binary files a/docs/bad_project/images/muse_books1.png and /dev/null differ diff --git a/docs/bad_project/images/muse_books2.png b/docs/bad_project/images/muse_books2.png deleted file mode 100644 index 92c9d20f..00000000 Binary files a/docs/bad_project/images/muse_books2.png and /dev/null differ diff --git a/docs/bad_project/images/muse_books3.png b/docs/bad_project/images/muse_books3.png deleted file mode 100644 index adefb1b8..00000000 Binary files a/docs/bad_project/images/muse_books3.png and /dev/null differ diff --git a/docs/bad_project/images/muse_country1.png b/docs/bad_project/images/muse_country1.png deleted file mode 100644 index c3464f15..00000000 Binary files a/docs/bad_project/images/muse_country1.png and /dev/null differ diff --git a/docs/bad_project/images/muse_country2.png b/docs/bad_project/images/muse_country2.png deleted file mode 100644 index 4baf31bc..00000000 Binary files a/docs/bad_project/images/muse_country2.png and /dev/null differ diff --git a/docs/bad_project/images/muse_filters1.png b/docs/bad_project/images/muse_filters1.png deleted file mode 100644 index c9cd769f..00000000 Binary files a/docs/bad_project/images/muse_filters1.png and /dev/null differ diff --git a/docs/bad_project/images/muse_filters2.png b/docs/bad_project/images/muse_filters2.png deleted file mode 100644 index 0d22d8f1..00000000 Binary files a/docs/bad_project/images/muse_filters2.png and /dev/null differ diff --git a/docs/bad_project/images/muse_filters3.png b/docs/bad_project/images/muse_filters3.png deleted file mode 100644 index 8b008c79..00000000 Binary files a/docs/bad_project/images/muse_filters3.png and /dev/null differ diff --git a/docs/bad_project/images/muse_institutions1.png b/docs/bad_project/images/muse_institutions1.png deleted file mode 100644 index de572aa8..00000000 Binary files a/docs/bad_project/images/muse_institutions1.png and /dev/null differ diff --git a/docs/bad_project/images/overview_intro1.png b/docs/bad_project/images/overview_intro1.png deleted file mode 100644 index ab8062e7..00000000 Binary files a/docs/bad_project/images/overview_intro1.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary1.png b/docs/bad_project/images/overview_summary1.png deleted file mode 100644 index b6a33d9c..00000000 Binary files a/docs/bad_project/images/overview_summary1.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary2.png b/docs/bad_project/images/overview_summary2.png deleted file mode 100644 index 00ecae1b..00000000 Binary files a/docs/bad_project/images/overview_summary2.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary3.png b/docs/bad_project/images/overview_summary3.png deleted file mode 100644 index 9ab6b44c..00000000 Binary files a/docs/bad_project/images/overview_summary3.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary4.png b/docs/bad_project/images/overview_summary4.png deleted file mode 100644 index ff8b363f..00000000 Binary files a/docs/bad_project/images/overview_summary4.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary5.png b/docs/bad_project/images/overview_summary5.png deleted file mode 100644 index 66f269f6..00000000 Binary files a/docs/bad_project/images/overview_summary5.png and /dev/null differ diff --git a/docs/bad_project/images/overview_summary6.png b/docs/bad_project/images/overview_summary6.png deleted file mode 100644 index a4aeb5b2..00000000 Binary files a/docs/bad_project/images/overview_summary6.png and /dev/null differ diff --git a/docs/bad_project/images/quality_isbns1.png b/docs/bad_project/images/quality_isbns1.png deleted file mode 100644 index a8be71f4..00000000 Binary files a/docs/bad_project/images/quality_isbns1.png and /dev/null differ diff --git a/docs/bad_project/images/quality_isbns2.png b/docs/bad_project/images/quality_isbns2.png deleted file mode 100644 index 9cb12fd7..00000000 Binary files a/docs/bad_project/images/quality_isbns2.png and /dev/null differ diff --git a/docs/bad_project/images/quality_isbns3.png b/docs/bad_project/images/quality_isbns3.png deleted file mode 100644 index 293e901e..00000000 Binary files a/docs/bad_project/images/quality_isbns3.png and /dev/null differ diff --git a/docs/bad_project/images/quality_isbns4.png b/docs/bad_project/images/quality_isbns4.png deleted file mode 100644 index 4bd7576e..00000000 Binary files a/docs/bad_project/images/quality_isbns4.png and /dev/null differ diff --git a/docs/bad_project/images/quality_isbns5.png b/docs/bad_project/images/quality_isbns5.png deleted file mode 100644 index 596ce62d..00000000 Binary files a/docs/bad_project/images/quality_isbns5.png and /dev/null differ diff --git a/docs/bad_project/overview/how_works.md b/docs/bad_project/overview/how_works.md index d91abd93..1dcebcf3 100644 --- a/docs/bad_project/overview/how_works.md +++ b/docs/bad_project/overview/how_works.md @@ -3,9 +3,9 @@ How the BAD Project Works The BAD project technology stack uses the book industry metadata interchange standard ONIX, combined with open bibliographic metadata (Crossref and OAPEN) to integrate usage data from multiple sources. Data integration through the Book Usage Data Workflows code base is built on an open-source workflow system written in Python. Data workflows (or telescopes) fetch, process, disambiguate and analyse data about open access eBooks from multiple sources. This data is saved to Google Cloud’s BigQuery data warehouse. The multiple workflows include: - 1. Ingesting data via telescope workflows from Crossref-Metadata, Crossref-Events, Google Analytics, Google Books, JSTOR, IRUS Fulcrum, IRUS OAPEN, ONIX, Thoth, UCL Discovery, and + 1. Ingesting data via telescope workflows from Crossref Event Data, Crossref Metadata, Google Analytics, Google Books, IRUS Fulcrum, IRUS OAPEN, JSTOR, OAPEN metadata, ONIX, Thoth, UCL Discovery, and - 2. A series of analytic workflows to process and combine the data ingested by the telescope workflows. + 2. A series of analytic workflows to process and combine the data ingested by the telescope workflows. The processed data in the Google Cloud BiqQuery data warehouse is then visualised in dashboards provided by Looker Studio, a dashboarding solution offered by Google. The information from our data sources is refreshed on a regular basis, keeping the Dashboard up-to-date. Data access permissions flow through from the underlying sources into the cloud database and the Dashboard. Stakeholder data is sandboxed into separate areas with user access permissions controlling access for each area, providing strong security and privacy. \ No newline at end of file diff --git a/docs/bad_project/overview/partner_data.md b/docs/bad_project/overview/partner_data.md index 05d15e07..21c115b9 100644 --- a/docs/bad_project/overview/partner_data.md +++ b/docs/bad_project/overview/partner_data.md @@ -1,25 +1,31 @@ BAD Project Dashboard Partner Data Sources ========================== -The BAD project dashboard partner data sources are where permission has been granted for the BAD project to access usage data about publishers’ open access eBook collections. +The BAD project dashboard partner data sources are where permission has been granted for the BAD project to access usage data about publishers’ open access eBook collections, or is publicly available but specific to a particular publisher. -## ONIX-FTP feed from publishers -[ONIX](https://www.editeur.org/83/Overview/) is a standard that book publishers use to share information about the books that they have published. BAD project dashboard partners that have ONIX feeds are given credentials and access to their own upload folder on the Mellon SFTP server. The BAD project dashboard partner uploads their ONIX feed to their upload folder on a weekly, fortnightly or monthly basis. The Book Usage Data Workflows ONIX telescope downloads, transforms (with the ONIX parser Java command line tool) and then loads the ONIX data into BigQuery for further processing. +## Google Analytics +Google Analytics monitors and records web traffic for specific websites. If a BAD project dashboard partner has configured Google Analytics on their publisher website, the Google Analytics data can be used to find out which countries and territories website visitors are from. + +## Google Books +The Google Books Partner program hosts eBooks, including some free open access eBooks. eBook publishers can then download usage reports from [Google Books](https://play.google.com/books/publish/). The BAD project uses data from the Google Play sales transaction report and the Google Books Traffic Report. + +## IRUS Fulcrum +IRUS provides COUNTER standard access reports for eBooks hosted on the Fulcrum platform. [Fulcrum](https://www.press.umich.edu/librarians) is a “community-developed, open source platform for digital scholarship” which provides “users the ability to read books with associated digital enhancements, such as: 3-D models, embedded audio, video, and databases; zoomable online images, and interactive media”. ## IRUS OAPEN -IRUS provides OAPEN COUNTER standard access reports. Almost all eBooks on OAPEN are provided as a PDF file for the whole book. The reports show access figures for each month, and the location (IP address) of the access. Within the OAPEN Google Cloud project (located in Europe), IP addresses are replaced with geographical information (city and country). This means that IP addresses are not stored within the BAD project data, and only de-identified geographical information transferred to the BAD project. +IRUS provides COUNTER standard access reports for eBooks hosted on the [OAPEN library and platform](https://oapen.org/). OAPEN "promotes and supports the transition to open access for academic books by providing open infrastructure services to stakeholders in scholarly communication". Almost all eBooks on OAPEN are provided as a PDF file for the whole book. The reports show access figures for each month, and the location (IP address) of the access. Within the OAPEN Google Cloud project (located in Europe), IP addresses are replaced with geographical information (city and country). This means that IP addresses are not stored within the BAD project data, and only de-identified geographical information transferred to the BAD project. ## JSTOR [JSTOR](https://about.jstor.org/librarians/books/open-access-books-jstor/) is a digital library, offering over 7000 open access eBooks. Publisher usage reports offer details about the use (views and downloads) of eBooks by institution, and country. -## Google Books -The Google Books Partner program hosts eBooks, including some free open access eBooks. eBook publishers can then download usage reports from [Google Books](https://play.google.com/books/publish/). The BAD project uses data from the Google Play sales transaction report and the Google Books Traffic Report. +## ONIX-FTP feed from publishers +[ONIX](https://www.editeur.org/83/Overview/) is a standard that book publishers use to share information about the books that they have published. BAD project dashboard partners that have ONIX feeds are given credentials and access to their own upload folder on the Mellon SFTP server. The BAD project dashboard partner uploads their ONIX feed to their upload folder on a weekly, fortnightly or monthly basis. The Book Usage Data Workflows ONIX telescope downloads, transforms (with the ONIX parser Java command line tool) and then loads the ONIX data into BigQuery for further processing. -## Google Analytics -Google Analytics monitors and records web traffic for specific websites. If a BAD project dashboard partner has configured Google Analytics on their publisher website, the Google Analytics data can be used to find out which countries and territories website visitors are from. +## OAPEN Metadata +OAPEN enables libraries and aggregators to use the metadata of all available titles in the OAPEN Library. The metadata is available in different formats and the BAD project harvests the data in the XML format to obtain an file in ONIX format for the OAPEN platform. -## IRUS Fulcrum -IRUS provides OAPEN COUNTER standard access reports. [Fulcrum](https://www.press.umich.edu/librarians) is a “community-developed, open source platform for digital scholarship” which provides “users the ability to read books with associated digital enhancements, such as: 3-D models, embedded audio, video, and databases; zoomable online images, and interactive media”. +## Thoth +Thoth is a free, open metadata service that publishers can choose to utilise as a solution for metadata storage. Thoth can provide metadata upon request in a number of formats. The BAD project uses the [Thoth Export API](https://export.thoth.pub/#get-/formats/-format_id-) to download metadata for publishers in an ONIX format. ## UCL Discovery University College London (UCL) is an eBook publisher, and partner in the BAD project. UCL Discovery is UCL's open access repository, showcasing and providing access to the full texts of UCL research publications. \ No newline at end of file diff --git a/docs/bad_project/overview/public_data.md b/docs/bad_project/overview/public_data.md index 1202ac7e..1e5e0eb3 100644 --- a/docs/bad_project/overview/public_data.md +++ b/docs/bad_project/overview/public_data.md @@ -3,8 +3,8 @@ Public Data Sources The public data sources are where data is publicly available, rather than data provided by a specific pilot project dashboard partner. -## Crossref – Metadata -Crossref is a non-for-profit membership organization, and an official Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation. They make metadata available for all DOIs registered with [Crossref](https://www.crossref.org/community/). +## Crossref Event Data +Crossref Event Data captures online discussion about research outputs, such as [‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’](https://www.crossref.org/services/event-data/). The event data is retrieved using the [Crossref Events API](https://www.eventdata.crossref.org/guide/service/query-api/). Crossref Event data must be queried using a DOI, which the BAD project obtains from Crossref Metadata. -## Crossref – Events -Crossref Event Data captures online discussion about research outputs, such as [‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’](https://www.crossref.org/services/event-data/). The event data is retrieved using the [Crossref Events API](https://www.eventdata.crossref.org/guide/service/query-api/). \ No newline at end of file +## Crossref Metadata +Crossref is a non-for-profit membership organization, and an official Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation. They make metadata available for all DOIs registered with [Crossref](https://www.crossref.org/community/). The BAD project uses Crossref Metadata to match ISBNs obtained from a publishers Onix feed to DOIs to query Crossref Event Data. \ No newline at end of file diff --git a/docs/bad_project/user_guides/authors_db.md b/docs/bad_project/user_guides/authors_db.md deleted file mode 100644 index 6441106c..00000000 --- a/docs/bad_project/user_guides/authors_db.md +++ /dev/null @@ -1,88 +0,0 @@ -# Authors and Volume Editors Dashboard - -The ‘Authors/Volume Editors’ dashboard provides data about titles per author(s) or volume editor(s). -By default this dashboard shows data for all authors and editors of open access eBooks for this publisher. When specific author(s) and/or volume editor(s) are selected in the ‘Select Author(s)/Volume Editor(s)’ field, the ‘Downloads’, ‘Views’ and ‘Events referencing these books’ sections will be automatically updated. - -The sections of this dashboard are: -* [Filter](#filter) -* [Downloads](#downloads) -* [Views](#views) -* [Events referencing these books](#events-referencing-these-books) - -The ‘Authors/Volume Editors’ dashboard does not include a filter for titles. To see a list of titles with usage data for an individual author, use the [‘Book ISBNs’ dashboard](./book_isbns_db.md). Add the author in the ‘Select Author’ filter, then scroll down to view the ‘Downloads’, ‘Views’ and ‘Mentions and Events’ sections including the author’s titles. - -## Filter -There is one filter in this dashboard – ‘Select Author(s)/Volume Editor(s)’. By default, data will be shown for all authors and editors where usage data is available. - -### Select one value in the filter -Use the dropdown option in the filter to choose a value, or start typing in a filter field to search all values. There is no search button, the rest of the page will automatically update. - -### Select multiple values in the filter -To select multiple values in the filter, keep selecting values one by one. This can be either via the dropdown OR by typing directly in the field’s text box and selecting from the matches. - -### Remove value/values from the filter -Selected values of a filter can be removed one at a time, or all removed at the same time. -For example, to clear one author/editor, click on the X next to the author/editor: - -``` eval_rst -.. image:: ../images/authors_filters1.png - :width: 400 -``` - -To remove multiple authors at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/authors_filters2.png - :width: 400 -``` - -## Downloads -The ‘Downloads’ section contains three bar graphs: -* ‘# monthly downloads OAPEN’ -* ‘# monthly downloads GoogleBooks’ -* ‘# monthly downloads JSTOR’ - -In each bar graph, you can hover over a bar to see the number of downloads per month for the authors/editors selected in the filter ‘Select Author(s)/Volume Editor(s)’. If no authors/editors are selected in the filter, data will be shown for all authors/editors where usage data is available. - -To apply a month to the whole dashboard, click on any month in a bar graph. To remove the month filter, reload the dashboard by clicking on the ‘Authors/Volume Editors’ link in the navigation menu at the top right of this dashboard. - -The data contained in the graphs can be exported to CSV by using the ‘Options’ popup. Hover to the right of one of the bar graph titles e.g. ‘# monthly downloads OAPEN’. Three dots will appear. - - -``` eval_rst -.. image:: ../images/authors_downloads1.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -Note: The data shown in this section is from the following variables: OAPEN (title_requests, then total_item_requests from April 2020), Google Books (google_books_sales.qty). - -## Views -The ‘Views’ section contains one bar graph – ‘# monthly views from GoogleBooks’. You can hover over a bar to see the number of views per month from Google Books for the authors/editors selected in the filter ‘Select Author(s)/Volume Editor(s)’. If no authors/editors are selected in the filter, data will be shown for all authors and editors where usage data is available. - -To apply a month to the whole dashboard, click on any month in the bar graph. To remove the month filter, reload the dashboard by clicking on the ‘Authors/Volume Editors’ link in the navigation menu at the top right of this dashboard. - -The data contained in the graphs can be exported to CSV by using the ‘Options’ popup. Hover to the far right of the bar graph title ‘# monthly views from GoogleBooks’. Three dots will appear. - -``` eval_rst -.. image:: ../images/authors_views1.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. -Note: The data shown in this section is from the following variable: Google Books (Google Analytics unique_views). - -## Events referencing these books -The ‘Events referencing these books’ section contains one bar graph – ‘# Crossref events’. You can hover over a bar to see the number of Crossref Events for the authors/editors selected in the filter ‘Select Author(s)/Volume Editor(s)’. If no authors/editors are selected in the filter, data will be shown for all authors and editors where usage data is available. - -To apply a month to the whole dashboard, click on any month in the bar graph. To remove the month filter, reload the dashboard by clicking on the ‘Authors/Volume Editors’ link in the navigation menu at the top right of this dashboard. - -The data contained in the graphs can be exported to CSV by using the ‘Options’ popup. Hover to the far right of the bar graph title ‘# Crossref events’. Three dots will appear. - -``` eval_rst -.. image:: ../images/authors_events1.png - :width: 600 -``` - -Note: The data shown in this section is from the following variable: Mentions in social media etc. from Crossref Events (crossref_events.count) \ No newline at end of file diff --git a/docs/bad_project/user_guides/bisac_db.md b/docs/bad_project/user_guides/bisac_db.md deleted file mode 100644 index c20f2a96..00000000 --- a/docs/bad_project/user_guides/bisac_db.md +++ /dev/null @@ -1,140 +0,0 @@ -# BISAC Subjects Dashboard - -This dashboard shows usage data by Book Industry Standards and Communications (BISAC) subject code, where the BISAC subject code is a way of classifying what a book is about. Examples of BISAC subject codes are ‘Education’ and ‘Mathematics’. - -More information about the BISAC Subject Codes is available at these links: -* BISAC Subject Codes description - -* BISAC Subject Codes FAQ - -* List of codes - - -The sections of this dashboard are: -* [Filter](#filter) -* [Summary](#summary) -* [Downloads by BISAC subject](#downloads-by-bisac-subject) -* [Views by BISAC subject](#views-by-bisac-subject) -* [Mentions and Events](#mentions-and-events) - -## Filter -There is one filter in this dashboard – ‘Select BISAC Subject(s)’. By default, data will be shown for all BISAC subjects where usage data is available. - -### Select one value in a filter -Use the dropdown in the filter to choose a value, or start typing in a filter field to search all values. There is no search button, the rest of the page will automatically update. - -### Select multiple values in a filter -To select multiple values in the filter, keep selecting values one by one. This can be either using the dropdown OR by typing directly in the field’s text box and selecting from the matches. - -### Remove value/values from a filter -Selected values of a filter can be removed one at a time, or all removed at the same time. -For example, to clear one author, click on the X next to the author: - -``` eval_rst -.. image:: ../images/bisac_filters1.png - :width: 400 -``` - -To remove multiple authors at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/bisac_filters2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter field in the top left of the dashboard. Click on the grey x to remove the filter. - -``` eval_rst -.. image:: ../images/bisac_filters3.png - :width: 400 -``` - -## Summary -### List of BISAC subjects -The summary section contains a table ‘List of BISAC subjects with usage metrics’ showing the number of Products (eBooks) with usage data per BISAC subject. - -The data contained in the table can be exported to CSV. Hover to the right of the table name ‘List of BISAC subjects with usage metrics’. Three dots will appear. - -``` eval_rst -.. image:: ../images/bisac_summary1.png - :width: 400 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. Note that this CSV export also includes the number of ISBNs per BISAC subject. - -### Number of products with usage metrics -The graph ‘Monthly # Products with usage metrics by BISAC subject’ shows the number of products (eBooks) per BISAC subject for each month. By default the graph will show data from 2018. Each BISAC subject has a different coloured line in the graph. - -You can hover over the graph to see the BISAC subject breakdown per month. - -``` eval_rst -.. image:: ../images/bisac_summary2.png - :width: 400 -``` - -Clicking on any month and BISAC subject in the graph will filter on it and apply this month and subject to the whole dashboard. After you click in the graph, you will see a ‘Select filters to apply’ popup. Check these are the filters you require, and click on ‘Apply’. - -``` eval_rst -.. image:: ../images/bisac_summary3.png - :width: 400 -``` - -The whole dashboard will update to show data for one month and one BISAC subject. - -To remove the month and subject filters, go to the filter bar at the top left of the dashboard. Click on the grey X to remove the subject filter. - -``` eval_rst -.. image:: ../images/bisac_summary4.png - :width: 400 -``` - -Then, to remove the time filter, reload the dashboard by clicking on the ‘Subjects - BISAC’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/bisac_summary5.png - :width: 400 -``` - -The data contained in the graph can be exported to CSV by clicking on the three dots beside the graph title ‘Monthly # Products with usage metrics by BISAC subject’, and selecting ‘Download as CSV’ from the ‘Options’ popup. - -## Downloads by BISAC subject - -The ‘Downloads by BISAC subject’ section contains three bar graphs: -* ‘Top BISAC subjects by # downloads from OAPEN (Counter 5)’ - this usage data starts from April 2020. -* ‘Top BISAC subjects by # downloads from GoogleBooks’ - this usage data starts from September 2020. -* ‘Top BISAC subjects by # downloads from JSTOR’ - this usage data starts from May 2018. - -In each bar graph, you can hover over a section of a coloured bar to see the number of downloads per subject for each month. - -To apply a month and BISAC subject to the whole dashboard, click on any month and its BISAC subject section to filter on it. After you click in the graph, you will see a ‘Select filters to apply’ popup. Check these are the filters you require, and click on ‘Apply’. The whole dashboard will then update to show data for one month and one BISAC subject. - -To remove the month and subject filters, follow the instructions in the Summary section. - -The data in each bar graph can be exported to CSV by following the instructions in the -Summary section above. Note that the three dots to open the ‘Options’ popup are beside the graph title e.g. ‘Top BISAC subjects …’, not beside the source name e.g. OAPEN. - -``` eval_rst -.. image:: ../images/bisac_downloads1.png - :width: 400 -``` - -Note: The data shown in this section is from the following variables: OAPEN (total_item_requests), Google Books (google_books_sales.qty), JSTOR chapters (Total_Item_Requests). - -## Views by BISAC subject -The ‘Views by BISAC subject’ section contains one bar graph ‘Top BISAC subjects by # views from GoogleBooks’. This graph shows the number of views on Google Books per BISAC subject. You can hover over a section of a coloured bar to see the BISAC subject breakdown per month. - -To apply a month and BISAC subject to the whole dashboard, click on any month and its BISAC subject section to filter on it. After you click in the graph, you will see a ‘Select filters to apply’ popup. Check these are the filters you require, and click on ‘Apply’. The whole dashboard will then update to show the number of views from Google Books for one month and one BISAC subject. - -To remove the month and subject filters, follow the instructions in the Summary section. - -Note: The data shown in this section is from the following variables: Google Books (BV_with_Pages_Viewed). - -## Mentions and Events -The ‘Mentions and Events’ section contains one bar graph, ‘Top BISAC subjects by # Crossref Events’. -Crossref Event Data captures online discussion about research outputs, such as ‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’ () - -This graph shows the number of mentions and events captured by Crossref Events per BISAC subject. You can hover over a section of a coloured bar to see the BISAC subject breakdown for Crossref Events per month. - -To apply a month and BISAC subject to the whole dashboard, click on any month and its BISAC subject section to filter on it. After you click in the graph, you will see a ‘Select filters to apply’ popup. Check these are the filters you require, and click on ‘Apply’. The whole dashboard will update to show the number of Crossref Events for one month and one BISAC subject. - -To remove the month and subject filters, follow the instructions in the Summary section. - -Note: The data shown in this section is from the following variables: Crossref Events (crossref_events.count). - diff --git a/docs/bad_project/user_guides/book_isbns_db.md b/docs/bad_project/user_guides/book_isbns_db.md deleted file mode 100644 index 3608e7db..00000000 --- a/docs/bad_project/user_guides/book_isbns_db.md +++ /dev/null @@ -1,422 +0,0 @@ -# Book ISBNs Dashboard -The ‘Book ISBNs’ dashboard shows usage data for all eBooks. This usage data comes from Google Books, JSTOR, OAPEN and Crossref Events. For more information about these data sources, see the [partner data](../overview/partner_data) and [public data](../overview/public_data) sections of this help guide. - -Data sources have usage data for different time periods. For example, the University of Michigan Press has data sources for the following time periods: -* Usage data is from September 2020 for Google Books -* Usage data is from January 2018 onwards for JSTOR -* Usage data is from January 2018 to March 2020 for OAPEN (Counter 4), and from April 2020 for OAPEN (Counter 5) -* Event data is from May 2018 onwards for Crossref Events - -The sections of this dashboard are: -* [Filters](#filters) -* [Summary](#summary) -* [Downloads](#downloads) -* [Views](#views) -* [Mentions and Events](#mentions-and-events) - -## Filters -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Title(s)’ OR ‘Select ISBN(s)’ OR ‘Select Author’. -``` -If you select a filter then add a second filter, the first filter will be automatically removed. For example, if you use ‘Select Title(s)’, then ‘Select ISBN(s)’, the ‘Select Title(s)’ filter will be automatically removed. Note that on this ‘Book ISBNs’ dashboard only one author can be selected, not multiple authors. - -To look at the downloads, views and events for multiple University of Michigan Press authors or volume editors, go to the [Author/Volume Editor dashboard](./authors_db). - -### Select one value in a filter -Use the dropdown in any filter field to choose a value e.g. a Title in the ‘Select Title(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter -To select multiple values in a specific filter field, keep selecting values one by one. This can be either using the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. - -For example, to clear one Title, click on the X next to the Title: - -``` eval_rst -.. image:: ../images/book_isbns_filter1.png - :width: 400 -``` - -To remove multiple titles at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/book_isbns_filter2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/book_isbns_filter3.png - :width: 600 -``` - -## Summary -The ‘Summary’ section shows the number of ISBNs with usage data from Google Books, JSTOR, OAPEN and Crossref Events in two bar graphs: -* ‘# ISBNs per month with usage data’ - this is not the same as the number of downloads across ISBNs/Titles or Authors, for this data see the ‘Downloads’ section in this dashboard. -* ‘# ISBNs per year with usage data’ - this data is not the same as the individual months added together, as it shows the number of ISBNs across a whole year that had usage data recorded about them. - -Hover over any of the months/years in these bar graphs to see the number of ISBNs in that month/year that have usage data. - -### View month or year in detail -To look at a specific month/year in more detail in the Summary bar graphs, click on a month/year in the bar graph to add a time filter. The bar graph visualisation will now display the number of ISBNs for which there is usage data, for one month/year. The rest of the dashboard will automatically update to use this time filter. - -To remove a time filter in the ‘Summary’ section, force the dashboard to reload by clicking on the Book ISBNs link at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/bisac_summary5.png - :width: 300 -``` -The dashboard will automatically refresh to show usage data for all ISBNs. - -## Options popup -The Options popup contains the following features: -* Customize time range -* Inspect -* Maximise panel -* Download as CSV - -To open the ‘Options’ popup, hover over and click on the three dots on the right of a graph or table name. For example, in the ‘Summary’ section of the ‘Book ISBNs’ dashboard, hover over and click on the three dots on the right of ‘# ISBNs per month with usage data’ (or ‘# ISBNs per year with usage data): - -``` eval_rst -.. image:: ../images/book_isbns_summary2.png - :width: 400 -``` - -#### Customize time range -In the ‘Options’ popup, click on ‘Customize time range’. - -``` eval_rst -.. image:: ../images/book_isbns_summary3.png - :width: 400 -``` - -A ‘Customize panel time range’ popup will be shown, with a ‘Time range’ of ‘All Years Usage data’ by default. - -``` eval_rst -.. image:: ../images/book_isbns_summary4.png - :width: 400 -``` - -Click on ‘Show dates’ to see the current date range. - -``` eval_rst -.. image:: ../images/book_isbns_summary5.png - :width: 400 -``` - -To change the time span of the range, click on the first date. The calendar popup will be shown, with a blue line under the ‘From date’. There are three ways to choose a ‘From’ date: -* Absolute -* Relative -* Now - -``` eval_rst -.. image:: ../images/book_isbns_summary6.png - :width: 400 -``` - -For example, to change the time range to 2018 – 2019 only, click in the ‘Time range’ field to change the dates to this range: - -``` eval_rst -.. image:: ../images/book_isbns_summary7.png - :width: 400 -``` - -Select ‘Add to panel’ to apply the new time range to the bar graph. - -To clear the time range in the bar graph, go back into ‘Options’ and then ‘Customize time range’. Click on ‘Remove’. - -``` eval_rst -.. image:: ../images/book_isbns_summary8.png - :width: 400 -``` - -The time range will be removed, and the bar graph will return to the default of showing ISBNs with usage data after January 2018 - -If the entered time range is not possible or has no data, for example ‘in 2 months to in 3 months’, the message ‘No results found’ will be displayed: - -``` eval_rst -.. image:: ../images/book_isbns_summary9.png - :width: 400 -``` - -#### Inspect - -In the ‘Options’ popup, click on ‘Inspect’: - -``` eval_rst -.. image:: ../images/book_isbns_summary10.png - :width: 400 -``` - -A table of the data used in the bar graph (data visualisation) will be displayed. For example, for ‘# ISBNs per month with usage data’: - -``` eval_rst -.. image:: ../images/book_isbns_summary12.png - :width: 400 -``` - -You can change the number of rows displayed via ‘Rows per page’ (10, 20 or 50), and navigating to other rows. - -``` eval_rst -.. image:: ../images/book_isbns_summary13.png - :width: 400 -``` - -##### Downloading the data using Inspect - -Data can be exported in two ways: -* ‘Options’ -> ‘Inspect -> Download CSV’ (this section) or -* ‘Options’ -> [‘Download as CSV’](#download-as-csv) - -While still in the ‘Inspect’ popup, Select ‘Download CSV’ to download the displayed data either as a formatted CSV file (data already in table format), or as a raw CSV file (data that has not been formatted). - -``` eval_rst -.. image:: ../images/book_isbns_summary14.png - :width: 400 -``` - -In this example, the raw CSV contains the following format, where month is expressed as an epoch timestamp in milliseconds. Epoch time is ‘the number of seconds that have elapsed since the Unix epoch’ () - -``` -"Month ","# ISBNs" -1514736000000,27 -1517414400000,28 -``` - -Because the CSV file format does not require specific software, you can view, or import it in many other programs including LibreOffice’s Calc () and Microsoft Excel. - -##### Using filters in Inspect - -It is possible to use any of these months as a filter across the whole ‘Book ISBNs’ dashboard. To do this, hover beside the month you would like to filter on, and click on the plus in a circle icon to ‘Filter for value’. In this example, we are adding a filter for December 2021 in the ‘Book ISBNs’ dashboard. - -``` eval_rst -.. image:: ../images/book_isbns_summary15.png - :width: 400 -``` - -Each section of the whole ‘Book ISBNs’ dashboard (‘Summary’, ‘Downloads’, ‘Views’, and ‘Mentions and Events’) will automatically update to only show data for December 2021. - -``` eval_rst -.. image:: ../images/book_isbns_summary16.png - :width: 400 -``` - -This filter is also displayed at in the filter bar at the top left of the of ‘Book ISBNs’ dashboard. - -``` eval_rst -.. image:: ../images/book_isbns_summary17.png - :width: 400 -``` - -To remove this filter, go to the filter bar at the top left of the ‘Book ISBNs’ dashboard, and click on the X. - -``` eval_rst -.. image:: ../images/book_isbns_summary18.png - :width: 400 -``` - -The filter will be removed, and the ‘Book ISBNs’ dashboard will return to showing the data for all ISBNs for this publisher. - - -##### View Requests via Inspect - -By default, the data is displayed when ‘Options – Inspect’ is selected in the ‘Book ISBNs’ dashboard. The requests used to collect the data can also be viewed, by clicking on ‘View: Data’ and selecting ‘Requests’. Note, the documents here refer to how the data is stored in Elasticsearch, and is not a summary of the usage metrics that are visualised in the dashboards. The Requests menu may be of interest if you wish to view details of the Elasticsearch queries that power the visualisations. - -``` eval_rst -.. image:: ../images/book_isbns_summary19.png - :width: 400 -``` - -The three tabs are: -* Statistics -* Request -* Response - -``` eval_rst -.. image:: ../images/book_isbns_summary20.png - :width: 400 -``` - -Here the word ‘document’ means the number of months where there is usage data for the specified ISBN(s), title(s) or author. -The ‘Statistics’ tab shows: -* Hits = The number of documents returned by the query -* Hits (total) = The number of documents that match the query -* Index pattern = The index pattern that connected to the Elasticsearch indices -* Index pattern ID = The ID in the .kibana index -* Query time = The time it took to process the query. Does not include the time to send the request or parse it in the browser -* Request timestamp = Time when the start of the request has been logged - -The ‘Request’ tab shows the Kibana Query Language (KQL) query that was used in Elasticsearch to find the number of ISBNs per month with usage data. You can find out more about KQL at - -The ‘Response’ tab shows the response to the KQL request. - -To return to viewing the data, click on ‘View: Requests’ and select ‘Data’. - - -``` eval_rst -.. image:: ../images/book_isbns_summary21.png - :width: 400 -``` - -#### Maximise panel -Use ‘Maximise panel’ to expand a visualisation to full screen view. In the ‘Options’ popup, click on ‘Maximise panel’: - -``` eval_rst -.. image:: ../images/book_isbns_summary22.png - :width: 400 -``` - -The bar graph visualisation will then be shown as full screen within your browser. - -``` eval_rst -.. image:: ../images/book_isbns_summary23.png - :width: 400 -``` - -To minimise the panel and return to the default view, hover over and click on the three dots on the right of ‘# ISBNs per month with usage data’, and select ‘Minimize’. - -``` eval_rst -.. image:: ../images/book_isbns_summary24.png - :width: 400 -``` - -#### Download as CSV - -To export the data from the bar graph or table to a .CSV file, go to the ‘Options’ popup and click on ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -## Downloads - -The ‘Downloads’ section of the ‘Book ISBNs’ dashboard shows the total number of downloads for the selected ISBN(s) from OAPEN, Google Books and JSTOR combined. The separate totals are also presented for each data source - OAPEN, Google Books and JSTOR. - -``` eval_rst -.. tip:: To select specific ISBNs, only one filter should be used at a time – ‘Select Title(s)’ OR ‘Select ISBN(s)’ OR ‘Select Author’. -``` - -There are three monthly downloads bar graphs, and three top titles tables in the ‘Downloads’ section: -* ‘# monthly downloads OAPEN’ -* ‘# monthly downloads GoogleBooks’ -* ‘# monthly downloads JSTOR’ -* ‘Top titles by # downloads from OAPEN from Counter 4 and 5’ -* ‘Top titles by # downloads from GoogleBooks’ -* ‘Top titles by # chapter downloads from JSTOR’ - -### Monthly downloads bar graphs - -For example, when the title ‘Embodied Archive’ is selected in the ‘Select Title(s)’ filter, no downloads were reported in the OAPEN data: - -``` eval_rst -.. image:: ../images/book_isbns_downloads1.png - :width: 300 -``` - -For the same title, ‘Embodied Archive’ (a single ISBN), two downloads were reported in the Google Books data – one download in April 2021, and one download in October 2021. - -``` eval_rst -.. image:: ../images/book_isbns_downloads2.png - :width: 300 -``` - -For the same title, ‘Embodied Archive’ (a single ISBN), 285 chapter downloads were reported in the JSTOR data, spread between April 2021 and December 2021. - -``` eval_rst -.. image:: ../images/book_isbns_downloads3.png - :width: 300 -``` - -To open the ‘Options’ popup for the OAPEN, Google Books or JSTOR bar graphs, hover over and click on the three dots on the right of ‘# monthly downloads OAPEN’, OR ‘# monthly downloads GoogleBooks’ OR ‘# monthly downloads JSTOR’. - -``` eval_rst -.. image:: ../images/book_isbns_downloads4.png - :width: 500 -``` - -See the [‘Summary’](#summary) section in this help page for how to use the [‘Options’ popup](#options-popup) functions including exporting data from graphs and tables. - -### Top titles tables - -This table shows the top titles reported by OAPEN, Google Books and JSTOR from the specified title(s)/ISBN(s). By default these are top titles for all books, but it will also show top titles for whatever subset of books you have selected via the Filter section of the top of the ‘Books ISBN’ dashboard. - -Filters at the top of the dashboard can also be used to view a specific author’s top titles in this table, for example see the following screenshot for all eBooks published by the University of Michigan Press by the author Robert E. Cole. - -``` eval_rst -.. image:: ../images/book_isbns_top_titles_1.png - :width: 600 -``` - -Note that if only one ISBN is selected via the filters at the top of the ‘Book ISBNs’ dashboard, it will be the top title of one ISBN. For example, see the following screenshot for ‘Embodied Archive’. - -``` eval_rst -.. image:: ../images/book_isbns_top_titles_2.png - :width: 600 -``` - -Note: The data shown in this section is from the following variables: OAPEN (title_requests, then total_item_requests from April 2020), Google Books (google_books_sales.qty) and JSTOR chapters (Total_Item_Requests). - -## Views - -The ‘Views’ section shows the number of views from Google Books for the selected ISBNs. The ‘Select Title(s)’ OR ‘Select ISBN(s)’ OR ‘Select Author’ filters at the top of the dashboard can be used to select specific ISBNs. By default the views across all ISBNs from this publisher will be shown. - -To open the ‘Options’ popup in the ‘Views’ section, hover over and click on the three dots on the right of ‘# monthly views from GoogleBooks’ or ‘Top ISBNs by # views from Google Books’. - -See the [‘Summary’](#summary) section in this help page for how to use the [‘Options’ popup](#options-popup) functions including exporting data from graphs and tables. - -### Monthly views - -Hover over any of the months in the ‘# monthly views from GoogleBooks’ bar graph to see the number of views from Google Books for the selected ISBN(s) in that month. - -See the section [‘View month or year in detail’](#view-month-or-year-in-detail) for how to use a time filter in this bar graph. - -### Top ISBNs by number of views - -The table ‘Top ISBNs by # views from Google Books’ shows which ISBNs had the most views from Google Books, for the ISBN(s), Title(s) or author that were selected at the top of the ‘Book ISBNs’ dashboard. This list is ranked in descending order of views. Note that ordering in this list is just based on data from Google Books, and other titles may have higher downloads (see the ‘Downloads’ section), or higher mentions and events (see the ‘Mentions and Events’ section). - -Any of the ‘ISBN’, ‘Title’ and ‘# views’ columns can be hidden. Click on this symbol in the column header: ̌ - -``` eval_rst -.. image:: ../images/book_isbns_views1.png - :width: 400 -``` - -In the popup menu, click on ‘Hide’ to hide this column. - -``` eval_rst -.. image:: ../images/book_isbns_views2.png - :width: 400 -``` - -Note that you cannot hide all columns, the last column remaining will not have a ‘Hide’ option. - -``` eval_rst -.. image:: ../images/book_isbns_views3.png - :width: 400 -``` - -To restore the hidden columns, reload the dashboard by clicking on the ‘Book ISBNs’ link in the navigation menu at the top right of this dashboard. - -Note: The data shown in this section is from the following variable: Google Books (BV_with_Pages_Viewed). - -## Mentions and events - -Crossref Event Data captures online discussion about research outputs including eBooks, such as ‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’ (). - -The ‘Mentions and Events’ section shows Crossref Event data for the ISBN(s), Title(s) or Author selected at the top of the ‘Book ISBNs’ dashboard. By default the mentions and events from across all ISBNs from this publisher will be shown. - -See the [‘Summary’](#summary) section in this help page for how to use the [‘Options’ popup](#options-popup) functions including exporting data from graphs and tables. - -### Number of Crossref events - -The bar graph ‘# Crossref events’ shows the number of Crossref events per month. By default this data will be shown for all ISBNs, if filters are selected the data will be displayed for those filters. - -Hover on a bar in the bar graph to show the number of Crossref events for a specific month. To apply a time filter by clicking on a bar in the bar graph, see the section ‘View month or year in detail’ on this help page. - -### Top ISBNs - -The table ‘Top ISBNs by # Crossref events’ shows which ISBNs had the most mentions and events from Crossref, for the ISBN(s), Title(s) or Author that were selected at the top of the ‘Book ISBNs’ dashboard. This list is ranked in descending order of mentions and events. Note that the ordering in this list is based on data from Crossref Events, and other titles may have higher downloads (see the [‘Downloads’](#downloads) section), or higher views (see the [‘Views’](#views) section). - -Any of the ‘ISBN’, ‘Title’ and ‘# events’ columns can be hidden. See the instructions on how to hide and restore table columns in the [‘Options’ popup](#options-popup) section of this help page. - -Note: The data shown in this section is from the following variable: crossref_events.count. diff --git a/docs/bad_project/user_guides/countries_db.md b/docs/bad_project/user_guides/countries_db.md deleted file mode 100644 index 79fe9781..00000000 --- a/docs/bad_project/user_guides/countries_db.md +++ /dev/null @@ -1,230 +0,0 @@ -# Countries and Territories Dashboard - -The ‘Countries and Territories’ dashboard shows usage data or downloads for eBooks by countries and territories. By default, this dashboard will automatically show usage data for all eBooks from this publisher. This data comes from Google Books, JSTOR, OAPEN and Crossref Events. For more information about these data sources, see the [partner data](../overview/partner_data) and [public data](../overview/public_data) sections of this help guide. - -* Usage data is available from September 2020 for Google Books -* Usage data is available from January 2018 onwards for JSTOR -* Usage data is available from January 2018 to March 2020 for OAPEN (Counter 4), and from April 2020 for OAPEN (Counter 5) -* Event data is from May 2018 onwards for Crossref Events - -The sections of this dashboard are: -* [Filters](#filters) -* [Downloads](#downloads) - -## Filters -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Country(s)/Territory(s)’ OR ‘Select Title(s)’ OR ‘Select ISBN(s)’. -``` - -If you select a filter then and add a second filter, the first filter will be automatically removed. For example, if you ‘Select Country(s)/Territory(s)’, then ‘Select Title(s)’, the ‘Select Country(s)/Territory(s)’ filter will be automatically removed. - -Three filters can be used in this dashboard: -* ‘Select Country(s)/Territory(s)’ - filter for downloads from one or multiple countries and territories OR -* ‘Select Title(s)’ - filter for downloads of one or multiple Titles across countries and territories OR -* ‘Select ISBN(s)’ - filter for downloads for one or multiple ISBNs across countries and territories - -### Select one value in a filter -Use the dropdown option in any filter field to choose a value e.g. a Title in the ‘Select Title(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter -To select multiple values in a specific filter field, keep selecting values one by one. This can be either via the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. - -For example, to clear one Title, click on the X next to the Title: - -``` eval_rst -.. image:: ../images/countries_filters1.png - :width: 400 -``` - -To remove multiple titles at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/countries_filters2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/countries_filters3.png - :width: 600 -``` - -## Downloads -In the ‘Downloads’ section, data is shown for the three data sources – OAPEN, JSTOR and Google Books. For each data source, there are four ways to view the download data: on a map, in a donut graph, in a bar graph and in a table. -* Downloads map - ‘Downloads by country or territory’ -* Top countries/territories (donut graph) e.g. ‘Top countries/territories for chapter downloads from JSTOR’ -* Number of monthly downloads (bar graph) e.g. ‘# monthly downloads from OAPEN by country/territory’ -* All countries/territories (table) e.g. ‘All countries/territories for downloads from GoogleBooks’ - -By default, the dashboard shows download data for all open access eBooks by this publisher. If any of the country/territory, title(s) or ISBN(s) filters are used, the map, donut graphs, bar graphs and tables will automatically update to show downloads for the specified filters. For example a specific Title can be added in the ‘Select Title(s)’ filter at the top of the dashboard. The downloads donut graphs, bar charts and tables will automatically update to display downloads for that title only. - -Note: The data shown in this section is from the following variables: OAPEN (total_item_requests), JSTOR chapters (Total_Item_Requests), Google Books (google_books_sales.qty). - -### Downloads map -In the map ‘Downloads by country or territory’, you can hover over each country to view information on the number of downloads for the titles specified in the filters at the top of the ‘Countries and Territories’ dashboard. If no filters are selected, the map will default to showing downloads for all titles from this publisher. - -``` eval_rst -.. image:: ../images/countries_downloads1.png - :width: 400 -``` - -The downloads map uses different shades of grey to show different download numbers. See the section [Layers popup](#layers-popup) (Show/hide downloads layer details) for more about this map feature. - -#### Layers popup -To open the ‘Layers’ popup, click on the layers panel icon in the top right corner of the map: - -``` eval_rst -.. image:: ../images/countries_downloads2.png - :width: 400 -``` - -The ‘Layers’ popup will be displayed: - -``` eval_rst -.. image:: ../images/countries_downloads3.png - :width: 300 -``` - -##### Show or hide downloads layer - -In the ‘Layers’ popup, click on the eye icon near ‘Downloads’ to show/hide the downloads layer of the map. - -``` eval_rst -.. image:: ../images/countries_downloads4.png - :width: 300 -``` - -##### Show or hide country labels layer - -In the ‘Layers’ popup, click on the eye icon near ‘Country labels’ to show/hide the country labels layer of the map. - -``` eval_rst -.. image:: ../images/countries_downloads5.png - :width: 300 -``` - -##### Show or hide downloads layer details - -In the ‘Layers’ popup, click on the expand symbol in between ‘Downloads’ and ‘Country Labels’ to show/hide the details of the downloads layer. - -``` eval_rst -.. image:: ../images/countries_downloads6.png - :width: 300 -``` - -The key will be displayed for the different shades of grey used to show different download numbers. - -``` eval_rst -.. image:: ../images/countries_downloads7.png - :width: 400 -``` - -### Downloads donut graphs -There are three donut graphs showing the top countries/territories for downloads from different sources: -* ‘Top countries/territories for downloads from OAPEN (C5)’ -* ‘Top countries/territories for chapter downloads from JSTOR’ -* ‘Top countries/territories for downloads from GoogleBooks’ - -Hover over a country/territory in a donut graph to see the number of downloads per country/territory from that data source. - -``` eval_rst -.. image:: ../images/countries_downloads8.png - :width: 500 -``` - -Click on any country/territory in a donut graph to update the dashboard to show downloads for just that country/territory. - -To clear a country/territory that was selected in a donut graph, go to the filter bar at the top left of the dashboard. Click on the X to remove the filter. - - -``` eval_rst -.. image:: ../images/countries_downloads9.png - :width: 400 -``` - -The data contained in the donut graphs can be exported to CSV by using the ‘Options’ popup. Hover to the right of one of the donut graph titles e.g. ‘Top countries/territories for downloads from OAPEN (C5)’. Three dots will appear. - - -``` eval_rst -.. image:: ../images/countries_downloads10.png - :width: 400 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Downloads bar graphs - -There are three bar graphs showing the number of monthly downloads by country/territory: -* ‘# monthly downloads from OAPEN by country/territory’ -* ‘# monthly chapter downloads from JSTOR by country/territory’ -* ‘# monthly downloads from GoogleBooks by country/territory’ - -Hover over a country/territory in the bar graphs to see the number of downloads per month, from different countries/territories from that data source. - - -``` eval_rst -.. image:: ../images/countries_downloads11.png - :width: 400 -``` - -To look at a specific month/year in more detail, click on a month in the bar graph to add time and country/territory filters. In the ‘Select filters to apply’ popup, select ‘Apply’. - -``` eval_rst -.. image:: ../images/countries_downloads12.png - :width: 400 -``` - -The bar graph visualisation will now display the number of downloads by country/territory for the selected month. The rest of the dashboard will automatically update to use these filters. - -To remove the Country filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the Country filter. - -``` eval_rst -.. image:: ../images/countries_downloads13.png - :width: 400 -``` - -To remove the time filter, reload the dashboard by clicking on the ‘Countries and Territories’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/countries_downloads14.png - :width: 300 -``` - -The dashboard will automatically refresh to show usage data by country for all ISBNs from this publisher. - -The data contained in the bar graphs can be exported to CSV by using the ‘Options’ popup. Hover to the right of one of the bar graph titles e.g. ‘# monthly downloads from OAPEN by country/territory’. Three dots will appear. - -``` eval_rst -.. image:: ../images/countries_downloads15.png - :width: 400 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Downloads tables -There are three tables showing the number of downloads by country/territory: -* ‘All countries/territories for downloads from OAPEN (C5)’ -* ‘All countries/territories for chapter downloads from JSTOR’ -* ‘All countries/territories for downloads from GoogleBooks’ - -Use the slider bar on the right of a table to scroll through the countries and territories. - -``` eval_rst -.. image:: ../images/countries_downloads16.png - :width: 400 -``` - -The data contained in the tables can be exported to CSV by using the ‘Options’ popup. Hover to the right of one of the table titles e.g. ‘All countries/territories for downloads from OAPEN (C5)’. Three dots will appear. - -``` eval_rst -.. image:: ../images/countries_downloads17.png - :width: 400 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. diff --git a/docs/bad_project/user_guides/data_qual_db.md b/docs/bad_project/user_guides/data_qual_db.md deleted file mode 100644 index 18222c19..00000000 --- a/docs/bad_project/user_guides/data_qual_db.md +++ /dev/null @@ -1,60 +0,0 @@ -# Data Quality Dashboard - -The ‘Data Quality’ dashboard can be used by a publisher to identify issues with data quality, and discrepancies between data from different sources. - -This dashboard has two sections: -* ISBNs with no usage data -* ISBNs with usage data that are not in ONIX - -## ISBNs with no usage data - -This section shows a table containing any ISBNs from the University of Michigan Press that are in ONIX, but which have no usage data. The table is scrollable by using the scrollbar on the right-hand side. - -``` eval_rst -.. image:: ../images/quality_isbns1.png - :width: 500 -``` - -The data in the table can be exported. Click on the three dots beside the ‘Title’ column. - -``` eval_rst -.. image:: ../images/quality_isbns2.png - :width: 500 -``` - -Select ‘Download as CSV’ from the ‘Options’ popup menu. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -``` eval_rst -.. tip:: If you open the CSV in Excel, the ISBN columns may have odd values of something similar to ‘9.78047E+12’. To correct this and show the actual ISBNs in Excel: 1. highlight the ISBN column in Excel, 2. right-click and select ‘Format Cells’ 3. change the ‘Category’ to ‘Number’ with 0 decimal places. The ISBNs will now be displayed as numbers. -``` - -## ISBNs with usage data that are not in ONIX -A comparison of ONIX data to three data sources is provided so that publishers can review their data: -* Number of ISBNs in the OAPEN data source but not in ONIX - ‘# ISBNs in OAPEN that are not in ONIX’ -* Number of ISBNs in the JSTOR data source but not in ONIX - ‘# ISBNs in JSTOR that are not in ONIX’ -* Number of ISBNs in the Google Books data source but not in ONIX - '# ISBNs in Google Books that are not in ONIX.’ - -Note, as data sources such as JSTOR and Google Books are not restricted to only open access titles, ISBNs present in this section (that are not present in the Open Access ONIX feed) may belong to gated or closed titles. The OAPEN metadata source also contains both chapters and books, and there may be some inconsistencies in titles between Counter 4 and Counter 5 schemas that cause a title to be presented twice in the data with the same ISBN. - -These three tables are scrollable by using the scrollbar on the right-hand side of each table: - -``` eval_rst -.. image:: ../images/quality_isbns3.png - :width: 400 -``` - -The three tables can also be exported. Click on the three dots beside the ISBN column in any of the tables. - -``` eval_rst -.. image:: ../images/quality_isbns4.png - :width: 400 -``` - -Select ‘Download as CSV’ from the ‘Options’ popup menu: - -``` eval_rst -.. image:: ../images/quality_isbns5.png - :width: 400 -``` - -Depending on your internet browser, the CSV file may be saved to the Downloads folder. \ No newline at end of file diff --git a/docs/bad_project/user_guides/ebsco_db.md b/docs/bad_project/user_guides/ebsco_db.md deleted file mode 100644 index f97053ba..00000000 --- a/docs/bad_project/user_guides/ebsco_db.md +++ /dev/null @@ -1,157 +0,0 @@ -# EBSCO Dashboard - -EBSCO hosts collections of different publications including eBooks, some of which are open access. They provide data about the usage of these eBooks to publishers, such as University of Michigan Press. The eISBN used in the EBSCO data source is one of the identifiers for eBooks used by some publishers and platforms. To access EBSCO open access book titles, libraries can sign up here: - - -For the pilot project, the EBSCO data source is a manual data upload that was developed specifically for the University of Michigan Press. Usage data is available from January 2021. - -The sections of this dashboard are: -* [Filters](#filters) -* [EBSCO - Summary](#ebsco-summary) -* [EBSCO - Usage](#ebsco-usage) -* [EBSCO - Imprints](#ebsco-imprints) -* [EBSCO - Subjects](#ebsco-subjects) -* [EBSCO - Customers](#ebsco-customers) -* [EBSCO - Markets](#ebsco-markets) - -## Filters -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Book Title(s)’ OR ‘Select ISBN(s)’ OR ‘Select EISBN(s)’. -``` - -If you select a filter then add a second filter, the first filter will be automatically removed. For example, if you ‘Select Book Title(s)’, then ‘Select ISBN(s)’, the ‘Select Book Title(s)’ filter will be automatically removed. - -``` eval_rst -.. note:: A warning message may appear on the filters: ‘Terms list might be incomplete because the request is taking too long ...’. This is expected, and these filters can still be used but the full list of options may not be displayed. -``` - -### Select one value in a filter -Use the dropdown option in any filter field to choose a value e.g. an ISBN in the ‘Select ISBN(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter -To select multiple values in a specific filter field, keep selecting values one by one. This can be either via the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. - -For example, to clear one ISBN, click on the X next to the ISBN: - -``` eval_rst -.. image:: ../images/ebsco_filters1.png - :width: 400 -``` - -To remove multiple ISBNs at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/ebsco_filters2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the grey X to remove the filter. - -``` eval_rst -.. image:: ../images/ebsco_filters3.png - :width: 400 -``` - -## EBSCO Summary -The ‘EBSCO - Summary’ section shows the number of Titles, ISBNs, EISBNs and ‘RetrievalCount’ from EBSCO for any selected filters. When no filters are selected, a summary from the EBSCO data source for all University of Michigan Press titles is displayed in the dashboard. - -The ‘Book Titles’ table shows all book titles from the University of Michigan Press that are available from EBSCO. - -To look at a specific Title, scroll to it in the ‘Book Titles’ table, and hover beside the Title to show the +. Click on the + to apply a Title filter to the ‘EBSCO’ dashboard. All sections in this dashboard will automatically update to show data about this Title. - - -``` eval_rst -.. image:: ../images/ebsco_summary1.png - :width: 500 -``` - -To remove the Title filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the title filter. - -``` eval_rst -.. image:: ../images/ebsco_summary2.png - :width: 400 -``` - -Note: The data shown in this section is from the following variables: title, isbn, eisbn, RetrievalCount. - -### Exporting the data -To export the data from the table to a CSV file, hover in the top right corner of the graph or table to find three dots: - -``` eval_rst -.. image:: ../images/ebsco_summary3.png - :width: 500 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -## EBSCO Usage -Usage is the total number of times that a title has been requested from EBSCO. The ‘EBSCO - Usage' section shows the usage per month (in a graph and a table). - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘EBSCO - Summary’ section above. - -To look at a specific month in the bar graph in more detail, click on a bar in the bar graph. The dashboard will automatically update to use this time filter. To remove this time filter, reload the dashboard by clicking on the ‘EBSCO’ link in the navigation menu at the top right of this dashboard. - -Note: The data shown in this section is from the following variable: RetrievalCount. - -## EBSCO Imprints -The ‘EBSCO - Imprints' section shows the top imprints by usage in a donut graph and a table. - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘EBSCO - Summary’ section above. - -Note: The data shown in this section is from the following variable: ImprintPublisher. - -## EBSCO Subjects -The ‘EBSCO - Subjects' section shows the top imprints by usage in a donut graph and a table. - -Click on a subject in the donut graph to select a specific subject. All sections in this dashboard will automatically update. - -Alternatively, scroll to a subject in the table ‘Top Subjects by usage’, and hover beside the ‘Subject’ to show the +. Click on the + to apply a subject filter to the whole ‘EBSCO’ dashboard. - -``` eval_rst -.. image:: ../images/ebsco_subjects1.png - :width: 500 -``` - -To remove the Subject filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the Subject filter. - -``` eval_rst -.. image:: ../images/ebsco_subjects2.png - :width: 400 -``` - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘EBSCO - Summary’ section above. - -Note: The data shown in this section is from the following variable: Subjects. - - -## EBSCO Customers -The ‘EBSCO – Customers’ section shows downloads by EBSCO customers as a donut graph and table. -In the ‘Top Countries of Customers by usage’ donut graph, click on any of the EBSCO customers to apply a filter to the whole dashboard. - -Alternatively, scroll to a customer in the table ‘Top Customers by usage’, and hover beside the ‘Customer’ to show the +. Click on the + to apply a customer filter to the whole ‘EBSCO’ dashboard. - -To remove this filter by Customer, go to the filter bar at the top left of the dashboard. Click on the X to remove the Customer filter. - -Note that some Customer usage in the EBSCO data is for ‘Other’ customers. - -The donut graph and table can be exported to .CSV, see the ‘Exporting the data’ instructions in the ‘EBSCO -Summary’ section above. - -Note: The data shown in this section is from the following variables: Cust_Name, CustId, CountryName, Cust_StateProv. - -## EBSCO Markets -The ‘EBSCO – Markets’ section shows top markets by usage from the EBSCO data, as a donut graph and a table. - -In the ‘Top Markets by usage’ donut graph, click on any of the EBSCO markets to apply a filter to the whole dashboard. - -Alternatively, scroll to a market in the table ‘Top Markets by usage’, and hover beside the ‘Market’ to show the +. Click on the + to apply a market filter to the whole ‘EBSCO’ dashboard. - -To remove this filter by Market, go to the filter bar at the top left of the dashboard. Click on the X to remove the Market filter. - -The donut graph and table can be exported to .CSV, see the ‘Exporting the data’ instructions in the ‘EBSCO -Summary’ section above. - -Note: The data shown in this section is from the following variable: Market. diff --git a/docs/bad_project/user_guides/fulcrum_db.md b/docs/bad_project/user_guides/fulcrum_db.md deleted file mode 100644 index f00fdc62..00000000 --- a/docs/bad_project/user_guides/fulcrum_db.md +++ /dev/null @@ -1,170 +0,0 @@ -# Fulcrum Dashboard - -Fulcrum is a “community-developed, open-source platform for digital scholarship” which provides “users the ability to read books with associated digital enhancements, such as: 3-D models, embedded audio, video, and databases; zoomable online images, and interactive media.” (). - -In Fulcrum, DOIs are allocated to any digital content within eBooks, for example images and audio files. The Parent Title and Parent DOI are used for the eBook, and one Parent DOI can include multiple DOIs. In the default dashboard displayed, the number of downloads is the sum of downloads across all DOIs in Fulcrum, not just the parent DOIs. - -For the pilot project, the Fulcrum data source is a manual data upload for the University of Michigan Press, and usage data is available from April 2020 for both paid and open access eBooks. - -The University of Michigan Press Ebook Collection can be accessed at: - - -The sections of this dashboard are: -* [Filters](#fulcrum-filters) -* [Fulcrum - Summary](#fulcrum-summary) -* [Fulcrum - Books](#fulcrum-books) -* [Fulcrum - Usage](#fulcrum-usage) -* [Fulcrum - Institutions](#fulcrum-institutions) -* [Fulcrum - Publishers](#fulcrum-publishers) - -## Fulcrum - Filters - -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Parent Title(s)’ OR ‘Select Authors(s)’ OR ‘Select Parent ISBN(s)’ OR ‘Select Parent DOI(s)’ OR ‘Select ISBN(s)’ OR ‘Select DOI Handle(s)’ OR ‘Select Access type(s). -``` -If you select a filter then add a second filter, the first filter will be automatically removed. For example, if you ‘Select Parent Title(s)’, then ‘Select Author(s)’, the ‘Select Parent Title(s)’ filter will be automatically removed. - -``` eval_rst -.. note:: A warning message may appear on the filters: ‘Terms list might be incomplete because the request is taking too long ...’. This is expected, and these filters can still be used but the full list of options may not be displayed. -``` - -### Select one value in a filter - -Use the dropdown in any filter field to choose a value e.g. an Author in the ‘Select Author(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter - -To select multiple values in a specific filter field, keep selecting values one by one. This can be either using the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter - -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. -For example, to clear one author, click on the X next to the author: - -``` eval_rst -.. image:: ../images/fulcrum_select_authors1.png - :width: 400 -``` - -To remove multiple authors at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/fulcrum_select_authors2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter field in the top left of the dashboard. Click on the grey x to remove the filter. - -``` eval_rst -.. image:: ../images/fulcrum_select_authors3.png - :width: 400 -``` - -## Fulcrum - Summary - -The ‘Fulcrum - Summary’ section shows the number of Parent Titles, Parent DOIs, DOIs and downloads from Fulcrum for any selected filters. When no filters are selected, a summary of all titles and associated DOIs in the Fulcrum data source for the University of Michigan Press is displayed in the dashboard. - -There may be more DOIs than Titles because DOIs can be allocated to extra digital content associated with a single eBook, for example images and audio files. The number of downloads is across all University of Michigan Press DOIs in Fulcrum, not just the parent DOIs. - -Note: The data shown in this section is from the following variables: Total_Item_Requests, parent_title, parent_doi, doi. - -## Fulcrum - Books - -In the ‘Fulcrum - Books' section, the ‘# publications published each year’ bar graph shows the number of titles published by the University of Michigan Press per year (from the Fulcrum data). Hover over any of the months in this bar graph to see the number of titles published in that month. - -The ‘# Parent Titles per year’ table shows the same data as the ‘# publications published each year’ bar graph, and can be scrolled to view all years. - -``` eval_rst -.. image:: ../images/fulcrum_books1.png - :width: 400 -``` - -### Exporting the data - -To export the data from the bar graph or table to a .CSV file, hover in the top right corner of the graph or table to find three dots: - -``` eval_rst -.. image:: ../images/fulcrum_books2.png - :width: 500 -``` - -``` eval_rst -.. image:: ../images/fulcrum_books3.png - :width: 400 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’: - -``` eval_rst -.. image:: ../images/fulcrum_books4.png - :width: 400 -``` - -Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Using time filters - -To look at a specific month in the bar graph ‘# publications published each year’ in more detail, click on this month in the bar graph to add a time filter. The bar graph visualisation will now display the number of publications for one year. The rest of the dashboard will automatically update to use this time filter. - -To remove the time filter of a specific month, click on the grey x in the top left corner: - -``` eval_rst -.. image:: ../images/fulcrum_books5.png - :width: 400 -``` - -The dashboard will automatically refresh to show all titles and associated DOIs. - -Note: The data shown in this section is from the following variable: parent_title. - -## Fulcrum - Usage - -Usage is the total number of times that a Title has been requested from Fulcrum. The ‘Fulcrum - Usage' section shows the usage per month (in a graph and a table), and top authors by usage. - -All data in this section can be exported to CSV by following the ['Exporting the data'](#exporting-the-data) instructions in the ‘Fulcrum - Books’ section above: - -* ‘Monthly usage’ bar graph (hover over any month to see the number of downloads in that month) -* ‘Monthly usage’ table (usage per month in chronological order) -* ‘Top Authors by usage’ table (all authors, ranked by highest usage first) -* ‘Top Parent Titles by usage’ table (all titles, ranked by highest usage first) - -To look at a specific month in a bar graph in more detail, see the section above on [‘Using time filters’](#using-time-filters). - -``` eval_rst -.. tip:: In the ‘Top Parent Titles by usage’ table, click on any DOI in the Parent DOI column to go to the Fulcrum website where that open access eBook title can be viewed and downloaded. -``` - -Note: The data shown in this section is from the following variable: Total_Item_Requests. - -## Fulcrum - Institutions - -The ‘Fulcrum - Institutions’ section shows downloads by institution as a word cloud and table. - -In the ‘Top Institutions by usage’ word cloud – click on any of the institutions to apply a filter to the whole dashboard. To remove this filter by institution, click on the grey x in the top left corner of the dashboard: - -``` eval_rst -.. image:: ../images/fulcrum_institutions.png - :width: 400 -``` -Note that some usage in the Fulcrum data is for ‘Unknown Institution’. - -The ‘Top Institutions by usage’ table can be exported to .CSV, see the ['Exporting the data'](#exporting-the-data) instructions in the ‘Fulcrum - Books’ section above. - -Note: The data shown in this section is from the following variable: institution. - -## Fulcrum - Publishers - -The ‘Fulcrum - Institutions’ section shows downloads for different publishers which make up the University of Michigan Press, as a donut graph and a table. - -In the ‘Top Publishers by usage’ donut graph, hover over a publisher to view the number and % of downloads per publisher. Click on a publisher section of the donut graph to apply this as a filter to the whole dashboards. To remove this filter by publisher, click on the grey x in the top left corner of the dashboard. - -``` eval_rst -.. image:: ../images/fulcrum_publishers.png - :width: 400 -``` - -The ‘Top Publishers by usage’ table can be exported to .CSV by following the ['Exporting the data'](#exporting-the-data) instructions in the ‘Fulcrum - Books’ section above. - -Note: The data shown in this section is from the following variable: publisher. \ No newline at end of file diff --git a/docs/bad_project/user_guides/index.rst b/docs/bad_project/user_guides/index.rst deleted file mode 100644 index 2fd0a447..00000000 --- a/docs/bad_project/user_guides/index.rst +++ /dev/null @@ -1,16 +0,0 @@ - -.. toctree:: - :maxdepth: 1 - - overview_db - book_isbns_db - countries_db - institutions_db - mentions_db - authors_db - bisac_db - data_qual_db - fulcrum_db - muse_db - ebsco_db - diff --git a/docs/bad_project/user_guides/institutions_db.md b/docs/bad_project/user_guides/institutions_db.md deleted file mode 100644 index 8708f91c..00000000 --- a/docs/bad_project/user_guides/institutions_db.md +++ /dev/null @@ -1,202 +0,0 @@ -# Institutions Dashboard -The ‘Institutions’ dashboard shows usage data (or downloads of eBook chapters) by institutions. For the pilot project, this data comes from JSTOR, and the data is available from January 2018 onwards. For more information about JSTOR -see the [partner data](../overview/partner_data) section of this help guide. - -The sections of this dashboard are: -* [Filters](#filters) -* [Downloads](#downloads) - -## Filters -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Institution(s)’ OR ‘Select Title(s’) OR ‘Select ISBN(s)’. -``` - -If you use select a filter then and add a second filter, the first filter will be automatically removed. For example, if you ‘Select Institution(s)’, then ‘Select Title(s)’, the ‘Select Institution(s)’ filter will be automatically removed. - -The ‘Downloads’ section will automatically show the number of chapter downloads by institutions for all eBook chapters available from this publisher. Filters can be used to look at: -* Chapter downloads by institution(s) OR -* Chapter downloads for one or multiple Titles across institution(s) OR -* Chapter downloads for one or multiple ISBNs across institution(s). - -### Select one value in a filter -Use the dropdown option in any filter field to choose a value e.g. a Title in the ‘Select Title(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter -To select multiple values in a specific filter field, keep selecting values one by one. This can be either via the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. - -For example, to clear one Title, click on the X next to the Title: - -``` eval_rst -.. image:: ../images/institutions_filters1.png - :width: 400 -``` - -To remove multiple titles at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/institutions_filters2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/institutions_filters3.png - :width: 500 -``` - -## Downloads -In the ‘Downloads’ section, data about chapter downloads and institutions is presented in different ways: -* Monthly chapter downloads bar graph - ‘# monthly chapter downloads from JSTOR by the top 10 institutions’ -* Top 1000 institutions table - ‘Top 1000 institutions for # chapter downloads from JSTOR’ -* Top 20 institutions word cloud - ‘Top 20 institutions for # chapter downloads from JSTOR from Jan 2018’ -* Top institutions and titles table - ‘Top institutions and Titles by # chapter downloads JSTOR’ - -Note: The data shown in this section is from the following variable: JSTOR chapters (Total_Item_Requests). - -### Monthly chapter downloads bar graph -The bar graph ‘# monthly chapter downloads from JSTOR by the top 10 institutions’ shows the number of chapter downloads per institution by month, for the 10 institutions with the highest total of chapter downloads from JSTOR since January 2018. - -Hover over a month bar in this bar graph to see the number of chapter downloads for each of the top 10 institutions. - -``` eval_rst -.. image:: ../images/institutions_downloads1.png - :width: 400 -``` - -The data contained in the graph can be exported to CSV by using the ‘Options’ popup. Hover to the right of the bar graph title ‘# monthly chapter downloads from JSTOR by the top 10 institutions’. Three dots will appear. - -``` eval_rst -.. image:: ../images/institutions_downloads2.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -To look at a specific month and institution in more detail, click on a month and institution in the bar graph. In the example in the next screenshot, the ‘Macquarie University’ part of the 2021-08-01 bar was clicked on. - -``` eval_rst -.. image:: ../images/institutions_downloads3.png - :width: 400 -``` - -In the ‘Select filters to apply’ popup, select ‘Apply’. The dashboard will update to show data for this month and institution. - -``` eval_rst -.. image:: ../images/institutions_downloads4.png - :width: 400 -``` - -``` eval_rst -.. note:: Clicking on the ‘Other’ section of a bar graph will cause incorrect filters to be applied, as ‘Other’ represents combined data from multiple institutions. -``` - -To remove the institution filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/institutions_downloads5.png - :width: 400 -``` - -To remove the time filter, reload the dashboard by clicking on the ‘Institutions’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/institutions_downloads6.png - :width: 300 -``` - -The dashboard will automatically refresh to show downloads for all institutions from this publisher. - -### Top 1000 institutions table -The ‘Top 1000 institutions for # chapter downloads from JSTOR’ table shows the top institutions ranked in order of highest downloads for this publisher. By default this shows downloads for all titles from this publisher, but it will also show top institutions for whatever subset of books you have selected via the Filter section of the top of the ‘Institutions’ dashboard. - -In this table, you can select one institution to see downloads for just that institution. Click on the + sign in a circle beside an institution. - -``` eval_rst -.. image:: ../images/institutions_downloads7.png - :width: 500 -``` - -To clear an institution that was selected in the table, go to the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/institutions_downloads8.png - :width: 400 -``` - -Do not use the - sign in a circle to try to remove the filter. This will add another filter that excludes the selected institution. - -``` eval_rst -.. image:: ../images/institutions_downloads9.png - :width: 500 -``` - -The data contained in the table can be exported to CSV by using the ‘Options’ popup. Hover to the right of the table title ‘Top 1000 institutions for # chapter downloads from JSTOR’. Three dots will appear. - -``` eval_rst -.. image:: ../images/institutions_downloads10.png - :width: 500 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Top 20 institutions word cloud -The word cloud ‘Top 20 institutions for # chapter downloads from JSTOR’ shows the top 20 institutions that had the highest number of chapter downloads, using JSTOR as a data source. Note that this data source starts from January 2018. - -If specific institutions have been selected in the filters at the top of the dashboard, then this section will reflect only those institutions. - -Click on any of the institution names to see downloads for just that single institution applied across this ‘Institutions’ dashboard. For example, click on ‘University of Edinburgh’ in the word cloud: - -``` eval_rst -.. image:: ../images/institutions_downloads11.png - :width: 600 -``` - -The dashboard will update to show data for only the selected institution, including monthly chapter downloads from JSTOR and the top titles. - -To return to seeing data for all institutions, go to the top of the dashboard. In the filter bar at the top left of the dashboard, click on X to remove the filter being ‘University of Edinburgh’ in this example. - -``` eval_rst -.. image:: ../images/institutions_downloads12.png - :width: 400 -``` - -### Top institutions and titles table -The ‘Top institutions and Titles by # chapter downloads JSTOR’ table shows the Titles with the highest chapter downloads per institution. This means that there are duplicate ISBNs and titles in the ‘ISBN’ and ‘Title’ columns of this table that show downloads for different institutions. - -In this table, you can select one institution to see downloads for just that institution. Click on the + sign in a circle beside an institution. - -``` eval_rst -.. image:: ../images/institutions_downloads13.png - :width: 500 -``` - -It is also possible to select a particular ISBN or title by clicking on the + sign in a circle beside an ISBN or Title. - -To clear an institution, ISBN or title that was selected in the table, go to the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/institutions_downloads14.png - :width: 400 -``` - -Do not use the - sign in a circle to try to remove the filter. This will add another filter that excludes the selected institution, ISBN or title. - -``` eval_rst -.. image:: ../images/institutions_downloads15.png - :width: 400 -``` - -The data contained in the table can be exported to CSV by using the ‘Options’ popup. Hover to the right of the table title ‘Top institutions and Titles by # chapter downloads JSTOR’. Three dots will appear. - -``` eval_rst -.. image:: ../images/institutions_downloads16.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. diff --git a/docs/bad_project/user_guides/mentions_db.md b/docs/bad_project/user_guides/mentions_db.md deleted file mode 100644 index b45b89e2..00000000 --- a/docs/bad_project/user_guides/mentions_db.md +++ /dev/null @@ -1,137 +0,0 @@ -# Mentions and Events Dashboard - -Crossref Event Data captures online discussion about research outputs including eBooks, such as ‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’ (). - -For the pilot project, the Crossref Event data for the University of Michigan Press is from May 2018 onwards, and the event sources are: -* Crossref - relationships, references, and links to DataCite registered content -* Datacite - links to Crossref registered content -* Twitter - Mentions in tweets -* Wikipedia - References on Wikipedia pages - -The ‘Mentions and Events’ dashboard shows Crossref Event data from this publisher for all ISBNs by default. The filters can be used to select specific event sources, Title(s) or ISBN(s). - -The sections of this dashboard are: -* [Filters](#filters) -* [Mentions and Events](#mentions-and-events) - -## Filters - -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select event source(s) from Crossref Events’ OR ‘Select Title(s)’ OR ‘Select ISBN(s)’. -``` - -If you select a filter then add a second filter, the first filter will be automatically removed. For example, if you use ‘Select Title(s)’, then ‘Select ISBN(s)’, the ‘Select Title(s)’ filter will be automatically removed. - -### Select one value in a filter -Use the dropdown option in any filter field to choose a value e.g. a Title in the ‘Select Title(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter -To select multiple values in a specific filter field, keep selecting values one by one. This can be either via the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. -For example, to clear one Title, click on the X next to the Title: - -``` eval_rst -.. image:: ../images/mentions_filters1.png - :width: 400 -``` - -To remove multiple titles at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/mentions_filters2.png - :width: 400 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/mentions_filters3.png - :width: 600 -``` - -## Mentions and Events - -The ‘Mentions and Events’ section shows data from Crossref in four ways: -* Events as a donut graph - ‘# events from Crossref’ -* Events by source as a table - ‘# Crossref events by source’ -* Events per month as a bar graph - ‘# events per month from Crossref’ -* Top titles table - ‘Top titles by # Crossref events’ - -### Events as a donut graph - -Hover over a Crossref Event source in the ‘# events from Crossref’ donut graph to see the number of mentions and events from that event source. The event sources are Crossref, Datacite, Twitter and Wikipedia. - -Click on any Crossref source e.g. Twitter in a donut graph to update the dashboard to show mentions and events for just that Crossref source. - -To clear a Crossref source that was selected in the donut graph, go to the filter bar at the top left of the dashboard. Click on the X to remove the filter. - -``` eval_rst -.. image:: ../images/mentions_events1.png - :width: 400 -``` - -The data contained in the donut graph can be exported to CSV by clicking on the three dots beside the graph title ‘# events from Crossref’, and selecting ‘Download as CSV’ from the ‘Options’ popup. - -``` eval_rst -.. image:: ../images/mentions_events2.png - :width: 600 -``` - -### Events by source as a table - -The ‘# Crossref events by source’ table shows a breakdown of the Crossref events from different sources (Crossref, Datacite, Twitter and Wikipedia). - -The data contained in this table can be exported to CSV by clicking on the three dots beside the table title ‘# Crossref events by source’, and selecting ‘Download as CSV’ from the ‘Options’ popup. - -``` eval_rst -.. image:: ../images/mentions_events3.png - :width: 500 -``` - -### Events per month as a bar graph - -The ‘# events per month from Crossref’ bar graph shows the Crossref events per month from different sources (Crossref, Datacite, Twitter and Wikipedia). - -Click on a month and bar section in ‘# events per month from Crossref’ to see more detail for a specific month and source of Crossref events (Crossref, Datacite, Twitter and Wikipedia). - -``` eval_rst -.. image:: ../images/mentions_events4.png - :width: 300 -``` - -In the popup ‘Select filters to apply’, select ‘Apply’. - -``` eval_rst -.. image:: ../images/mentions_events5.png - :width: 400 -``` - -To clear a month filter, reload the dashboard by clicking on the ‘Mentions and Events’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/mentions_events6.png - :width: 300 -``` - -The data contained in this table can be exported to CSV by clicking on the three dots beside the graph title ‘# events per month from Crossref’, and selecting ‘Download as CSV’ from the ‘Options’ popup. - -``` eval_rst -.. image:: ../images/mentions_events7.png - :width: 600 -``` - - -### Top titles table - -The ‘Top titles by # Crossref events’ table shows the titles ranked by the highest number of Crossref events per title (includes Crossref, Datacite, Twitter and Wikipedia). - -The data contained in this table can be exported to CSV by clicking on the three dots beside the table title ‘Top titles by # Crossref events’, and selecting ‘Download as CSV’ from the ‘Options’ popup. - -``` eval_rst -.. image:: ../images/mentions_events8.png - :width: 600 -``` diff --git a/docs/bad_project/user_guides/muse_db.md b/docs/bad_project/user_guides/muse_db.md deleted file mode 100644 index 1b628baf..00000000 --- a/docs/bad_project/user_guides/muse_db.md +++ /dev/null @@ -1,169 +0,0 @@ -# MUSE Dashboard -Project MUSE provides a platform which hosts journals and books from multiple publishers including University of Michigan Press and University College London. Some of the MUSE offerings are open access eBooks (). To see open access book titles on Project MUSE, go to - -For the pilot project, the MUSE data source is a manual data upload for the University of Michigan Press, and usage data is available from January 2020. - -The sections of this dashboard are: -* [Filters](#filters) -* [MUSE Summary](#muse-summary) -* [MUSE Books and Items](#muse-books-and-items) -* [MUSE Usage](#muse-usage) -* [MUSE Access](#muse-access) -* [MUSE Country](#muse-country) -* [MUSE Institutions](#muse-institutions) -* [MUSE Format](#muse-format) - -## Filters -A filter is a way to narrow a search and look for more specific information. - -``` eval_rst -.. tip:: Only one filter can be used at a time – ‘Select Book Title(s)’ OR ‘Select ISSN/ISBN(s)’ OR ‘Select Item Title(s)’ OR ‘Select Resource_ID(s)’ OR ‘Select Authors(s)’ OR ‘Select Access type(s)'. -``` - -If you select a filter then add a second filter, the first filter will be automatically removed. For example, if you ‘Select Book Title(s)’, then ‘Select Author(s)’, the ‘Select Book Title(s)’ filter will be automatically removed. - -``` eval_rst -.. note:: A warning message may appear on the filters: ‘Terms list might be incomplete because the request is taking too long ...’. This is expected, and these filters can still be used but the full list of options may not be displayed. -``` -### Select one value in a filter - -Use the dropdown in any filter field to choose a value e.g. an Author in the ‘Select Author(s)’ field, or start typing in a filter field to search all values by specific text. There is no search button to click, and the rest of the dashboard will automatically update. - -### Select multiple values in a filter - -To select multiple values in a specific filter field, keep selecting values one by one. This can be either using the dropdown OR by typing directly in the field and selecting from the matches. - -### Remove value/values from a filter - -Selected value(s) of a filter can be removed one at a time, or all removed at the same time. -For example, to clear one author, click on the X next to the author: - -``` eval_rst -.. image:: ../images/muse_filters1.png - :width: 300 -``` - -To remove multiple authors at the same time, click on the grey circle icon with a white X inside it: - -``` eval_rst -.. image:: ../images/muse_filters2.png - :width: 300 -``` - -Another way to remove applied filters in a dashboard, is to use the filter bar at the top left of the dashboard. Click on the grey X to remove the filter. - -``` eval_rst -.. image:: ../images/muse_filters3.png - :width: 400 -``` - -## MUSE Summary -The ‘MUSE - Summary’ section shows the number of Book titles, Resource IDs, ISSN/ISBNs, Item titles and requests from MUSE for any selected filters. When no filters are selected, a summary in the MUSE data source for all University of Michigan Press titles is displayed in the dashboard. Note that there can be multiple ISBNs per title for different formats, for example pdf, html or hard cover. - -Note: The data shown in this section is from the following variables: resource, resource_id, issn_isbn, fulltext_title. - -## MUSE Books and items -In the ‘MUSE – Books and Items' section, the number of books and the number of items per launch year are shown in two bar graphs and a table: -* ‘# Book titles per launch year’ (book titles only) -* ‘# Item titles per launch year’ (item titles only) -* ‘# Books and Items per launch year’ (books and items combined) - -Hover over a year in the bar graphs to see the number of book or item titles per launch year. -Note: The data shown in this section is from the following variables: resource, resource_id, issn_isbn, fulltext_title. - -### Using time filters -To look at a specific launch year in more detail, click on a year in the bar graph. The dashboard will be updated to show usage data from MUSE for that year only. - -To remove the launch year filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the launch filter. - -``` eval_rst -.. image:: ../images/muse_books1.png - :width: 300 -``` - -### Exporting the data -To export the data from the bar graphs or table to a .CSV file, hover in the top right corner of the graph or table to find three dots: - -``` eval_rst -.. image:: ../images/muse_books2.png - :width: 600 -``` - -``` eval_rst -.. image:: ../images/muse_books3.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -## MUSE Usage -Usage is the total number of times that a title or item has been requested from MUSE. The ‘MUSE - Usage' section shows the usage per month (in a graph and a table), and top book and item titles by usage. - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘MUSE – Books and Items’ section above: -* ‘Monthly usage’ bar graph (hover over any month to see the number of downloads in that month) -* ‘Monthly usage’ table (usage per month in chronological order) -* ‘Top 10 Book titles by usage’ table (all titles ranked by highest usage first) -* ‘Top 10 Item titles by usage’ table (all items ranked by highest usage first) - -To look at a specific month in the bar graph in more detail, click on a bar in the bar graph. The dashboard will automatically update to use this time filter. To remove this time filter, reload the dashboard by clicking on the ‘MUSE’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. tip:: Tip: In the ‘Top 10 Book titles by usage’ and ‘Top 10 Item titles by usage’ table, click on any URL in the [resource_url] or [fulltext_url] column to go to the MUSE website where that open access eBook title/item can be viewed and downloaded. -``` - -Note: The data shown in this section is from the following variable: requests - -## MUSE Access -Access is how the title or item was requested from MUSE, and can be one of the following types: ‘open_access’ or ‘gated’. The ‘MUSE - Access’ section shows the top access types in a donut graph and table. - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘MUSE - Books and Items’ section above. - -Note: The data shown in this section is from the following variable: access - -## MUSE Country -Country is the source of the request for a title or item from MUSE. The ‘MUSE - Country’ section shows the top countries in a donut graph ‘Top Countries by usage’ and table ‘Top Countries by usage’. - -Click on a country in the donut graph to select a specific country. All sections in this dashboard will automatically update, for example the ‘MUSE - Institutions’ section will show all institutions in the selected country only. - -Alternatively, scroll to a country in the table ‘Top Countries by usage’, and hover beside the ‘Country’ name to show the +. Click on the + to apply a country filter to the ‘MUSE’ dashboard. - -``` eval_rst -.. image:: ../images/muse_country1.png - :width: 400 -``` - -To remove the Country filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the Country filter. - -``` eval_rst -.. image:: ../images/muse_country2.png - :width: 200 -``` - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘MUSE - Books and Items’ section above. - -Note: The data shown in this section is from the following variable: country - -## MUSE Institutions -Institution refers to the institution where the request for a title or item from MUSE originated. The ‘MUSE - Institution’ section shows the top institutions in a donut graph and table. - -Click on an institution in the donut graph to select a specific institution. All sections in this dashboard will automatically update. - -Alternatively, scroll to an institution in the table ‘Top Institutions by usage’, and hover beside the institution name to show the +. Click on the + to apply an Institution filter to the ‘MUSE’ dashboard. - -To remove the Institution filter, go to the filter bar at the top left of the dashboard. Click on the X to remove the Institution filter. - -``` eval_rst -.. image:: ../images/muse_institutions1.png - :width: 250 -``` - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘MUSE - Books and Items’ section above. - -Note: The data shown in this section is from the following variable: institution - -## MUSE Format -Format is the format of how the title or item was downloaded from MUSE, and can be of the types ‘pdf’ or ‘html’. The ‘MUSE - Format’ section shows the top format types in a donut graph and table. - -All data in this section can be exported to CSV by following the ‘Exporting the data’ instructions in the ‘MUSE - Books and Items’ section above. - -Note: The data shown in this section is from the following variable: format diff --git a/docs/bad_project/user_guides/overview_db.md b/docs/bad_project/user_guides/overview_db.md deleted file mode 100644 index a43abc0e..00000000 --- a/docs/bad_project/user_guides/overview_db.md +++ /dev/null @@ -1,123 +0,0 @@ -# Overview Dashboard - -The ‘Overview’ dashboard presents a summary of all open access eBooks published by the University of Michigan Press. To look at other dashboards showing usage of specific titles and authors by country, institution, or subject, use the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/overview_intro1.png - :width: 200 -``` - -For the pilot project, data sources of usage data cover different time periods. For example, the University of Michigan Press has data sources for the following time periods: -* Usage data is from September 2020 for Google Books -* Usage data is from January 2018 onwards for JSTOR -* Usage data is from January 2018 to March 2020 for OAPEN (Counter 4), and from April 2020 for OAPEN (Counter 5) -* Event data is from May 2018 onwards for Crossref Events - -The following sections are included in the ‘Overview’ dashboard: -* [Summary ONIX data](#summary-onix-data) -* [Summary Usage data](#summary-usage-data) - -## Summary ONIX data -The ‘Summary - ONIX data’ section shows a summary of the ONIX feed from the University of Michigan Press. ONIX for Books (ONline Information eXchange) is a standard format that book publishers use to share information about the books that they have published (). - -This section contains: -* Number of unique ISBNs in a bar graph and table - ‘# unique ISBNs in the ONIX feed’ -* List of all titles in the ONIX feed as a table - ‘All titles in the ONIX feed’ - -### Number of unique ISBNs -Hover over a year in the ‘# unique ISBNs in the ONIX feed’ bar graph to see the number of ISBNs published in that year. - -Click on any bar in the bar graph, and the dashboard will be updated to show ONIX and usage data for the specified year, including a list of titles published that year in the ‘All titles in the ONIX feed’ table. Note that the ‘Summary - Usage data’ section will show the years after the publication year, as usage of an eBook generally occurs after the publication year. - -To remove the year filter, use the filter bar at the top left of the dashboard. Click on the X to remove the year filter. - -``` eval_rst -.. image:: ../images/overview_summary1.png - :width: 200 -``` - -The data contained in the ‘# unique ISBNs in the ONIX feed’ table can be exported to CSV by using the ‘Options’ popup. Hover to the right of the ‘ISBNs’ column to show three dots. - -``` eval_rst -.. image:: ../images/overview_summary2.png - :width: 300 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### List of all titles in the ONIX feed -Navigate the ‘All titles in the ONIX feed’ table to view more Titles, by using the scrollbar and page numbers. You can also change the number of rows displayed via ‘Rows per page’. - -``` eval_rst -.. image:: ../images/overview_summary3.png - :width: 600 -``` - -The data contained in the ‘All titles in the ONIX feed’ table can be exported to CSV by using the ‘Options’ popup. Hover to the right of the ‘Book ISBNs’ column to show three dots. - -``` eval_rst -.. image:: ../images/overview_summary4.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -## Summary Usage data -The ‘Summary - Usage data’ section usage data comes from different sources (OAPEN, Google Books, JSTOR and Crossref): -* ISBN summary -* Downloads from all sources -* Views from all sources -* Events from all sources - -### ISBN summary -The start of the ‘Summary - Usage data’ section shows a summary of the number of ISBNs with usage data for each data source (OAPEN, Google Books, JSTOR and Crossref). - -The two bar graphs show the number of ISBNs with usage data from Google Books, JSTOR, OAPEN and Crossref Events combined: -* ‘# ISBNs per month with usage data’ -* ‘# ISBNs per year with usage data’ - note that this data is not the same as the individual months added together, as it shows the number of ISBNs across a whole year that had usage data recorded about them. - -Hover over any of the months/years in these bar graphs to see the number of ISBNs in that month/year that have usage data. - -Click on any bar in the bar graphs, the dashboard will be updated to show only the ONIX and usage data for the specified month/year, including a list of titles published that month/year in the ‘All titles in the ONIX feed’ table. Note that the ‘Summary - Usage data’ section will show only the years after the publication year, where usage of an eBook occurred after the publication year. - -To remove the month/year filter, reload the dashboard by clicking on the ‘Overview’ link in the navigation menu at the top right of this dashboard. - -``` eval_rst -.. image:: ../images/overview_intro1.png - :width: 200 -``` - -The data contained in the ‘Summary - Usage data’ bar graphs can be exported to CSV by using the ‘Options’ popup. Hover to the right of the bar graph title to show three dots e.g. for ‘# ISBNs per year with usage data’: - -``` eval_rst -.. image:: ../images/overview_summary5.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Downloads from all sources -This section shows downloads from the following sources combined together - book downloads from OAPEN and GoogleBooks, and chapter downloads from JSTOR. - -Hover over any of the months in the bar graph to see the number of downloads for that month. - -Click on any bar in the bar graph, the dashboard will be updated to show the downloads, views and events data for the specified month. - -To remove the month filter, reload the dashboard by clicking on the ‘Overview’ link in the navigation menu at the top right of this dashboard. - -The data contained in this bar graph can be exported to CSV by using the ‘Options’ popup. Hover in the top right corner of the bar graph to show three dots. - -``` eval_rst -.. image:: ../images/overview_summary6.png - :width: 600 -``` - -Click on the three dots to open the ‘Options’ popup, and select ‘Download as CSV’. Depending on your internet browser, the CSV file may be saved to the Downloads folder. - -### Views from all sources -This section shows views from Google Books. See the ‘Downloads from all sources’ section on this help page for how to add and remove a month filter, and export data to CSV. - -### Events from all sources -This section shows events and mentions from Crossref Events. Crossref Event Data captures online discussion about research outputs including eBooks, such as ‘a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, or discussion and comment on social media’ (). - -See the [‘Downloads from all sources’](#downloads-from-all-sources) section on this help page for how to add and remove a month filter, and export data to CSV. diff --git a/docs/conf.py b/docs/conf.py index c47d3279..877a6a93 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -10,26 +10,31 @@ # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # -# import os -# import sys # sys.path.insert(0, os.path.abspath('.')) +import sys import os import shutil from pathlib import Path from recommonmark.transform import AutoStructify -from generate_schema_csv import generate_csv, generate_latest_files +from generate_schema_csv import generate_csv, generate_csv_pdf, generate_latest_files -# -- Formatting -------------------------------------------------------------- +# -- Project information ----------------------------------------------------- + +project = "Book Usage Data Workflows" +copyright = "2020-2022 Curtin University" +author = "Curtin University" + + +# -- Options for PDFL output ------------------------------------------------- latex_elements = { 'sphinxsetup': "verbatimforcewraps, verbatimmaxunderfull=0", + 'extraclassoptions': 'openany,oneside', 'preamble': r''' \usepackage[none]{hyphenat} \usepackage{makeidx} - \usepackage{ragged2e} - \RaggedRight \makeindex \makeatletter \renewenvironment{theindex} @@ -41,12 +46,6 @@ } -# -- Project information ----------------------------------------------------- - -project = "Book Usage Data Workflows" -copyright = "2022-2025 Curtin University" -author = "Curtin University" - # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be @@ -58,8 +57,11 @@ "sphinx.ext.intersphinx", "autoapi.extension", "recommonmark", + "sphinx.ext.autodoc", ] +autodoc_typehints = 'description' + # Auto API settings: https://github.com/readthedocs/sphinx-autoapi autoapi_type = "python" autoapi_dirs = ["../oaebu_workflows"] @@ -67,6 +69,8 @@ autoapi_python_use_implicit_namespaces = True autoapi_root = "oaebu_workflows/api" +autoapi_python_use_implicit_namespaces = True +autoapi_python_class_content = 'both' # Add any paths that contain templates here, relative to this directory. templates_path = ["templates"] @@ -99,8 +103,23 @@ def setup(app): app.add_config_value("recommonmark_config", {"enable_eval_rst": True, "auto_toc_tree_section": "Contents"}, True) app.add_transform(AutoStructify) - -generate_csv(schema_dir="../oaebu_workflows/database/schema") -generate_latest_files() -html_build_dir = "_build/html" -Path(html_build_dir).mkdir(exist_ok=True, parents=True) +# -- Build options to format tables for html or pdf output ------------------------------------------------- + +# Determine the command used to build the documentation +build_command = ' '.join(sys.argv) + +if 'html' in build_command: + generate_csv(schema_dir="../oaebu_workflows/database/schema") + generate_latest_files() + html_build_dir = "_build/html" + Path(html_build_dir).mkdir(exist_ok=True, parents=True) +elif 'latexpdf' in build_command: + generate_csv_pdf(schema_dir="../oaebu_workflows/database/schema") + generate_latest_files() + latex_build_dir = "_build/latex" + Path(latex_build_dir).mkdir(exist_ok=True, parents=True) +else: + generate_csv(schema_dir="../oaebu_workflows/database/schema") + generate_latest_files() + html_build_dir = "_build/html" + Path(html_build_dir).mkdir(exist_ok=True, parents=True) diff --git a/docs/generate_schema_csv.py b/docs/generate_schema_csv.py index 54214c88..cff041ab 100644 --- a/docs/generate_schema_csv.py +++ b/docs/generate_schema_csv.py @@ -49,9 +49,9 @@ def schema_to_csv(*, schema: List, output: List, prefix: str = ""): ffield = field["fields"] schema_to_csv(schema=ffield, output=output, prefix=f"{prefix}{fname}.") - def generate_csv(*, schema_dir): """Convert all observatory schema files in JSON format to CSV for inclusion in Sphinx. + Split long text for pdf :param schema_dir: Path to schema directory. """ @@ -62,6 +62,7 @@ def generate_csv(*, schema_dir): shutil.rmtree(dst_dir) Path(dst_dir).mkdir(exist_ok=True, parents=True) + max_length = 30 for schema_file in schema_files: filename = os.path.basename(schema_file) @@ -109,6 +110,46 @@ def generate_latest_files(): src_file = table_schemas[table][-1] shutil.copyfile(src_file, dst_path) + +def wrap_long_first_words(text, max_length): + """ + Break long strings so they will wrap on a new line in tables + text = text to split + max_length = number of characters to split text + """ + if len(text) > max_length: + wrapped_text = "\n".join([text[i:i+max_length] for i in range(0, len(text), max_length)]) + return wrapped_text + else: + return text + +def generate_csv_pdf(*, schema_dir): + """Convert all observatory schema files in JSON format to CSV for inclusion in Sphinx. + Split long text for pdf output + :param schema_dir: Path to schema directory. + """ + + schema_files = glob(os.path.join(schema_dir, "*.json")) + dst_dir = "schemas" + + if os.path.exists(dst_dir): + shutil.rmtree(dst_dir) + + Path(dst_dir).mkdir(exist_ok=True, parents=True) + max_length = 30 + + for schema_file in schema_files: + filename = os.path.basename(schema_file) + filename = filename[:-4] + "csv" # Remove json and add csv suffix + with open(schema_file, "r", encoding="utf-8") as f: + data = f.read() + schema = json.loads(data) + rows = list() + schema_to_csv(schema=schema, output=rows) + df = pd.DataFrame(rows) + df.iloc[:, 0] = df.iloc[:, 0].apply(wrap_long_first_words, args=(max_length,)) + df.to_csv(os.path.join(dst_dir, filename), index=False) + if __name__ == "__main__": generate_csv(schema_dir=os.path.join("..", "oaebu_workflows", "database", "schema")) diff --git a/docs/logos/crossref-events.svg b/docs/logos/crossref-events.svg new file mode 100644 index 00000000..6eeb033b --- /dev/null +++ b/docs/logos/crossref-events.svg @@ -0,0 +1,68 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/logos/crossref-metadata.svg b/docs/logos/crossref-metadata.svg new file mode 100644 index 00000000..2e4cdd32 --- /dev/null +++ b/docs/logos/crossref-metadata.svg @@ -0,0 +1,69 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/logos/fulcrum.png b/docs/logos/fulcrum.png new file mode 100644 index 00000000..4e1c4017 Binary files /dev/null and b/docs/logos/fulcrum.png differ diff --git a/docs/logos/irus.png b/docs/logos/irus.png new file mode 100644 index 00000000..7fd25b39 Binary files /dev/null and b/docs/logos/irus.png differ diff --git a/docs/logos/thoth.png b/docs/logos/thoth.png new file mode 100644 index 00000000..f80aeaf7 Binary files /dev/null and b/docs/logos/thoth.png differ diff --git a/docs/oaebu_workflows/static/onix_workflow_1.png b/docs/oaebu_workflows/static/onix_workflow_1.png index c4e7605a..b557cf62 100644 Binary files a/docs/oaebu_workflows/static/onix_workflow_1.png and b/docs/oaebu_workflows/static/onix_workflow_1.png differ diff --git a/docs/oaebu_workflows/static/onix_workflow_2.png b/docs/oaebu_workflows/static/onix_workflow_2.png index f1a7708a..b42f880a 100644 Binary files a/docs/oaebu_workflows/static/onix_workflow_2.png and b/docs/oaebu_workflows/static/onix_workflow_2.png differ diff --git a/docs/oaebu_workflows/static/onix_workflow_3.png b/docs/oaebu_workflows/static/onix_workflow_3.png index e5cba15c..72e5d01e 100644 Binary files a/docs/oaebu_workflows/static/onix_workflow_3.png and b/docs/oaebu_workflows/static/onix_workflow_3.png differ diff --git a/docs/oaebu_workflows/static/onix_workflow_all.png b/docs/oaebu_workflows/static/onix_workflow_all.png index 59539d70..56db952e 100644 Binary files a/docs/oaebu_workflows/static/onix_workflow_all.png and b/docs/oaebu_workflows/static/onix_workflow_all.png differ diff --git a/docs/oaebu_workflows/telescopes/crossref_events.md b/docs/oaebu_workflows/telescopes/crossref_events.md new file mode 100644 index 00000000..f54ef202 --- /dev/null +++ b/docs/oaebu_workflows/telescopes/crossref_events.md @@ -0,0 +1,62 @@ +# Crossref Events + +> When someone links their data online, or mentions research on a social media site, we capture that event and make it + available for anyone to use in their own way. We provide the unprocessed data—you decide how to use it. + +Before the expansion of the Internet, most discussion about scholarly content stayed within scholarly content, +with articles citing each other. +With the growth of online platforms for discussion, publication and social media, +we have seen discussions extend into new, non-traditional venues. +Crossref Event Data captures this activity and acts as a hub for the storage and distribution of this data. +An event may be a citation in a dataset or patent, a mention in a news article, Wikipedia page or on a blog, +or discussion and comment on social media. + +When someone links their data online, or mentions research on, for example, Twitter, +Wikipedia, or Reddit, Crossref’s uses a set of APIs to captures and records those events in +their ‘Event dataset’. Events are tracked via their DOI and URLs, which enables Crossref to +monitor where it’s been shared, linked, bookmarked, referenced or commented on. +Crossref Event Data currently contains events from a range of data sources, including +Crossref Metadata, DataCite Metadata, F1000Prime (Recommendations of research publications, +Hypothes.is, The Lens (Cambia), Newsfeed, Reddit, Reddit Links, Stack Exchange Network, +Twitter, Wikipedia, and Wordpress.com + +See the crossref events [page](https://www.crossref.org/services/event-data/), and [data details](https://www.eventdata.crossref.org/guide/data/events/), for more information. + +The corresponding table created in BigQuery is `crossref.crossref_eventsYYYYMMDD`. + +```eval_rst ++------------------------------+---------+ +| Summary | | ++==============================+=========+ +| Average runtime | 2 hours | ++------------------------------+---------+ +| Average download size | 10 GB | ++------------------------------+---------+ +| Harvest Type | API | ++------------------------------+---------+ +| Harvest Frequency | Weekly | ++------------------------------+---------+ +| Runs on remote worker | True | ++------------------------------+---------+ +| Catchup missed runs | False | ++------------------------------+---------+ +| Table Write Disposition | Append | ++------------------------------+---------+ +| Update Frequency | Daily | ++------------------------------+---------+ +| Credentials Required | No | ++------------------------------+---------+ +| Uses Telescope Template | Stream | ++------------------------------+---------+ +``` + +## Latest schema + +``` eval_rst +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| +.. csv-table:: + :file: ../../schemas/crossref_events_latest.csv + :width: 100% + :header-rows: 1 + :class: longtable +``` \ No newline at end of file diff --git a/docs/oaebu_workflows/telescopes/crossref_metadata.md b/docs/oaebu_workflows/telescopes/crossref_metadata.md new file mode 100644 index 00000000..50ffc612 --- /dev/null +++ b/docs/oaebu_workflows/telescopes/crossref_metadata.md @@ -0,0 +1,27 @@ +# Crossref Metadata + +Crossref is a non-for-profit membership organisation working on making scholarly communications better. +It is an official Digital Object Identifier (DOI) Registration Agency of the International DOI Foundation. +They provide metadata for every DOI that is registered with Crossref. + +Crossref Members send Crossref scholarly metadata on research which is collated and +standardised into the Crossref metadata dataset. This dataset is made available through +services and tools for manuscript tracking, searching, bibliographic management, +library systems, author profiling, specialist subject databases, scholarly sharing networks +. _- source: [Crossref Metadata](https://www.crossref.org/services/metadata-retrieval/)_ +and [schema details](https://github.com/Crossref/rest-api-doc/blob/master/api_format.md). + +The BigQuery table created by the Crossref Metadata telescope from the [Academic Observatory workflows](https://academic-observatory-workflows.readthedocs.io/en/latest/telescopes/crossref_metadata.html) is queried +with the list of ISBNs from a publisher's Onix feed to create a filtered table in BigQuery called `crossref.crossref_metadataYYYYMMDD`. + + +## Latest schema + +``` eval_rst +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| +.. csv-table:: + :file: ../../schemas/crossref_metadata_latest.csv + :width: 100% + :header-rows: 1 + :class: longtable +``` \ No newline at end of file diff --git a/docs/oaebu_workflows/telescopes/doab.md b/docs/oaebu_workflows/telescopes/doab.md deleted file mode 100644 index ad2cf356..00000000 --- a/docs/oaebu_workflows/telescopes/doab.md +++ /dev/null @@ -1,49 +0,0 @@ -# Directory of Open Access Books -The Directory of Open Access Books (DOAB) is a directory of open-access peer reviewed scholarly books. -Its aim is to increase discoverability of books. Currently, there are two requirements to take part in DOAB: -* Academic books in DOAB shall be available under an Open Access license (such as a Creative Commons license) -* Academic books in DOAB shall be subjected to independent and external peer review prior to publication -Data is downloaded from a csv file. - -```eval_rst -+--------------------------+---------+ -| Summary | | -+==========================+=========+ -| Average runtime | 15m | -+--------------------------+---------+ -| Average download size | 50MB | -+--------------------------+---------+ -| Harvest Type | API | -+--------------------------+---------+ -| Harvest Frequency | Weekly | -+--------------------------+---------+ -| Runs on remote worker | False | -+--------------------------+---------+ -| Catchup missed runs | False | -+--------------------------+---------+ -| Table Write Disposition | Append | -+--------------------------+---------+ -| Update Frequency | Daily | -+--------------------------+---------+ -| Credentials Required | No | -+--------------------------+---------+ -``` - -## Schedule -The csv is updated daily and this telescope is scheduled to harvest the data weekly. - -## Results -There are two tables containing data related to this telescope: - * The main table which contains all the up-to-date DOAB data. - * A partitioned table, where each partition contains the data of one run. The table is partitioned by the ingestion time. - -If there are any edited or deleted entries, the old versions of these entries will first be deleted from the main table -(using a MERGE statement) after which the updated versions are appended to this table. - -## Latest schema -``` eval_rst -.. csv-table:: - :file: ../../schemas/doab_latest.csv - :width: 100% - :header-rows: 1 -``` \ No newline at end of file diff --git a/docs/oaebu_workflows/telescopes/google_analytics.md b/docs/oaebu_workflows/telescopes/google_analytics.md index 080f9b05..4d1ce708 100644 --- a/docs/oaebu_workflows/telescopes/google_analytics.md +++ b/docs/oaebu_workflows/telescopes/google_analytics.md @@ -108,7 +108,7 @@ oaebu_service_account: google-cloud-platform://?type=service_account&private_key ## Latest schema ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/google_analytics_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/telescopes/google_books.md b/docs/oaebu_workflows/telescopes/google_books.md index 3d630215..eef62655 100644 --- a/docs/oaebu_workflows/telescopes/google_books.md +++ b/docs/oaebu_workflows/telescopes/google_books.md @@ -246,7 +246,7 @@ sftp_service: ssh://:@:?host_key= ### Google Books Sales ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/google_books_sales_latest.csv :width: 100% @@ -257,7 +257,7 @@ sftp_service: ssh://:@:?host_key= ### Google Books Traffic ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/google_books_traffic_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/telescopes/index.rst b/docs/oaebu_workflows/telescopes/index.rst index 16bd9120..b2b8e764 100644 --- a/docs/oaebu_workflows/telescopes/index.rst +++ b/docs/oaebu_workflows/telescopes/index.rst @@ -4,6 +4,8 @@ output data to other places. Workflows are built on top of Apache Airflow's DAGs .. toctree:: :maxdepth: 1 + crossref_events + crossref_metadata google_analytics google_books jstor diff --git a/docs/oaebu_workflows/telescopes/irus_fulcrum.md b/docs/oaebu_workflows/telescopes/irus_fulcrum.md index e7005fd3..970aca0c 100644 --- a/docs/oaebu_workflows/telescopes/irus_fulcrum.md +++ b/docs/oaebu_workflows/telescopes/irus_fulcrum.md @@ -6,7 +6,6 @@ The earliest available data for the Fulcrum platform is April 2022. It follows t The corresponding table created in BigQuery is `irus.irus_fulcrumYYYYMMDD`. - ```eval_rst +------------------------------+--------------+ | Summary | | @@ -80,10 +79,10 @@ The transformed data is loaded from the Google Cloud bucket into a partitioned B ## Latest schema ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: - :file: ../../schemas/irus_fulcrum.csv + :file: ../../schemas/irus_fulcrum_latest.csv :width: 100% :header-rows: 1 :class: longtable -``` +``` \ No newline at end of file diff --git a/docs/oaebu_workflows/telescopes/irus_oapen.md b/docs/oaebu_workflows/telescopes/irus_oapen.md index 94e4f5aa..341d954d 100644 --- a/docs/oaebu_workflows/telescopes/irus_oapen.md +++ b/docs/oaebu_workflows/telescopes/irus_oapen.md @@ -97,22 +97,15 @@ Similar to the publisher names described above, multiple publisher UUIDs can be '|' character. ## Cloud Function -<<<<<<< HEAD:docs/oaebu_workflows/telescopes/irus_oapen.md - -The OAPEN IRUS-UK telescope makes use of a Google Cloud Function that resides in the OAPEN Google project. -======= The IRUS OAPEN telescope makes use of a Google Cloud Function that resides in the OAPEN Google project. ->>>>>>> 1da38aa (COK-531 supporting rendering of pdf in progress. Changes to conf.py to be improved for longtable and api reference still not line breaking correctly. onix, thoth and book product tale schemas with long names still need to be accounted for.):docs/oaebu_workflows/telescopes/oapen_irus_uk.md + There is a specific airflow task that will create the Cloud Function if it does not exist yet, or update it if the source code has changed. + The source code for the Cloud Function can be found inside a separate repository that is part of the same organization (https://github.com/The-Academic-Observatory/oapen-irus-uk-cloud-function). ### Download access stats data -<<<<<<< HEAD:docs/oaebu_workflows/telescopes/irus_oapen.md - -The Cloud Function downloads OAPEN IRUS-UK access stats data for 1 month and for a single publisher. Usage data after April 2020 is hosted on a new platform. -======= The Cloud Function downloads IRUS OAPEN access stats data for 1 month and for a single publisher. Usage data after April 2020 is hosted on a new platform. ->>>>>>> 1da38aa (COK-531 supporting rendering of pdf in progress. Changes to conf.py to be improved for longtable and api reference still not line breaking correctly. onix, thoth and book product tale schemas with long names still need to be accounted for.):docs/oaebu_workflows/telescopes/oapen_irus_uk.md + The newer data is obtained by using their API, this requires a `requestor_id` and an `api_key`. Data before April 2020 is obtained from an URL, this requires an `email` and a `password`. @@ -164,19 +157,10 @@ Note that all values need to be urlencoded. In the config.yaml file, the following airflow connections are required: ### oapen_irus_uk_login -<<<<<<< HEAD:docs/oaebu_workflows/telescopes/irus_oapen.md - -To get the email address/password combination, contact OAPEN IRUS-UK. - -### oapen_irus_uk_api - -To get the requestor_id/api_key, contact OAPEN IRUS-UK. -======= To get the email address/password combination, contact IRUS. ### oapen_irus_uk_api To get the requestor_id/api_key, contact IRUS. ->>>>>>> 1da38aa (COK-531 supporting rendering of pdf in progress. Changes to conf.py to be improved for longtable and api reference still not line breaking correctly. onix, thoth and book product tale schemas with long names still need to be accounted for.):docs/oaebu_workflows/telescopes/oapen_irus_uk.md ### geoip_license_key @@ -192,20 +176,11 @@ geoip_license_key: mysql://user_id:license_key@ ``` ## Latest schema -<<<<<<< HEAD:docs/oaebu_workflows/telescopes/irus_oapen.md - ```eval_rst -======= -``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| ->>>>>>> 1da38aa (COK-531 supporting rendering of pdf in progress. Changes to conf.py to be improved for longtable and api reference still not line breaking correctly. onix, thoth and book product tale schemas with long names still need to be accounted for.):docs/oaebu_workflows/telescopes/oapen_irus_uk.md +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/irus_oapen.csv :width: 100% :header-rows: 1 -<<<<<<< HEAD:docs/oaebu_workflows/telescopes/irus_oapen.md -``` -======= :class: longtable ``` ->>>>>>> 1da38aa (COK-531 supporting rendering of pdf in progress. Changes to conf.py to be improved for longtable and api reference still not line breaking correctly. onix, thoth and book product tale schemas with long names still need to be accounted for.):docs/oaebu_workflows/telescopes/oapen_irus_uk.md diff --git a/docs/oaebu_workflows/telescopes/jstor.md b/docs/oaebu_workflows/telescopes/jstor.md index a098498e..10a02458 100644 --- a/docs/oaebu_workflows/telescopes/jstor.md +++ b/docs/oaebu_workflows/telescopes/jstor.md @@ -13,7 +13,7 @@ In the telescope the Gmail of the G-suite account is parsed for messages with a The production server of the observatory-platform has been white listed by JSTOR to avoid bot detection. -The corresponding tables created in BigQuery are `jstor_countryYYYYMMDD` and `jstor_institutionYYYYMMDD`. +The corresponding tables created in BigQuery are `jstor.jstor_countryYYYYMMDD` and `jstor.jstor_institutionYYYYMMDD`. ```eval_rst +------------------------------+---------+ @@ -352,7 +352,7 @@ gmail_api: google-cloud-platform://?token=&refresh_token=& ### JSTOR Institution ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/jstor_institution_latest.csv :width: 100% @@ -363,7 +363,7 @@ gmail_api: google-cloud-platform://?token=&refresh_token=& ### JSTOR Country ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/jstor_country_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/telescopes/oapen_metadata.md b/docs/oaebu_workflows/telescopes/oapen_metadata.md index ec40f2dd..4521fb43 100644 --- a/docs/oaebu_workflows/telescopes/oapen_metadata.md +++ b/docs/oaebu_workflows/telescopes/oapen_metadata.md @@ -5,6 +5,8 @@ OAPEN enables libraries and aggregators to use the metadata of all available tit The metadata is available in different formats and this telescope harvests the data in the XML format. See the [OAPEN Metadata webpage](https://www.oapen.org/resources/15635975-metadata) for more information. +The corresponding table in BigQuery is `onix.onixYYYYMMDD`. + ```eval_rst +------------------------------+------------+ | Summary | | @@ -43,7 +45,7 @@ The XML file containing metadata is updated daily at +0000GMT. This telescope is ## Results -The resulting ONIX table will be stored in BigQuery - `oaebu-oapen.onix.onixYYYYMMDD` +The resulting ONIX table will be stored in BigQuery - `onix.onixYYYYMMDD` ## Tasks @@ -64,3 +66,13 @@ The transform step modifies the downloaded metadata into a valid ONIX format. Th ### Load to BigQuery The valid ONIX feed can now be loaded from the transform bucket into a BigQuery sharded table. + +## Latest schema +``` eval_rst +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| +.. csv-table:: + :file: ../../schemas/onix_latest.csv + :width: 100% + :header-rows: 1 + :class: longtable +``` diff --git a/docs/oaebu_workflows/telescopes/onix.md b/docs/oaebu_workflows/telescopes/onix.md index 7a1d515d..c22e38a0 100644 --- a/docs/oaebu_workflows/telescopes/onix.md +++ b/docs/oaebu_workflows/telescopes/onix.md @@ -10,6 +10,8 @@ The ONIX telescope downloads the ONIX files from the SFTP server. It then transf for loading into BigQuery with the [ONIX parser](https://github.com/The-Academic-Observatory/onix-parser) Java command line tool. The data is loaded into BigQuery and then used by the ONIX Workflow. +The corresponding table in BigQuery is `onix.onixYYYYMMDD`. + ```eval_rst +------------------------------+--------------+ | Summary | | @@ -59,7 +61,7 @@ sftp_service: ssh://user-name:password@host-name:port?host_key=host-key ## Latest schema ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/telescopes/thoth.md b/docs/oaebu_workflows/telescopes/thoth.md index 20869f40..1835a455 100644 --- a/docs/oaebu_workflows/telescopes/thoth.md +++ b/docs/oaebu_workflows/telescopes/thoth.md @@ -6,6 +6,8 @@ Thoth is a free, open metadata service that publishers can choose to utilise as The Thoth telescope downloads the ONIX metadata files and then transforms the data into a format suitable for loading into BigQuery with the [ONIX parser](https://github.com/The-Academic-Observatory/onix-parser) Java command line tool. This is a near-identical process to how the [ONIX telescope's](onix.md) data-transformation step is executed. The transformed data is loaded into BigQuery, where it can be picked up and used by the [ONIX Workflow](../workflows/onix_workflow_intro.md). +The corresponding table in BigQuery is `onix.onixYYYYMMDD`. + ```eval_rst +------------------------------+--------------+ | Summary | | @@ -45,10 +47,10 @@ The Thoth telescope does not require any airflow connections to run, as the Thot ## Latest schema ```eval_rst -.. tabularcolumns:: |p{6cm}|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_latest.csv :width: 100% :header-rows: 1 - :class: longtable + :class: longtable ``` diff --git a/docs/oaebu_workflows/telescopes/ucl_discovery.md b/docs/oaebu_workflows/telescopes/ucl_discovery.md index afefbdcc..186850f5 100644 --- a/docs/oaebu_workflows/telescopes/ucl_discovery.md +++ b/docs/oaebu_workflows/telescopes/ucl_discovery.md @@ -6,6 +6,7 @@ The metadata for all eprints is obtained from the publicly available CSV file (h Additionally for each eprint the total downloads and downloads per country is gathered from the publicly available stats (https://discovery.ucl.ac.uk/cgi/stats/report). +The corresponding table in BigQuery is `ucl.ucl_discoveryYYYYMMDD`. ```eval_rst +------------------------------+---------+ @@ -37,7 +38,7 @@ Additionally for each eprint the total downloads and downloads per country is ga ## Latest schema ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/ucl_discovery_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/workflows/onix_workflow_step_1.md b/docs/oaebu_workflows/workflows/onix_workflow_step_1.md index cbf7bc5f..c678b533 100644 --- a/docs/oaebu_workflows/workflows/onix_workflow_step_1.md +++ b/docs/oaebu_workflows/workflows/onix_workflow_step_1.md @@ -28,7 +28,7 @@ The ONIX workflow is dependent on the ONIX telescope. It waits for the ONIX tel The Work ID will be an arbitrary ISBN representative from a product in the equivalence class. ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_workid_isbn_latest.csv :width: 100% @@ -40,7 +40,7 @@ The Work ID will be an arbitrary ISBN representative from a product in the equiv The Work Family ID will be an arbitrary Work ID (ISBN) representative from a work in the equivalence class. ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_workfamilyid_isbn_latest.csv :width: 100% @@ -51,7 +51,7 @@ The Work Family ID will be an arbitrary Work ID (ISBN) representative from a wor ## ONIX Work ID ISBN Errors ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_workid_isbn_errors_latest.csv :width: 100% @@ -59,10 +59,10 @@ The Work Family ID will be an arbitrary Work ID (ISBN) representative from a wor :class: longtable ``` ## Create Crossref metadata table -Crossref data is required to proceed. The ISBNs for each work is obtained from the publisher's Onix table. For each of these ISBNs, the Crossref [metadata REST API](https://api.crossref.org/) is queried. +Crossref data is required to proceed. The ISBNs for each work is obtained from the publisher's Onix table. For each of these ISBNs, the Crossref metadata table produced by the [Academic Observatory workflows](https://github.com/The-Academic-Observatory/academic-observatory-workflows/tree/develop) is queried. ## Create Crossref events table -Similarly to the metadata, Crossref event data is retrieved through Crossref's dedicated [event REST API](https://www.eventdata.crossref.org/guide/service/query-api/). The API accepts queries based on DOI only, which we retrienve by matching the appropriate ISBN13 from the metadata. +Similarly to the metadata, Crossref event data is retrieved through Crossref's dedicated [event REST API](https://www.eventdata.crossref.org/guide/service/query-api/). The API accepts queries based on DOI only, which we retrieve by matching the appropriate ISBN13 from the metadata. ## Create book table The book table is a collection of works and their relevant details for the relative publisher. The table accommodates a title's Crossref metadata, events and separate chapters. @@ -88,7 +88,7 @@ For each data source, including the intermediate tables, we perform basic qualit [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/onix_aggregate_metrics.sql.jinja2) ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_aggregate_metrics_latest.csv :width: 100% @@ -100,7 +100,7 @@ For each data source, including the intermediate tables, we perform basic qualit Details ISBN13s in the ONIX feed that are not valid. ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/onix_invalid_isbn_latest.csv :width: 100% @@ -113,7 +113,7 @@ Details ISBN13s in the ONIX feed that are not valid. Details ISBN13s in the data source that are not valid. An example schema is below, as data platforms may use different name fields (e.g, 'ISBN', 'publication_id', 'Primary_ISBN'). ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/platform_invalid_isbn_latest.csv :width: 100% @@ -126,7 +126,7 @@ Details ISBN13s in the data source that are not valid. An example schema is belo Details ISBN-13s in the data source that were not matched to ISBN-13s in the ONIX feed. ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/platform_unmatched_isbn_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/workflows/onix_workflow_step_2.md b/docs/oaebu_workflows/workflows/onix_workflow_step_2.md index 335c9648..217430da 100644 --- a/docs/oaebu_workflows/workflows/onix_workflow_step_2.md +++ b/docs/oaebu_workflows/workflows/onix_workflow_step_2.md @@ -15,7 +15,7 @@ The output is the book_product table, containing one row per unique book, with a [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/create_book_products.sql.jinja2) ``` eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/book_product_latest.csv :width: 100% diff --git a/docs/oaebu_workflows/workflows/onix_workflow_step_3.md b/docs/oaebu_workflows/workflows/onix_workflow_step_3.md index 968fe022..e268d0f7 100644 --- a/docs/oaebu_workflows/workflows/onix_workflow_step_3.md +++ b/docs/oaebu_workflows/workflows/onix_workflow_step_3.md @@ -18,7 +18,7 @@ This table is a list of each Book Product. It is primarily used for drop-down fi [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_list.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_list_latest.csv :width: 100% @@ -33,7 +33,7 @@ This table contains metrics, organised by month, that are linked to each book. T [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_latest.csv :width: 100% @@ -48,7 +48,7 @@ This table contains metrics, organised by month and author, that are linked to e [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_author_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_author_metrics_latest.csv :width: 100% @@ -63,7 +63,7 @@ This table contains metrics, organised by published year and month, that are lin [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_year_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_year_metrics_latest.csv :width: 100% @@ -78,7 +78,7 @@ This table contains metrics, organised by month and crossref event type, that ar [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics_event.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_events_latest.csv :width: 100% @@ -93,7 +93,7 @@ This table contains metrics, organised by month and city of measured usage, that [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics_city.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_city_latest.csv :width: 100% @@ -108,7 +108,7 @@ This table contains metrics, organised by month and country of measured usage, t [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics_country.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_country_latest.csv :width: 100% @@ -116,19 +116,6 @@ This table contains metrics, organised by month and country of measured usage, t :class: longtable ``` -### Country List Schema - -This table is a list of each unique Country or Territory. It is primarily used for drop-down fields. - -```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| -.. csv-table:: - :file: ../../schemas/oaebu_public_data_country_list_latest.csv - :width: 100% - :header-rows: 1 - :class: longtable -``` - ### Book Product Metrics Events Schema This table contains metrics, organised by month and crossref event type, that are linked to each book. @@ -136,7 +123,7 @@ This table contains metrics, organised by month and crossref event type, that ar [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics_event.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_events_latest.csv :width: 100% @@ -151,7 +138,7 @@ This table is a list of each unique Institution where metrics are linked too. It [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_metrics_institution.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_institution_list_latest.csv :width: 100% @@ -166,7 +153,7 @@ This table contains metrics, organised by month and institution for which there [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_institution_list.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_metrics_institution_latest.csv :width: 100% @@ -181,7 +168,7 @@ This index contains a summary of metrics, organised by month that are linked to [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_publisher_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_publisher_metrics_latest.csv :width: 100% @@ -196,7 +183,7 @@ This table contains metrics, organised by month and BIC subject type, that are l [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_subject_bic_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_subject_bic_metrics_latest.csv :width: 100% @@ -211,7 +198,7 @@ This table contains metrics, organised by month and BISAC subject type, that are [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_subject_bisac_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_subject_bisac_metrics_latest.csv :width: 100% @@ -226,7 +213,7 @@ This table contains metrics, organised by month and THEMA subject type, that are [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_subject_thema_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_subject_thema_metrics_latest.csv :width: 100% @@ -241,7 +228,7 @@ This table contains metrics, organised by published year and month and currently [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_book_subject_year_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_book_product_subject_year_metrics_latest.csv :width: 100% @@ -258,7 +245,7 @@ This dataset is helpful for understanding where metrics and books defined in the [Link to Query](https://github.com/The-Academic-Observatory/oaebu-workflows/blob/develop/oaebu_workflows/database/sql/export_unmatched_metrics.sql.jinja2) ```eval_rst -.. tabularcolumns:: |l|l|l|p{6cm}| +.. tabularcolumns:: |p{4.5cm}|l|l|p{6cm}| .. csv-table:: :file: ../../schemas/oaebu_publisher_unmatched_book_metrics_latest.csv :width: 100% diff --git a/requirements.txt b/requirements.txt index b40a2830..7e65eedf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,4 +3,5 @@ faker>=8.12.1,<9 Markdown==3.3.4 # prevent error: INSTALLED_EXTENSIONS = metadata.entry_points(group='markdown.extensions') TypeError: entry_points() got an unexpected keyword argument 'group' responses==0.20.* onixcheck==0.9.7 -ratelimit==2.2.1 \ No newline at end of file +ratelimit==2.2.1 +