Merge pull request #1443 from cmu-delphi/release/delphi-epidata-4.1.21

Release Delphi Epidata 4.1.21

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 4.1.20
+current_version = 4.1.21
 commit = False
 tag = False
 
@@ -9,10 +9,10 @@ tag = False
 
 [bumpversion:file:src/client/delphi_epidata.R]
 
+[bumpversion:file:src/client/delphi_epidata.py]
+
 [bumpversion:file:src/client/packaging/npm/package.json]
 
 [bumpversion:file:src/client/packaging/pypi/setup.py]
 
-[bumpversion:file:src/client/packaging/pypi/delphi_epidata/__init__.py]
-
 [bumpversion:file:dev/local/setup.cfg]

dev/local/setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = Delphi Development
-version = 4.1.20
+version = 4.1.21
 
 [options]
 packages =
@@ -24,6 +24,7 @@ packages =
     delphi.epidata.acquisition.twtr
     delphi.epidata.acquisition.wiki
     delphi.epidata.client
+    delphi.epidata.common
     delphi.epidata.server
     delphi.epidata.server.admin
     delphi.epidata.server.admin.templates

docs/Gemfile.lock

@@ -205,14 +205,14 @@ GEM
       rb-fsevent (~> 0.10, >= 0.10.3)
       rb-inotify (~> 0.9, >= 0.9.10)
     mercenary (0.3.6)
-    mini_portile2 (2.8.5)
+    mini_portile2 (2.8.6)
     minima (2.5.1)
       jekyll (>= 3.5, < 5.0)
       jekyll-feed (~> 0.9)
       jekyll-seo-tag (~> 2.1)
     minitest (5.17.0)
     multipart-post (2.1.1)
-    nokogiri (1.16.2)
+    nokogiri (1.16.5)
       mini_portile2 (~> 2.8.2)
       racc (~> 1.4)
     octokit (4.20.0)
@@ -225,7 +225,8 @@ GEM
     rb-fsevent (0.10.4)
     rb-inotify (0.10.1)
       ffi (~> 1.0)
-    rexml (3.2.5)
+    rexml (3.2.8)
+      strscan (>= 3.0.9)
     rouge (3.26.0)
     ruby-enum (0.9.0)
       i18n
@@ -242,6 +243,7 @@ GEM
       faraday (> 0.8, < 2.0)
     simpleidn (0.2.1)
       unf (~> 0.1.4)
+    strscan (3.1.0)
     terminal-table (1.8.0)
       unicode-display_width (~> 1.1, >= 1.1.1)
     thread_safe (0.3.6)

docs/api/covidcast-signals/jhu-csse.md

@@ -1,6 +1,6 @@
 ---
 title: JHU Cases and Deaths
-parent: Data Sources and Signals
+parent: Inactive Signals
 grand_parent: COVIDcast Main Endpoint
 ---
 
@@ -23,12 +23,12 @@ University.
 
 | Signal and 7-day average signal | Description |
 |---|---|
-| `confirmed_cumulative_num` | Cumulative number of confirmed COVID-19 cases <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
-| `confirmed_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
+| `confirmed_cumulative_num` and `confirmed_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
+| `confirmed_cumulative_prop` and `confirmed_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
 | `confirmed_incidence_num` and `confirmed_7dav_incidence_num` | Number of new confirmed COVID-19 cases, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
 | `confirmed_incidence_prop` and `confirmed_7dav_incidence_prop` | Number of new confirmed COVID-19 cases per 100,000 population, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
-| `deaths_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19 <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
-| `deaths_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
+| `deaths_cumulative_num` and `deaths_7dav_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19 <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
+| `deaths_cumulative_prop` and `deaths_7dav_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
 | `deaths_incidence_num` and `deaths_7dav_incidence_num` | Number of new confirmed deaths due to COVID-19, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
 | `deaths_incidence_prop` and `deaths_7dav_incidence_prop` | Number of new confirmed deaths due to COVID-19 per 100,000 population, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
 

docs/api/covidcast-signals/nchs-mortality.md

@@ -68,7 +68,7 @@ York State in our reports.
 
 We report the NCHS Mortality data in a weekly format (`time_type=week` &
 `time_value={YYYYWW}`, where `YYYYWW` refers to an epiweek). The CDC defines
-the [epiweek](https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as
+the [epiweek](https://web.archive.org/web/20210623224758/https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as
 seven days, from Sunday to Saturday. We check the week-ending dates provided in
 the NCHS morality data and use Python package
 [epiweeks](https://pypi.org/project/epiweeks/) to convert them into epiweek

docs/symptom-survey/publications.md

@@ -26,6 +26,23 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*:
 
 Research publications using the survey data include:
 
+- Z. Yang, R. Krishnan, and B. Li (2024). [The interplay between individual
+  mobility, health risk, and economic choice: A holistic model for COVID-19
+  policy intervention](https://doi.org/10.1287/ijds.2023.0013). *INFORMS
+  Journal on Data Science*.
+- A. Srivastava, J. M. Ramirez, S. Díaz-Aranda, J. Aguilar, A. F. Anta, A. Ortega,
+  and R. E. Lillo (2024). [Nowcasting temporal trends using indirect surveys](https://doi.org/10.1609/aaai.v38i20.30242).
+  In *Proceedings of the 38th AAAI Conference on Artificial Intelligence* 38,
+  22359–22367.
+- P. Porebski, S. Venkatramanan, A. Adiga, B. Klahn, B. Hurt, M. L. Wilson,
+  J. Chen, A. Vullikanti, M. Marathe & B. Lewis (2024). [Data-driven
+  mechanistic framework with stratified immunity and effective transmissibility
+  for COVID-19 scenario projections](https://doi.org/10.1016/j.epidem.2024.100761).
+  *Epidemics* 100761.
+- V. Nelson, B. Bashyal, P.-N. Tan & Y. A. Argyris (2024). [Vaccine rhetoric
+  on social media and COVID-19 vaccine uptake rates: A triangulation using
+  self-reported vaccine acceptance](https://doi.org/10.1016/j.socscimed.2024.116775).
+  *Social Science & Medicine* 116775.
 - R.R. Andridge (2024). [Using proxy pattern-mixture models to explain bias in
   estimates of COVID-19 vaccine uptake from two large surveys](https://doi.org/10.1093/jrsssa/qnae005).
   *Journal of the Royal Statistical Society Series A: Statistics in Society*.
@@ -34,7 +51,7 @@ Research publications using the survey data include:
   *IISE Transactions*.
 - de Vries, M., Kim, J.Y. & Han, H. (2023). [The unequal landscape of civic
   opportunity in America](https://doi.org/10.1038/s41562-023-01743-1). *Nature
-  Human Behavior*. 
+  Human Behavior* 8, 256-263. 
 - E. Tuzhilina, T. J. Hastie, D. J. McDonald, J. K. Tay & R. Tibshirani (2023).
   [Smooth multi-period forecasting with application to prediction of COVID-19
   cases](https://doi.org/10.1080/10618600.2023.2285337). *Journal of Computational

integrations/client/test_delphi_epidata.py

@@ -3,6 +3,7 @@
 # standard library
 import time
 from json import JSONDecodeError
+from requests.models import Response
 from unittest.mock import MagicMock, patch
 
 # first party
@@ -41,6 +42,8 @@ def localSetUp(self):
     # use the local instance of the Epidata API
     Epidata.BASE_URL = 'http://delphi_web_epidata/epidata'
     Epidata.auth = ('epidata', 'key')
+    Epidata.debug = False
+    Epidata.sandbox = False
 
     # use the local instance of the epidata database
     secrets.db.host = 'delphi_database_epidata'
@@ -221,6 +224,82 @@ def test_retry_request(self, get):
                        {'result': 0, 'message': 'error: Expecting value: line 1 column 1 (char 0)'}
                        )
 
+  @patch('requests.post')
+  @patch('requests.get')
+  def test_debug(self, get, post):
+    """Test that in debug mode request params are correctly logged."""
+    class MockResponse:
+      def __init__(self, content, status_code):
+          self.content = content
+          self.status_code = status_code
+      def raise_for_status(self): pass
+
+    Epidata.debug = True
+
+    try:
+      with self.subTest(name='test multiple GET'):
+        with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
+          get.reset_mock()
+          get.return_value = MockResponse(b'{"key": "value"}', 200)
+          Epidata._request_with_retry("test_endpoint1", params={"key1": "value1"})
+          Epidata._request_with_retry("test_endpoint2", params={"key2": "value2"})
+
+        output = logs.output
+        self.assertEqual(len(output), 4) # [request, response, request, response]
+        self.assertIn("Sending GET request", output[0])
+        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint1/\"", output[0])
+        self.assertIn("\"params\": {\"key1\": \"value1\"}", output[0])
+        self.assertIn("Received response", output[1])
+        self.assertIn("\"status_code\": 200", output[1])
+        self.assertIn("\"len\": 16", output[1])
+        self.assertIn("Sending GET request", output[2])
+        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint2/\"", output[2])
+        self.assertIn("\"params\": {\"key2\": \"value2\"}", output[2])
+        self.assertIn("Received response", output[3])
+        self.assertIn("\"status_code\": 200", output[3])
+        self.assertIn("\"len\": 16", output[3])
+
+      with self.subTest(name='test GET and POST'):
+        with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
+          get.reset_mock()
+          get.return_value = MockResponse(b'{"key": "value"}', 414)
+          post.reset_mock()
+          post.return_value = MockResponse(b'{"key": "value"}', 200)
+          Epidata._request_with_retry("test_endpoint3", params={"key3": "value3"})
+
+        output = logs.output
+        self.assertEqual(len(output), 3) # [request, response, request, response]
+        self.assertIn("Sending GET request", output[0])
+        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[0])
+        self.assertIn("\"params\": {\"key3\": \"value3\"}", output[0])
+        self.assertIn("Received 414 response, retrying as POST request", output[1])
+        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[1])
+        self.assertIn("\"params\": {\"key3\": \"value3\"}", output[1])
+        self.assertIn("Received response", output[2])
+        self.assertIn("\"status_code\": 200", output[2])
+        self.assertIn("\"len\": 16", output[2])
+    finally: # make sure this global is always reset
+      Epidata.debug = False
+
+  @patch('requests.post')
+  @patch('requests.get')
+  def test_sandbox(self, get, post):
+    """Test that in debug + sandbox mode request params are correctly logged, but no queries are sent."""
+    Epidata.debug = True
+    Epidata.sandbox = True
+    try:
+      with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
+        Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
+      output = logs.output
+      self.assertEqual(len(output), 1)
+      self.assertIn("Sending GET request", output[0])
+      self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/covidcast/\"", output[0])
+      get.assert_not_called()
+      post.assert_not_called()
+    finally: # make sure these globals are always reset
+      Epidata.debug = False
+      Epidata.sandbox = False
+
   def test_geo_value(self):
     """test different variants of geo types: single, *, multi."""
 

requirements.api.txt

@@ -2,7 +2,7 @@ delphi_utils==0.3.15
 epiweeks==2.1.2
 Flask==2.2.5
 Flask-Limiter==3.3.0
-jinja2==3.1.3
+jinja2==3.1.4
 more_itertools==8.4.0
 mysqlclient==2.1.1
 orjson==3.9.15

requirements.dev.txt

@@ -1,4 +1,4 @@
-aiohttp==3.9.2
+aiohttp==3.9.4
 black>=20.8b1
 bump2version==1.0.1
 covidcast==0.1.5

src/client/delphi_epidata.R

@@ -15,7 +15,7 @@ Epidata <- (function() {
   # API base url
   BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/')
 
-  client_version <- '4.1.20'
+  client_version <- '4.1.21'
 
   auth <- getOption("epidata.auth", default = NA)
 

src/client/delphi_epidata.js

@@ -22,7 +22,7 @@
   }
 })(this, function (exports, fetchImpl, jQuery) {
   const BASE_URL = "https://api.delphi.cmu.edu/epidata/";
-  const client_version = "4.1.20";
+  const client_version = "4.1.21";
 
   // Helper function to cast values and/or ranges to strings
   function _listitem(value) {

src/client/delphi_epidata.py

@@ -10,21 +10,17 @@
 
 # External modules
 import requests
+import time
 import asyncio
 from tenacity import retry, stop_after_attempt
 
 from aiohttp import ClientSession, TCPConnector, BasicAuth
-from importlib.metadata import version, PackageNotFoundError
 
-# Obtain package version for the user-agent. Uses the installed version by
-# preference, even if you've installed it and then use this script independently
-# by accident.
-try:
-    _version = version("delphi-epidata")
-except PackageNotFoundError:
-    _version = "0.script"
+from delphi.epidata.common.logger import get_structured_logger
 
-_HEADERS = {"user-agent": "delphi_epidata/" + _version + " (Python)"}
+__version__ = "4.1.21"
+
+_HEADERS = {"user-agent": "delphi_epidata/" + __version__ + " (Python)"}
 
 
 class EpidataException(Exception):
@@ -47,7 +43,11 @@ class Epidata:
     BASE_URL = "https://api.delphi.cmu.edu/epidata"
     auth = None
 
-    client_version = _version
+    client_version = __version__
+
+    logger = get_structured_logger('delphi_epidata_client')
+    debug = False # if True, prints extra logging statements
+    sandbox = False # if True, will not execute any queries
 
     # Helper function to cast values and/or ranges to strings
     @staticmethod
@@ -71,9 +71,25 @@ def _list(values):
     def _request_with_retry(endpoint, params={}):
         """Make request with a retry if an exception is thrown."""
         request_url = f"{Epidata.BASE_URL}/{endpoint}/"
+        if Epidata.debug:
+            Epidata.logger.info("Sending GET request", url=request_url, params=params, headers=_HEADERS, auth=Epidata.auth)
+        if Epidata.sandbox:
+            resp = requests.Response()
+            resp._content = b'true'
+            return resp
+        start_time = time.time()
         req = requests.get(request_url, params, auth=Epidata.auth, headers=_HEADERS)
         if req.status_code == 414:
+            if Epidata.debug:
+                Epidata.logger.info("Received 414 response, retrying as POST request", url=request_url, params=params, headers=_HEADERS)
             req = requests.post(request_url, params, auth=Epidata.auth, headers=_HEADERS)
+        if Epidata.debug:
+            Epidata.logger.info(
+                "Received response",
+                status_code=req.status_code,
+                len=len(req.content),
+                time=round(time.time() - start_time, 4)
+            )
         # handle 401 and 429
         req.raise_for_status()
         return req

src/client/packaging/npm/package.json

@@ -2,7 +2,7 @@
   "name": "delphi_epidata",
   "description": "Delphi Epidata API Client",
   "authors": "Delphi Group",
-  "version": "4.1.20",
+  "version": "4.1.21",
   "license": "MIT",
   "homepage": "https://github.com/cmu-delphi/delphi-epidata",
   "bugs": {

src/client/packaging/pypi/CHANGELOG.md

@@ -3,6 +3,38 @@
 All notable future changes to the `delphi_epidata` python client will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/).
 
+## [4.1.21] - 2024-05-20
+
+### Includes
+- https://github.com/cmu-delphi/delphi-epidata/pull/1418
+- https://github.com/cmu-delphi/delphi-epidata/pull/1436
+
+### Added
+- Adds two debug flags:
+  - `debug` logs info about HTTP requests and responses
+  - `sandbox` prevents any HTTP requests from actually executing, allowing for tests that do not incur server load.
+- Fixes the `user-agent` version so that it is correctly set to match the current client release.
+
+## [4.1.17] - 2024-01-30
+
+### Includes
+- https://github.com/cmu-delphi/delphi-epidata/pull/1363
+
+### Changed
+- Replaced use of deprecated setuptools' `pkg_resources` library with the native `importlib.metadata` library.
+
+## [4.1.13] - 2023-11-04
+
+### Includes
+- https://github.com/cmu-delphi/delphi-epidata/pull/1323
+- https://github.com/cmu-delphi/delphi-epidata/pull/1330
+
+### Changed
+- Appends a trailing slash to URLs requested by the Python client, which should prevent an automatic redirect and an extra request to the server.
+
+### Removed
+- Removed the `covidcast_nowcast()` method, as the associated API endpoint is no longer available.
+
 ## [4.1.11] - 2023-10-12
 
 ### Includes

src/client/packaging/pypi/delphi_epidata/__init__.py

@@ -1,4 +1,3 @@
-from .delphi_epidata import Epidata
+from .delphi_epidata import Epidata, __version__
 
 name = "delphi_epidata"
-__version__ = "4.1.20"