From d27a69bbb350c268291a0691c0f375804ff395a3 Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Thu, 2 Jan 2025 09:04:35 -0800 Subject: [PATCH] Change pixels to grid cells --- docs/deployment/time_series_api_limits.ipynb | 38 +++++++++++--------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/docs/deployment/time_series_api_limits.ipynb b/docs/deployment/time_series_api_limits.ipynb index ee5c75e..a1a95dd 100644 --- a/docs/deployment/time_series_api_limits.ipynb +++ b/docs/deployment/time_series_api_limits.ipynb @@ -11,11 +11,15 @@ "\n", "The `titiler-cmr` API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.\n", "\n", + "## Terminology\n", + "\n", + "**Grid cells** refer to the smallest spatial units in a gridded dataset, representing the area over which a single data value is aggregated or measured. These cells collectively form a regular grid that defines the spatial structure of the data.\n", + "\n", "## Highlights\n", - "- Maximum of 995 discrete points or intervals in a time series request (due to Lambda concurrency limits)\n", - "- You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total pixels (`x_pixels * y_pixels * n_time`) which is helpful for determining if a request will succeed\n", - "- The `/timeseries/bbox` endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to **100,000,000 (`1e8`) total pixels**. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.\n", - "- The `/timeseries/statistics` endpoint can handle larger requests than the `/timeseries/bbox` endpoint Based on a coarse evaluation of the API, requests are limited to **15,000,000,000 (`1.5e10`) total pixels** as long as the individual images read by the `/statistics` endpoint are smaller than 56,250,000 (`5.625e7`) pixels.\n", + "- Maximum of 995 discrete points or intervals in a time series request (due to [AWS Lambda concurrency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html) limits)\n", + "- You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total grid cells (`x_dimension * y_dimension * n_time`) which is helpful for determining if a request will succeed\n", + "- The `/timeseries/bbox` endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to **100,000,000 (`1e8`) total grid cells**. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.\n", + "- The `/timeseries/statistics` endpoint can handle larger requests than the `/timeseries/bbox` endpoint Based on a coarse evaluation of the API, requests are limited to **15,000,000,000 (`1.5e10`) total grid cells** as long as the individual images read by the `/statistics` endpoint are smaller than 56,250,000 (`5.625e7`) grid cells.\n", "\n", "## Tips\n", "\n", @@ -52,13 +56,13 @@ "\n", "The limits for the `xarray` backend are:\n", "- `/timeseries/bbox`\n", - " - individual image size: `5.6e7` pixels (~7500x7500)\n", - " - total image size (`x_pixels * y_pixels * n_time`): `1e8` pixels\n", + " - individual image size: `5.6e7` grid cells (~7500x7500)\n", + " - total image size (`x_dimension * y_dimension * n_time`): `1e8` grid cells\n", "- `/timeseries/statistics`\n", - " - individual image size: `5.6e7` pixels (~7500x7500)\n", - " - total image size: `1.5e10` pixels\n", + " - individual image size: `5.6e7` grid cells (~7500x7500)\n", + " - total image size: `1.5e10` grid cells\n", "\n", - "For low-resolution datasets (e.g. 28km or 0.25 degree) you will not run into any issues (unless you request too many time points!) because a request for the full dataset will be reading arrays that are ~1440x720 pixels. \n", + "For low-resolution datasets (e.g. 28km or 0.25 degree) you will not run into any issues (unless you request too many time points!) because a request for the full dataset will be reading arrays that are ~1440x720 grid cells. \n", "\n", "For higher-resolution datasets (e.g. 1km or 0.01 degree), you will start to run into problems as the size of the raw arrays that titiler-cmr is processing increases (and the number of discrete points or intervals increases). " ] @@ -70,7 +74,7 @@ "source": [ "### Examples\n", "\n", - "The MUR-SST dataset is good for demonstrating the limits of the time series endpoints with the `xarray` backend. It has high resolution (1 km, 0.01 degree) daily global sea surface temperature observations! With this resolution it is easy to craft a request that will break the `/timeseries` endpoints. Here are some examples of how to manipulate the time series parameters to achieve success with the `/timeseries/bbox` endpoint.\n", + "The [MUR SST](https://podaac.jpl.nasa.gov/dataset/MUR-JPL-L4-GLOB-v4.1) dataset is good for demonstrating the limits of the time series endpoints with the `xarray` backend. MUR SST delivers high resolution (1 km, 0.01 degree) daily global sea surface temperature observations! With this resolution it is easy to craft a request that will break the `/timeseries` endpoints. Here are some examples of how to manipulate the time series parameters to achieve success with the `/timeseries/bbox` endpoint.\n", "\n", "```python\n", "from datetime import datetime, timedelta\n", @@ -107,7 +111,7 @@ "```\n", "\n", "That request is about half of the maximum request size for the `/timeseries/bbox` endpoint. We can push it to the limit by doubling the length of the time series:\n", - "- 5x5 degree bounding box (500 x 500 pixels)\n", + "- 5x5 degree bounding box (500 x 500 grid cells)\n", "- 360 daily observations (`360 / P1D`)\n", "- total size: `500 * 500 * 360 = 9.0e7`\n", "\n", @@ -135,7 +139,7 @@ "```\n", "\n", "If we increase the length of the time series such that the request exceeds the maximum size, the API will return an error:\n", - "- 5x5 degree bounding box (500 x 500 pixels)\n", + "- 5x5 degree bounding box (500 x 500 grid cells)\n", "- 540 daily observations (`540 / P1D`)\n", "- total size: `500 * 500 * 540 = 1.35e8` (greater than maximum of `1.0e8`!)\n", "\n", @@ -163,7 +167,7 @@ "```\n", "\n", "We can get get a successful response for the larger time window if we reduce the temporal resolution:\n", - "- 5x5 degree bounding box (500 x 500 pixels)\n", + "- 5x5 degree bounding box (500 x 500 grid cells)\n", "- 77 weekly observations (`540 / P7D`)\n", "- total size: `500 * 500 * 77 = 1.925e7`\n", "\n", @@ -191,7 +195,7 @@ "```\n", "\n", "With the weekly temporal resolution we have some room to increase the size of the bounding box!\n", - "- 10x10 degree bounding box (1000 x 1000 pixels)\n", + "- 10x10 degree bounding box (1000 x 1000 grid cells)\n", "- 77 weekly observations (`540 / P7D`)\n", "- total size: `1000 * 1000 * 77 = 7.7e7`\n", "\n", @@ -219,7 +223,7 @@ "```\n", "\n", "If we double the AOI size again, we will break exceed the request size limit:\n", - "- 20x20 degree bounding box (1000 x 1000 pixels)\n", + "- 20x20 degree bounding box (1000 x 1000 grid cells)\n", "- 77 weekly observations (`540 / P7D`)\n", "- total size: `2000 * 2000 * 77 = 3.08e8` (greater than maximum of `1e8`\n", "\n", @@ -247,7 +251,7 @@ "```\n", "\n", "But if we reduce the temporal resolution from weekly to monthly, it will work!\n", - "- 20x20 degree bounding box (1000 x 1000 pixels)\n", + "- 20x20 degree bounding box (1000 x 1000 grid cells)\n", "- 18 monthly observations (`540 / P1M`)\n", "- total size: `2000 * 2000 * 18 = 3.08e8`\n", "\n", @@ -274,7 +278,7 @@ ")\n", "```\n", "\n", - "However, there is a maximum image size that we can read with the `xarray` backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is `5.6e7` pixels (7500 x 7500 pixels). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees." + "However, there is a maximum image size that we can read with the `xarray` backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is `5.6e7` grid cells (7500 x 7500 grid cells). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees." ] } ],