Skip to content

Commit

Permalink
Merge pull request #43 from developmentseed/ab/use_grid_cells
Browse files Browse the repository at this point in the history 
Change pixels to grid cells
  • Loading branch information
@abarciauskas-bgse
abarciauskas-bgse authored Jan 2, 2025
2 parents ea7d467 + d27a69b commit 0f158df
Showing 1 changed file with 21 additions and 17 deletions.
38 changes: 21 additions & 17 deletions docs/deployment/time_series_api_limits.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,15 @@
"\n",
"The `titiler-cmr` API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.\n",
"\n",
"## Terminology\n",
"\n",
"**Grid cells** refer to the smallest spatial units in a gridded dataset, representing the area over which a single data value is aggregated or measured. These cells collectively form a regular grid that defines the spatial structure of the data.\n",
"\n",
"## Highlights\n",
"- Maximum of 995 discrete points or intervals in a time series request (due to Lambda concurrency limits)\n",
"- You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total pixels (`x_pixels * y_pixels * n_time`) which is helpful for determining if a request will succeed\n",
"- The `/timeseries/bbox` endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to **100,000,000 (`1e8`) total pixels**. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.\n",
"- The `/timeseries/statistics` endpoint can handle larger requests than the `/timeseries/bbox` endpoint Based on a coarse evaluation of the API, requests are limited to **15,000,000,000 (`1.5e10`) total pixels** as long as the individual images read by the `/statistics` endpoint are smaller than 56,250,000 (`5.625e7`) pixels.\n",
"- Maximum of 995 discrete points or intervals in a time series request (due to [AWS Lambda concurrency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html) limits)\n",
"- You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total grid cells (`x_dimension * y_dimension * n_time`) which is helpful for determining if a request will succeed\n",
"- The `/timeseries/bbox` endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to **100,000,000 (`1e8`) total grid cells**. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.\n",
"- The `/timeseries/statistics` endpoint can handle larger requests than the `/timeseries/bbox` endpoint Based on a coarse evaluation of the API, requests are limited to **15,000,000,000 (`1.5e10`) total grid cells** as long as the individual images read by the `/statistics` endpoint are smaller than 56,250,000 (`5.625e7`) grid cells.\n",
"\n",
"## Tips\n",
"\n",
Expand Down Expand Up @@ -52,13 +56,13 @@
"\n",
"The limits for the `xarray` backend are:\n",
"- `/timeseries/bbox`\n",
" - individual image size: `5.6e7` pixels (~7500x7500)\n",
" - total image size (`x_pixels * y_pixels * n_time`): `1e8` pixels\n",
" - individual image size: `5.6e7` grid cells (~7500x7500)\n",
" - total image size (`x_dimension * y_dimension * n_time`): `1e8` grid cells\n",
"- `/timeseries/statistics`\n",
" - individual image size: `5.6e7` pixels (~7500x7500)\n",
" - total image size: `1.5e10` pixels\n",
" - individual image size: `5.6e7` grid cells (~7500x7500)\n",
" - total image size: `1.5e10` grid cells\n",
"\n",
"For low-resolution datasets (e.g. 28km or 0.25 degree) you will not run into any issues (unless you request too many time points!) because a request for the full dataset will be reading arrays that are ~1440x720 pixels. \n",
"For low-resolution datasets (e.g. 28km or 0.25 degree) you will not run into any issues (unless you request too many time points!) because a request for the full dataset will be reading arrays that are ~1440x720 grid cells. \n",
"\n",
"For higher-resolution datasets (e.g. 1km or 0.01 degree), you will start to run into problems as the size of the raw arrays that titiler-cmr is processing increases (and the number of discrete points or intervals increases). "
]
Expand All @@ -70,7 +74,7 @@
"source": [
"### Examples\n",
"\n",
"The MUR-SST dataset is good for demonstrating the limits of the time series endpoints with the `xarray` backend. It has high resolution (1 km, 0.01 degree) daily global sea surface temperature observations! With this resolution it is easy to craft a request that will break the `/timeseries` endpoints. Here are some examples of how to manipulate the time series parameters to achieve success with the `/timeseries/bbox` endpoint.\n",
"The [MUR SST](https://podaac.jpl.nasa.gov/dataset/MUR-JPL-L4-GLOB-v4.1) dataset is good for demonstrating the limits of the time series endpoints with the `xarray` backend. MUR SST delivers high resolution (1 km, 0.01 degree) daily global sea surface temperature observations! With this resolution it is easy to craft a request that will break the `/timeseries` endpoints. Here are some examples of how to manipulate the time series parameters to achieve success with the `/timeseries/bbox` endpoint.\n",
"\n",
"```python\n",
"from datetime import datetime, timedelta\n",
Expand Down Expand Up @@ -107,7 +111,7 @@
"```\n",
"\n",
"That request is about half of the maximum request size for the `/timeseries/bbox` endpoint. We can push it to the limit by doubling the length of the time series:\n",
"- 5x5 degree bounding box (500 x 500 pixels)\n",
"- 5x5 degree bounding box (500 x 500 grid cells)\n",
"- 360 daily observations (`360 / P1D`)\n",
"- total size: `500 * 500 * 360 = 9.0e7`\n",
"\n",
Expand Down Expand Up @@ -135,7 +139,7 @@
"```\n",
"\n",
"If we increase the length of the time series such that the request exceeds the maximum size, the API will return an error:\n",
"- 5x5 degree bounding box (500 x 500 pixels)\n",
"- 5x5 degree bounding box (500 x 500 grid cells)\n",
"- 540 daily observations (`540 / P1D`)\n",
"- total size: `500 * 500 * 540 = 1.35e8` (greater than maximum of `1.0e8`!)\n",
"\n",
Expand Down Expand Up @@ -163,7 +167,7 @@
"```\n",
"\n",
"We can get get a successful response for the larger time window if we reduce the temporal resolution:\n",
"- 5x5 degree bounding box (500 x 500 pixels)\n",
"- 5x5 degree bounding box (500 x 500 grid cells)\n",
"- 77 weekly observations (`540 / P7D`)\n",
"- total size: `500 * 500 * 77 = 1.925e7`\n",
"\n",
Expand Down Expand Up @@ -191,7 +195,7 @@
"```\n",
"\n",
"With the weekly temporal resolution we have some room to increase the size of the bounding box!\n",
"- 10x10 degree bounding box (1000 x 1000 pixels)\n",
"- 10x10 degree bounding box (1000 x 1000 grid cells)\n",
"- 77 weekly observations (`540 / P7D`)\n",
"- total size: `1000 * 1000 * 77 = 7.7e7`\n",
"\n",
Expand Down Expand Up @@ -219,7 +223,7 @@
"```\n",
"\n",
"If we double the AOI size again, we will break exceed the request size limit:\n",
"- 20x20 degree bounding box (1000 x 1000 pixels)\n",
"- 20x20 degree bounding box (1000 x 1000 grid cells)\n",
"- 77 weekly observations (`540 / P7D`)\n",
"- total size: `2000 * 2000 * 77 = 3.08e8` (greater than maximum of `1e8`\n",
"\n",
Expand Down Expand Up @@ -247,7 +251,7 @@
"```\n",
"\n",
"But if we reduce the temporal resolution from weekly to monthly, it will work!\n",
"- 20x20 degree bounding box (1000 x 1000 pixels)\n",
"- 20x20 degree bounding box (1000 x 1000 grid cells)\n",
"- 18 monthly observations (`540 / P1M`)\n",
"- total size: `2000 * 2000 * 18 = 3.08e8`\n",
"\n",
Expand All @@ -274,7 +278,7 @@
")\n",
"```\n",
"\n",
"However, there is a maximum image size that we can read with the `xarray` backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is `5.6e7` pixels (7500 x 7500 pixels). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees."
"However, there is a maximum image size that we can read with the `xarray` backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is `5.6e7` grid cells (7500 x 7500 grid cells). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees."
]
}
],
Expand Down

0 comments on commit 0f158df

Please sign in to comment.