Implemented enhancements:
A REST API server acting as a gateway for databases with an OPTIMADE API, handling the distribution and collection of a single query to several different OPTIMADE databases.
The design outline is available here.
"},{"location":"#known-limitations","title":"Known limitations","text":"Here follows a list of known limitations and oddities of the current OPTIMADE gateway code.
"},{"location":"#pagination","title":"Pagination","text":"Pagination is a bit awkward in its current implementation state.
When using the page_limit query parameter for a gateway query for gateways with multiple databases, i.e., for GET /gateways/{gateway ID}/structures and GET /queries/{query ID}, the resulting entry-resource number is the product of the page_limit value and the number of databases in the gateway (maximum). This is because the page_limit query parameter is passed straight through to the external database requests, and the returned entries are stitched together for the gateway response.
So effectively, when querying GET /gateways/{gateway with N databases}/structures?page_limit=5 the resulting (maximum) number of entries returned in the response (the size of the data array in the response) will be N x 5, and not 5 as would otherwise be expected.
The intention is to fix this in the future, either through short-time caching of external database responses, or figuring out if there is a usable algorithm that doesn't extend the number of external requests (and therefore the gateway response times) by too much.
"},{"location":"#sorting","title":"Sorting","text":"Sorting is supported for all the gateway's own resources, i.e., in the /gateways, /databases, and /queries endpoints. But sorting is not supported for the results from external OPTIMADE databases. This means the sort query parameter has no effect in the GET /gateways/{gateway ID}/structures and GET /queries/{query ID} endpoints.
This shortcoming is a direct result of the current page_limit query parameter handling, and the limitation of the same.
All code in this repository was originally written by Casper Welzel Andersen (@CasperWA). The design for the gateway as outlined in design.md was a joint effort between Casper Welzel Andersen & Carl Simon Adorf (@csadorf).
All files in this repository are licensed under the MIT license with copyright \u00a9 2021 Casper Welzel Andersen & THEOS, EPFL.
"},{"location":"#funding-support","title":"Funding support","text":"This work was funded by THEOS, EPFL and the MarketPlace project.
The MarketPlace project is funded by Horizon 2020 under H2020-NMBP-25-2017 call with Grant agreement number: 760173.
"},{"location":"CHANGELOG/","title":"Changelog","text":""},{"location":"CHANGELOG/#unreleased-changes-2024-08-22","title":"Unreleased changes (2024-08-22)","text":"Full Changelog
Implemented enhancements:
Fixed bugs:
pytest-asyncio #447pre-commit issue for the mypy hook #385Closed issues:
Merged pull requests:
pyproject.toml) #430 (CasperWA)Full Changelog
Merged pull requests:
Full Changelog
Fixed bugs:
ci/dependabot-updates branch failing #174Closed issues:
main push CI job #184pre-commit hooks autoupdate to CI #183Merged pull requests:
ID! type instead of String! #239 (CasperWA)pre-commit hooks in dependabot CI #186 (CasperWA)ref instead of sha #178 (CasperWA)Full Changelog
Fixed bugs:
ci/dependabot-updates after merging ci/update-dependencies #131Closed issues:
ci/update-dependencies PR to Tuesday or Friday #160Merged pull requests:
ci/dependabot-updates branch upon merge to main #161 (CasperWA)Full Changelog
Implemented enhancements:
bandit, pylint, safety, and mypy #119 (CasperWA)Fixed bugs:
main docs deployment #152gh api in workflow #150mike deploy #144 (CasperWA)git push instead of action #136 (CasperWA)Closed issues:
gh-pages in documentation deploy workflows #142Merged pull requests:
gh-pages branch #157 (CasperWA)main build #153 (CasperWA)env outside of usable scope #126 (CasperWA)Full Changelog
Implemented enhancements:
Fixed bugs:
Merged pull requests:
Full Changelog
Fixed bugs:
Merged pull requests:
Full Changelog
Fixed bugs:
Merged pull requests:
Full Changelog
Implemented enhancements:
Fixed bugs:
Closed issues:
Merged pull requests:
* This Changelog was automatically generated by github_changelog_generator
"},{"location":"LICENSE/","title":"License","text":"MIT License
Copyright (c) 2021 Casper Welzel Andersen & THEOS, EPFL
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"},{"location":"design/","title":"Design of the OPTIMADE gateway","text":"The OPTIMADE gateway is intended to be implemented into the MarketPlace platform. Therefore, it should implement the MarketPlace Data Source API, as well as endpoints needed for the gateway capabilities themselves. To this end, the following sections defines/recaps these APIs and capabilities.
"},{"location":"design/#marketplace-data-source-api","title":"MarketPlace Data Source API","text":"The MarketPlace Data Source API developed in T2.2 of the MarketPlace project. It can be found on the Fraunhofer GitLab here.
Outline of the currently defined endpoints. Note, if there is no HTTP method next to the endpoint, it is not an available and reachable endpoint.
/marketplace/
/schemas/ (GET)
/{schema_id}/
/attributes (GET)/export (POST)/search (POST)The suggested OPTIMADE gateway API.
This API is based on the expected capabilities outlined below.
/optimade/ Methods: GET Behavior: Introspective/static metadata overview of server.
/query/ Methods: POST or GET Behavior: Orchestrate an OPTIMADE query.
/gateways/ Methods: GET Behavior: Standard reponse: Introspective/static metadata overview of all gateways. Using special query parameter: Create/retrieve and return unique gateway ID.
/{gateway_id}/ Methods: POST or GET Behavior: Create/retrieve search ID and return unique search ID. Start asynchronous search task.
Either:
/queries/ Methods: None Behavior: Disallowed. Note: This endpoint could support GET requests with similar functionality and behavior as for /gateways/? This would move some functionality away from /{gateway_id}/ to this endpoint. Making /{gateway_id}/ act as a mix of /query/ and /gateways/ in terms of orchestrating the search and returning introspective/static metadata about the gateway.
/{search_id}/ Methods: GET Behavior: Return current results according to state of asynchronous search task.
or:
/{search_id}/ Methods: GET Behavior: Return current results according to state of asynchronous search task.I think the way you would achieve the \u201cselection\u201d of databases is by creating provider-specific endpoints like this:
GET\n/gateway?providers=abc,def,xyz\nThis will return a deterministic gateway id related to specific set of providers, which you will then use for further queries like this:
GET\n/gateway/{gateway_id}/structures/\netc.
The gateway id would provide introspection, so /gateway/{gateway_id} returns some information about the gateway (supported OPTIMADE API, list of providers) etc. You would cache the gateway id in the client, so you don\u2019t have to make two requests for each query. If you don\u2019t provide a list of providers, the current default set is used. But this ensures that the REST API is actually stateless, because one gateway is always tied to a specific set of providers even if the default list is changed. Obviously, if you use a gateway that includes providers that are no longer available you would respond with code 503 or so.
This design solves the issue of how to provide a gateway that implements the OPTIMADE API and allows for the selection of providers. I assume your results are paginated, so IMO \u2014 unless you request a specific order \u2014 you should just return results as they come in. You need to implement this gateway asynchronously anyways so it really does not matter whether you include slow providers or not.
Of course, this changes if the user requests a specific order, but that\u2019s just how it is. From a user perspective it would make sense to me that such a query across multiple providers may take a while.
You should definitely define a timeout for each gateway where if a provider does not respond by then, the result is returned regardless of whether the provider has responded. Or you respond with a time out code.
"},{"location":"design/#searching","title":"Searching","text":"Taking Simon's comments into account, the search capability should be:
The asynchronicity comes from creating web calls (possibly using CORS) to each (chosen) database asynchronously, collating the results in a single (gateway) endpoint.
The dynamics here relate to the suggested dynamic creation (and possible deletion) of gateway IDs under a /gateway-endpoint.
Essentially, for each search, a new gateway will be created (if needed) with a unique ID. This unique ID will constitue the content of the initial response after performing a search, so that the user can go to the new gateway ID-specific endpoint to retrieve the results. To make this easier for the user, the server could automatically redirect the user after creating the endpoint. Here the response will contain the currently retrieved results as well as som metadata information about how the search is going and a general overview.
This would ideally result in the following search sequence:
The final GET request can be repeated to retrieve more results during the timeline of the search happening, and to retrieve the final list of results in a set time period after the search has finished.
One could also think of using POST requests instead, containing the OPTIMADE query parameters alongside with other information, mainly utilized for the /gateway/{unique ID}-endpoints. The response could contain a link or simply redirect to a /gateway/{unique ID}/{search unique ID}-endpoint. The latter part could also be done for the GET approach, since a specific gateway should support multiple unique simultaneous searched. Since the searches are asynchronous, the results don't come back from all resources simultaneously, thus demanding an extra endpoint, where the continuously updating results can be found - as well as the final list of results for a specific search.
This differs from the section above, where a GET request should contain query parameters in the URL and this will be correlated with an ongoing (unique) search in the backend, which would potentially allow different users to experience the same loading of results if they performed the same search in the same gateway, even at slightly different times during the searching period.
A sequence would ideally look this:
"},{"location":"design/#conclusion","title":"Conclusion","text":"The best approach here would be to create unique search IDs under each unique gateway, pertaining to a specific search. In the same way that gateways may be reused, search results may be reused. However, to ensure the \"freshness\" of the data, the \"live\"-period for any unique search should be significantly smaller than that of any unique gateway.
POST requests may be preferred due to the ability of combining OPTIMADE-specific query data and gateway-specific data.
Suggested search sequence diagram:
"},{"location":"design/#design-discussions-17122020","title":"Design discussions (17.12.2020)","text":"To be backwards compatible (where each gateway may represent a fully fledged OPTIMADE database), make /gateways/{unique ID}/ redirect to /gateways/{unique ID}/structures/.
Note, remove CUDS as a required capability, content negotiation might be with different means than a URL query parameter.
"},{"location":"design/#caching","title":"Caching","text":"Caching should be segmented for each database. For each new user query that retrieves and caches individual resources from a database, the lifetime of the cached resource should be updated to the set default (or what is determined by caching headers from the side of the database). Either the CacheControl or requests-cache packages will be utilized for caching.
Since the time it takes for an OPTIMADE database to change its content varies, but is mainly quite long, individual search life times (/gateways/{gateway ID}(/queries)/{search ID}/) can be \"long\", e.g., a couple of hours. However, these two ways of \"caching\" should be separate.
It should always be possible to forcefully ensure a \"fresh\" search.
"},{"location":"design/#optimade-filter-language","title":"OPTIMADE filter language","text":"The filter language will be reused as the filter language for any search in any gateway.
The filter language is defined in the OPTIMADE specification.
"},{"location":"design/#retrieval-formats","title":"Retrieval formats","text":"All responses will be in JSON (for now).
To choose the retrieval format of the structure, a query parameter will be dedicated for the /{search unique ID} endpoint.
The standard OPTIMADE format for defining structures will be reused for listing the structure entries.
See the OPTIMADE specification for a list of properties defining the structures entry.
When returning the results in this format, the whole response should be compliant with a standard OPTIMADE response as is expected in the /structures-endpoint.
Utilizing the optimade2cuds Python package in the SimOPTIMADE repository on the Fraunhofer GitLab for the MarketPlace project, the resulting OPTIMADE structure can be converted to Python CUDS objects. From there they can be serialized to JSON representations (using the OSP-Core package) and returned as a search result response.
When making external API calls, i.e., requesting the various OPTIMADE databases, this is technically done in a concurrent.futures.ThreadPoolExecutor. This is mainly done to not block the main OS thread, where the asyncio event loop is running. This is the event loop that handles incoming gateway requests. While the number of databases may not be significant, the response times can still vary and by using a ThreadPoolExecutor, the gateway is ready for more heavy use out-of-the-box.
Another key reason to use a ThreadPoolExecutor (instead of Starlette's - and therefore FastAPI's - BackgroundTask) is for testing with the pytest framework. When using BackgroundTask the response cannot be properly mocked and instead blocks the main OS thread. Perhaps this could be solved by implementing the same solution as has been done for now, namely running a time.sleep function call in a ThreadPoolExecutor, in the mocked response callback, but the benefits of using a ThreadPoolExecutor also for the actual queries outweigh this in the long run.
For further considerations a ProcessPoolExecutor might even be considered, but it shouldn't be necessary as the work done is IO blocking, not CPU blocking. The possible speed-up should not be significant.
Further reading and considerations on this subject Multithreading vs. Multiprocessing in Python by Amine Baatout is a good read. Another source of inspiration was found in this StackOverflow post response.
"},{"location":"design/#other-ideas-a-queue","title":"Other ideas - a queue","text":"Throughout the process of figuring this out, other ideas were on the table. One was to setup an asyncio.Queue - either a single \"unbuffered channel\" queue for the whole lifetime of the server, or one each per request. This would effectively split up the perform_query in producer/worker functions.
For some nice reading on this, check out Latency in Asynchronous Python by Chris Wellons (null program).
Since the ThreadPoolExecutor solution solves the issue of the analogous \"heartbeat\" function not losing its responsivenes, i.e., the asyncio event loop not being blocked, and it would work with the current code implementation, I opted for this solution instead. But I recon a queue solution would work similarly, but perhaps with slightly less gateway API responsiveness during heavy load, since it all still runs in the same event loop.
ASGI app events.
These events can be run at application startup or shutdown. The specific events are listed in EVENTS along with their respected proper invocation time.
EVENTS: Sequence[tuple[str, Callable[[], Coroutine[Any, Any, None]]]] = (('startup', ci_dev_startup), ('startup', load_optimade_providers_databases)) module-attribute","text":"A tuple of all pairs of events and event functions.
To use this tuple of tuples:
from fastapi import FastAPI\nAPP = FastAPI()\nfor event, func in EVENTS:\n APP.add_event_handler(event, func)\nci_dev_startup() async","text":"Function to run at app startup - only relevant for CI or development to add test data.
Source code inoptimade_gateway/events.py async def ci_dev_startup() -> None:\n \"\"\"Function to run at app startup - only relevant for CI or development to add test\n data.\"\"\"\n if bool(os.getenv(\"CI\", \"\")):\n LOGGER.info(\n \"CI detected - Will load test gateways (after dropping the collection)!\"\n )\n elif os.getenv(\"OPTIMADE_MONGO_DATABASE\", \"\") == \"optimade_gateway_dev\":\n LOGGER.info(\n \"Running in development mode - Will load test gateways (after dropping the\"\n \" collection)!\"\n )\n else:\n LOGGER.debug(\"Not in CI or development mode - will start normally.\")\n return\n\n # Add test gateways\n import json\n from pathlib import Path\n\n from optimade_gateway.mongo.database import MONGO_DB\n\n test_data = (\n Path(__file__).parent.parent.joinpath(\".ci/test_gateways.json\").resolve()\n )\n\n await MONGO_DB[CONFIG.gateways_collection].drop()\n\n if await MONGO_DB[CONFIG.gateways_collection].count_documents({}) != 0:\n raise RuntimeError(\n f\"Unexpectedly found documents in the {CONFIG.gateways_collection!r} Mongo\"\n \" collection after dropping it ! Found number of documents: \"\n f\"{await MONGO_DB[CONFIG.gateways_collection].count_documents({})}\"\n )\n\n if not test_data.exists():\n raise FileNotFoundError(\n f\"Could not find test data file with test gateways at {test_data} !\"\n )\n\n data = json.loads(test_data.read_bytes())\n\n await MONGO_DB[CONFIG.gateways_collection].insert_many(data)\nload_optimade_providers_databases() async","text":"Load in the providers' OPTIMADE databases from Materials-Consortia
Utilize the Materials-Consortia list of OPTIMADE providers at https://providers.optimade.org. Load in all databases with a valid base URL.
Source code inoptimade_gateway/events.py async def load_optimade_providers_databases() -> None:\n \"\"\"Load in the providers' OPTIMADE databases from Materials-Consortia\n\n Utilize the Materials-Consortia list of OPTIMADE providers at\n [https://providers.optimade.org](https://providers.optimade.org).\n Load in all databases with a valid base URL.\n \"\"\"\n import asyncio\n\n import httpx\n from optimade import __api_version__\n from optimade.models import LinksResponse\n from optimade.models.links import LinkType\n from optimade.server.routers.utils import BASE_URL_PREFIXES\n\n from optimade_gateway.common.utils import clean_python_types, get_resource_attribute\n from optimade_gateway.models.databases import DatabaseCreate\n from optimade_gateway.queries.perform import db_get_all_resources\n from optimade_gateway.routers.utils import resource_factory\n\n if not CONFIG.load_optimade_providers_databases:\n LOGGER.debug(\n \"Will not load databases from Materials-Consortia list of providers.\"\n )\n return\n\n if TYPE_CHECKING or bool(os.getenv(\"MKDOCS_BUILD\", \"\")): # pragma: no cover\n providers: httpx.Response | LinksResponse\n\n async with httpx.AsyncClient() as client:\n providers = await client.get(\n \"https://providers.optimade.org/v\"\n f\"{__api_version__.split('.', maxsplit=1)[0]}/links\"\n )\n\n if providers.is_error:\n LOGGER.warning(\n \"Response from Materials-Consortia's list of OPTIMADE providers was not \"\n \"successful (status code != 200). No databases will therefore be added at \"\n \"server startup.\"\n )\n return\n\n LOGGER.info(\n \"Registering Materials-Consortia list of OPTIMADE providers' databases.\"\n )\n\n providers = LinksResponse(**providers.json())\n\n valid_providers = []\n for provider in providers.data:\n if get_resource_attribute(provider, \"id\") in (\"exmpl\", \"optimade\"):\n LOGGER.info(\n \"- %s (id=%r) - Skipping: Not a real provider.\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n continue\n\n if not get_resource_attribute(provider, \"attributes.base_url\"):\n LOGGER.info(\n \"- %s (id=%r) - Skipping: No base URL information.\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n continue\n\n valid_providers.append(provider)\n\n # Run queries to each database using the supported major versioned base URL to get a\n # list of the provider's databases.\n # There is no need to use ThreadPoolExecutor here, since we want this to block\n # everything and then finish, before the server actually starts up.\n provider_queries = [\n asyncio.create_task(\n db_get_all_resources(\n database=provider,\n endpoint=\"links\",\n response_model=LinksResponse,\n )\n )\n for provider in valid_providers\n ]\n\n for query in asyncio.as_completed(provider_queries):\n provider_databases, provider = await query\n\n LOGGER.info(\n \"- %s (id=%r) - Processing\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n if not provider_databases:\n LOGGER.info(\" - No OPTIMADE databases found.\")\n continue\n\n provider_databases = [\n db\n for db in provider_databases\n if await clean_python_types(\n get_resource_attribute(db, \"attributes.link_type\", \"\")\n )\n == LinkType.CHILD.value\n ]\n\n if not provider_databases:\n LOGGER.info(\" - No OPTIMADE databases found.\")\n continue\n\n for database in provider_databases:\n if not get_resource_attribute(database, \"attributes.base_url\"):\n LOGGER.info(\n \" - %s (id=%r) - Skipping: No base URL information.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n )\n continue\n\n LOGGER.info(\n \" - %s (id=%r) - Checking versioned base URL and /structures\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n )\n\n async with httpx.AsyncClient() as client:\n try:\n db_response = await client.get(\n f\"{str(get_resource_attribute(database, 'attributes.base_url')).rstrip('/')}\" # noqa: E501\n f\"{BASE_URL_PREFIXES['major']}/structures\",\n )\n except httpx.ReadTimeout:\n LOGGER.info(\n \" - %s (id=%r) - Skipping: Timeout while requesting \"\n \"%s/structures.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n BASE_URL_PREFIXES[\"major\"],\n )\n continue\n if db_response.status_code != 200:\n LOGGER.info(\n \" - %s (id=%r) - Skipping: Response from %s/structures is not \"\n \"200 OK.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n BASE_URL_PREFIXES[\"major\"],\n )\n continue\n\n new_id = (\n f\"{get_resource_attribute(provider, 'id')}\"\n f\"/{get_resource_attribute(database, 'id')}\"\n if len(provider_databases) > 1\n else get_resource_attribute(database, \"id\")\n )\n registered_database, _ = await resource_factory(\n DatabaseCreate(\n id=new_id,\n **await clean_python_types(\n get_resource_attribute(database, \"attributes\", {})\n ),\n )\n )\n LOGGER.info(\n \" - %s (id=%r) - Registered database with id=%r\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n registered_database.id,\n )\nASGI app exception handlers.
These are in addition to the exception handlers available in OPTIMADE Python tools. For more information see https://www.optimade.org/optimade-python-tools/api_reference/server/exception_handlers/.
"},{"location":"api_reference/exception_handlers/#optimade_gateway.exception_handlers.request_validation_exception_handler","title":"request_validation_exception_handler(request, exc) async","text":"Special handler if a RequestValidationError comes from wrong POST data
optimade_gateway/exception_handlers.py async def request_validation_exception_handler(\n request: Request, exc: RequestValidationError\n) -> JSONResponse:\n \"\"\"Special handler if a `RequestValidationError` comes from wrong `POST` data\"\"\"\n status_code = 500\n if request.method in (\"POST\", \"post\"):\n status_code = 400\n\n errors = set()\n for error in exc.errors():\n pointer = \"/\" + \"/\".join([str(_) for _ in error[\"loc\"]])\n source = ErrorSource(pointer=pointer)\n code = error[\"type\"]\n detail = error[\"msg\"]\n errors.add(\n OptimadeError(\n detail=detail,\n status=status_code,\n title=str(exc.__class__.__name__),\n source=source,\n code=code,\n )\n )\n\n return general_exception(request, exc, status_code=status_code, errors=list(errors))\nThe initialization of the ASGI FastAPI application.
"},{"location":"api_reference/main/#optimade_gateway.main.APP","title":"APP = FastAPI(title='OPTIMADE Gateway', description='A gateway server to query multiple OPTIMADE databases.', version=__version__) module-attribute","text":"The FastAPI ASGI application.
"},{"location":"api_reference/main/#optimade_gateway.main.get_root","title":"get_root(request) async","text":"GET /
Introspective overview of gateway server.
Note
Temporarily redirecting to GET /docs.
optimade_gateway/main.py @APP.get(\"/\", include_in_schema=False)\nasync def get_root(request: Request) -> RedirectResponse:\n \"\"\"`GET /`\n\n Introspective overview of gateway server.\n\n !!! note\n Temporarily redirecting to `GET /docs`.\n \"\"\"\n return RedirectResponse(\n request.url.replace(path=f\"{request.url.path.strip('/')}/docs\")\n )\nASGI app middleware.
These are in addition to the middleware available in OPTIMADE Python tools. For more information see https://www.optimade.org/optimade-python-tools/api_reference/server/middleware/.
"},{"location":"api_reference/middleware/#optimade_gateway.middleware.CheckWronglyVersionedBaseUrlsGateways","title":"CheckWronglyVersionedBaseUrlsGateways","text":" Bases: BaseHTTPMiddleware
If a non-supported versioned base URL is supplied to a gateway return 553 Version Not Supported.
optimade_gateway/middleware.py class CheckWronglyVersionedBaseUrlsGateways(BaseHTTPMiddleware):\n \"\"\"If a non-supported versioned base URL is supplied to a gateway\n return `553 Version Not Supported`.\"\"\"\n\n @staticmethod\n async def check_url(url: URL):\n \"\"\"Check URL path for versioned part.\n\n Parameters:\n url: A complete `urllib`-parsed raw URL.\n\n Raises:\n VersionNotSupported: If the URL represents an OPTIMADE versioned base URL\n and the version part is not supported by the implementation.\n\n \"\"\"\n base_url = get_base_url(url)\n optimade_path = f\"{url.scheme}://{url.netloc}{url.path}\"[len(base_url) :]\n match = re.match(\n r\"^/gateways/[^/\\s]+(?P<version>/v[0-9]+(\\.[0-9]+){0,2}).*\", optimade_path\n )\n if (\n match is not None\n and match.group(\"version\") not in BASE_URL_PREFIXES.values()\n ):\n raise VersionNotSupported(\n detail=(\n f\"The parsed versioned base URL {match.group('version')!r} \"\n f\"from {url} is not supported by this implementation. \"\n \"Supported versioned base URLs are: \"\n f\"{', '.join(BASE_URL_PREFIXES.values())}\"\n )\n )\n\n async def dispatch(self, request: Request, call_next):\n if request.url.path:\n await self.check_url(request.url)\n return await call_next(request)\ncheck_url(url) async staticmethod","text":"Check URL path for versioned part.
Parameters:
Name Type Description Defaulturl URL A complete urllib-parsed raw URL.
Raises:
Type DescriptionVersionNotSupported If the URL represents an OPTIMADE versioned base URL and the version part is not supported by the implementation.
Source code inoptimade_gateway/middleware.py @staticmethod\nasync def check_url(url: URL):\n \"\"\"Check URL path for versioned part.\n\n Parameters:\n url: A complete `urllib`-parsed raw URL.\n\n Raises:\n VersionNotSupported: If the URL represents an OPTIMADE versioned base URL\n and the version part is not supported by the implementation.\n\n \"\"\"\n base_url = get_base_url(url)\n optimade_path = f\"{url.scheme}://{url.netloc}{url.path}\"[len(base_url) :]\n match = re.match(\n r\"^/gateways/[^/\\s]+(?P<version>/v[0-9]+(\\.[0-9]+){0,2}).*\", optimade_path\n )\n if (\n match is not None\n and match.group(\"version\") not in BASE_URL_PREFIXES.values()\n ):\n raise VersionNotSupported(\n detail=(\n f\"The parsed versioned base URL {match.group('version')!r} \"\n f\"from {url} is not supported by this implementation. \"\n \"Supported versioned base URLs are: \"\n f\"{', '.join(BASE_URL_PREFIXES.values())}\"\n )\n )\nServer warnings.
The warnings in this module will all be caught by middleware and added to the response under meta.warnings.
OptimadeGatewayWarning","text":" Bases: OptimadeWarning
Base Warning for the optimade-gateway package.
optimade_gateway/warnings.py class OptimadeGatewayWarning(OptimadeWarning):\n \"\"\"Base Warning for the `optimade-gateway` package.\"\"\"\nSortNotSupported","text":" Bases: OptimadeGatewayWarning
Sorting (the sort query parameter) is currently not supported for gateway queries to external OPTIMADE databases. See https://optimade.org/optimade-gateway#sorting for more information.
optimade_gateway/warnings.py class SortNotSupported(OptimadeGatewayWarning):\n \"\"\"Sorting (the `sort` query parameter) is currently not supported for gateway\n queries to external OPTIMADE databases. See\n https://optimade.org/optimade-gateway#sorting for more information.\"\"\"\nConfiguration of the FastAPI server.
"},{"location":"api_reference/common/config/#optimade_gateway.common.config.ServerConfig","title":"ServerConfig","text":" Bases: ServerConfig
This class stores server config parameters in a way that can be easily extended for new config file types.
Source code inoptimade_gateway/common/config.py class ServerConfig(OptimadeServerConfig):\n \"\"\"This class stores server config parameters in a way that\n can be easily extended for new config file types.\n\n \"\"\"\n\n databases_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/databases` endpoint resources.\",\n ),\n ] = \"databases\"\n\n gateways_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/gateways` endpoint resources.\",\n ),\n ] = \"gateways\"\n\n queries_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/queries` endpoint resources.\",\n ),\n ] = \"queries\"\n\n load_optimade_providers_databases: Annotated[\n bool,\n Field(\n description=(\n \"Whether or not to load all valid OPTIMADE providers' databases from \"\n \"the [Materials-Consortia list of OPTIMADE providers]\"\n \"(https://providers.optimade.org) on server startup.\"\n ),\n ),\n ] = True\n\n mongo_certfile: Annotated[\n Path,\n Field(\n description=\"Path to the MongoDB certificate file.\",\n ),\n ] = Path(\"/certs/mongodb.pem\")\n\n mongo_atlas_pem_content: Annotated[\n SecretStr | None,\n Field(\n description=\"PEM content for MongoDB Atlas certificate.\",\n ),\n ] = None\n\n @field_validator(\"mongo_uri\", mode=\"after\")\n @classmethod\n def replace_with_env_vars(cls, value: str) -> str:\n \"\"\"Replace string variables with environment variables, if possible\"\"\"\n res = value\n for match in re.finditer(r\"\\{[^{}]+\\}\", value):\n string_var = match.group()[1:-1]\n env_var = os.getenv(\n string_var, os.getenv(string_var.upper(), os.getenv(string_var.lower()))\n )\n if env_var is not None:\n res = res.replace(match.group(), env_var)\n else:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Could not find an environment variable for \"\n f\"{match.group()!r} from mongo_uri: {value}\"\n )\n )\n )\n return res\n\n @model_validator(mode=\"after\")\n def write_pem_content_to_file(self) -> ServerConfig:\n \"\"\"Write the MongoDB Atlas PEM content to a file\"\"\"\n if self.mongo_atlas_pem_content:\n self.mongo_certfile.parent.mkdir(parents=True, exist_ok=True)\n self.mongo_certfile.write_text(\n self.mongo_atlas_pem_content.get_secret_value()\n )\n\n return self\nreplace_with_env_vars(value) classmethod","text":"Replace string variables with environment variables, if possible
Source code inoptimade_gateway/common/config.py @field_validator(\"mongo_uri\", mode=\"after\")\n@classmethod\ndef replace_with_env_vars(cls, value: str) -> str:\n \"\"\"Replace string variables with environment variables, if possible\"\"\"\n res = value\n for match in re.finditer(r\"\\{[^{}]+\\}\", value):\n string_var = match.group()[1:-1]\n env_var = os.getenv(\n string_var, os.getenv(string_var.upper(), os.getenv(string_var.lower()))\n )\n if env_var is not None:\n res = res.replace(match.group(), env_var)\n else:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Could not find an environment variable for \"\n f\"{match.group()!r} from mongo_uri: {value}\"\n )\n )\n )\n return res\nwrite_pem_content_to_file()","text":"Write the MongoDB Atlas PEM content to a file
Source code inoptimade_gateway/common/config.py @model_validator(mode=\"after\")\ndef write_pem_content_to_file(self) -> ServerConfig:\n \"\"\"Write the MongoDB Atlas PEM content to a file\"\"\"\n if self.mongo_atlas_pem_content:\n self.mongo_certfile.parent.mkdir(parents=True, exist_ok=True)\n self.mongo_certfile.write_text(\n self.mongo_atlas_pem_content.get_secret_value()\n )\n\n return self\nSpecific OPTIMADE Gateway Python exceptions.
"},{"location":"api_reference/common/exceptions/#optimade_gateway.common.exceptions.OptimadeGatewayError","title":"OptimadeGatewayError","text":" Bases: Exception
General OPTIMADE Gateway exception.
Source code inoptimade_gateway/common/exceptions.py class OptimadeGatewayError(Exception):\n \"\"\"General OPTIMADE Gateway exception.\"\"\"\nLogging to both file and console
"},{"location":"api_reference/common/logger/#optimade_gateway.common.logger.disable_logging","title":"disable_logging()","text":"Temporarily disable logging.
Usage:
from optimade_gateway.common.logger import disable_logging\n\n# Do stuff, logging to all handlers.\n# ...\nwith disable_logging():\n # Do stuff, without logging to any handlers.\n # ...\n# Do stuff, logging to all handlers now re-enabled.\n# ...\noptimade_gateway/common/logger.py @contextmanager\ndef disable_logging():\n \"\"\"Temporarily disable logging.\n\n Usage:\n\n ```python\n from optimade_gateway.common.logger import disable_logging\n\n # Do stuff, logging to all handlers.\n # ...\n with disable_logging():\n # Do stuff, without logging to any handlers.\n # ...\n # Do stuff, logging to all handlers now re-enabled.\n # ...\n ```\n\n \"\"\"\n try:\n # Disable logging lower than CRITICAL level\n logging.disable(logging.CRITICAL)\n yield\n finally:\n # Re-enable logging to desired levels\n logging.disable(logging.NOTSET)\nCommon utility functions.
These functions may be used in general throughout the OPTIMADE Gateway Python code.
"},{"location":"api_reference/common/utils/#optimade_gateway.common.utils.clean_python_types","title":"clean_python_types(data, **dump_kwargs) async","text":"Turn any types into MongoDB-friendly Python types.
Use model_dump() method for Pydantic models. Use value property for Enums. Turn tuples and sets into lists.
optimade_gateway/common/utils.py async def clean_python_types(data: Any, **dump_kwargs: Any) -> Any:\n \"\"\"Turn any types into MongoDB-friendly Python types.\n\n Use `model_dump()` method for Pydantic models.\n Use `value` property for Enums.\n Turn tuples and sets into lists.\n \"\"\"\n if isinstance(data, (list, tuple, set)):\n res_list = []\n for datum in data:\n res_list.append(await clean_python_types(datum, **dump_kwargs))\n return res_list\n\n if isinstance(data, dict):\n res_dict = {}\n for key in list(data.keys()):\n res_dict[key] = await clean_python_types(data[key], **dump_kwargs)\n return res_dict\n\n if isinstance(data, BaseModel):\n # Pydantic model\n return await clean_python_types(data.model_dump(**dump_kwargs))\n\n if isinstance(data, Enum):\n return await clean_python_types(data.value, **dump_kwargs)\n\n if isinstance(data, type):\n return await clean_python_types(\n f\"{data.__module__}.{data.__name__}\", **dump_kwargs\n )\n\n if isinstance(data, AnyUrl):\n return await clean_python_types(str(data), **dump_kwargs)\n\n # Unknown or other basic type, e.g., str, int, etc.\n return data\nget_resource_attribute(resource, field, default=None, disambiguate=True)","text":"Return a resource's field's value
Get the field value no matter if the resource is a pydantic model or a Python dictionary.
Determine ambiguous field values and return them if desired (disambiguate). For example, if \"attributes.base_url\" is requested for a LinksResource it can be either a string, a Link model or a dictionary resembling the Link model.
Parameters:
Name Type Description Defaultresource BaseModel | dict[str, Any] | None The resource, from which to get the field value.
requiredfield str The resource field. This can be a dot-separated nested field, e.g., \"attributes.base_url\".
default Any The default value to return if field does not exist.
None disambiguate bool Whether or not to \"shortcut\" a field value. For example, for attributes.base_url, if True, this would return the string value or the value of it's \"href\" key.
True Returns:
Type DescriptionAny The resource's field's value.
Source code inoptimade_gateway/common/utils.py def get_resource_attribute(\n resource: BaseModel | dict[str, Any] | None,\n field: str,\n default: Any = None,\n disambiguate: bool = True,\n) -> Any:\n \"\"\"Return a resource's field's value\n\n Get the field value no matter if the resource is a pydantic model or a Python\n dictionary.\n\n Determine ambiguous field values and return them if desired (`disambiguate`).\n For example, if\n [`\"attributes.base_url\"`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes.base_url)\n is requested for a\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n it can be either a string, a\n [`Link`](https://www.optimade.org/optimade-python-tools/api_reference/models/jsonapi/#optimade.models.jsonapi.Link)\n model or a dictionary resembling the `Link` model.\n\n Parameters:\n resource: The resource, from which to get the field value.\n field: The resource field. This can be a dot-separated nested field, e.g.,\n `\"attributes.base_url\"`.\n default: The default value to return if `field` does not exist.\n disambiguate: Whether or not to \"shortcut\" a field value.\n For example, for `attributes.base_url`, if `True`, this would return the\n string value or the value of it's `\"href\"` key.\n\n Returns:\n The resource's field's value.\n\n \"\"\"\n if isinstance(resource, BaseModel):\n _get_attr = getattr\n elif isinstance(resource, dict):\n\n def _get_attr(mapping: dict, key: str, default: Any) -> Any: # type: ignore[misc]\n return mapping.get(key, default)\n\n elif resource is None:\n # Allow passing `None`, but simply return `default`\n return default\n else:\n raise TypeError(\n \"resource must be either a pydantic model or a Python dictionary, it was \"\n f\"of type {type(resource)!r}\"\n )\n\n fields = field.split(\".\")\n for _ in fields[:-1]:\n resource = _get_attr(resource, _, {})\n field = fields[-1]\n value = _get_attr(resource, field, default)\n\n if (\n disambiguate\n and field in (\"base_url\", \"next\", \"prev\", \"last\", \"first\")\n and not isinstance(value, (str, AnyUrl))\n ):\n value = _get_attr(value, \"href\", default)\n\n return value\nBase resource mapper.
Based on the BaseResourceMapper in OPTIMADE Python tools.
BaseResourceMapper","text":" Bases: BaseResourceMapper
Generic Resource Mapper that defines and performs the mapping between objects in the database and the resource objects defined by the specification.
NoteThis is a \"wrapped\" sub-class to make certain methods asynchronous.
Attributes:
Name Type DescriptionALIASES a tuple of aliases between OPTIMADE field names and the field names in the database , e.g. ((\"elements\", \"custom_elements_field\")).
LENGTH_ALIASES a tuple of aliases between a field name and another field that defines its length, to be used when querying, e.g. ((\"elements\", \"nelements\")). e.g. ((\"elements\", \"custom_elements_field\")).
ENTRY_RESOURCE_CLASS The entry type that this mapper corresponds to.
PROVIDER_FIELDS a tuple of extra field names that this mapper should support when querying with the database prefix.
TOP_LEVEL_NON_ATTRIBUTES_FIELDS the set of top-level field names common to all endpoints.
SUPPORTED_PREFIXES The set of prefixes registered by this mapper.
ALL_ATTRIBUTES The set of attributes defined across the entry resource class and the server configuration.
ENTRY_RESOURCE_ATTRIBUTES A dictionary of attributes and their definitions defined by the schema of the entry resource class.
ENDPOINT The expected endpoint name for this resource, as defined by the type in the schema of the entry resource class.
optimade_gateway/mappers/base.py class BaseResourceMapper(OptimadeBaseResourceMapper):\n \"\"\"\n Generic Resource Mapper that defines and performs the mapping\n between objects in the database and the resource objects defined by\n the specification.\n\n Note:\n This is a \"wrapped\" sub-class to make certain methods asynchronous.\n\n Attributes:\n ALIASES: a tuple of aliases between\n OPTIMADE field names and the field names in the database ,\n e.g. `((\"elements\", \"custom_elements_field\"))`.\n LENGTH_ALIASES: a tuple of aliases between\n a field name and another field that defines its length, to be used\n when querying, e.g. `((\"elements\", \"nelements\"))`.\n e.g. `((\"elements\", \"custom_elements_field\"))`.\n ENTRY_RESOURCE_CLASS: The entry type that this mapper corresponds to.\n PROVIDER_FIELDS: a tuple of extra field names that this\n mapper should support when querying with the database prefix.\n TOP_LEVEL_NON_ATTRIBUTES_FIELDS: the set of top-level\n field names common to all endpoints.\n SUPPORTED_PREFIXES: The set of prefixes registered by this mapper.\n ALL_ATTRIBUTES: The set of attributes defined across the entry\n resource class and the server configuration.\n ENTRY_RESOURCE_ATTRIBUTES: A dictionary of attributes and their definitions\n defined by the schema of the entry resource class.\n ENDPOINT: The expected endpoint name for this resource, as defined by\n the `type` in the schema of the entry resource class.\n\n \"\"\"\n\n @classmethod\n async def adeserialize(\n cls, results: dict | Iterable[dict]\n ) -> list[EntryResource] | EntryResource:\n \"\"\"Asynchronous version of the `deserialize()` class method.\n\n Parameters:\n results: A list of or a single dictionary, representing an entry-endpoint\n resource.\n\n Returns:\n The deserialized list of or single pydantic resource model for the input\n `results`.\n\n \"\"\"\n return super().deserialize(results)\n\n @classmethod\n def map_back(cls, doc: dict) -> dict:\n from optimade.server.routers.utils import BASE_URL_PREFIXES\n\n if \"_id\" in doc:\n _id = str(doc.pop(\"_id\"))\n if \"id\" not in doc:\n doc[\"id\"] = _id\n\n doc[\"links\"] = {\n \"self\": AnyUrl(\n url=(\n f\"{CONFIG.base_url.strip('/')}{BASE_URL_PREFIXES['major']}\"\n f\"/{cls.ENDPOINT}/{doc['id']}\"\n ),\n )\n }\n return super().map_back(doc)\nadeserialize(results) async classmethod","text":"Asynchronous version of the deserialize() class method.
Parameters:
Name Type Description Defaultresults dict | Iterable[dict] A list of or a single dictionary, representing an entry-endpoint resource.
requiredReturns:
Type Descriptionlist[EntryResource] | EntryResource The deserialized list of or single pydantic resource model for the input
list[EntryResource] | EntryResource results.
optimade_gateway/mappers/base.py @classmethod\nasync def adeserialize(\n cls, results: dict | Iterable[dict]\n) -> list[EntryResource] | EntryResource:\n \"\"\"Asynchronous version of the `deserialize()` class method.\n\n Parameters:\n results: A list of or a single dictionary, representing an entry-endpoint\n resource.\n\n Returns:\n The deserialized list of or single pydantic resource model for the input\n `results`.\n\n \"\"\"\n return super().deserialize(results)\nResource mapper for resources under /databases.
These resources are LinksResources.
DatabasesMapper","text":" Bases: LinksMapper
/databases-endpoint resources mapper.
optimade_gateway/mappers/databases.py class DatabasesMapper(LinksMapper):\n \"\"\"`/databases`-endpoint resources mapper.\"\"\"\n\n ENDPOINT = \"databases\"\nResource mapper for GatewayResource.
GatewaysMapper","text":" Bases: BaseResourceMapper
GatewayResource mapper.
optimade_gateway/mappers/gateways.py class GatewaysMapper(BaseResourceMapper):\n \"\"\"[`GatewayResource`][optimade_gateway.models.gateways.GatewayResource] mapper.\"\"\"\n\n ENDPOINT = \"gateways\"\n ENTRY_RESOURCE_CLASS = GatewayResource\nReplicate of LinksMapper in OPTIMADE Python tools.
LinksMapper","text":" Bases: BaseResourceMapper
Replicate of LinksMapper in OPTIMADE Python tools.
This is based on the OPTIMADE Gateway BaseResourceMapper however.
optimade_gateway/mappers/links.py class LinksMapper(BaseResourceMapper):\n \"\"\"Replicate of\n [`LinksMapper`](https://www.optimade.org/optimade-python-tools/api_reference/server/mappers/links/#optimade.server.mappers.links.LinksMapper)\n in OPTIMADE Python tools.\n\n This is based on the OPTIMADE Gateway\n [`BaseResourceMapper`][optimade_gateway.mappers.base.BaseResourceMapper] however.\n \"\"\"\n\n ENDPOINT = \"links\"\n ENTRY_RESOURCE_CLASS = LinksResource\n\n @classmethod\n def map_back(cls, doc: dict) -> dict:\n type_ = doc.get(\"type\") or \"links\"\n newdoc = super().map_back(doc)\n newdoc[\"type\"] = type_\n return newdoc\nResource mapper for QueryResource.
QueryMapper","text":" Bases: BaseResourceMapper
QueryResource mapper.
optimade_gateway/mappers/queries.py class QueryMapper(BaseResourceMapper):\n \"\"\"[`QueryResource`][optimade_gateway.models.queries.QueryResource] mapper.\"\"\"\n\n ENDPOINT = \"queries\"\n ENTRY_RESOURCE_CLASS = QueryResource\nPydantic models/schemas for the LinksResource used in /databases
"},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate","title":"DatabaseCreate","text":" Bases: EntryResourceCreate, LinksResourceAttributes
Model for creating new LinksResources representing /databases resources in the MongoDB.
Required fields:
namebase_urlOriginal required fields for a LinksResourceAttributes model:
namedescriptionlink_typeoptimade_gateway/models/databases.py class DatabaseCreate(EntryResourceCreate, LinksResourceAttributes):\n \"\"\"Model for creating new LinksResources representing `/databases` resources in the\n MongoDB.\n\n Required fields:\n\n - `name`\n - `base_url`\n\n Original required fields for a\n [`LinksResourceAttributes`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes)\n model:\n\n - `name`\n - `description`\n - `link_type`\n\n \"\"\"\n\n description: Annotated[\n str | None,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"description\"].description\n ),\n ] = None\n\n base_url: Annotated[\n JsonLinkType,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"base_url\"].description\n ),\n ]\n\n homepage: Annotated[\n JsonLinkType | None,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"homepage\"].description,\n ),\n ] = None\n\n link_type: Annotated[\n LinkType | None,\n StrictField(\n title=LinksResourceAttributes.model_fields[\"link_type\"].title,\n description=LinksResourceAttributes.model_fields[\"link_type\"].description,\n ),\n ] = None\n\n @field_validator(\"link_type\", mode=\"after\")\n @classmethod\n def ensure_database_link_type(cls, value: LinkType) -> LinkType:\n \"\"\"Ensure databases are not index meta-database-only types\n\n I.e., ensure they're not of type `\"root\"` or `\"providers\"`.\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n if value in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway-usable database resources. Given link_type: {value}\"\n )\n return value\nbase_url: Annotated[JsonLinkType, StrictField(description=LinksResourceAttributes.model_fields['base_url'].description)] instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.description","title":"description: Annotated[str | None, StrictField(description=LinksResourceAttributes.model_fields['description'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.homepage","title":"homepage: Annotated[JsonLinkType | None, StrictField(description=LinksResourceAttributes.model_fields['homepage'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.link_type","title":"link_type: Annotated[LinkType | None, StrictField(title=LinksResourceAttributes.model_fields['link_type'].title, description=LinksResourceAttributes.model_fields['link_type'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.ensure_database_link_type","title":"ensure_database_link_type(value) classmethod","text":"Ensure databases are not index meta-database-only types
I.e., ensure they're not of type \"root\" or \"providers\".
Note
Both \"external\" and \"child\" can still represent index meta-dbs, but \"root\" and \"providers\" can not represent \"regular\" dbs.
optimade_gateway/models/databases.py @field_validator(\"link_type\", mode=\"after\")\n@classmethod\ndef ensure_database_link_type(cls, value: LinkType) -> LinkType:\n \"\"\"Ensure databases are not index meta-database-only types\n\n I.e., ensure they're not of type `\"root\"` or `\"providers\"`.\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n if value in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway-usable database resources. Given link_type: {value}\"\n )\n return value\nPydantic models/schemas for the Gateways resource.
"},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate","title":"GatewayCreate","text":" Bases: EntryResourceCreate, GatewayResourceAttributes
Model for creating new Gateway resources in the MongoDB
Source code inoptimade_gateway/models/gateways.py class GatewayCreate(EntryResourceCreate, GatewayResourceAttributes):\n \"\"\"Model for creating new Gateway resources in the MongoDB\"\"\"\n\n id: Annotated[\n str | None,\n OptimadeField(\n description=EntryResource.model_fields[\"id\"].description,\n support=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-support\"\n ],\n queryable=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-queryable\"\n ],\n pattern=r\"^[^/]*$\", # This pattern is the special addition\n ),\n ] = None\n\n database_ids: Annotated[\n set[str] | None,\n Field(description=\"A unique list of database IDs for registered databases.\"),\n ] = None\n\n databases: Annotated[\n list[LinksResource] | None,\n Field(\n description=GatewayResourceAttributes.model_fields[\"databases\"].description\n ),\n ] = None # type: ignore[assignment]\n\n @model_validator(mode=\"after\")\n def specify_databases(self) -> GatewayCreate:\n \"\"\"Either `database_ids` or `databases` must be non-empty.\n Both together is also fine.\n \"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"databases\")):\n raise ValueError(\"Either 'database_ids' or 'databases' MUST be specified\")\n return self\ndatabase_ids: Annotated[set[str] | None, Field(description='A unique list of database IDs for registered databases.')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.databases","title":"databases: Annotated[list[LinksResource] | None, Field(description=GatewayResourceAttributes.model_fields['databases'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.id","title":"id: Annotated[str | None, OptimadeField(description=EntryResource.model_fields['id'].description, support=EntryResource.model_fields['id'].json_schema_extra['x-optimade-support'], queryable=EntryResource.model_fields['id'].json_schema_extra['x-optimade-queryable'], pattern='^[^/]*$')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.specify_databases","title":"specify_databases()","text":"Either database_ids or databases must be non-empty. Both together is also fine.
optimade_gateway/models/gateways.py @model_validator(mode=\"after\")\ndef specify_databases(self) -> GatewayCreate:\n \"\"\"Either `database_ids` or `databases` must be non-empty.\n Both together is also fine.\n \"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"databases\")):\n raise ValueError(\"Either 'database_ids' or 'databases' MUST be specified\")\n return self\nGatewayResource","text":" Bases: EntryResource
OPTIMADE gateway
A resource representing a dynamic collection of OPTIMADE databases. The gateway can be treated as any other OPTIMADE gateway, but the entries are an aggregate of multiple databases. The id of each aggregated resource will reflect the originating database.
optimade_gateway/models/gateways.py class GatewayResource(EntryResource):\n \"\"\"OPTIMADE gateway\n\n A resource representing a dynamic collection of OPTIMADE databases.\n The gateway can be treated as any other OPTIMADE gateway, but the entries are an\n aggregate of multiple databases. The `id` of each aggregated resource will reflect\n the originating database.\n \"\"\"\n\n id: Annotated[\n str,\n OptimadeField(\n description=EntryResource.model_fields[\"id\"].description,\n support=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-support\"\n ],\n queryable=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-queryable\"\n ],\n pattern=r\"^[^/]*$\",\n ),\n ]\n\n type: Annotated[\n Literal[\"gateways\"],\n Field(description=\"The name of the type of an entry.\"),\n ] = \"gateways\"\n\n attributes: Annotated[\n GatewayResourceAttributes,\n Field(description=EntryResource.model_fields[\"attributes\"].description),\n ]\nattributes: Annotated[GatewayResourceAttributes, Field(description=EntryResource.model_fields['attributes'].description)] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResource.id","title":"id: Annotated[str, OptimadeField(description=EntryResource.model_fields['id'].description, support=EntryResource.model_fields['id'].json_schema_extra['x-optimade-support'], queryable=EntryResource.model_fields['id'].json_schema_extra['x-optimade-queryable'], pattern='^[^/]*$')] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResource.type","title":"type: Annotated[Literal['gateways'], Field(description='The name of the type of an entry.')] = 'gateways' class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResourceAttributes","title":"GatewayResourceAttributes","text":" Bases: EntryResourceAttributes
Attributes for an OPTIMADE gateway
Source code inoptimade_gateway/models/gateways.py class GatewayResourceAttributes(EntryResourceAttributes):\n \"\"\"Attributes for an OPTIMADE gateway\"\"\"\n\n databases: Annotated[\n list[LinksResource],\n Field(\n description=(\n \"List of databases (OPTIMADE 'links') to be queried in this gateway.\"\n ),\n ),\n ]\n\n @field_validator(\"databases\", mode=\"after\")\n @classmethod\n def unique_base_urls(cls, value: list[LinksResource]) -> list[LinksResource]:\n \"\"\"Remove extra entries with repeated base_urls.\n\n Also, ensure databases are not of type `\"root\"` or `\"providers\"`\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n for resource in value:\n if resource.attributes.link_type in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway resources. Given database: {resource}\"\n )\n\n db_base_urls = [_.attributes.base_url for _ in value]\n unique_base_urls = set(db_base_urls)\n if len(db_base_urls) == len(unique_base_urls):\n return value\n\n repeated_base_urls = [_ for _ in unique_base_urls if db_base_urls.count(_) > 1]\n new_databases = [\n _ for _ in value if _.attributes.base_url not in repeated_base_urls\n ]\n for base_url in repeated_base_urls:\n new_databases.append(\n next(_ for _ in value if _.attributes.base_url == base_url)\n )\n warnings.warn(\n \"Removed extra database entries for a gateway, because the base_url was \"\n \"repeated. The first found database entry was kept, while the others were \"\n f\"removed. Original number of databases: {len(value)}. New number of \"\n f\"databases: {len(new_databases)} Repeated base_urls (number of repeats): \"\n \"{}\".format(\n [\n f\"{base_url} ({db_base_urls.count(base_url)})\"\n for base_url in repeated_base_urls\n ]\n ),\n OptimadeGatewayWarning,\n )\n return new_databases\ndatabases: Annotated[list[LinksResource], Field(description=\"List of databases (OPTIMADE 'links') to be queried in this gateway.\")] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResourceAttributes.unique_base_urls","title":"unique_base_urls(value) classmethod","text":"Remove extra entries with repeated base_urls.
Also, ensure databases are not of type \"root\" or \"providers\"
Note
Both \"external\" and \"child\" can still represent index meta-dbs, but \"root\" and \"providers\" can not represent \"regular\" dbs.
optimade_gateway/models/gateways.py @field_validator(\"databases\", mode=\"after\")\n@classmethod\ndef unique_base_urls(cls, value: list[LinksResource]) -> list[LinksResource]:\n \"\"\"Remove extra entries with repeated base_urls.\n\n Also, ensure databases are not of type `\"root\"` or `\"providers\"`\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n for resource in value:\n if resource.attributes.link_type in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway resources. Given database: {resource}\"\n )\n\n db_base_urls = [_.attributes.base_url for _ in value]\n unique_base_urls = set(db_base_urls)\n if len(db_base_urls) == len(unique_base_urls):\n return value\n\n repeated_base_urls = [_ for _ in unique_base_urls if db_base_urls.count(_) > 1]\n new_databases = [\n _ for _ in value if _.attributes.base_url not in repeated_base_urls\n ]\n for base_url in repeated_base_urls:\n new_databases.append(\n next(_ for _ in value if _.attributes.base_url == base_url)\n )\n warnings.warn(\n \"Removed extra database entries for a gateway, because the base_url was \"\n \"repeated. The first found database entry was kept, while the others were \"\n f\"removed. Original number of databases: {len(value)}. New number of \"\n f\"databases: {len(new_databases)} Repeated base_urls (number of repeats): \"\n \"{}\".format(\n [\n f\"{base_url} ({db_base_urls.count(base_url)})\"\n for base_url in repeated_base_urls\n ]\n ),\n OptimadeGatewayWarning,\n )\n return new_databases\nPydantic models/schemas for the Queries resource.
"},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QUERY_PARAMETERS","title":"QUERY_PARAMETERS: QueryParameters = {'annotations': {name: FieldInfo.from_annotation(parameter.annotation)for (name, parameter) in inspect.signature(EntryListingQueryParams).parameters.items()}, 'defaults': EntryListingQueryParams()} module-attribute","text":"Entry listing URL query parameters from the optimade package (EntryListingQueryParams).
EndpointEntryType","text":" Bases: Enum
Entry endpoint resource types, mapping to their pydantic models from the optimade package.
optimade_gateway/models/queries.py class EndpointEntryType(Enum):\n \"\"\"Entry endpoint resource types, mapping to their pydantic models from the\n `optimade` package.\"\"\"\n\n REFERENCES = \"references\"\n STRUCTURES = \"structures\"\n\n def get_resource_model(self) -> ReferenceResource | StructureResource:\n \"\"\"Get the matching pydantic model for a resource.\"\"\"\n return {\n \"references\": ReferenceResource,\n \"structures\": StructureResource,\n }[self.value]\n\n def get_response_model(\n self, single: bool = False\n ) -> (\n ReferenceResponseMany\n | ReferenceResponseOne\n | StructureResponseMany\n | StructureResponseOne\n ):\n \"\"\"Get the matching pydantic model for a successful response.\"\"\"\n if single:\n return {\n \"references\": ReferenceResponseOne,\n \"structures\": StructureResponseOne,\n }[self.value]\n return {\n \"references\": ReferenceResponseMany,\n \"structures\": StructureResponseMany,\n }[self.value]\nREFERENCES = 'references' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.EndpointEntryType.STRUCTURES","title":"STRUCTURES = 'structures' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.EndpointEntryType.get_resource_model","title":"get_resource_model()","text":"Get the matching pydantic model for a resource.
Source code inoptimade_gateway/models/queries.py def get_resource_model(self) -> ReferenceResource | StructureResource:\n \"\"\"Get the matching pydantic model for a resource.\"\"\"\n return {\n \"references\": ReferenceResource,\n \"structures\": StructureResource,\n }[self.value]\nget_response_model(single=False)","text":"Get the matching pydantic model for a successful response.
Source code inoptimade_gateway/models/queries.py def get_response_model(\n self, single: bool = False\n) -> (\n ReferenceResponseMany\n | ReferenceResponseOne\n | StructureResponseMany\n | StructureResponseOne\n):\n \"\"\"Get the matching pydantic model for a successful response.\"\"\"\n if single:\n return {\n \"references\": ReferenceResponseOne,\n \"structures\": StructureResponseOne,\n }[self.value]\n return {\n \"references\": ReferenceResponseMany,\n \"structures\": StructureResponseMany,\n }[self.value]\nEntryResource","text":" Bases: EntryResource
Entry Resource ensuring datetimes are not naive.
Source code inoptimade_gateway/models/queries.py class EntryResource(OptimadeEntryResource):\n \"\"\"Entry Resource ensuring datetimes are not naive.\"\"\"\n\n @field_validator(\"attributes\", mode=\"after\")\n @classmethod\n def ensure_non_naive_datetime(\n cls, value: EntryResourceAttributes\n ) -> EntryResourceAttributes:\n \"\"\"Set timezone to UTC if datetime is naive.\"\"\"\n if value.last_modified and value.last_modified.tzinfo is None:\n value.last_modified = value.last_modified.replace(tzinfo=timezone.utc)\n return value\nensure_non_naive_datetime(value) classmethod","text":"Set timezone to UTC if datetime is naive.
Source code inoptimade_gateway/models/queries.py @field_validator(\"attributes\", mode=\"after\")\n@classmethod\ndef ensure_non_naive_datetime(\n cls, value: EntryResourceAttributes\n) -> EntryResourceAttributes:\n \"\"\"Set timezone to UTC if datetime is naive.\"\"\"\n if value.last_modified and value.last_modified.tzinfo is None:\n value.last_modified = value.last_modified.replace(tzinfo=timezone.utc)\n return value\nGatewayQueryResponse","text":" Bases: Response
Response from a Gateway Query.
Source code inoptimade_gateway/models/queries.py class GatewayQueryResponse(Response):\n \"\"\"Response from a Gateway Query.\"\"\"\n\n data: Annotated[\n dict[str, list[EntryResource] | list[dict[str, Any]]],\n StrictField(uniqueItems=True, description=\"Outputted Data.\"),\n ]\n\n meta: Annotated[\n ResponseMeta,\n StrictField(description=\"A meta object containing non-standard information.\"),\n ]\n\n errors: Annotated[\n list[OptimadeError] | None,\n StrictField(\n description=(\n \"A list of OPTIMADE-specific JSON API error objects, where the field \"\n \"detail MUST be present.\"\n ),\n uniqueItems=True,\n ),\n ] = [] # noqa: RUF012\n\n included: Annotated[\n list[EntryResource] | list[dict[str, Any]] | None,\n StrictField(\n description=\"A list of unique included OPTIMADE entry resources.\",\n uniqueItems=True,\n union_mode=\"left_to_right\",\n ),\n ] = None\n\n @model_validator(mode=\"after\")\n def either_data_meta_or_errors_must_be_set(self) -> GatewayQueryResponse:\n \"\"\"Overwrite `either_data_meta_or_errors_must_be_set`.\n\n `errors` should be allowed to be present always for this special response.\n \"\"\"\n return self\ndata: Annotated[dict[str, list[EntryResource] | list[dict[str, Any]]], StrictField(uniqueItems=True, description='Outputted Data.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.errors","title":"errors: Annotated[list[OptimadeError] | None, StrictField(description='A list of OPTIMADE-specific JSON API error objects, where the field detail MUST be present.', uniqueItems=True)] = [] class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.included","title":"included: Annotated[list[EntryResource] | list[dict[str, Any]] | None, StrictField(description='A list of unique included OPTIMADE entry resources.', uniqueItems=True, union_mode='left_to_right')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.meta","title":"meta: Annotated[ResponseMeta, StrictField(description='A meta object containing non-standard information.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.either_data_meta_or_errors_must_be_set","title":"either_data_meta_or_errors_must_be_set()","text":"Overwrite either_data_meta_or_errors_must_be_set.
errors should be allowed to be present always for this special response.
optimade_gateway/models/queries.py @model_validator(mode=\"after\")\ndef either_data_meta_or_errors_must_be_set(self) -> GatewayQueryResponse:\n \"\"\"Overwrite `either_data_meta_or_errors_must_be_set`.\n\n `errors` should be allowed to be present always for this special response.\n \"\"\"\n return self\nOptimadeQueryParameters","text":" Bases: BaseModel
Common OPTIMADE entry listing endpoint query parameters.
Source code inoptimade_gateway/models/queries.py class OptimadeQueryParameters(BaseModel):\n \"\"\"Common OPTIMADE entry listing endpoint query parameters.\"\"\"\n\n filter: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"filter\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].filter\n\n response_format: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"response_format\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].response_format\n\n email_address: Annotated[\n EmailStr | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"email_address\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].email_address\n\n response_fields: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"response_fields\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"response_fields\"]\n .metadata[0]\n .pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].response_fields\n\n sort: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"sort\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"sort\"].metadata[0].pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].sort\n\n page_limit: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_limit\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_limit\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_limit\n\n page_offset: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_offset\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_offset\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_offset\n\n page_number: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_number\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_number\n\n page_cursor: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_cursor\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_cursor\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_cursor\n\n page_above: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_above\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_above\n\n page_below: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_below\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_below\n\n include: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"include\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].include\n\n api_hint: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"api_hint\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"api_hint\"].metadata[0].pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].api_hint\napi_hint: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['api_hint'].description, pattern=QUERY_PARAMETERS['annotations']['api_hint'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].api_hint class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.email_address","title":"email_address: Annotated[EmailStr | None, Field(description=QUERY_PARAMETERS['annotations']['email_address'].description)] = QUERY_PARAMETERS['defaults'].email_address class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.filter","title":"filter: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['filter'].description)] = QUERY_PARAMETERS['defaults'].filter class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.include","title":"include: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['include'].description)] = QUERY_PARAMETERS['defaults'].include class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_above","title":"page_above: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_above'].description)] = QUERY_PARAMETERS['defaults'].page_above class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_below","title":"page_below: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_below'].description)] = QUERY_PARAMETERS['defaults'].page_below class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_cursor","title":"page_cursor: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_cursor'].description, ge=QUERY_PARAMETERS['annotations']['page_cursor'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_cursor class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_limit","title":"page_limit: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_limit'].description, ge=QUERY_PARAMETERS['annotations']['page_limit'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_limit class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_number","title":"page_number: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_number'].description)] = QUERY_PARAMETERS['defaults'].page_number class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_offset","title":"page_offset: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_offset'].description, ge=QUERY_PARAMETERS['annotations']['page_offset'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_offset class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.response_fields","title":"response_fields: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['response_fields'].description, pattern=QUERY_PARAMETERS['annotations']['response_fields'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].response_fields class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.response_format","title":"response_format: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['response_format'].description)] = QUERY_PARAMETERS['defaults'].response_format class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.sort","title":"sort: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['sort'].description, pattern=QUERY_PARAMETERS['annotations']['sort'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].sort class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate","title":"QueryCreate","text":" Bases: EntryResourceCreate, QueryResourceAttributes
Model for creating new Query resources in the MongoDB
Source code inoptimade_gateway/models/queries.py class QueryCreate(EntryResourceCreate, QueryResourceAttributes):\n \"\"\"Model for creating new Query resources in the MongoDB\"\"\"\n\n state: Annotated[\n QueryState | None,\n Field(\n title=QueryResourceAttributes.model_fields[\"state\"].title,\n description=QueryResourceAttributes.model_fields[\"state\"].description,\n json_schema_extra=QueryResourceAttributes.model_fields[\n \"state\"\n ].json_schema_extra,\n ),\n ] = None # type: ignore[assignment]\n endpoint: Annotated[\n EndpointEntryType | None,\n Field(\n title=QueryResourceAttributes.model_fields[\"endpoint\"].title,\n description=QueryResourceAttributes.model_fields[\"endpoint\"].description,\n json_schema_extra=QueryResourceAttributes.model_fields[\n \"endpoint\"\n ].json_schema_extra,\n ),\n ] = None # type: ignore[assignment]\n\n @field_validator(\"query_parameters\", mode=\"after\")\n @classmethod\n def sort_not_supported(\n cls, value: OptimadeQueryParameters\n ) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nendpoint: Annotated[EndpointEntryType | None, Field(title=QueryResourceAttributes.model_fields['endpoint'].title, description=QueryResourceAttributes.model_fields['endpoint'].description, json_schema_extra=QueryResourceAttributes.model_fields['endpoint'].json_schema_extra)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate.state","title":"state: Annotated[QueryState | None, Field(title=QueryResourceAttributes.model_fields['state'].title, description=QueryResourceAttributes.model_fields['state'].description, json_schema_extra=QueryResourceAttributes.model_fields['state'].json_schema_extra)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate.sort_not_supported","title":"sort_not_supported(value) classmethod","text":"Warn and reset value if sort is supplied.
optimade_gateway/models/queries.py @field_validator(\"query_parameters\", mode=\"after\")\n@classmethod\ndef sort_not_supported(\n cls, value: OptimadeQueryParameters\n) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nQueryParameters","text":" Bases: TypedDict
Type definition for QUERY_PARAMETERS.
optimade_gateway/models/queries.py class QueryParameters(TypedDict):\n \"\"\"Type definition for `QUERY_PARAMETERS`.\"\"\"\n\n annotations: dict[str, FieldInfo]\n defaults: EntryListingQueryParams\nannotations: dict[str, FieldInfo] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryParameters.defaults","title":"defaults: EntryListingQueryParams instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource","title":"QueryResource","text":" Bases: EntryResource
OPTIMADE query resource for a gateway
Source code inoptimade_gateway/models/queries.py class QueryResource(EntryResource):\n \"\"\"OPTIMADE query resource for a gateway\"\"\"\n\n type: Annotated[\n Literal[\"queries\"],\n Field(\n description=\"The name of the type of an entry.\",\n ),\n ] = \"queries\"\n\n attributes: QueryResourceAttributes\n\n async def response_as_optimade(\n self,\n url: None | (\n urllib.parse.ParseResult | urllib.parse.SplitResult | StarletteURL | str\n ) = None,\n ) -> EntryResponseMany | ErrorResponse:\n \"\"\"Return `attributes.response` as a valid OPTIMADE entry listing response.\n\n Note, this method disregards the state of the query and will simply return the\n query results as they currently are (if there are any at all).\n\n Parameters:\n url: Optionally, update the `meta.query.representation` value with this.\n\n Returns:\n A valid OPTIMADE entry-listing response according to the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints)\n or an error response, if errors were returned or occurred during the query.\n\n \"\"\"\n from optimade.server.routers.utils import (\n meta_values,\n )\n\n async def _update_id(\n entry_: EntryResource | dict[str, Any], database_provider_: str\n ) -> EntryResource | dict[str, Any]:\n \"\"\"Internal utility function to prepend the entries' `id` with\n `provider/database/`.\n\n Parameters:\n entry_: The entry as a model or a dictionary.\n database_provider_: `provider/database` string.\n\n Returns:\n The entry with an updated `id` value.\n\n \"\"\"\n if isinstance(entry_, dict):\n _entry = deepcopy(entry_)\n _entry[\"id\"] = f\"{database_provider_}/{entry_['id']}\"\n return _entry\n\n return entry_.model_copy(\n update={\"id\": f\"{database_provider_}/{entry_.id}\"},\n deep=True,\n ).model_dump(exclude_unset=True, exclude_none=True)\n\n if not self.attributes.response:\n # The query has not yet been initiated\n return ErrorResponse(\n errors=[\n {\n \"detail\": (\n \"Can not return as a valid OPTIMADE response as the query \"\n \"has not yet been initialized.\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_QUERY_NOT_INITIALIZED\",\n }\n ],\n meta=meta_values(\n url=url or f\"/queries/{self.id}?\",\n data_returned=0,\n data_available=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n meta_ = self.attributes.response.meta\n\n if url:\n meta_ = meta_.model_dump(exclude_unset=True)\n for repeated_key in (\n \"query\",\n \"api_version\",\n \"time_stamp\",\n \"provider\",\n \"implementation\",\n ):\n meta_.pop(repeated_key, None)\n meta_ = meta_values(url=url, **meta_)\n\n # Error response\n if self.attributes.response.errors:\n return ErrorResponse(\n errors=self.attributes.response.errors,\n meta=meta_,\n )\n\n # Data response\n results = []\n for database_provider, entries in self.attributes.response.data.items():\n results.extend(\n [await _update_id(entry, database_provider) for entry in entries]\n )\n\n return self.attributes.endpoint.get_response_model()(\n data=results,\n meta=meta_,\n links=self.attributes.response.links,\n )\nattributes: QueryResourceAttributes instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource.type","title":"type: Annotated[Literal['queries'], Field(description='The name of the type of an entry.')] = 'queries' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource.response_as_optimade","title":"response_as_optimade(url=None) async","text":"Return attributes.response as a valid OPTIMADE entry listing response.
Note, this method disregards the state of the query and will simply return the query results as they currently are (if there are any at all).
Parameters:
Name Type Description Defaulturl None | ParseResult | SplitResult | URL | str Optionally, update the meta.query.representation value with this.
None Returns:
Type DescriptionEntryResponseMany | ErrorResponse A valid OPTIMADE entry-listing response according to the
EntryResponseMany | ErrorResponse OPTIMADE specification
EntryResponseMany | ErrorResponse or an error response, if errors were returned or occurred during the query.
Source code inoptimade_gateway/models/queries.py async def response_as_optimade(\n self,\n url: None | (\n urllib.parse.ParseResult | urllib.parse.SplitResult | StarletteURL | str\n ) = None,\n) -> EntryResponseMany | ErrorResponse:\n \"\"\"Return `attributes.response` as a valid OPTIMADE entry listing response.\n\n Note, this method disregards the state of the query and will simply return the\n query results as they currently are (if there are any at all).\n\n Parameters:\n url: Optionally, update the `meta.query.representation` value with this.\n\n Returns:\n A valid OPTIMADE entry-listing response according to the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints)\n or an error response, if errors were returned or occurred during the query.\n\n \"\"\"\n from optimade.server.routers.utils import (\n meta_values,\n )\n\n async def _update_id(\n entry_: EntryResource | dict[str, Any], database_provider_: str\n ) -> EntryResource | dict[str, Any]:\n \"\"\"Internal utility function to prepend the entries' `id` with\n `provider/database/`.\n\n Parameters:\n entry_: The entry as a model or a dictionary.\n database_provider_: `provider/database` string.\n\n Returns:\n The entry with an updated `id` value.\n\n \"\"\"\n if isinstance(entry_, dict):\n _entry = deepcopy(entry_)\n _entry[\"id\"] = f\"{database_provider_}/{entry_['id']}\"\n return _entry\n\n return entry_.model_copy(\n update={\"id\": f\"{database_provider_}/{entry_.id}\"},\n deep=True,\n ).model_dump(exclude_unset=True, exclude_none=True)\n\n if not self.attributes.response:\n # The query has not yet been initiated\n return ErrorResponse(\n errors=[\n {\n \"detail\": (\n \"Can not return as a valid OPTIMADE response as the query \"\n \"has not yet been initialized.\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_QUERY_NOT_INITIALIZED\",\n }\n ],\n meta=meta_values(\n url=url or f\"/queries/{self.id}?\",\n data_returned=0,\n data_available=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n meta_ = self.attributes.response.meta\n\n if url:\n meta_ = meta_.model_dump(exclude_unset=True)\n for repeated_key in (\n \"query\",\n \"api_version\",\n \"time_stamp\",\n \"provider\",\n \"implementation\",\n ):\n meta_.pop(repeated_key, None)\n meta_ = meta_values(url=url, **meta_)\n\n # Error response\n if self.attributes.response.errors:\n return ErrorResponse(\n errors=self.attributes.response.errors,\n meta=meta_,\n )\n\n # Data response\n results = []\n for database_provider, entries in self.attributes.response.data.items():\n results.extend(\n [await _update_id(entry, database_provider) for entry in entries]\n )\n\n return self.attributes.endpoint.get_response_model()(\n data=results,\n meta=meta_,\n links=self.attributes.response.links,\n )\nQueryResourceAttributes","text":" Bases: EntryResourceAttributes
Attributes for an OPTIMADE gateway query.
Source code inoptimade_gateway/models/queries.py class QueryResourceAttributes(EntryResourceAttributes):\n \"\"\"Attributes for an OPTIMADE gateway query.\"\"\"\n\n gateway_id: Annotated[\n str,\n Field(\n description=\"The OPTIMADE gateway ID for this query.\",\n ),\n ]\n\n query_parameters: Annotated[\n OptimadeQueryParameters,\n Field(\n description=(\n \"OPTIMADE query parameters for entry listing endpoints used for this \"\n \"query.\"\n ),\n json_schema_extra={\n \"type\": \"object\",\n },\n ),\n ]\n\n state: Annotated[\n QueryState,\n Field(\n description=\"Current state of Gateway Query.\",\n title=\"State\",\n json_schema_extra={\n \"type\": \"enum\",\n },\n ),\n ] = QueryState.CREATED\n\n response: Annotated[\n GatewayQueryResponse | None,\n Field(\n description=\"Response from gateway query.\",\n ),\n ] = None\n\n endpoint: Annotated[\n EndpointEntryType,\n Field(\n description=\"The entry endpoint queried, e.g., 'structures'.\",\n title=\"Endpoint\",\n json_schema_extra={\n \"type\": \"enum\",\n },\n ),\n ] = EndpointEntryType.STRUCTURES\n\n @field_validator(\"endpoint\", mode=\"after\")\n @classmethod\n def only_allow_structures(cls, value: EndpointEntryType) -> EndpointEntryType:\n \"\"\"Temporarily only allow queries to \"structures\" endpoints.\"\"\"\n if value != EndpointEntryType.STRUCTURES:\n raise NotImplementedError(\n 'OPTIMADE Gateway temporarily only supports queries to \"structures\" '\n 'endpoints, i.e.: endpoint=\"structures\"'\n )\n return value\nendpoint: Annotated[EndpointEntryType, Field(description=\"The entry endpoint queried, e.g., 'structures'.\", title='Endpoint', json_schema_extra={'type': 'enum'})] = EndpointEntryType.STRUCTURES class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.gateway_id","title":"gateway_id: Annotated[str, Field(description='The OPTIMADE gateway ID for this query.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.query_parameters","title":"query_parameters: Annotated[OptimadeQueryParameters, Field(description='OPTIMADE query parameters for entry listing endpoints used for this query.', json_schema_extra={'type': 'object'})] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.response","title":"response: Annotated[GatewayQueryResponse | None, Field(description='Response from gateway query.')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.state","title":"state: Annotated[QueryState, Field(description='Current state of Gateway Query.', title='State', json_schema_extra={'type': 'enum'})] = QueryState.CREATED class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.only_allow_structures","title":"only_allow_structures(value) classmethod","text":"Temporarily only allow queries to \"structures\" endpoints.
Source code inoptimade_gateway/models/queries.py @field_validator(\"endpoint\", mode=\"after\")\n@classmethod\ndef only_allow_structures(cls, value: EndpointEntryType) -> EndpointEntryType:\n \"\"\"Temporarily only allow queries to \"structures\" endpoints.\"\"\"\n if value != EndpointEntryType.STRUCTURES:\n raise NotImplementedError(\n 'OPTIMADE Gateway temporarily only supports queries to \"structures\" '\n 'endpoints, i.e.: endpoint=\"structures\"'\n )\n return value\nQueryState","text":" Bases: Enum
Enumeration of possible states for a Gateway Query.
The states are enumerated here in the expected evolvement.
Source code inoptimade_gateway/models/queries.py class QueryState(Enum):\n \"\"\"Enumeration of possible states for a Gateway Query.\n\n The states are enumerated here in the expected evolvement.\n \"\"\"\n\n CREATED = \"created\"\n STARTED = \"started\"\n IN_PROGRESS = \"in progress\"\n FINISHED = \"finished\"\nCREATED = 'created' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.FINISHED","title":"FINISHED = 'finished' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.IN_PROGRESS","title":"IN_PROGRESS = 'in progress' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.STARTED","title":"STARTED = 'started' class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/","title":"resources","text":"Pydantic models/schemas for entry-endpoint resources.
This module is mainly used for a special pydantic base model, which can be used as a mix-in class when creating entry-endpoint resources.
"},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate","title":"EntryResourceCreate","text":" Bases: EntryResourceAttributes
Generic model for creating new entry resources in the MongoDB
Source code inoptimade_gateway/models/resources.py class EntryResourceCreate(EntryResourceAttributes):\n \"\"\"Generic model for creating new entry resources in the MongoDB\"\"\"\n\n model_config = ConfigDict(extra=\"ignore\")\n\n last_modified: datetime | None = None\n\n id: str | None = None\n\n @model_validator(mode=\"after\")\n def check_illegal_attributes_fields(self) -> EntryResourceCreate:\n \"\"\"Overwrite parental `check_illegal_attributes_fields` class validators.\"\"\"\n return self\nid: str | None = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.last_modified","title":"last_modified: datetime | None = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.model_config","title":"model_config = ConfigDict(extra='ignore') class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.check_illegal_attributes_fields","title":"check_illegal_attributes_fields()","text":"Overwrite parental check_illegal_attributes_fields class validators.
optimade_gateway/models/resources.py @model_validator(mode=\"after\")\ndef check_illegal_attributes_fields(self) -> EntryResourceCreate:\n \"\"\"Overwrite parental `check_illegal_attributes_fields` class validators.\"\"\"\n return self\nPydantic models/schemas for the API responses.
"},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.DatabasesResponse","title":"DatabasesResponse","text":" Bases: EntryResponseMany
Successful response for GET /databases
This model is essentially equal to LinksResponse with the exception of the data field's description.
optimade_gateway/models/responses.py class DatabasesResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /databases`\n\n This model is essentially equal to\n [`LinksResponse`](https://www.optimade.org/optimade-python-tools/api_reference/models/responses/#optimade.models.responses.LinksResponse)\n with the exception of the `data` field's description.\n \"\"\"\n\n data: Annotated[\n list[LinksResource],\n StrictField(\n description=(\n \"List of unique OPTIMADE links resource objects.\\nThese links resource \"\n \"objects represents OPTIMADE databases that can be used for queries in \"\n \"gateways.\"\n ),\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[LinksResource], StrictField(description='List of unique OPTIMADE links resource objects.\\nThese links resource objects represents OPTIMADE databases that can be used for queries in gateways.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.DatabasesResponseSingle","title":"DatabasesResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /databases and GET /databases/{database_id}
optimade_gateway/models/responses.py class DatabasesResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /databases` and `GET /databases/{database_id}`\"\"\"\n\n data: Annotated[\n LinksResource | None,\n Field(\n description=(\n \"A unique OPTIMADE links resource object.\\nThe OPTIMADE links resource \"\n \"object has just been created or found according to the specific query \"\n \"parameter(s) or URL id.\\nIt represents an OPTIMADE database that can \"\n \"be used for queries in gateways.\"\n ),\n ),\n ]\ndata: Annotated[LinksResource | None, Field(description='A unique OPTIMADE links resource object.\\nThe OPTIMADE links resource object has just been created or found according to the specific query parameter(s) or URL id.\\nIt represents an OPTIMADE database that can be used for queries in gateways.')] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.GatewaysResponse","title":"GatewaysResponse","text":" Bases: EntryResponseMany
Successful response for GET /gateways
optimade_gateway/models/responses.py class GatewaysResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /gateways`\"\"\"\n\n data: Annotated[\n list[GatewayResource],\n StrictField(\n description=\"List of unique OPTIMADE gateway resource objects.\",\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[GatewayResource], StrictField(description='List of unique OPTIMADE gateway resource objects.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.GatewaysResponseSingle","title":"GatewaysResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /gateways and GET /gateways/{gateway_id}.
optimade_gateway/models/responses.py class GatewaysResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /gateways` and `GET /gateways/{gateway_id}`.\"\"\"\n\n data: Annotated[\n GatewayResource | None,\n Field(\n description=(\n \"A unique OPTIMADE gateway resource object.\\nThe OPTIMADE gateway \"\n \"resource object has just been created or found according to the \"\n \"specific query parameter(s) or URL id.\"\n ),\n ),\n ]\ndata: Annotated[GatewayResource | None, Field(description='A unique OPTIMADE gateway resource object.\\nThe OPTIMADE gateway resource object has just been created or found according to the specific query parameter(s) or URL id.')] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.QueriesResponse","title":"QueriesResponse","text":" Bases: EntryResponseMany
Successful response for GET /gateways/{gateway_ID}/queries.
optimade_gateway/models/responses.py class QueriesResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /gateways/{gateway_ID}/queries`.\"\"\"\n\n data: Annotated[\n list[QueryResource],\n StrictField(\n description=\"List of unique OPTIMADE gateway query resource objects.\",\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[QueryResource], StrictField(description='List of unique OPTIMADE gateway query resource objects.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.QueriesResponseSingle","title":"QueriesResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /gateways/{gateway_ID}/queries and GET /gateways/{gateway_ID}/queries/{query_id}.
optimade_gateway/models/responses.py class QueriesResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /gateways/{gateway_ID}/queries`\n and `GET /gateways/{gateway_ID}/queries/{query_id}`.\"\"\"\n\n data: Annotated[\n QueryResource | None,\n Field(\n description=(\n \"A unique OPTIMADE gateway query resource object.\\nThe OPTIMADE \"\n \"gateway query resource object has just been created or found \"\n \"according to the specific query parameter(s) or URL id.\"\n ),\n ),\n ]\ndata: Annotated[QueryResource | None, Field(description='A unique OPTIMADE gateway query resource object.\\nThe OPTIMADE gateway query resource object has just been created or found according to the specific query parameter(s) or URL id.')] instance-attribute","text":""},{"location":"api_reference/models/search/","title":"search","text":"Pydantic models/schemas for the Search resource.
"},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search","title":"Search","text":" Bases: BaseModel
A general coordinated OPTIMADE search
Important
Either database_ids or optimade_urls MUST be specified.
optimade_gateway/models/search.py class Search(BaseModel):\n \"\"\"A general coordinated OPTIMADE search\n\n !!! important\n Either `database_ids` or `optimade_urls` MUST be specified.\n\n \"\"\"\n\n query_parameters: Annotated[\n OptimadeQueryParameters,\n Field(\n description=(\n \"OPTIMADE query parameters for entry listing endpoints used for this \"\n \"query.\"\n ),\n ),\n ] = OptimadeQueryParameters()\n\n database_ids: Annotated[\n set[str],\n Field(\n description=(\n \"A list of registered database IDs. Go to `/databases` to get all \"\n \"registered databases.\"\n ),\n ),\n ] = set()\n\n optimade_urls: Annotated[\n set[AnyUrl],\n Field(\n description=(\n \"A list of OPTIMADE base URLs. If a versioned base URL is supplied it \"\n \"will be used as is, as long as it represents a supported version. If \"\n \"an un-versioned base URL, standard version negotiation will be \"\n \"conducted to get the versioned base URL, which will be used as long \"\n \"as it represents a supported version. Note, a single URL can be \"\n \"supplied as well, and it will automatically be wrapped in a list in \"\n \"the server logic.\"\n ),\n ),\n ] = set()\n\n endpoint: Annotated[\n str,\n Field(\n description=(\n \"The entry endpoint queried. According to the OPTIMADE specification, \"\n \"this is the same as the resource's type.\"\n ),\n ),\n ] = \"structures\"\n\n @model_validator(mode=\"after\")\n def either_ids_or_urls(self) -> Search:\n \"\"\"Either `database_ids` or `optimade_urls` must be defined\"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"optimade_urls\")):\n raise ValueError(\n \"Either 'database_ids' or 'optimade_urls' MUST be specified.\"\n )\n return self\n\n @field_validator(\"query_parameters\", mode=\"after\")\n @classmethod\n def sort_not_supported(\n cls, value: OptimadeQueryParameters\n ) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\ndatabase_ids: Annotated[set[str], Field(description='A list of registered database IDs. Go to `/databases` to get all registered databases.')] = set() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.endpoint","title":"endpoint: Annotated[str, Field(description=\"The entry endpoint queried. According to the OPTIMADE specification, this is the same as the resource's type.\")] = 'structures' class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.optimade_urls","title":"optimade_urls: Annotated[set[AnyUrl], Field(description='A list of OPTIMADE base URLs. If a versioned base URL is supplied it will be used as is, as long as it represents a supported version. If an un-versioned base URL, standard version negotiation will be conducted to get the versioned base URL, which will be used as long as it represents a supported version. Note, a single URL can be supplied as well, and it will automatically be wrapped in a list in the server logic.')] = set() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.query_parameters","title":"query_parameters: Annotated[OptimadeQueryParameters, Field(description='OPTIMADE query parameters for entry listing endpoints used for this query.')] = OptimadeQueryParameters() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.either_ids_or_urls","title":"either_ids_or_urls()","text":"Either database_ids or optimade_urls must be defined
optimade_gateway/models/search.py @model_validator(mode=\"after\")\ndef either_ids_or_urls(self) -> Search:\n \"\"\"Either `database_ids` or `optimade_urls` must be defined\"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"optimade_urls\")):\n raise ValueError(\n \"Either 'database_ids' or 'optimade_urls' MUST be specified.\"\n )\n return self\nsort_not_supported(value) classmethod","text":"Warn and reset value if sort is supplied.
optimade_gateway/models/search.py @field_validator(\"query_parameters\", mode=\"after\")\n@classmethod\ndef sort_not_supported(\n cls, value: OptimadeQueryParameters\n) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nMongoDB collection for entry-endpoint resources.
The AsyncMongoCollection represents an asynchronous version of the equivalent MongoDB collection in optimade: MongoCollection.
AsyncMongoCollection","text":" Bases: EntryCollection
MongoDB Collection for use with asyncio
The asynchronicity is implemented using motor and asyncio.
optimade_gateway/mongo/collection.py class AsyncMongoCollection(EntryCollection):\n \"\"\"MongoDB Collection for use with `asyncio`\n\n The asynchronicity is implemented using [`motor`](https://motor.readthedocs.io) and\n [`asyncio`](https://asyncio.readthedocs.io/).\n \"\"\"\n\n def __init__(\n self,\n name: str,\n resource_cls: EntryResource,\n resource_mapper: BaseResourceMapper,\n ):\n \"\"\"Initialize the AsyncMongoCollection for the given parameters.\n\n Parameters:\n name: The name of the collection.\n resource_cls: The `EntryResource` model that is stored by the collection.\n resource_mapper: A resource mapper object that handles aliases and format\n changes between deserialization and response.\n\n \"\"\"\n from optimade_gateway.mongo.database import (\n MONGO_DB,\n )\n\n super().__init__(\n resource_cls=resource_cls,\n resource_mapper=resource_mapper,\n transformer=MongoTransformer(mapper=resource_mapper),\n )\n\n self.parser = LarkParser(version=(1, 0, 0), variant=\"default\")\n self.collection: MongoCollection = MONGO_DB[name]\n\n # Check aliases do not clash with mongo operators\n self._check_aliases(self.resource_mapper.all_aliases())\n self._check_aliases(self.resource_mapper.all_length_aliases())\n\n def __str__(self) -> str:\n \"\"\"Standard printing result for an instance.\"\"\"\n return (\n f\"<{self.__class__.__name__}: resource={self.resource_cls.__name__} \"\n f\"endpoint(mapper)={self.resource_mapper.ENDPOINT} \"\n f\"DB_collection={self.collection.name}>\"\n )\n\n def __repr__(self) -> str:\n \"\"\"Representation of instance.\"\"\"\n return (\n f\"{self.__class__.__name__}(name={self.collection.name!r}, \"\n f\"resource_cls={self.resource_cls!r}, \"\n f\"resource_mapper={self.resource_mapper!r})\"\n )\n\n def __len__(self) -> int:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Cannot calculate length of collection using `len()`. Use \"\n \"`count()` instead.\"\n )\n )\n )\n return 0\n\n def insert(self, data: list[EntryResource]) -> None:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `ainsert(data: \"\n \"List[EntryResource])`.\"\n )\n\n async def ainsert(self, data: list[EntryResource]) -> None:\n \"\"\"Add the given entries to the underlying database.\n\n This is the asynchronous version of the parent class method named `insert()`.\n\n Arguments:\n data: The entry resource objects to add to the database.\n\n \"\"\"\n await self.collection.insert_many(await clean_python_types(data))\n\n def count(self, **kwargs) -> int:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `acount(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"**kwargs)`.\"\n )\n\n async def acount(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n **kwargs: Any,\n ) -> int:\n \"\"\"Count documents in Collection.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n **kwargs: Query parameters as keyword arguments. Valid keys will be passed\n to the\n [`AsyncIOMotorCollection.count_documents`](https://motor.readthedocs.io/en/stable/api-asyncio/asyncio_motor_collection.html#motor.motor_asyncio.AsyncIOMotorCollection.count_documents)\n method.\n\n Returns:\n int: The number of entries matching the query specified by the keyword\n arguments.\n\n \"\"\"\n if params is not None and kwargs:\n raise ValueError(\n \"When 'params' is supplied, no other parameters can be supplied.\"\n )\n\n if params is not None:\n kwargs = await self.ahandle_query_params(params)\n\n valid_method_keys = (\n \"filter\",\n \"skip\",\n \"limit\",\n \"hint\",\n \"maxTimeMS\",\n \"collation\",\n \"session\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n criteria[\"filter\"] = {}\n\n return await self.collection.count_documents(**criteria)\n\n def find(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n ]:\n \"\"\"\n Fetches results and indicates if more data is available.\n\n Also gives the total number of data available in the absence of `page_limit`.\n See\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n for more information.\n\n Parameters:\n params: Entry listing URL query params.\n\n Returns:\n A tuple of various relevant values:\n (`results`, `data_returned`, `more_data_available`, `exclude_fields`,\n `include_fields`).\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `afind(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"criteria: Optional[Dict[str, Any]])`.\"\n )\n\n async def afind(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n criteria: dict[str, Any] | None = None,\n ) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n ]:\n \"\"\"Perform the query on the underlying MongoDB Collection, handling projection\n and pagination of the output.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Either provide `params` or `criteria`. Not both, but at least one.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of entry resource objects, how much data was returned for the query,\n whether more data is available with pagination, and fields (excluded,\n included).\n\n \"\"\"\n if (params is None and criteria is None) or (\n params is not None and criteria is not None\n ):\n raise ValueError(\n \"Exacly one of either `params` and `criteria` must be specified.\"\n )\n\n # Set single_entry to False, this is done since if criteria is defined,\n # this is an unknown factor - better to then get a list of results.\n single_entry = False\n if criteria is None:\n criteria = await self.ahandle_query_params(params)\n else:\n single_entry = isinstance(params, SingleEntryQueryParams)\n\n response_fields = criteria.pop(\"fields\", self.all_fields)\n\n results, data_returned, more_data_available = await self._arun_db_query(\n criteria=criteria,\n single_entry=single_entry,\n )\n\n if single_entry:\n results = results[0] if results else None # type: ignore[assignment]\n\n if data_returned > 1:\n raise NotFound(\n detail=(\n f\"Instead of a single entry, {data_returned} entries were found\"\n ),\n )\n\n include_fields = (\n response_fields - self.resource_mapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS\n )\n bad_optimade_fields = set()\n bad_provider_fields = set()\n for field in include_fields:\n if field not in self.resource_mapper.ALL_ATTRIBUTES:\n if field.startswith(\"_\"):\n if any(\n field.startswith(f\"_{prefix}_\")\n for prefix in self.resource_mapper.SUPPORTED_PREFIXES\n ):\n bad_provider_fields.add(field)\n else:\n bad_optimade_fields.add(field)\n\n if bad_provider_fields:\n warn(\n UnknownProviderProperty(\n detail=(\n \"Unrecognised field(s) for this provider requested in \"\n f\"`response_fields`: {bad_provider_fields}.\"\n )\n )\n )\n\n if bad_optimade_fields:\n raise BadRequest(\n detail=(\n \"Unrecognised OPTIMADE field(s) in requested `response_fields`: \"\n f\"{bad_optimade_fields}.\"\n )\n )\n\n if results:\n results = await self.resource_mapper.adeserialize(results)\n\n return ( # type: ignore[return-value]\n results,\n data_returned,\n more_data_available,\n self.all_fields - response_fields,\n include_fields,\n )\n\n def handle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields\n or response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`ahandle_query_params(params: Union[EntryListingQueryParams, \"\n \"SingleEntryQueryParams])`.\"\n )\n\n async def ahandle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n This is the asynchronous version of the parent class method named\n `handle_query_params()`.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields or\n response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n return super().handle_query_params(params)\n\n def _run_db_query(\n self, criteria: dict[str, Any], single_entry: bool = False\n ) -> tuple[list[dict[str, Any]], int, bool]:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`_arun_db_query(criteria: Dict[str, Any], single_entry: bool)`.\"\n )\n\n async def _arun_db_query(\n self, criteria: dict[str, Any], single_entry: bool = False\n ) -> tuple[list[dict[str, Any]], int, bool]:\n \"\"\"Run the query on the backend and collect the results.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Arguments:\n criteria: A dictionary representation of the query parameters.\n single_entry: Whether or not the caller is expecting a single entry\n response.\n\n Returns:\n The list of entries from the database (without any re-mapping), the total\n number of entries matching the query and a boolean for whether or not there\n is more data available.\n\n \"\"\"\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n if criteria.get(\"projection\", {}).get(\"_id\"):\n document[\"_id\"] = str(document[\"_id\"])\n results.append(document)\n\n if single_entry:\n data_returned = len(results)\n more_data_available = False\n else:\n criteria_nolimit = criteria.copy()\n criteria_nolimit.pop(\"limit\", None)\n data_returned = await self.acount(params=None, **criteria_nolimit)\n more_data_available = len(results) < data_returned\n\n return results, data_returned, more_data_available\n\n @staticmethod\n def _check_aliases(aliases: tuple[tuple[str, str]]) -> None:\n \"\"\"Check that aliases do not clash with mongo keywords.\n\n Parameters:\n aliases: Tuple of tuple of aliases to be checked.\n\n Raises:\n RuntimeError: If any alias starts with the dollar (`$`) character.\n\n \"\"\"\n if any(\n alias[0].startswith(\"$\") or alias[1].startswith(\"$\") for alias in aliases\n ):\n raise RuntimeError(f\"Cannot define an alias starting with a '$': {aliases}\")\n\n async def get_one(self, **criteria: Any) -> EntryResource:\n \"\"\"Get one resource based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a single resource.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A single resource from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one(**self._valid_find_keys(**criteria))\n )\n )\n\n async def get_multiple(self, **criteria: Any) -> list[EntryResource]:\n \"\"\"Get a list of resources based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a list of resources.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of resources from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n results.append(self.resource_cls(**self.resource_mapper.map_back(document)))\n\n return results\n\n async def create_one(self, resource: EntryResourceCreate) -> EntryResource:\n \"\"\"Create a new document in the MongoDB collection based on query parameters.\n\n Update the newly created document with an `\"id\"` field.\n The value will be the string representation of the `\"_id\"` field.\n This will only be done if `\"id\"` is not already present in `resource`.\n\n Parameters:\n resource: The resource to be created.\n\n Returns:\n The newly created document as a pydantic model entry resource.\n\n \"\"\"\n resource.last_modified = datetime.now(timezone.utc)\n result = await self.collection.insert_one(\n await clean_python_types(resource.model_dump(exclude_unset=True))\n )\n LOGGER.debug(\n \"Inserted resource %r in DB collection %s with ID %s\",\n resource,\n self.collection.name,\n result.inserted_id,\n )\n\n if not resource.id:\n LOGGER.debug(\"Updating resource with an `id` field equal to str(id_).\")\n await self.collection.update_one(\n {\"_id\": result.inserted_id}, {\"$set\": {\"id\": str(result.inserted_id)}}\n )\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one({\"_id\": result.inserted_id})\n )\n )\n\n async def exists(self, entry_id: str) -> bool:\n \"\"\"Assert whether entry_id exists in the collection (value of `\"id\"`)\n\n Parameters:\n entry_id: The `\"id\"` value of the entry.\n\n \"\"\"\n return bool(await self.collection.count_documents({\"id\": entry_id}))\n\n @staticmethod\n def _valid_find_keys(**kwargs: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Return valid MongoDB find() keys with values from kwargs\n\n Note, not including deprecated flags\n (see https://pymongo.readthedocs.io/en/3.11.0/api/pymongo/collection.html#pymongo.collection.Collection.find).\n \"\"\"\n valid_method_keys = (\n \"filter\",\n \"projection\",\n \"session\",\n \"skip\",\n \"limit\",\n \"no_cursor_timeout\",\n \"cursor_type\",\n \"sort\",\n \"allow_partial_results\",\n \"batch_size\",\n \"collation\",\n \"return_key\",\n \"show_record_id\",\n \"hint\",\n \"max_time_ms\",\n \"min\",\n \"max\",\n \"comment\",\n \"allow_disk_use\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n # Ensure documents are included in the result set\n criteria[\"filter\"] = {}\n\n return criteria\n__init__(name, resource_cls, resource_mapper)","text":"Initialize the AsyncMongoCollection for the given parameters.
Parameters:
Name Type Description Defaultname str The name of the collection.
requiredresource_cls EntryResource The EntryResource model that is stored by the collection.
resource_mapper BaseResourceMapper A resource mapper object that handles aliases and format changes between deserialization and response.
required Source code inoptimade_gateway/mongo/collection.py def __init__(\n self,\n name: str,\n resource_cls: EntryResource,\n resource_mapper: BaseResourceMapper,\n):\n \"\"\"Initialize the AsyncMongoCollection for the given parameters.\n\n Parameters:\n name: The name of the collection.\n resource_cls: The `EntryResource` model that is stored by the collection.\n resource_mapper: A resource mapper object that handles aliases and format\n changes between deserialization and response.\n\n \"\"\"\n from optimade_gateway.mongo.database import (\n MONGO_DB,\n )\n\n super().__init__(\n resource_cls=resource_cls,\n resource_mapper=resource_mapper,\n transformer=MongoTransformer(mapper=resource_mapper),\n )\n\n self.parser = LarkParser(version=(1, 0, 0), variant=\"default\")\n self.collection: MongoCollection = MONGO_DB[name]\n\n # Check aliases do not clash with mongo operators\n self._check_aliases(self.resource_mapper.all_aliases())\n self._check_aliases(self.resource_mapper.all_length_aliases())\nacount(params=None, **kwargs) async","text":"Count documents in Collection.
This is the asynchronous version of the parent class method named count().
Parameters:
Name Type Description Defaultparams None | EntryListingQueryParams | SingleEntryQueryParams URL query parameters, either from a general entry endpoint or a single-entry endpoint.
None **kwargs Any Query parameters as keyword arguments. Valid keys will be passed to the AsyncIOMotorCollection.count_documents method.
{} Returns:
Name Type Descriptionint int The number of entries matching the query specified by the keyword arguments.
Source code inoptimade_gateway/mongo/collection.py async def acount(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n **kwargs: Any,\n) -> int:\n \"\"\"Count documents in Collection.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n **kwargs: Query parameters as keyword arguments. Valid keys will be passed\n to the\n [`AsyncIOMotorCollection.count_documents`](https://motor.readthedocs.io/en/stable/api-asyncio/asyncio_motor_collection.html#motor.motor_asyncio.AsyncIOMotorCollection.count_documents)\n method.\n\n Returns:\n int: The number of entries matching the query specified by the keyword\n arguments.\n\n \"\"\"\n if params is not None and kwargs:\n raise ValueError(\n \"When 'params' is supplied, no other parameters can be supplied.\"\n )\n\n if params is not None:\n kwargs = await self.ahandle_query_params(params)\n\n valid_method_keys = (\n \"filter\",\n \"skip\",\n \"limit\",\n \"hint\",\n \"maxTimeMS\",\n \"collation\",\n \"session\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n criteria[\"filter\"] = {}\n\n return await self.collection.count_documents(**criteria)\nafind(params=None, criteria=None) async","text":"Perform the query on the underlying MongoDB Collection, handling projection and pagination of the output.
This is the asynchronous version of the parent class method named count().
Either provide params or criteria. Not both, but at least one.
Parameters:
Name Type Description Defaultparams None | EntryListingQueryParams | SingleEntryQueryParams URL query parameters, either from a general entry endpoint or a single-entry endpoint.
None criteria dict[str, Any] | None Already handled/parsed URL query parameters.
None Returns:
Type Descriptionlist[EntryResource] | EntryResource | None A list of entry resource objects, how much data was returned for the query,
int whether more data is available with pagination, and fields (excluded,
bool included).
Source code inoptimade_gateway/mongo/collection.py async def afind(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n criteria: dict[str, Any] | None = None,\n) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n]:\n \"\"\"Perform the query on the underlying MongoDB Collection, handling projection\n and pagination of the output.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Either provide `params` or `criteria`. Not both, but at least one.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of entry resource objects, how much data was returned for the query,\n whether more data is available with pagination, and fields (excluded,\n included).\n\n \"\"\"\n if (params is None and criteria is None) or (\n params is not None and criteria is not None\n ):\n raise ValueError(\n \"Exacly one of either `params` and `criteria` must be specified.\"\n )\n\n # Set single_entry to False, this is done since if criteria is defined,\n # this is an unknown factor - better to then get a list of results.\n single_entry = False\n if criteria is None:\n criteria = await self.ahandle_query_params(params)\n else:\n single_entry = isinstance(params, SingleEntryQueryParams)\n\n response_fields = criteria.pop(\"fields\", self.all_fields)\n\n results, data_returned, more_data_available = await self._arun_db_query(\n criteria=criteria,\n single_entry=single_entry,\n )\n\n if single_entry:\n results = results[0] if results else None # type: ignore[assignment]\n\n if data_returned > 1:\n raise NotFound(\n detail=(\n f\"Instead of a single entry, {data_returned} entries were found\"\n ),\n )\n\n include_fields = (\n response_fields - self.resource_mapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS\n )\n bad_optimade_fields = set()\n bad_provider_fields = set()\n for field in include_fields:\n if field not in self.resource_mapper.ALL_ATTRIBUTES:\n if field.startswith(\"_\"):\n if any(\n field.startswith(f\"_{prefix}_\")\n for prefix in self.resource_mapper.SUPPORTED_PREFIXES\n ):\n bad_provider_fields.add(field)\n else:\n bad_optimade_fields.add(field)\n\n if bad_provider_fields:\n warn(\n UnknownProviderProperty(\n detail=(\n \"Unrecognised field(s) for this provider requested in \"\n f\"`response_fields`: {bad_provider_fields}.\"\n )\n )\n )\n\n if bad_optimade_fields:\n raise BadRequest(\n detail=(\n \"Unrecognised OPTIMADE field(s) in requested `response_fields`: \"\n f\"{bad_optimade_fields}.\"\n )\n )\n\n if results:\n results = await self.resource_mapper.adeserialize(results)\n\n return ( # type: ignore[return-value]\n results,\n data_returned,\n more_data_available,\n self.all_fields - response_fields,\n include_fields,\n )\nahandle_query_params(params) async","text":"Parse and interpret the backend-agnostic query parameter models into a dictionary that can be used by the specific backend.
This is the asynchronous version of the parent class method named handle_query_params().
Currently this method returns the pymongo interpretation of the parameters, which will need modification for modified for other backends.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams The initialized query parameter model from the server.
requiredRaises:
Type DescriptionForbidden If too large of a page limit is provided.
BadRequest If an invalid request is made, e.g., with incorrect fields or response format.
Returns:
Type Descriptiondict[str, Any] A dictionary representation of the query parameters.
Source code inoptimade_gateway/mongo/collection.py async def ahandle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n This is the asynchronous version of the parent class method named\n `handle_query_params()`.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields or\n response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n return super().handle_query_params(params)\nainsert(data) async","text":"Add the given entries to the underlying database.
This is the asynchronous version of the parent class method named insert().
Parameters:
Name Type Description Defaultdata list[EntryResource] The entry resource objects to add to the database.
required Source code inoptimade_gateway/mongo/collection.py async def ainsert(self, data: list[EntryResource]) -> None:\n \"\"\"Add the given entries to the underlying database.\n\n This is the asynchronous version of the parent class method named `insert()`.\n\n Arguments:\n data: The entry resource objects to add to the database.\n\n \"\"\"\n await self.collection.insert_many(await clean_python_types(data))\ncreate_one(resource) async","text":"Create a new document in the MongoDB collection based on query parameters.
Update the newly created document with an \"id\" field. The value will be the string representation of the \"_id\" field. This will only be done if \"id\" is not already present in resource.
Parameters:
Name Type Description Defaultresource EntryResourceCreate The resource to be created.
requiredReturns:
Type DescriptionEntryResource The newly created document as a pydantic model entry resource.
Source code inoptimade_gateway/mongo/collection.py async def create_one(self, resource: EntryResourceCreate) -> EntryResource:\n \"\"\"Create a new document in the MongoDB collection based on query parameters.\n\n Update the newly created document with an `\"id\"` field.\n The value will be the string representation of the `\"_id\"` field.\n This will only be done if `\"id\"` is not already present in `resource`.\n\n Parameters:\n resource: The resource to be created.\n\n Returns:\n The newly created document as a pydantic model entry resource.\n\n \"\"\"\n resource.last_modified = datetime.now(timezone.utc)\n result = await self.collection.insert_one(\n await clean_python_types(resource.model_dump(exclude_unset=True))\n )\n LOGGER.debug(\n \"Inserted resource %r in DB collection %s with ID %s\",\n resource,\n self.collection.name,\n result.inserted_id,\n )\n\n if not resource.id:\n LOGGER.debug(\"Updating resource with an `id` field equal to str(id_).\")\n await self.collection.update_one(\n {\"_id\": result.inserted_id}, {\"$set\": {\"id\": str(result.inserted_id)}}\n )\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one({\"_id\": result.inserted_id})\n )\n )\nexists(entry_id) async","text":"Assert whether entry_id exists in the collection (value of \"id\")
Parameters:
Name Type Description Defaultentry_id str The \"id\" value of the entry.
optimade_gateway/mongo/collection.py async def exists(self, entry_id: str) -> bool:\n \"\"\"Assert whether entry_id exists in the collection (value of `\"id\"`)\n\n Parameters:\n entry_id: The `\"id\"` value of the entry.\n\n \"\"\"\n return bool(await self.collection.count_documents({\"id\": entry_id}))\nfind(params)","text":"Fetches results and indicates if more data is available.
Also gives the total number of data available in the absence of page_limit. See EntryListingQueryParams for more information.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams Entry listing URL query params.
requiredReturns:
Type Descriptionlist[EntryResource] | EntryResource | None A tuple of various relevant values:
int (results, data_returned, more_data_available, exclude_fields,
bool include_fields).
optimade_gateway/mongo/collection.py def find(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n]:\n \"\"\"\n Fetches results and indicates if more data is available.\n\n Also gives the total number of data available in the absence of `page_limit`.\n See\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n for more information.\n\n Parameters:\n params: Entry listing URL query params.\n\n Returns:\n A tuple of various relevant values:\n (`results`, `data_returned`, `more_data_available`, `exclude_fields`,\n `include_fields`).\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `afind(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"criteria: Optional[Dict[str, Any]])`.\"\n )\nget_multiple(**criteria) async","text":"Get a list of resources based on criteria
WarningThis is not to be used for creating a REST API response, but is rather a utility function to easily retrieve a list of resources.
Parameters:
Name Type Description Default**criteria Any Already handled/parsed URL query parameters.
{} Returns:
Type Descriptionlist[EntryResource] A list of resources from the MongoDB (mapped to pydantic models).
Source code inoptimade_gateway/mongo/collection.py async def get_multiple(self, **criteria: Any) -> list[EntryResource]:\n \"\"\"Get a list of resources based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a list of resources.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of resources from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n results.append(self.resource_cls(**self.resource_mapper.map_back(document)))\n\n return results\nget_one(**criteria) async","text":"Get one resource based on criteria
WarningThis is not to be used for creating a REST API response, but is rather a utility function to easily retrieve a single resource.
Parameters:
Name Type Description Default**criteria Any Already handled/parsed URL query parameters.
{} Returns:
Type DescriptionEntryResource A single resource from the MongoDB (mapped to pydantic models).
Source code inoptimade_gateway/mongo/collection.py async def get_one(self, **criteria: Any) -> EntryResource:\n \"\"\"Get one resource based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a single resource.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A single resource from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one(**self._valid_find_keys(**criteria))\n )\n )\nhandle_query_params(params)","text":"Parse and interpret the backend-agnostic query parameter models into a dictionary that can be used by the specific backend.
NoteCurrently this method returns the pymongo interpretation of the parameters, which will need modification for modified for other backends.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams The initialized query parameter model from the server.
requiredRaises:
Type DescriptionForbidden If too large of a page limit is provided.
BadRequest If an invalid request is made, e.g., with incorrect fields or response format.
Returns:
Type Descriptiondict[str, Any] A dictionary representation of the query parameters.
Source code inoptimade_gateway/mongo/collection.py def handle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields\n or response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`ahandle_query_params(params: Union[EntryListingQueryParams, \"\n \"SingleEntryQueryParams])`.\"\n )\nInitialize the MongoDB database.
"},{"location":"api_reference/mongo/database/#optimade_gateway.mongo.database.MONGO_CLIENT","title":"MONGO_CLIENT = AsyncIOMotorClient(CONFIG.mongo_uri, **mongo_client_configuration) module-attribute","text":"The MongoDB motor client.
"},{"location":"api_reference/mongo/database/#optimade_gateway.mongo.database.MONGO_DB","title":"MONGO_DB = MONGO_CLIENT[CONFIG.mongo_database] module-attribute","text":"The MongoDB motor database. This is a representation of the database used for the gateway service.
"},{"location":"api_reference/queries/params/","title":"params","text":"URL query parameters.
"},{"location":"api_reference/queries/params/#optimade_gateway.queries.params.SearchQueryParams","title":"SearchQueryParams","text":"URL query parameters for GET /search
This is an extension of the EntryListingQueryParams class in optimade, which defines the standard entry listing endpoint query parameters.
The extra query parameters are as follows.
Attributes:
Name Type Descriptiondatabase_ids set[str] List of possible database IDs that are already known by the gateway. To be known they need to be registered with the gateway (currently not possible).
optimade_urls set[AnyUrl] A list of OPTIMADE base URLs. If a versioned base URL is supplied it will be used as is, as long as it represents a supported version. If an un-versioned base URL, standard version negotiation will be conducted to get the versioned base URL, which will be used as long as it represents a supported version.
Example: http://example.org/optimade/v1/search?optimade_urls=\"https://example.org/optimade_db/v1\",\"https://optimade.herokuapp.com\"
endpoint str The entry endpoint queried. According to the OPTIMADE specification, this is the same as the resource's type.
Example: structures
timeout int Timeout time (in seconds) to wait for a query to finish before redirecting (after starting the query). Note, if the query has not finished after the timeout time, a redirection will still be performed, but to a zero-results page, which can be refreshed to get the finished query (once it has finished).
as_optimade bool Return the response as a standard OPTIMADE entry listing endpoint response. Otherwise, the response will be based on the QueriesResponseSingle model.
optimade_gateway/queries/params.py class SearchQueryParams:\n \"\"\"URL query parameters for `GET /search`\n\n This is an extension of the\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n class in `optimade`, which defines the standard entry listing endpoint query\n parameters.\n\n The extra query parameters are as follows.\n\n Attributes:\n database_ids (set[str]): List of possible database IDs that are already known by\n the gateway. To be known they need to be registered with the gateway\n (currently not possible).\n\n optimade_urls (set[AnyUrl]): A list of OPTIMADE base URLs. If a versioned base\n URL is supplied it will be used as is, as long as it represents a supported\n version. If an un-versioned base URL, standard version negotiation will be\n conducted to get the versioned base URL, which will be used as long as it\n represents a supported version.\n\n **Example**: `http://example.org/optimade/v1/search?optimade_urls=\"https://example.org/optimade_db/v1\",\"https://optimade.herokuapp.com\"`\n\n endpoint (str): The entry endpoint queried. According to the OPTIMADE\n specification, this is the same as the resource's type.\n\n **Example**: `structures`\n\n timeout (int): Timeout time (in seconds) to wait for a query to finish before\n redirecting (*after* starting the query). Note, if the query has not\n finished after the timeout time, a redirection will still be performed, but\n to a zero-results page, which can be refreshed to get the finished query\n (once it has finished).\n\n as_optimade (bool): Return the response as a standard OPTIMADE entry listing\n endpoint response. Otherwise, the response will be based on the\n [`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\n model.\n\n \"\"\"\n\n def __init__(\n self,\n *,\n database_ids: Annotated[\n set[str],\n Query(\n description=(\n \"Unique list of possible database IDs that are already known by \"\n \"the gateway. To be known they need to be registered with the \"\n \"gateway (currently not possible).\"\n ),\n ),\n ] = set(),\n optimade_urls: Annotated[\n set[AnyUrl],\n Query(\n description=(\n \"A unique list of OPTIMADE base URLs. If a versioned base URL is \"\n \"supplied it will be used as is, as long as it represents a \"\n \"supported version. If an un-versioned base URL, standard version \"\n \"negotiation will be conducted to get the versioned base URL, \"\n \"which will be used as long as it represents a supported version.\"\n ),\n ),\n ] = set(),\n endpoint: Annotated[\n str,\n Query(\n description=(\n \"The entry endpoint queried. According to the OPTIMADE \"\n \"specification, this is the same as the resource's type.\"\n ),\n ),\n ] = \"structures\",\n timeout: Annotated[\n int,\n Query(\n description=(\n \"Timeout time (in seconds) to wait for a query to finish before \"\n \"redirecting (*after* starting the query). Note, if the query has \"\n \"not finished after the timeout time, a redirection will still be \"\n \"performed, but to a zero-results page, which can be refreshed to \"\n \"get the finished query (once it has finished).\"\n ),\n ),\n ] = 15,\n as_optimade: Annotated[\n bool,\n Query(\n description=(\n \"Return the response as a standard OPTIMADE entry listing endpoint \"\n \"response. Otherwise, the response will be based on the \"\n \"[`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\"\n \" model.\"\n ),\n ),\n ] = False,\n ) -> None:\n self.database_ids = database_ids\n self.optimade_urls = optimade_urls\n self.endpoint = endpoint\n self.timeout = timeout\n self.as_optimade = as_optimade\nPerform OPTIMADE queries
"},{"location":"api_reference/queries/perform/#optimade_gateway.queries.perform.db_find","title":"db_find(database, endpoint, response_model, query_params='', raw_url=None)","text":"Imitate Collection.find() for any given database for entry-resource endpoints
Parameters:
Name Type Description Defaultdatabase LinksResource | dict[str, Any] The OPTIMADE implementation to be queried. It must have a valid base URL and id.
requiredendpoint str The entry-listing endpoint, e.g., \"structures\".
response_model EntryResponseMany | EntryResponseOne The expected OPTIMADE pydantic response model, e.g., optimade.models.StructureResponseMany.
query_params str URL query parameters to pass to the database.
'' raw_url AnyUrl | str | None A raw URL to use straight up instead of deriving a URL from database, endpoint, and query_params.
None Returns:
Type Descriptiontuple[ErrorResponse | EntryResponseMany | EntryResponseOne, str] Response as an optimade pydantic model and the database's ID.
optimade_gateway/queries/perform.py def db_find(\n database: LinksResource | dict[str, Any],\n endpoint: str,\n response_model: EntryResponseMany | EntryResponseOne,\n query_params: str = \"\",\n raw_url: AnyUrl | str | None = None,\n) -> tuple[ErrorResponse | EntryResponseMany | EntryResponseOne, str]:\n \"\"\"Imitate `Collection.find()` for any given database for entry-resource endpoints\n\n Parameters:\n database: The OPTIMADE implementation to be queried.\n It **must** have a valid base URL and id.\n endpoint: The entry-listing endpoint, e.g., `\"structures\"`.\n response_model: The expected OPTIMADE pydantic response model, e.g.,\n `optimade.models.StructureResponseMany`.\n query_params: URL query parameters to pass to the database.\n raw_url: A raw URL to use straight up instead of deriving a URL from `database`,\n `endpoint`, and `query_params`.\n\n Returns:\n Response as an `optimade` pydantic model and the `database`'s ID.\n\n \"\"\"\n if TYPE_CHECKING or bool(os.getenv(\"MKDOCS_BUILD\", \"\")): # pragma: no cover\n response: (\n httpx.Response\n | dict[str, Any]\n | EntryResponseMany\n | EntryResponseOne\n | ErrorResponse\n )\n\n if raw_url:\n url = str(raw_url)\n else:\n url = \"\"\n\n base_url = str(get_resource_attribute(database, \"attributes.base_url\")).rstrip(\n \"/\"\n )\n url += base_url\n\n # Check whether base_url is a versioned base URL\n if not any(base_url.endswith(_) for _ in BASE_URL_PREFIXES.values()):\n # Unversioned base URL - add the currently supported major version\n url += BASE_URL_PREFIXES[\"major\"]\n\n url += f\"/{endpoint.strip('/')}?{query_params}\"\n\n response = httpx.get(url, timeout=60)\n\n try:\n response = response.json()\n except json.JSONDecodeError:\n return (\n ErrorResponse(\n errors=[\n {\n \"detail\": f\"Could not JSONify response from {url}\",\n \"id\": \"OPTIMADE_GATEWAY_DB_FIND_MANY_JSONDECODEERROR\",\n }\n ],\n meta={\n \"query\": {\n \"representation\": f\"/{endpoint.strip('/')}?{query_params}\"\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n try:\n response = response_model(**response)\n except ValidationError:\n try:\n response = ErrorResponse(**response)\n except ValidationError as exc:\n # If it's an error and `meta` is missing, it is not a valid OPTIMADE\n # response, but this happens a lot, and is therefore worth having an\n # edge-case for.\n if \"errors\" in response:\n errors = list(response[\"errors\"])\n errors.append(\n {\n \"detail\": (\n f\"Could not pass response from {url} as either a \"\n f\"{response_model.__name__!r} or 'ErrorResponse'. \"\n f\"ValidationError: {exc}\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_DB_FINDS_MANY_VALIDATIONERRORS\",\n }\n )\n return (\n ErrorResponse(\n errors=errors,\n meta={\n \"query\": {\n \"representation\": (\n f\"/{endpoint.strip('/')}?{query_params}\"\n )\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n return (\n ErrorResponse(\n errors=[\n {\n \"detail\": (\n f\"Could not pass response from {url} as either a \"\n f\"{response_model.__name__!r} or 'ErrorResponse'. \"\n f\"ValidationError: {exc}\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_DB_FINDS_MANY_VALIDATIONERRORS\",\n }\n ],\n meta={\n \"query\": {\n \"representation\": f\"/{endpoint.strip('/')}?{query_params}\"\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n return response, get_resource_attribute(database, \"id\")\ndb_get_all_resources(database, endpoint, response_model, query_params='', raw_url=None) async","text":"Recursively retrieve all resources from an entry-listing endpoint
This function keeps pulling the links.next link if meta.more_data_available is True to ultimately retrieve all entries for endpoint.
Warning
This function can be dangerous if an endpoint with hundreds or thousands of entries is requested.
Parameters:
Name Type Description Defaultdatabase LinksResource | dict[str, Any] The OPTIMADE implementation to be queried. It must have a valid base URL and id.
requiredendpoint str The entry-listing endpoint, e.g., \"structures\".
response_model EntryResponseMany The expected OPTIMADE pydantic response model, e.g., optimade.models.StructureResponseMany.
query_params str URL query parameters to pass to the database.
'' raw_url AnyUrl | str | None A raw URL to use straight up instead of deriving a URL from database, endpoint, and query_params.
None Returns:
Type Descriptiontuple[list[EntryResource | dict[str, Any]], LinksResource | dict[str, Any]] A collected list of successful responses' data value and the database's ID.
optimade_gateway/queries/perform.py async def db_get_all_resources(\n database: LinksResource | dict[str, Any],\n endpoint: str,\n response_model: EntryResponseMany,\n query_params: str = \"\",\n raw_url: AnyUrl | str | None = None,\n) -> tuple[list[EntryResource | dict[str, Any]], LinksResource | dict[str, Any]]:\n \"\"\"Recursively retrieve all resources from an entry-listing endpoint\n\n This function keeps pulling the `links.next` link if `meta.more_data_available` is\n `True` to ultimately retrieve *all* entries for `endpoint`.\n\n !!! warning\n This function can be dangerous if an endpoint with hundreds or thousands of\n entries is requested.\n\n Parameters:\n database: The OPTIMADE implementation to be queried.\n It **must** have a valid base URL and id.\n endpoint: The entry-listing endpoint, e.g., `\"structures\"`.\n response_model: The expected OPTIMADE pydantic response model, e.g.,\n `optimade.models.StructureResponseMany`.\n query_params: URL query parameters to pass to the database.\n raw_url: A raw URL to use straight up instead of deriving a URL from `database`,\n `endpoint`, and `query_params`.\n\n Returns:\n A collected list of successful responses' `data` value and the `database`'s ID.\n\n \"\"\"\n resulting_resources = []\n\n response, _ = db_find(\n database=database,\n endpoint=endpoint,\n response_model=response_model,\n query_params=query_params,\n raw_url=raw_url,\n )\n\n if isinstance(response, ErrorResponse):\n # An errored response will result in no databases from a provider.\n LOGGER.error(\n \"Error while querying database (id=%r). Full response: %s\",\n get_resource_attribute(database, \"id\"),\n response.model_dump_json(indent=2),\n )\n return [], database\n\n resulting_resources.extend(response.data)\n\n if response.meta.more_data_available:\n next_page = get_resource_attribute(response, \"links.next\")\n if next_page is None:\n LOGGER.error(\n \"Could not find a 'next' link for an OPTIMADE query request to %r \"\n \"(id=%r). Cannot get all resources from /%s, even though this was \"\n \"asked and `more_data_available` is `True` in the response.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n endpoint,\n )\n return resulting_resources, database\n\n more_resources, _ = await db_get_all_resources(\n database=database,\n endpoint=endpoint,\n response_model=response_model,\n query_params=query_params,\n raw_url=next_page,\n )\n resulting_resources.extend(more_resources)\n\n return resulting_resources, database\nperform_query(url, query) async","text":"Perform OPTIMADE query with gateway.
Parameters:
Name Type Description Defaulturl URL Original request URL.
requiredquery QueryResource The query to be performed.
requiredReturns:
Type DescriptionEntryResponseMany | ErrorResponse | GatewayQueryResponse This function returns the final response; a
EntryResponseMany | ErrorResponse | GatewayQueryResponse GatewayQueryResponse.
optimade_gateway/queries/perform.py async def perform_query(\n url: URL,\n query: QueryResource,\n) -> EntryResponseMany | ErrorResponse | GatewayQueryResponse:\n \"\"\"Perform OPTIMADE query with gateway.\n\n Parameters:\n url: Original request URL.\n query: The query to be performed.\n\n Returns:\n This function returns the final response; a\n [`GatewayQueryResponse`][optimade_gateway.models.queries.GatewayQueryResponse].\n\n \"\"\"\n await update_query(query, \"state\", QueryState.STARTED)\n\n gateway: GatewayResource = await get_valid_resource(\n await collection_factory(CONFIG.gateways_collection),\n query.attributes.gateway_id,\n )\n\n filter_queries = await prepare_query_filter(\n database_ids=[_.id for _ in gateway.attributes.databases],\n filter_query=query.attributes.query_parameters.filter,\n )\n\n url = url.replace(path=f\"{url.path.rstrip('/')}/{query.id}\")\n await update_query(\n query,\n \"response\",\n GatewayQueryResponse(\n data={},\n links=ToplevelLinks(next=None),\n meta=meta_values(\n url=url,\n data_available=0,\n data_returned=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n ),\n operator=None,\n **{\"$set\": {\"state\": QueryState.IN_PROGRESS}},\n )\n\n loop = asyncio.get_running_loop()\n with ThreadPoolExecutor(\n max_workers=min(\n 32, (os.cpu_count() or 0) + 4, len(gateway.attributes.databases)\n )\n ) as executor:\n # Run OPTIMADE DB queries in a thread pool, i.e., not using the main OS thread,\n # where the asyncio event loop is running.\n query_tasks = []\n for database in gateway.attributes.databases:\n query_params = await get_query_params(\n query_parameters=query.attributes.query_parameters,\n database_id=database.id,\n filter_mapping=filter_queries,\n )\n query_tasks.append(\n loop.run_in_executor(\n executor=executor,\n func=functools.partial(\n db_find,\n database=database,\n endpoint=query.attributes.endpoint.value,\n response_model=query.attributes.endpoint.get_response_model(),\n query_params=query_params,\n ),\n )\n )\n\n for query_task in query_tasks:\n (db_response, db_id) = await query_task\n\n await process_db_response(\n response=db_response,\n database_id=db_id,\n query=query,\n gateway=gateway,\n )\n\n # Pagination\n #\n # if isinstance(results, list) and get_resource_attribute(\n # query,\n # \"attributes.response.meta.more_data_available\",\n # False,\n # disambiguate=False, # Extremely minor speed-up\n # ):\n # # Deduce the `next` link from the current request\n # query_string = urllib.parse.parse_qs(url.query)\n # query_string[\"page_offset\"] = [\n # int(query_string.get(\"page_offset\", [0])[0]) # type: ignore[list-item]\n # + len(results[: query.attributes.query_parameters.page_limit])\n # ]\n # urlencoded = urllib.parse.urlencode(query_string, doseq=True)\n # base_url = get_base_url(url)\n\n # links = ToplevelLinks(next=f\"{base_url}{url.path}?{urlencoded}\")\n\n # await update_query(query, \"response.links\", links)\n\n await update_query(query, \"state\", QueryState.FINISHED)\n return query.attributes.response\nPrepare OPTIMADE queries.
"},{"location":"api_reference/queries/prepare/#optimade_gateway.queries.prepare.get_query_params","title":"get_query_params(query_parameters, database_id, filter_mapping) async","text":"Construct the parsed URL query parameters
Source code inoptimade_gateway/queries/prepare.py async def get_query_params(\n query_parameters: OptimadeQueryParameters,\n database_id: str,\n filter_mapping: Mapping[str, str | None],\n) -> str:\n \"\"\"Construct the parsed URL query parameters\"\"\"\n query_params = {\n param: value for param, value in query_parameters.model_dump().items() if value\n }\n if filter_mapping[database_id]:\n query_params.update({\"filter\": filter_mapping[database_id]})\n return urllib.parse.urlencode(query_params)\nprepare_query_filter(database_ids, filter_query) async","text":"Update the query parameter filter value to be database-specific
This is needed due to the served change of id values. If someone searches for a gateway-changed id, it needs to be reverted to be database-specific.
Parameters:
Name Type Description Defaultdatabase_ids list[str] List of the databases to create updated filter values for. These values are part of the gateway-changed id values and are essential.
filter_query str | None The submitted filter query parameter value. Can be None if not supplied.
Returns:
Type DescriptionMapping[str, str | None] A mapping for database IDs to database-specific filter query parameter values.
optimade_gateway/queries/prepare.py async def prepare_query_filter(\n database_ids: list[str], filter_query: str | None\n) -> Mapping[str, str | None]:\n \"\"\"Update the query parameter `filter` value to be database-specific\n\n This is needed due to the served change of `id` values.\n If someone searches for a gateway-changed `id`, it needs to be reverted to be\n database-specific.\n\n Parameters:\n database_ids: List of the databases to create updated filter values for.\n These values are part of the gateway-changed `id` values and are essential.\n filter_query: The submitted `filter` query parameter value. Can be `None` if not\n supplied.\n\n Returns:\n A mapping for database IDs to database-specific `filter` query parameter values.\n\n \"\"\"\n updated_filter = {}.fromkeys(database_ids, filter_query)\n\n if not filter_query:\n return updated_filter\n\n for id_match in re.finditer(\n r'\"(?P<id_value_l>[^\\s]*)\"[\\s]*'\n r\"(<|>|<=|>=|=|!=|CONTAINS|STARTS WITH|ENDS WITH|STARTS|ENDS)\"\n r\"[\\s]*id|[^_]+id[\\s]*\"\n r'(<|>|<=|>=|=|!=|CONTAINS|STARTS WITH|ENDS WITH|STARTS|ENDS)[\\s]*\"'\n r'(?P<id_value_r>[^\\s]*)\"',\n f\"={filter_query}\" if filter_query else \"\",\n ):\n matched_id: str = id_match.group(\"id_value_l\") or id_match.group(\"id_value_r\")\n for database_id in database_ids:\n if matched_id.startswith(f\"{database_id}/\"):\n updated_filter_query = updated_filter[database_id]\n if not updated_filter_query or not isinstance(\n updated_filter_query, str\n ):\n raise TypeError(\n \"Expected a string for filter query, got \"\n f\"{type(updated_filter_query)}\"\n )\n\n # Database found\n updated_filter[database_id] = updated_filter_query.replace(\n f\"{database_id}/\", \"\", 1\n )\n break\n else:\n warn(\n OptimadeGatewayWarning(\n title=\"Non-Unique Entry ID\",\n detail=(\n f\"The passed entry ID <id={matched_id}> may be ambiguous! To \"\n \"get a specific structures entry, one can prepend the ID with \"\n \"a database ID belonging to the gateway, followed by a forward\"\n f\" slash, e.g., '{database_ids[0]}/<local_database_ID>'. \"\n f\"Available databases for this gateway: {database_ids}\"\n ),\n )\n )\n return updated_filter\nProcess performed OPTIMADE queries.
"},{"location":"api_reference/queries/process/#optimade_gateway.queries.process.process_db_response","title":"process_db_response(response, database_id, query, gateway) async","text":"Process an OPTIMADE database response.
The passed query will be updated with the top-level meta information: data_available, data_returned, and more_data_available.
Since, only either data or errors should ever be present, one or the other will be either an empty list or None.
Parameters:
Name Type Description Defaultresponse ErrorResponse | EntryResponseMany | EntryResponseOne The OPTIMADE database response to be processed.
requireddatabase_id str The database's id under which the returned resources or errors will be delivered.
query QueryResource A resource representing the performed query.
requiredgateway GatewayResource A resource representing the gateway that was queried.
requiredReturns:
Type Descriptionlist[EntryResource] | list[dict[str, Any]] | EntryResource | dict[str, Any] | None The response's data.
optimade_gateway/queries/process.py async def process_db_response(\n response: ErrorResponse | EntryResponseMany | EntryResponseOne,\n database_id: str,\n query: QueryResource,\n gateway: GatewayResource,\n) -> list[EntryResource] | list[dict[str, Any]] | EntryResource | dict[str, Any] | None:\n \"\"\"Process an OPTIMADE database response.\n\n The passed `query` will be updated with the top-level `meta` information:\n `data_available`, `data_returned`, and `more_data_available`.\n\n Since, only either `data` or `errors` should ever be present, one or the other will\n be either an empty list or `None`.\n\n Parameters:\n response: The OPTIMADE database response to be processed.\n database_id: The database's `id` under which the returned resources or errors\n will be delivered.\n query: A resource representing the performed query.\n gateway: A resource representing the gateway that was queried.\n\n Returns:\n The response's `data`.\n\n \"\"\"\n results = []\n errors = []\n\n LOGGER.debug(\"Starting to process database_id: %s\", database_id)\n\n if isinstance(response, ErrorResponse):\n for error in response.errors:\n if isinstance(error.id, str) and error.id.startswith(\"OPTIMADE_GATEWAY\"):\n warn(error.detail, OptimadeGatewayWarning)\n else:\n # The model `ErrorResponse` does not allow the objects in the top-level\n # `errors` list to be parsed as dictionaries - they must be a pydantic\n # model.\n meta_error = {}\n if error.meta:\n meta_error = error.meta.model_dump()\n meta_error.update(\n {\n f\"_{CONFIG.provider.prefix}_source_gateway\": {\n \"id\": gateway.id,\n \"type\": gateway.type,\n \"links\": {\"self\": gateway.links.self},\n },\n f\"_{CONFIG.provider.prefix}_source_database\": {\n \"id\": database_id,\n \"type\": \"links\",\n \"links\": {\n \"self\": (\n str(gateway.links.self).split(\n \"gateways\", maxsplit=1\n )[0]\n + f\"databases/{database_id}\"\n )\n },\n },\n }\n )\n error.meta = Meta(**meta_error)\n errors.append(error)\n data_returned = 0\n more_data_available = False\n else:\n results = response.data\n\n if isinstance(results, list):\n data_returned = response.meta.data_returned or len(results)\n else:\n data_returned = response.meta.data_returned or (0 if not results else 1)\n\n more_data_available = response.meta.more_data_available or False\n\n data_available = response.meta.data_available or 0\n\n extra_updates = {\n \"$inc\": {\n \"response.meta.data_available\": data_available,\n \"response.meta.data_returned\": data_returned,\n }\n }\n if not get_resource_attribute(\n query,\n \"attributes.response.meta.more_data_available\",\n False,\n disambiguate=False, # Extremely minor speed-up\n ):\n # Keep it True, if set to True once.\n extra_updates.update(\n {\"$set\": {\"response.meta.more_data_available\": more_data_available}}\n )\n\n # This ensures an empty list under `response.data.{database_id}` is returned if the\n # case is simply that there are no results to return.\n if errors:\n extra_updates.update({\"$addToSet\": {\"response.errors\": {\"$each\": errors}}})\n await update_query(\n query,\n f\"response.data.{database_id}\",\n results,\n operator=None,\n **extra_updates,\n )\n\n return results\nUtility functions for the queries module.
update_query(query, field, value, operator=None, **mongo_kwargs) async","text":"Update a query's field attribute with value.
If field is a dot-separated value, then only the last field part may be a non-pre-existing field. Otherwise a KeyError or AttributeError will be raised.
Note
This can only update a field for a query's attributes, i.e., this function cannot update id, type or any other top-level resource field.
Important
mongo_kwargs will not be considered for updating the pydantic model instance.
Parameters:
Name Type Description Defaultquery QueryResource The query to be updated.
requiredfield str The attributes field (key) to be set. This can be a dot-separated key value to signify embedded fields.
Example: response.meta.
value Any The (possibly) new value for field.
operator str | None A MongoDB operator to be used for updating field with value.
None **mongo_kwargs Any Further MongoDB update filters.
{} Source code in optimade_gateway/queries/utils.py async def update_query(\n query: QueryResource,\n field: str,\n value: Any,\n operator: str | None = None,\n **mongo_kwargs: Any,\n) -> None:\n \"\"\"Update a query's `field` attribute with `value`.\n\n If `field` is a dot-separated value, then only the last field part may be a\n non-pre-existing field. Otherwise a `KeyError` or `AttributeError` will be raised.\n\n !!! note\n This can *only* update a field for a query's `attributes`, i.e., this function\n cannot update `id`, `type` or any other top-level resource field.\n\n !!! important\n `mongo_kwargs` will not be considered for updating the pydantic model instance.\n\n Parameters:\n query: The query to be updated.\n field: The `attributes` field (key) to be set.\n This can be a dot-separated key value to signify embedded fields.\n\n **Example**: `response.meta`.\n value: The (possibly) new value for `field`.\n operator: A MongoDB operator to be used for updating `field` with `value`.\n **mongo_kwargs: Further MongoDB update filters.\n\n \"\"\"\n operator = operator or \"$set\"\n\n if operator and not operator.startswith(\"$\"):\n operator = f\"${operator}\"\n\n update_time = datetime.now(timezone.utc)\n\n update_kwargs = {\"$set\": {\"last_modified\": update_time}}\n\n if mongo_kwargs:\n update_kwargs.update(mongo_kwargs)\n\n if operator and operator == \"$set\":\n update_kwargs[\"$set\"].update({field: value})\n elif operator:\n if operator in update_kwargs:\n update_kwargs[operator].update({field: value})\n else:\n update_kwargs.update({operator: {field: value}})\n\n # MongoDB\n collection = await collection_factory(CONFIG.queries_collection)\n result: UpdateResult = await collection.collection.update_one(\n filter={\"id\": {\"$eq\": query.id}},\n update=await clean_python_types(update_kwargs),\n )\n if result.matched_count != 1:\n LOGGER.error(\n (\n \"matched_count should have been exactly 1, it was: %s. \"\n \"Returned update_one result: %s\"\n ),\n result.matched_count,\n result.raw_result,\n )\n\n # Pydantic model instance\n query.attributes.last_modified = update_time\n if \".\" in field:\n field_list = field.split(\".\")\n sub_field: BaseModel | dict[str, Any] = getattr(query.attributes, field_list[0])\n for field_part in field_list[1:-1]:\n if isinstance(sub_field, dict):\n sub_field = sub_field.get(field_part, {})\n else:\n sub_field = getattr(sub_field, field_part)\n if isinstance(sub_field, dict):\n sub_field[field_list[-1]] = value\n else:\n setattr(sub_field, field_list[-1], value)\n else:\n setattr(query.attributes, field, value)\n/databases/*
This file describes the router for:
/databases/{id}\nwhere, id may be left out.
Database resources represent the available databases that may be used for the gateways.
One can register a new database (by using POST /databases) or look through the available databases (by using GET /databases) using standard OPTIMADE filtering.
get_database(request, database_id, params) async","text":"GET /databases/{database ID}
Return a single LinksResource representing the database resource object with id={database ID}.
optimade_gateway/routers/databases.py @ROUTER.get(\n \"/databases/{database_id:path}\",\n response_model=DatabasesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_database(\n request: Request,\n database_id: str,\n params: Annotated[SingleEntryQueryParams, Depends()],\n) -> DatabasesResponseSingle:\n \"\"\"`GET /databases/{database ID}`\n\n Return a single\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n representing the database resource object with `id={database ID}`.\n \"\"\"\n collection = await collection_factory(CONFIG.databases_collection)\n\n params.filter = f'id=\"{database_id}\"'\n (\n result,\n data_returned,\n more_data_available,\n fields,\n include_fields,\n ) = await collection.afind(params=params)\n\n if fields or include_fields and result is not None:\n result = handle_response_fields(result, fields, include_fields)\n\n result = result[0] if isinstance(result, list) and data_returned else None\n\n return DatabasesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=data_returned,\n data_available=await collection.acount(),\n more_data_available=more_data_available,\n schema=CONFIG.schema_url,\n ),\n )\nget_databases(request, params) async","text":"GET /databases
Return overview of all (active) databases.
Source code inoptimade_gateway/routers/databases.py @ROUTER.get(\n \"/databases\",\n response_model=DatabasesResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_databases(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> DatabasesResponse:\n \"\"\"`GET /databases`\n\n Return overview of all (active) databases.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.databases_collection),\n response_cls=DatabasesResponse,\n request=request,\n params=params,\n )\npost_databases(request, database) async","text":"POST /databases
Create/Register or return an existing LinksResource, representing a database resource object, according to database.
optimade_gateway/routers/databases.py @ROUTER.post(\n \"/databases\",\n response_model=DatabasesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def post_databases(\n request: Request, database: DatabaseCreate\n) -> DatabasesResponseSingle:\n \"\"\"`POST /databases`\n\n Create/Register or return an existing\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource),\n representing a database resource object, according to `database`.\n \"\"\"\n result, created = await resource_factory(database)\n collection = await collection_factory(CONFIG.databases_collection)\n\n return DatabasesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\n/gateways/*
This file describes the router for:
/gateways/{id}\nwhere, id may be left out.
get_gateway(request, gateway_id) async","text":"GET /gateways/{gateway ID}
Return a single GatewayResource.
optimade_gateway/routers/gateways.py @ROUTER.get(\n \"/gateways/{gateway_id}\",\n response_model=GatewaysResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_gateway(request: Request, gateway_id: str) -> GatewaysResponseSingle:\n \"\"\"`GET /gateways/{gateway ID}`\n\n Return a single\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource].\n \"\"\"\n collection = await collection_factory(CONFIG.gateways_collection)\n result = await get_valid_resource(collection, gateway_id)\n\n return GatewaysResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\nget_gateways(request, params) async","text":"GET /gateways
Return overview of all (active) gateways.
Source code inoptimade_gateway/routers/gateways.py @ROUTER.get(\n \"/gateways\",\n response_model=GatewaysResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_gateways(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> GatewaysResponse:\n \"\"\"`GET /gateways`\n\n Return overview of all (active) gateways.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.gateways_collection),\n response_cls=GatewaysResponse,\n request=request,\n params=params,\n )\npost_gateways(request, gateway) async","text":"POST /gateways
Create or return existing gateway according to gateway.
optimade_gateway/routers/gateways.py @ROUTER.post(\n \"/gateways\",\n response_model=GatewaysResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def post_gateways(\n request: Request, gateway: GatewayCreate\n) -> GatewaysResponseSingle:\n \"\"\"`POST /gateways`\n\n Create or return existing gateway according to `gateway`.\n \"\"\"\n if gateway.database_ids:\n databases_collection = await collection_factory(CONFIG.databases_collection)\n\n databases = await databases_collection.get_multiple(\n filter={\"id\": {\"$in\": await clean_python_types(gateway.database_ids)}}\n )\n\n if not isinstance(gateway.databases, list):\n gateway.databases = []\n\n current_database_ids = [_.id for _ in gateway.databases]\n gateway.databases.extend(\n _ for _ in databases if _.id not in current_database_ids\n )\n\n result, created = await resource_factory(gateway)\n collection = await collection_factory(CONFIG.gateways_collection)\n\n return GatewaysResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\n/info/*
This file describes the router for:
/info/{entry}\nwhere, entry may be left out.
ENTRY_INFO_SCHEMAS: dict[str, type[EntryResource]] = {'databases': LinksResource, 'gateways': GatewayResource, 'queries': QueryResource} module-attribute","text":"This dictionary is used to define the /info/<entry_type> endpoints.
get_entry_info(request, entry) async","text":"GET /info/{entry}
Get information about the gateway service's entry-listing endpoints.
Source code inoptimade_gateway/routers/info.py @ROUTER.get(\n \"/info/{entry}\",\n response_model=EntryInfoResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Info\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_entry_info(request: Request, entry: str) -> EntryInfoResponse:\n \"\"\"`GET /info/{entry}`\n\n Get information about the gateway service's entry-listing endpoints.\n \"\"\"\n valid_entry_info_endpoints = ENTRY_INFO_SCHEMAS.keys()\n if entry not in valid_entry_info_endpoints:\n raise NotFound(\n detail=(\n f\"Entry info not found for {entry}, valid entry info endpoints are: \"\n f\"{', '.join(valid_entry_info_endpoints)}\"\n ),\n )\n\n schema = ENTRY_INFO_SCHEMAS[entry]\n queryable_properties = {\"id\", \"type\", \"attributes\"}\n properties = await aretrieve_queryable_properties(\n schema, queryable_properties, entry_type=entry\n )\n\n output_fields_by_format = {\"json\": list(properties)}\n\n return EntryInfoResponse(\n data=EntryInfoResource(\n formats=list(output_fields_by_format),\n description=getattr(schema, \"__doc__\", \"Entry Resources\"),\n properties=properties,\n output_fields_by_format=output_fields_by_format,\n ),\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=1,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\nget_info(request) async","text":"GET /info
An introspective endpoint for the gateway service.
Source code inoptimade_gateway/routers/info.py @ROUTER.get(\n \"/info\",\n response_model=InfoResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Info\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_info(request: Request) -> InfoResponse:\n \"\"\"`GET /info`\n\n An introspective endpoint for the gateway service.\n \"\"\"\n return InfoResponse(\n data=BaseInfoResource(\n id=BaseInfoResource.model_fields[\"id\"].default,\n type=BaseInfoResource.model_fields[\"type\"].default,\n attributes=BaseInfoAttributes(\n api_version=__api_version__,\n available_api_versions=[\n {\n \"url\": (\n f\"{get_base_url(request.url)}\"\n f\"/v{__api_version__.split('.', maxsplit=1)[0]}\"\n ),\n \"version\": __api_version__,\n }\n ],\n formats=[\"json\"],\n entry_types_by_format={\"json\": list(ENTRY_INFO_SCHEMAS.keys())},\n available_endpoints=sorted(\n [\n \"docs\",\n \"info\",\n \"links\",\n \"openapi.json\",\n \"redoc\",\n \"search\",\n *list(ENTRY_INFO_SCHEMAS.keys()),\n ]\n ),\n is_index=False,\n ),\n ),\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=1,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n/links/*
This file describes the router for:
/links\nget_links(request, params) async","text":"GET /links
Return a regular /links response for an OPTIMADE implementation.
optimade_gateway/routers/links.py @ROUTER.get(\n \"/links\",\n response_model=LinksResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Links\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_links(\n request: Request, params: Annotated[EntryListingQueryParams, Depends()]\n) -> LinksResponse:\n \"\"\"`GET /links`\n\n Return a regular `/links` response for an OPTIMADE implementation.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.links_collection),\n response_cls=LinksResponse,\n request=request,\n params=params,\n )\nGeneral /queries endpoint to handle gateway queries
This file describes the router for:
/queries/{id}\nwhere, id may be left out.
get_queries(request, params) async","text":"GET /queries
Return overview of all (active) queries.
Source code inoptimade_gateway/routers/queries.py @ROUTER.get(\n \"/queries\",\n response_model=QueriesResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_queries(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> QueriesResponse:\n \"\"\"`GET /queries`\n\n Return overview of all (active) queries.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.queries_collection),\n response_cls=QueriesResponse,\n request=request,\n params=params,\n )\nget_query(request, query_id, response) async","text":"GET /queries/{query_id}
Return a single QueryResource.
optimade_gateway/routers/queries.py @ROUTER.get(\n \"/queries/{query_id}\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_query(\n request: Request,\n query_id: str,\n response: Response,\n) -> QueriesResponseSingle:\n \"\"\"`GET /queries/{query_id}`\n\n Return a single [`QueryResource`][optimade_gateway.models.queries.QueryResource].\n \"\"\"\n collection = await collection_factory(CONFIG.queries_collection)\n query: QueryResource = await get_valid_resource(collection, query_id)\n\n if query.attributes.response and query.attributes.response.errors:\n for error in query.attributes.response.errors:\n if error.status:\n for part in error.status.split(\" \"):\n try:\n response.status_code = int(part)\n break\n except ValueError:\n pass\n if response.status_code and response.status_code >= 300:\n break\n else:\n response.status_code = 500\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\npost_queries(request, query) async","text":"POST /queries
Create or return existing gateway query according to query.
optimade_gateway/routers/queries.py @ROUTER.post(\n \"/queries\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n status_code=status.HTTP_202_ACCEPTED,\n responses=ERROR_RESPONSES,\n)\nasync def post_queries(\n request: Request,\n query: QueryCreate,\n) -> QueriesResponseSingle:\n \"\"\"`POST /queries`\n\n Create or return existing gateway query according to `query`.\n \"\"\"\n await validate_resource(\n await collection_factory(CONFIG.gateways_collection), query.gateway_id\n )\n\n result, created = await resource_factory(query)\n\n background_tasks: set[asyncio.Task] = set()\n\n if created:\n task = asyncio.create_task(perform_query(url=request.url, query=result))\n\n # Add task to the set. This creates a strong reference.\n background_tasks.add(task)\n\n # To prevent keeping references to finished tasks forever,\n # make each task remove its own reference from the set after\n # completion:\n task.add_done_callback(background_tasks.discard)\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\nGeneral /search endpoint to completely coordinate an OPTIMADE gateway query
This file describes the router for:
/search\nget_search(request, response, search_params, entry_params) async","text":"GET /search
Coordinate a new OPTIMADE query in multiple databases through a gateway:
Search POST data - calling POST /search.search_params.timeout seconds before returning the query, if it has not finished before.GET /queries/{query_id}.This endpoint works similarly to GET /queries/{query_id}, where one passes the query parameters directly in the URL, instead of first POSTing a query and then going to its URL. Hence, a QueryResponseSingle is the standard response model for this endpoint.
If the timeout time is reached and the query has not yet finished, the user is redirected to the specific URL for the query.
If the as_optimade query parameter is True, the response will be parseable as a standard OPTIMADE entry listing endpoint like, e.g., /structures. For more information see the OPTIMADE specification.
optimade_gateway/routers/search.py @ROUTER.get(\n \"/search\",\n response_model=Union[QueriesResponseSingle, ErrorResponse, EntryResponseMany],\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Search\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_search(\n request: Request,\n response: Response,\n search_params: Annotated[SearchQueryParams, Depends()],\n entry_params: Annotated[EntryListingQueryParams, Depends()],\n) -> QueriesResponseSingle | EntryResponseMany | ErrorResponse | RedirectResponse:\n \"\"\"`GET /search`\n\n Coordinate a new OPTIMADE query in multiple databases through a gateway:\n\n 1. Create a [`Search`][optimade_gateway.models.search.Search] `POST` data - calling\n `POST /search`.\n 1. Wait [`search_params.timeout`][optimade_gateway.queries.params.SearchQueryParams]\n seconds before returning the query, if it has not finished before.\n 1. Return query - similar to `GET /queries/{query_id}`.\n\n This endpoint works similarly to `GET /queries/{query_id}`, where one passes the\n query parameters directly in the URL, instead of first POSTing a query and then\n going to its URL. Hence, a\n [`QueryResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle] is\n the standard response model for this endpoint.\n\n If the timeout time is reached and the query has not yet finished, the user is\n redirected to the specific URL for the query.\n\n If the `as_optimade` query parameter is `True`, the response will be parseable as a\n standard OPTIMADE entry listing endpoint like, e.g., `/structures`.\n For more information see the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints).\n\n \"\"\"\n try:\n search = Search(\n query_parameters=OptimadeQueryParameters(\n **{\n field: getattr(entry_params, field)\n for field in OptimadeQueryParameters.model_fields\n if getattr(entry_params, field)\n }\n ),\n optimade_urls=search_params.optimade_urls,\n endpoint=search_params.endpoint,\n database_ids=search_params.database_ids,\n )\n except ValidationError as exc:\n raise BadRequest(\n detail=(\n \"A Search object could not be created from the given URL query \"\n f\"parameters. Error(s): {exc.errors}\"\n )\n ) from exc\n\n queries_response = await post_search(request, search=search)\n\n if not queries_response.data:\n LOGGER.error(\n \"QueryResource not found in POST /search response:\\n%s\", queries_response\n )\n raise RuntimeError(\n \"Expected the response from POST /search to return a QueryResource, it did \"\n \"not\"\n )\n\n once = True\n start_time = time()\n while time() < (start_time + search_params.timeout) or once:\n # Make sure to run this at least once (e.g., if timeout=0)\n once = False\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n query: QueryResource = await collection.get_one(\n filter={\"id\": queries_response.data.id}\n )\n\n if query.attributes.state == QueryState.FINISHED:\n if query.attributes.response and query.attributes.response.errors:\n for error in query.attributes.response.errors:\n if error.status:\n for part in error.status.split(\" \"):\n try:\n response.status_code = int(part)\n break\n except ValueError:\n pass\n if response.status_code and response.status_code >= 300:\n break\n else:\n response.status_code = 500\n\n if search_params.as_optimade:\n response = await query.response_as_optimade(url=request.url)\n LOGGER.debug(\n \"Returning response as OPTIMADE entry listing:\\n%s\", response\n )\n return response\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n await asyncio.sleep(0.1)\n\n # The query has not yet succeeded and we're past the timeout time -> Redirect to\n # /queries/<id>\n return RedirectResponse(query.links.self)\npost_search(request, search) async","text":"POST /search
Coordinate a new OPTIMADE query in multiple databases through a gateway:
optimade_urls and database_idsGatewayCreate modelPOST gateway resource to get ID - using functionality of POST /gatewaysPOST Query resource - using functionality of POST /queriesPOST /queries response - QueriesResponseSingleoptimade_gateway/routers/search.py @ROUTER.post(\n \"/search\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Search\"],\n status_code=status.HTTP_202_ACCEPTED,\n responses=ERROR_RESPONSES,\n)\nasync def post_search(request: Request, search: Search) -> QueriesResponseSingle:\n \"\"\"`POST /search`\n\n Coordinate a new OPTIMADE query in multiple databases through a gateway:\n\n 1. Search for gateway in DB using `optimade_urls` and `database_ids`\n 1. Create [`GatewayCreate`][optimade_gateway.models.gateways.GatewayCreate] model\n 1. `POST` gateway resource to get ID - using functionality of `POST /gateways`\n 1. Create new [Query][optimade_gateway.models.queries.QueryCreate] resource\n 1. `POST` Query resource - using functionality of `POST /queries`\n 1. Return `POST /queries` response -\n [`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\n\n \"\"\"\n databases_collection = await collection_factory(CONFIG.databases_collection)\n # NOTE: It may be that the final list of base URLs (`base_urls`) contains the same\n # provider(s), but with differring base URLS, if, for example, a versioned base URL\n # is supplied.\n base_urls: set[AnyUrl] = set()\n\n if search.database_ids:\n databases = await databases_collection.get_multiple(\n filter={\"id\": {\"$in\": await clean_python_types(search.database_ids)}}\n )\n base_urls |= {\n get_resource_attribute(database, \"attributes.base_url\")\n for database in databases\n if get_resource_attribute(database, \"attributes.base_url\") is not None\n }\n\n if search.optimade_urls:\n base_urls |= {_ for _ in search.optimade_urls if _ is not None}\n\n if not base_urls:\n msg = \"No (valid) OPTIMADE URLs with:\"\n if search.database_ids:\n msg += (\n f\"\\n Database IDs: {search.database_ids} and corresponding found \"\n \"URLs: \"\n f\"{[get_resource_attribute(database, 'attributes.base_url') for database in databases]}\" # noqa: E501\n )\n if search.optimade_urls:\n msg += f\"\\n Passed OPTIMADE URLs: {search.optimade_urls}\"\n raise BadRequest(detail=msg)\n\n # Ensure all URLs are `pydantic.AnyUrl`s\n if not all(isinstance(_, AnyUrl) for _ in base_urls):\n raise InternalServerError(\n \"Could unexpectedly not validate all base URLs as proper URLs.\"\n )\n\n databases = await databases_collection.get_multiple(\n filter={\"base_url\": {\"$in\": await clean_python_types(base_urls)}}\n )\n\n if len(databases) == len(base_urls):\n # At this point it is expected that the list of databases in `databases`\n # is a complete set of databases requested.\n pass\n\n elif len(databases) < len(base_urls):\n # There are unregistered databases, i.e., databases not in the local collection\n current_base_urls: set[AnyUrl] = {\n get_resource_attribute(database, \"attributes.base_url\")\n for database in databases\n }\n databases.extend(\n [\n LinksResource(\n id=str(url)\n .replace(\".\", \"__\")[len(url.scheme) + 3 :]\n .split(\"?\", maxsplit=1)[0]\n .split(\"#\", maxsplit=1)[0],\n type=\"links\",\n attributes=LinksResourceAttributes(\n name=str(url)[len(url.scheme) + 3 :]\n .split(\"?\", maxsplit=1)[0]\n .split(\"#\", maxsplit=1)[0],\n description=\"\",\n base_url=url,\n link_type=LinkType.CHILD,\n homepage=None,\n ),\n )\n for url in base_urls - current_base_urls\n ]\n )\n else:\n LOGGER.error(\n \"Found more database entries in MongoDB than then number of passed base \"\n \"URLs. This suggests ambiguity in the base URLs of databases stored in \"\n \"MongoDB.\\n base_urls: %s\\n databases %s\",\n base_urls,\n databases,\n )\n raise InternalServerError(\"Unambiguous base URLs. See logs for more details.\")\n\n gateway = GatewayCreate(databases=databases)\n gateway, created = await resource_factory(gateway)\n\n if created:\n LOGGER.debug(\"A new gateway was created for a query (id=%r)\", gateway.id)\n else:\n LOGGER.debug(\"A gateway was found and reused for a query (id=%r)\", gateway.id)\n\n query = QueryCreate(\n endpoint=search.endpoint,\n gateway_id=gateway.id,\n query_parameters=search.query_parameters,\n )\n query, created = await resource_factory(query)\n\n background_tasks: set[asyncio.Task] = set()\n\n if created:\n task = asyncio.create_task(perform_query(url=request.url, query=query))\n\n # Add task to the set. This creates a strong reference.\n background_tasks.add(task)\n\n # To prevent keeping references to finished tasks forever,\n # make each task remove its own reference from the set after\n # completion:\n task.add_done_callback(background_tasks.discard)\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\nUtility functions for all routers.
"},{"location":"api_reference/routers/utils/#optimade_gateway.routers.utils.COLLECTIONS","title":"COLLECTIONS: dict[str, AsyncMongoCollection] = {} module-attribute","text":"A lazy-loaded dictionary of asynchronous MongoDB entry-endpoint collections.
"},{"location":"api_reference/routers/utils/#optimade_gateway.routers.utils.aretrieve_queryable_properties","title":"aretrieve_queryable_properties(schema, queryable_properties, entry_type=None) async","text":"Asynchronous implementation of retrieve_queryable_properties() from optimade
Reference to the function in the optimade API documentation: retrieve_queryable_properties().
Recursively loops through the schema of a pydantic model and resolves all references, returning a dictionary of all the OPTIMADE-queryable properties of that model.
Parameters:
Name Type Description Defaultschema type[EntryResource] The schema of the pydantic model.
requiredqueryable_properties Iterable[str] The list of properties to find in the schema.
requiredentry_type str | None The entry type of the model, if any.
None Returns:
Type DescriptionQueryableProperties A flat dictionary with properties as keys, containing the field description,
QueryableProperties unit, sortability, support level, queryability and type, where provided.
Source code inoptimade_gateway/routers/utils.py async def aretrieve_queryable_properties(\n schema: type[EntryResource],\n queryable_properties: Iterable[str],\n entry_type: str | None = None,\n) -> QueryableProperties:\n \"\"\"Asynchronous implementation of `retrieve_queryable_properties()` from `optimade`\n\n Reference to the function in the `optimade` API documentation:\n [`retrieve_queryable_properties()`](https://www.optimade.org/optimade-python-tools/api_reference/server/schemas/#optimade.server.schemas.retrieve_queryable_properties).\n\n Recursively loops through the schema of a pydantic model and resolves all\n references, returning a dictionary of all the OPTIMADE-queryable properties of that\n model.\n\n Parameters:\n schema: The schema of the pydantic model.\n queryable_properties: The list of properties to find in the schema.\n entry_type: The entry type of the model, if any.\n\n Returns:\n A flat dictionary with properties as keys, containing the field description,\n unit, sortability, support level, queryability and type, where provided.\n\n \"\"\"\n return retrieve_queryable_properties(\n schema=schema,\n queryable_properties=queryable_properties,\n entry_type=entry_type,\n )\ncollection_factory(name) async","text":"Get or initiate an entry-endpoint resource collection.
This factory utilizes the global dictionary COLLECTIONS. It lazily instantiates the collections and then caches them in the dictionary.
Parameters:
Name Type Description Defaultname str The configured name for the entry-endpoint resource collection.
requiredReturns:
Type DescriptionAsyncMongoCollection The OPTIMADE Gateway asynchronous implementation of the
AsyncMongoCollection MongoCollection.
Raises:
Type DescriptionValueError If the supplied name is not one of the configured valid collection names.
optimade_gateway/routers/utils.py async def collection_factory(name: str) -> AsyncMongoCollection:\n \"\"\"Get or initiate an entry-endpoint resource collection.\n\n This factory utilizes the global dictionary\n [`COLLECTIONS`][optimade_gateway.routers.utils.COLLECTIONS].\n It lazily instantiates the collections and then caches them in the dictionary.\n\n Parameters:\n name: The configured name for the entry-endpoint resource collection.\n\n Returns:\n The OPTIMADE Gateway asynchronous implementation of the\n [`MongoCollection`](https://www.optimade.org/optimade-python-tools/api_reference/server/entry_collections/mongo/#optimade.server.entry_collections.mongo.MongoCollection).\n\n Raises:\n ValueError: If the supplied `name` is not one of the configured valid collection\n names.\n\n \"\"\"\n if name in COLLECTIONS:\n return COLLECTIONS[name]\n\n if name == CONFIG.databases_collection:\n from optimade_gateway.mappers.databases import DatabasesMapper as ResourceMapper\n elif name == CONFIG.gateways_collection:\n from optimade_gateway.mappers.gateways import ( # type: ignore[no-redef]\n GatewaysMapper as ResourceMapper,\n )\n elif name == CONFIG.queries_collection:\n from optimade_gateway.mappers.queries import ( # type: ignore[no-redef]\n QueryMapper as ResourceMapper,\n )\n elif name == CONFIG.links_collection:\n from optimade_gateway.mappers.links import ( # type: ignore[no-redef]\n LinksMapper as ResourceMapper,\n )\n else:\n raise ValueError(\n f\"{name!r} is not a valid entry-endpoint resource collection name. \"\n \"Configured valid names: \"\n f\"{(CONFIG.databases_collection, CONFIG.gateways_collection, CONFIG.queries_collection, CONFIG.links_collection)}\" # noqa: E501\n )\n\n COLLECTIONS[name] = AsyncMongoCollection(\n name=name,\n resource_cls=ResourceMapper.ENTRY_RESOURCE_CLASS,\n resource_mapper=ResourceMapper,\n )\n\n return COLLECTIONS[name]\nget_entries(collection, response_cls, request, params) async","text":"Generalized /{entries} endpoint getter
optimade_gateway/routers/utils.py async def get_entries(\n collection: AsyncMongoCollection,\n response_cls: EntryResponseMany,\n request: Request,\n params: EntryListingQueryParams,\n) -> EntryResponseMany:\n \"\"\"Generalized `/{entries}` endpoint getter\"\"\"\n (\n results,\n data_returned,\n more_data_available,\n fields,\n include_fields,\n ) = await collection.afind(params=params)\n\n if more_data_available:\n # Deduce the `next` link from the current request\n query = urllib.parse.parse_qs(request.url.query)\n query[\"page_offset\"] = [int(query.get(\"page_offset\", [0])[0]) + len(results)] # type: ignore[list-item, arg-type]\n urlencoded = urllib.parse.urlencode(query, doseq=True)\n base_url = get_base_url(request.url)\n\n links = ToplevelLinks(next=f\"{base_url}{request.url.path}?{urlencoded}\")\n else:\n links = ToplevelLinks(next=None)\n\n if fields or include_fields:\n results = handle_response_fields(results, fields, include_fields)\n\n return response_cls(\n links=links,\n data=results,\n meta=meta_values(\n url=request.url,\n data_returned=data_returned,\n data_available=await collection.acount(),\n more_data_available=more_data_available,\n schema=CONFIG.schema_url,\n ),\n )\nget_valid_resource(collection, entry_id) async","text":"Validate and retrieve a resource
Source code inoptimade_gateway/routers/utils.py async def get_valid_resource(\n collection: AsyncMongoCollection, entry_id: str\n) -> EntryResource:\n \"\"\"Validate and retrieve a resource\"\"\"\n await validate_resource(collection, entry_id)\n return await collection.get_one(filter={\"id\": entry_id})\nresource_factory(create_resource) async","text":"Get or create a resource
Currently supported resources:
\"databases\" (DatabaseCreate -> LinksResource)\"gateways\" (GatewayCreate -> GatewayResource)\"queries\" (QueryCreate -> QueryResource)For each of the resources, \"uniqueness\" is determined in the following way:
DatabasesThe base_url field is considered unique across all databases.
If a base_url is provided via a Link model, the base_url.href value is used to query the MongoDB.
The collected list of databases.attributes.base_url values is considered unique across all gateways.
In the database, the search is done as a combination of the length/size of the databases' Python list/MongoDB array and a match on all (using the MongoDB $all operator) of the databases.attributes.base_url element values, when compared with the create_resource.
Important
The database_ids attribute must not contain values that are not also included in the databases attribute, in the form of the IDs for the individual databases. If this should be the case an OptimadeGatewayError will be thrown.
The gateway_id, query_parameters, and endpoint fields are collectively considered to define uniqueness for a QueryResource in the MongoDB collection.
Attention
Only the /structures entry endpoint can be queried with multiple expected responses.
This means the endpoint field defaults to \"structures\", i.e., the StructureResource resource model.
Parameters:
Name Type Description Defaultcreate_resource DatabaseCreate | GatewayCreate | QueryCreate The resource to be retrieved or created anew.
requiredReturns:
Type Descriptiontuple[LinksResource | GatewayResource | QueryResource, bool] Two things in a tuple:
GatewayResource; a QueryResource; or a LinksResource andoptimade_gateway/routers/utils.py async def resource_factory(\n create_resource: DatabaseCreate | GatewayCreate | QueryCreate,\n) -> tuple[LinksResource | GatewayResource | QueryResource, bool]:\n \"\"\"Get or create a resource\n\n Currently supported resources:\n\n - `\"databases\"`\n ([`DatabaseCreate`][optimade_gateway.models.databases.DatabaseCreate]\n ->\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource))\n - `\"gateways\"` ([`GatewayCreate`][optimade_gateway.models.gateways.GatewayCreate] ->\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource])\n - `\"queries\"` ([`QueryCreate`][optimade_gateway.models.queries.QueryCreate] ->\n [`QueryResource`][optimade_gateway.models.queries.QueryResource])\n\n For each of the resources, \"uniqueness\" is determined in the following way:\n\n === \"Databases\"\n The `base_url` field is considered unique across all databases.\n\n If a `base_url` is provided via a\n [`Link`](https://www.optimade.org/optimade-python-tools/api_reference/models/jsonapi/#optimade.models.jsonapi.Link)\n model, the `base_url.href` value is used to query the MongoDB.\n\n === \"Gateways\"\n The collected list of `databases.attributes.base_url` values is considered\n unique across all gateways.\n\n In the database, the search is done as a combination of the length/size of the\n `databases`' Python list/MongoDB array and a match on all (using the MongoDB\n `$all` operator) of the\n [`databases.attributes.base_url`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes.base_url)\n element values, when compared with the `create_resource`.\n\n !!! important\n The `database_ids` attribute **must not** contain values that are not also\n included in the `databases` attribute, in the form of the IDs for the\n individual databases. If this should be the case an\n [`OptimadeGatewayError`][optimade_gateway.common.exceptions.OptimadeGatewayError]\n will be thrown.\n\n === \"Queries\"\n The `gateway_id`, `query_parameters`, and `endpoint` fields are collectively\n considered to define uniqueness for a\n [`QueryResource`][optimade_gateway.models.queries.QueryResource] in the MongoDB\n collection.\n\n !!! attention\n Only the `/structures` entry endpoint can be queried with multiple expected\n responses.\n\n This means the `endpoint` field defaults to `\"structures\"`, i.e., the\n [`StructureResource`](https://www.optimade.org/optimade-python-tools/all_models/#optimade.models.structures.StructureResource)\n resource model.\n\n Parameters:\n create_resource: The resource to be retrieved or created anew.\n\n Returns:\n Two things in a tuple:\n\n - Either a\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource];\n a [`QueryResource`][optimade_gateway.models.queries.QueryResource]; or a\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n and\n - whether or not the resource was newly created.\n\n \"\"\"\n created = False\n\n if isinstance(create_resource, DatabaseCreate):\n collection_name = CONFIG.databases_collection\n\n base_url = get_resource_attribute(create_resource, \"base_url\")\n\n mongo_query: dict[str, Any] = {\n \"$or\": [\n {\"base_url\": {\"$eq\": base_url}},\n {\"base_url.href\": {\"$eq\": base_url}},\n ]\n }\n elif isinstance(create_resource, GatewayCreate):\n collection_name = CONFIG.gateways_collection\n\n # One MUST have taken care of database_ids prior to calling `resource_factory()`\n database_attr_ids = {_.id for _ in create_resource.databases or []}\n unknown_ids = {\n database_id\n for database_id in create_resource.database_ids or []\n if database_id not in database_attr_ids\n }\n if unknown_ids:\n raise OptimadeGatewayError(\n \"When using `resource_factory()` for `GatewayCreate`, `database_ids` \"\n f\"MUST not include unknown IDs. Passed unknown IDs: {unknown_ids}\"\n )\n\n mongo_query = {\n \"databases\": {\"$size\": len(create_resource.databases or [])},\n \"databases.attributes.base_url\": {\n \"$all\": [_.attributes.base_url for _ in create_resource.databases or []]\n },\n }\n elif isinstance(create_resource, QueryCreate):\n collection_name = CONFIG.queries_collection\n\n # Currently only /structures entry endpoints can be queried with multiple\n # expected responses.\n create_resource.endpoint = (\n create_resource.endpoint\n if create_resource.endpoint is not None\n else EndpointEntryType(\"structures\")\n )\n\n mongo_query = {\n \"gateway_id\": {\"$eq\": create_resource.gateway_id},\n \"query_parameters\": {\"$eq\": create_resource.query_parameters},\n \"endpoint\": {\"$eq\": create_resource.endpoint},\n }\n else:\n raise TypeError(\n \"create_resource must be either a DatabaseCreate, GatewayCreate, or \"\n f\"QueryCreate object, not {type(create_resource)!r}\"\n )\n\n collection = await collection_factory(collection_name)\n result, data_returned, more_data_available, _, _ = await collection.afind(\n criteria={\"filter\": await clean_python_types(mongo_query)}\n )\n\n if more_data_available:\n raise OptimadeGatewayError(\n \"more_data_available MUST be False for a single entry response, however it \"\n f\"is {more_data_available}\"\n )\n\n if result:\n if data_returned > 1:\n raise OptimadeGatewayError(\n f\"More than one {result[0].type} were found. IDs of found \"\n f\"{result[0].type}: {[_.id for _ in result]}\"\n )\n if isinstance(result, list):\n result = result[0]\n else:\n if isinstance(create_resource, DatabaseCreate):\n # Set required `LinksResourceAttributes` values if not set\n if not create_resource.description:\n create_resource.description = (\n f\"{create_resource.name} created by OPTIMADE gateway database \"\n \"registration.\"\n )\n\n if not create_resource.link_type:\n create_resource.link_type = LinkType.EXTERNAL\n\n if not create_resource.homepage:\n create_resource.homepage = None\n\n elif (\n isinstance(create_resource, GatewayCreate)\n and \"database_ids\" in create_resource.model_fields_set\n ):\n # Do not store `database_ids`\n del create_resource.database_ids\n create_resource.model_fields_set.remove(\"database_ids\")\n\n elif isinstance(create_resource, QueryCreate):\n create_resource.state = QueryState.CREATED\n\n result = await collection.create_one(create_resource)\n LOGGER.debug(\"Created new %s: %r\", result.type, result)\n created = True\n\n return result, created\nvalidate_resource(collection, entry_id) async","text":"Validate whether a resource exists in a collection
Source code inoptimade_gateway/routers/utils.py async def validate_resource(collection: AsyncMongoCollection, entry_id: str) -> None:\n \"\"\"Validate whether a resource exists in a collection\"\"\"\n if not await collection.exists(entry_id):\n raise NotFound(\n detail=f\"Resource <id={entry_id}> not found in {collection}.\",\n )\nA REST API server acting as a gateway for databases with an OPTIMADE API, handling the distribution and collection of a single query to several different OPTIMADE databases.
The design outline is available here.
"},{"location":"#known-limitations","title":"Known limitations","text":"Here follows a list of known limitations and oddities of the current OPTIMADE gateway code.
"},{"location":"#pagination","title":"Pagination","text":"Pagination is a bit awkward in its current implementation state.
When using the page_limit query parameter for a gateway query for gateways with multiple databases, i.e., for GET /gateways/{gateway ID}/structures and GET /queries/{query ID}, the resulting entry-resource number is the product of the page_limit value and the number of databases in the gateway (maximum). This is because the page_limit query parameter is passed straight through to the external database requests, and the returned entries are stitched together for the gateway response.
So effectively, when querying GET /gateways/{gateway with N databases}/structures?page_limit=5 the resulting (maximum) number of entries returned in the response (the size of the data array in the response) will be N x 5, and not 5 as would otherwise be expected.
The intention is to fix this in the future, either through short-time caching of external database responses, or figuring out if there is a usable algorithm that doesn't extend the number of external requests (and therefore the gateway response times) by too much.
"},{"location":"#sorting","title":"Sorting","text":"Sorting is supported for all the gateway's own resources, i.e., in the /gateways, /databases, and /queries endpoints. But sorting is not supported for the results from external OPTIMADE databases. This means the sort query parameter has no effect in the GET /gateways/{gateway ID}/structures and GET /queries/{query ID} endpoints.
This shortcoming is a direct result of the current page_limit query parameter handling, and the limitation of the same.
All code in this repository was originally written by Casper Welzel Andersen (@CasperWA). The design for the gateway as outlined in design.md was a joint effort between Casper Welzel Andersen & Carl Simon Adorf (@csadorf).
All files in this repository are licensed under the MIT license with copyright \u00a9 2021 Casper Welzel Andersen & THEOS, EPFL.
"},{"location":"#funding-support","title":"Funding support","text":"This work was funded by THEOS, EPFL and the MarketPlace project.
The MarketPlace project is funded by Horizon 2020 under H2020-NMBP-25-2017 call with Grant agreement number: 760173.
"},{"location":"CHANGELOG/","title":"Changelog","text":""},{"location":"CHANGELOG/#unreleased-changes-2024-08-23","title":"Unreleased changes (2024-08-23)","text":"Full Changelog
Implemented enhancements:
Fixed bugs:
pytest-asyncio #447pre-commit issue for the mypy hook #385Closed issues:
Merged pull requests:
pyproject.toml) #430 (CasperWA)Full Changelog
Merged pull requests:
Full Changelog
Fixed bugs:
ci/dependabot-updates branch failing #174Closed issues:
main push CI job #184pre-commit hooks autoupdate to CI #183Merged pull requests:
ID! type instead of String! #239 (CasperWA)pre-commit hooks in dependabot CI #186 (CasperWA)ref instead of sha #178 (CasperWA)Full Changelog
Fixed bugs:
ci/dependabot-updates after merging ci/update-dependencies #131Closed issues:
ci/update-dependencies PR to Tuesday or Friday #160Merged pull requests:
ci/dependabot-updates branch upon merge to main #161 (CasperWA)Full Changelog
Implemented enhancements:
bandit, pylint, safety, and mypy #119 (CasperWA)Fixed bugs:
main docs deployment #152gh api in workflow #150mike deploy #144 (CasperWA)git push instead of action #136 (CasperWA)Closed issues:
gh-pages in documentation deploy workflows #142Merged pull requests:
gh-pages branch #157 (CasperWA)main build #153 (CasperWA)env outside of usable scope #126 (CasperWA)Full Changelog
Implemented enhancements:
Fixed bugs:
Merged pull requests:
Full Changelog
Fixed bugs:
Merged pull requests:
Full Changelog
Fixed bugs:
Merged pull requests:
Full Changelog
Implemented enhancements:
Fixed bugs:
Closed issues:
Merged pull requests:
* This Changelog was automatically generated by github_changelog_generator
"},{"location":"LICENSE/","title":"License","text":"MIT License
Copyright (c) 2021 Casper Welzel Andersen & THEOS, EPFL
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"},{"location":"design/","title":"Design of the OPTIMADE gateway","text":"The OPTIMADE gateway is intended to be implemented into the MarketPlace platform. Therefore, it should implement the MarketPlace Data Source API, as well as endpoints needed for the gateway capabilities themselves. To this end, the following sections defines/recaps these APIs and capabilities.
"},{"location":"design/#marketplace-data-source-api","title":"MarketPlace Data Source API","text":"The MarketPlace Data Source API developed in T2.2 of the MarketPlace project. It can be found on the Fraunhofer GitLab here.
Outline of the currently defined endpoints. Note, if there is no HTTP method next to the endpoint, it is not an available and reachable endpoint.
/marketplace/
/schemas/ (GET)
/{schema_id}/
/attributes (GET)/export (POST)/search (POST)The suggested OPTIMADE gateway API.
This API is based on the expected capabilities outlined below.
/optimade/ Methods: GET Behavior: Introspective/static metadata overview of server.
/query/ Methods: POST or GET Behavior: Orchestrate an OPTIMADE query.
/gateways/ Methods: GET Behavior: Standard reponse: Introspective/static metadata overview of all gateways. Using special query parameter: Create/retrieve and return unique gateway ID.
/{gateway_id}/ Methods: POST or GET Behavior: Create/retrieve search ID and return unique search ID. Start asynchronous search task.
Either:
/queries/ Methods: None Behavior: Disallowed. Note: This endpoint could support GET requests with similar functionality and behavior as for /gateways/? This would move some functionality away from /{gateway_id}/ to this endpoint. Making /{gateway_id}/ act as a mix of /query/ and /gateways/ in terms of orchestrating the search and returning introspective/static metadata about the gateway.
/{search_id}/ Methods: GET Behavior: Return current results according to state of asynchronous search task.
or:
/{search_id}/ Methods: GET Behavior: Return current results according to state of asynchronous search task.I think the way you would achieve the \u201cselection\u201d of databases is by creating provider-specific endpoints like this:
GET\n/gateway?providers=abc,def,xyz\nThis will return a deterministic gateway id related to specific set of providers, which you will then use for further queries like this:
GET\n/gateway/{gateway_id}/structures/\netc.
The gateway id would provide introspection, so /gateway/{gateway_id} returns some information about the gateway (supported OPTIMADE API, list of providers) etc. You would cache the gateway id in the client, so you don\u2019t have to make two requests for each query. If you don\u2019t provide a list of providers, the current default set is used. But this ensures that the REST API is actually stateless, because one gateway is always tied to a specific set of providers even if the default list is changed. Obviously, if you use a gateway that includes providers that are no longer available you would respond with code 503 or so.
This design solves the issue of how to provide a gateway that implements the OPTIMADE API and allows for the selection of providers. I assume your results are paginated, so IMO \u2014 unless you request a specific order \u2014 you should just return results as they come in. You need to implement this gateway asynchronously anyways so it really does not matter whether you include slow providers or not.
Of course, this changes if the user requests a specific order, but that\u2019s just how it is. From a user perspective it would make sense to me that such a query across multiple providers may take a while.
You should definitely define a timeout for each gateway where if a provider does not respond by then, the result is returned regardless of whether the provider has responded. Or you respond with a time out code.
"},{"location":"design/#searching","title":"Searching","text":"Taking Simon's comments into account, the search capability should be:
The asynchronicity comes from creating web calls (possibly using CORS) to each (chosen) database asynchronously, collating the results in a single (gateway) endpoint.
The dynamics here relate to the suggested dynamic creation (and possible deletion) of gateway IDs under a /gateway-endpoint.
Essentially, for each search, a new gateway will be created (if needed) with a unique ID. This unique ID will constitue the content of the initial response after performing a search, so that the user can go to the new gateway ID-specific endpoint to retrieve the results. To make this easier for the user, the server could automatically redirect the user after creating the endpoint. Here the response will contain the currently retrieved results as well as som metadata information about how the search is going and a general overview.
This would ideally result in the following search sequence:
The final GET request can be repeated to retrieve more results during the timeline of the search happening, and to retrieve the final list of results in a set time period after the search has finished.
One could also think of using POST requests instead, containing the OPTIMADE query parameters alongside with other information, mainly utilized for the /gateway/{unique ID}-endpoints. The response could contain a link or simply redirect to a /gateway/{unique ID}/{search unique ID}-endpoint. The latter part could also be done for the GET approach, since a specific gateway should support multiple unique simultaneous searched. Since the searches are asynchronous, the results don't come back from all resources simultaneously, thus demanding an extra endpoint, where the continuously updating results can be found - as well as the final list of results for a specific search.
This differs from the section above, where a GET request should contain query parameters in the URL and this will be correlated with an ongoing (unique) search in the backend, which would potentially allow different users to experience the same loading of results if they performed the same search in the same gateway, even at slightly different times during the searching period.
A sequence would ideally look this:
"},{"location":"design/#conclusion","title":"Conclusion","text":"The best approach here would be to create unique search IDs under each unique gateway, pertaining to a specific search. In the same way that gateways may be reused, search results may be reused. However, to ensure the \"freshness\" of the data, the \"live\"-period for any unique search should be significantly smaller than that of any unique gateway.
POST requests may be preferred due to the ability of combining OPTIMADE-specific query data and gateway-specific data.
Suggested search sequence diagram:
"},{"location":"design/#design-discussions-17122020","title":"Design discussions (17.12.2020)","text":"To be backwards compatible (where each gateway may represent a fully fledged OPTIMADE database), make /gateways/{unique ID}/ redirect to /gateways/{unique ID}/structures/.
Note, remove CUDS as a required capability, content negotiation might be with different means than a URL query parameter.
"},{"location":"design/#caching","title":"Caching","text":"Caching should be segmented for each database. For each new user query that retrieves and caches individual resources from a database, the lifetime of the cached resource should be updated to the set default (or what is determined by caching headers from the side of the database). Either the CacheControl or requests-cache packages will be utilized for caching.
Since the time it takes for an OPTIMADE database to change its content varies, but is mainly quite long, individual search life times (/gateways/{gateway ID}(/queries)/{search ID}/) can be \"long\", e.g., a couple of hours. However, these two ways of \"caching\" should be separate.
It should always be possible to forcefully ensure a \"fresh\" search.
"},{"location":"design/#optimade-filter-language","title":"OPTIMADE filter language","text":"The filter language will be reused as the filter language for any search in any gateway.
The filter language is defined in the OPTIMADE specification.
"},{"location":"design/#retrieval-formats","title":"Retrieval formats","text":"All responses will be in JSON (for now).
To choose the retrieval format of the structure, a query parameter will be dedicated for the /{search unique ID} endpoint.
The standard OPTIMADE format for defining structures will be reused for listing the structure entries.
See the OPTIMADE specification for a list of properties defining the structures entry.
When returning the results in this format, the whole response should be compliant with a standard OPTIMADE response as is expected in the /structures-endpoint.
Utilizing the optimade2cuds Python package in the SimOPTIMADE repository on the Fraunhofer GitLab for the MarketPlace project, the resulting OPTIMADE structure can be converted to Python CUDS objects. From there they can be serialized to JSON representations (using the OSP-Core package) and returned as a search result response.
When making external API calls, i.e., requesting the various OPTIMADE databases, this is technically done in a concurrent.futures.ThreadPoolExecutor. This is mainly done to not block the main OS thread, where the asyncio event loop is running. This is the event loop that handles incoming gateway requests. While the number of databases may not be significant, the response times can still vary and by using a ThreadPoolExecutor, the gateway is ready for more heavy use out-of-the-box.
Another key reason to use a ThreadPoolExecutor (instead of Starlette's - and therefore FastAPI's - BackgroundTask) is for testing with the pytest framework. When using BackgroundTask the response cannot be properly mocked and instead blocks the main OS thread. Perhaps this could be solved by implementing the same solution as has been done for now, namely running a time.sleep function call in a ThreadPoolExecutor, in the mocked response callback, but the benefits of using a ThreadPoolExecutor also for the actual queries outweigh this in the long run.
For further considerations a ProcessPoolExecutor might even be considered, but it shouldn't be necessary as the work done is IO blocking, not CPU blocking. The possible speed-up should not be significant.
Further reading and considerations on this subject Multithreading vs. Multiprocessing in Python by Amine Baatout is a good read. Another source of inspiration was found in this StackOverflow post response.
"},{"location":"design/#other-ideas-a-queue","title":"Other ideas - a queue","text":"Throughout the process of figuring this out, other ideas were on the table. One was to setup an asyncio.Queue - either a single \"unbuffered channel\" queue for the whole lifetime of the server, or one each per request. This would effectively split up the perform_query in producer/worker functions.
For some nice reading on this, check out Latency in Asynchronous Python by Chris Wellons (null program).
Since the ThreadPoolExecutor solution solves the issue of the analogous \"heartbeat\" function not losing its responsivenes, i.e., the asyncio event loop not being blocked, and it would work with the current code implementation, I opted for this solution instead. But I recon a queue solution would work similarly, but perhaps with slightly less gateway API responsiveness during heavy load, since it all still runs in the same event loop.
ASGI app events.
These events can be run at application startup or shutdown. The specific events are listed in EVENTS along with their respected proper invocation time.
EVENTS: Sequence[tuple[str, Callable[[], Coroutine[Any, Any, None]]]] = (('startup', ci_dev_startup), ('startup', load_optimade_providers_databases)) module-attribute","text":"A tuple of all pairs of events and event functions.
To use this tuple of tuples:
from fastapi import FastAPI\nAPP = FastAPI()\nfor event, func in EVENTS:\n APP.add_event_handler(event, func)\nci_dev_startup() async","text":"Function to run at app startup - only relevant for CI or development to add test data.
Source code inoptimade_gateway/events.py async def ci_dev_startup() -> None:\n \"\"\"Function to run at app startup - only relevant for CI or development to add test\n data.\"\"\"\n if bool(os.getenv(\"CI\", \"\")):\n LOGGER.info(\n \"CI detected - Will load test gateways (after dropping the collection)!\"\n )\n elif os.getenv(\"OPTIMADE_MONGO_DATABASE\", \"\") == \"optimade_gateway_dev\":\n LOGGER.info(\n \"Running in development mode - Will load test gateways (after dropping the\"\n \" collection)!\"\n )\n else:\n LOGGER.debug(\"Not in CI or development mode - will start normally.\")\n return\n\n # Add test gateways\n import json\n from pathlib import Path\n\n from optimade_gateway.mongo.database import MONGO_DB\n\n test_data = (\n Path(__file__).parent.parent.joinpath(\".ci/test_gateways.json\").resolve()\n )\n\n await MONGO_DB[CONFIG.gateways_collection].drop()\n\n if await MONGO_DB[CONFIG.gateways_collection].count_documents({}) != 0:\n raise RuntimeError(\n f\"Unexpectedly found documents in the {CONFIG.gateways_collection!r} Mongo\"\n \" collection after dropping it ! Found number of documents: \"\n f\"{await MONGO_DB[CONFIG.gateways_collection].count_documents({})}\"\n )\n\n if not test_data.exists():\n raise FileNotFoundError(\n f\"Could not find test data file with test gateways at {test_data} !\"\n )\n\n data = json.loads(test_data.read_bytes())\n\n await MONGO_DB[CONFIG.gateways_collection].insert_many(data)\nload_optimade_providers_databases() async","text":"Load in the providers' OPTIMADE databases from Materials-Consortia
Utilize the Materials-Consortia list of OPTIMADE providers at https://providers.optimade.org. Load in all databases with a valid base URL.
Source code inoptimade_gateway/events.py async def load_optimade_providers_databases() -> None:\n \"\"\"Load in the providers' OPTIMADE databases from Materials-Consortia\n\n Utilize the Materials-Consortia list of OPTIMADE providers at\n [https://providers.optimade.org](https://providers.optimade.org).\n Load in all databases with a valid base URL.\n \"\"\"\n import asyncio\n\n import httpx\n from optimade import __api_version__\n from optimade.models import LinksResponse\n from optimade.models.links import LinkType\n from optimade.server.routers.utils import BASE_URL_PREFIXES\n\n from optimade_gateway.common.utils import clean_python_types, get_resource_attribute\n from optimade_gateway.models.databases import DatabaseCreate\n from optimade_gateway.queries.perform import db_get_all_resources\n from optimade_gateway.routers.utils import resource_factory\n\n if not CONFIG.load_optimade_providers_databases:\n LOGGER.debug(\n \"Will not load databases from Materials-Consortia list of providers.\"\n )\n return\n\n if TYPE_CHECKING or bool(os.getenv(\"MKDOCS_BUILD\", \"\")): # pragma: no cover\n providers: httpx.Response | LinksResponse\n\n async with httpx.AsyncClient() as client:\n providers = await client.get(\n \"https://providers.optimade.org/v\"\n f\"{__api_version__.split('.', maxsplit=1)[0]}/links\"\n )\n\n if providers.is_error:\n LOGGER.warning(\n \"Response from Materials-Consortia's list of OPTIMADE providers was not \"\n \"successful (status code != 200). No databases will therefore be added at \"\n \"server startup.\"\n )\n return\n\n LOGGER.info(\n \"Registering Materials-Consortia list of OPTIMADE providers' databases.\"\n )\n\n providers = LinksResponse(**providers.json())\n\n valid_providers = []\n for provider in providers.data:\n if get_resource_attribute(provider, \"id\") in (\"exmpl\", \"optimade\"):\n LOGGER.info(\n \"- %s (id=%r) - Skipping: Not a real provider.\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n continue\n\n if not get_resource_attribute(provider, \"attributes.base_url\"):\n LOGGER.info(\n \"- %s (id=%r) - Skipping: No base URL information.\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n continue\n\n valid_providers.append(provider)\n\n # Run queries to each database using the supported major versioned base URL to get a\n # list of the provider's databases.\n # There is no need to use ThreadPoolExecutor here, since we want this to block\n # everything and then finish, before the server actually starts up.\n provider_queries = [\n asyncio.create_task(\n db_get_all_resources(\n database=provider,\n endpoint=\"links\",\n response_model=LinksResponse,\n )\n )\n for provider in valid_providers\n ]\n\n for query in asyncio.as_completed(provider_queries):\n provider_databases, provider = await query\n\n LOGGER.info(\n \"- %s (id=%r) - Processing\",\n get_resource_attribute(provider, \"attributes.name\", \"N/A\"),\n get_resource_attribute(provider, \"id\"),\n )\n if not provider_databases:\n LOGGER.info(\" - No OPTIMADE databases found.\")\n continue\n\n provider_databases = [\n db\n for db in provider_databases\n if await clean_python_types(\n get_resource_attribute(db, \"attributes.link_type\", \"\")\n )\n == LinkType.CHILD.value\n ]\n\n if not provider_databases:\n LOGGER.info(\" - No OPTIMADE databases found.\")\n continue\n\n for database in provider_databases:\n if not get_resource_attribute(database, \"attributes.base_url\"):\n LOGGER.info(\n \" - %s (id=%r) - Skipping: No base URL information.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n )\n continue\n\n LOGGER.info(\n \" - %s (id=%r) - Checking versioned base URL and /structures\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n )\n\n async with httpx.AsyncClient() as client:\n try:\n db_response = await client.get(\n f\"{str(get_resource_attribute(database, 'attributes.base_url')).rstrip('/')}\" # noqa: E501\n f\"{BASE_URL_PREFIXES['major']}/structures\",\n )\n except httpx.ReadTimeout:\n LOGGER.info(\n \" - %s (id=%r) - Skipping: Timeout while requesting \"\n \"%s/structures.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n BASE_URL_PREFIXES[\"major\"],\n )\n continue\n if db_response.status_code != 200:\n LOGGER.info(\n \" - %s (id=%r) - Skipping: Response from %s/structures is not \"\n \"200 OK.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n BASE_URL_PREFIXES[\"major\"],\n )\n continue\n\n new_id = (\n f\"{get_resource_attribute(provider, 'id')}\"\n f\"/{get_resource_attribute(database, 'id')}\"\n if len(provider_databases) > 1\n else get_resource_attribute(database, \"id\")\n )\n registered_database, _ = await resource_factory(\n DatabaseCreate(\n id=new_id,\n **await clean_python_types(\n get_resource_attribute(database, \"attributes\", {})\n ),\n )\n )\n LOGGER.info(\n \" - %s (id=%r) - Registered database with id=%r\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n registered_database.id,\n )\nASGI app exception handlers.
These are in addition to the exception handlers available in OPTIMADE Python tools. For more information see https://www.optimade.org/optimade-python-tools/api_reference/server/exception_handlers/.
"},{"location":"api_reference/exception_handlers/#optimade_gateway.exception_handlers.request_validation_exception_handler","title":"request_validation_exception_handler(request, exc) async","text":"Special handler if a RequestValidationError comes from wrong POST data
optimade_gateway/exception_handlers.py async def request_validation_exception_handler(\n request: Request, exc: RequestValidationError\n) -> JSONResponse:\n \"\"\"Special handler if a `RequestValidationError` comes from wrong `POST` data\"\"\"\n status_code = 500\n if request.method in (\"POST\", \"post\"):\n status_code = 400\n\n errors = set()\n for error in exc.errors():\n pointer = \"/\" + \"/\".join([str(_) for _ in error[\"loc\"]])\n source = ErrorSource(pointer=pointer)\n code = error[\"type\"]\n detail = error[\"msg\"]\n errors.add(\n OptimadeError(\n detail=detail,\n status=status_code,\n title=str(exc.__class__.__name__),\n source=source,\n code=code,\n )\n )\n\n return general_exception(request, exc, status_code=status_code, errors=list(errors))\nThe initialization of the ASGI FastAPI application.
"},{"location":"api_reference/main/#optimade_gateway.main.APP","title":"APP = FastAPI(title='OPTIMADE Gateway', description='A gateway server to query multiple OPTIMADE databases.', version=__version__) module-attribute","text":"The FastAPI ASGI application.
"},{"location":"api_reference/main/#optimade_gateway.main.get_root","title":"get_root(request) async","text":"GET /
Introspective overview of gateway server.
Note
Temporarily redirecting to GET /docs.
optimade_gateway/main.py @APP.get(\"/\", include_in_schema=False)\nasync def get_root(request: Request) -> RedirectResponse:\n \"\"\"`GET /`\n\n Introspective overview of gateway server.\n\n !!! note\n Temporarily redirecting to `GET /docs`.\n \"\"\"\n return RedirectResponse(\n request.url.replace(path=f\"{request.url.path.strip('/')}/docs\")\n )\nASGI app middleware.
These are in addition to the middleware available in OPTIMADE Python tools. For more information see https://www.optimade.org/optimade-python-tools/api_reference/server/middleware/.
"},{"location":"api_reference/middleware/#optimade_gateway.middleware.CheckWronglyVersionedBaseUrlsGateways","title":"CheckWronglyVersionedBaseUrlsGateways","text":" Bases: BaseHTTPMiddleware
If a non-supported versioned base URL is supplied to a gateway return 553 Version Not Supported.
optimade_gateway/middleware.py class CheckWronglyVersionedBaseUrlsGateways(BaseHTTPMiddleware):\n \"\"\"If a non-supported versioned base URL is supplied to a gateway\n return `553 Version Not Supported`.\"\"\"\n\n @staticmethod\n async def check_url(url: URL):\n \"\"\"Check URL path for versioned part.\n\n Parameters:\n url: A complete `urllib`-parsed raw URL.\n\n Raises:\n VersionNotSupported: If the URL represents an OPTIMADE versioned base URL\n and the version part is not supported by the implementation.\n\n \"\"\"\n base_url = get_base_url(url)\n optimade_path = f\"{url.scheme}://{url.netloc}{url.path}\"[len(base_url) :]\n match = re.match(\n r\"^/gateways/[^/\\s]+(?P<version>/v[0-9]+(\\.[0-9]+){0,2}).*\", optimade_path\n )\n if (\n match is not None\n and match.group(\"version\") not in BASE_URL_PREFIXES.values()\n ):\n raise VersionNotSupported(\n detail=(\n f\"The parsed versioned base URL {match.group('version')!r} \"\n f\"from {url} is not supported by this implementation. \"\n \"Supported versioned base URLs are: \"\n f\"{', '.join(BASE_URL_PREFIXES.values())}\"\n )\n )\n\n async def dispatch(self, request: Request, call_next):\n if request.url.path:\n await self.check_url(request.url)\n return await call_next(request)\ncheck_url(url) async staticmethod","text":"Check URL path for versioned part.
Parameters:
Name Type Description Defaulturl URL A complete urllib-parsed raw URL.
Raises:
Type DescriptionVersionNotSupported If the URL represents an OPTIMADE versioned base URL and the version part is not supported by the implementation.
Source code inoptimade_gateway/middleware.py @staticmethod\nasync def check_url(url: URL):\n \"\"\"Check URL path for versioned part.\n\n Parameters:\n url: A complete `urllib`-parsed raw URL.\n\n Raises:\n VersionNotSupported: If the URL represents an OPTIMADE versioned base URL\n and the version part is not supported by the implementation.\n\n \"\"\"\n base_url = get_base_url(url)\n optimade_path = f\"{url.scheme}://{url.netloc}{url.path}\"[len(base_url) :]\n match = re.match(\n r\"^/gateways/[^/\\s]+(?P<version>/v[0-9]+(\\.[0-9]+){0,2}).*\", optimade_path\n )\n if (\n match is not None\n and match.group(\"version\") not in BASE_URL_PREFIXES.values()\n ):\n raise VersionNotSupported(\n detail=(\n f\"The parsed versioned base URL {match.group('version')!r} \"\n f\"from {url} is not supported by this implementation. \"\n \"Supported versioned base URLs are: \"\n f\"{', '.join(BASE_URL_PREFIXES.values())}\"\n )\n )\nServer warnings.
The warnings in this module will all be caught by middleware and added to the response under meta.warnings.
OptimadeGatewayWarning","text":" Bases: OptimadeWarning
Base Warning for the optimade-gateway package.
optimade_gateway/warnings.py class OptimadeGatewayWarning(OptimadeWarning):\n \"\"\"Base Warning for the `optimade-gateway` package.\"\"\"\nSortNotSupported","text":" Bases: OptimadeGatewayWarning
Sorting (the sort query parameter) is currently not supported for gateway queries to external OPTIMADE databases. See https://optimade.org/optimade-gateway#sorting for more information.
optimade_gateway/warnings.py class SortNotSupported(OptimadeGatewayWarning):\n \"\"\"Sorting (the `sort` query parameter) is currently not supported for gateway\n queries to external OPTIMADE databases. See\n https://optimade.org/optimade-gateway#sorting for more information.\"\"\"\nConfiguration of the FastAPI server.
"},{"location":"api_reference/common/config/#optimade_gateway.common.config.ServerConfig","title":"ServerConfig","text":" Bases: ServerConfig
This class stores server config parameters in a way that can be easily extended for new config file types.
Source code inoptimade_gateway/common/config.py class ServerConfig(OptimadeServerConfig):\n \"\"\"This class stores server config parameters in a way that\n can be easily extended for new config file types.\n\n \"\"\"\n\n databases_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/databases` endpoint resources.\",\n ),\n ] = \"databases\"\n\n gateways_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/gateways` endpoint resources.\",\n ),\n ] = \"gateways\"\n\n queries_collection: Annotated[\n str,\n Field(\n description=\"Mongo collection name for `/queries` endpoint resources.\",\n ),\n ] = \"queries\"\n\n load_optimade_providers_databases: Annotated[\n bool,\n Field(\n description=(\n \"Whether or not to load all valid OPTIMADE providers' databases from \"\n \"the [Materials-Consortia list of OPTIMADE providers]\"\n \"(https://providers.optimade.org) on server startup.\"\n ),\n ),\n ] = True\n\n mongo_certfile: Annotated[\n Path,\n Field(\n description=\"Path to the MongoDB certificate file.\",\n ),\n ] = Path(\"/certs/mongodb.pem\")\n\n mongo_atlas_pem_content: Annotated[\n SecretStr | None,\n Field(\n description=\"PEM content for MongoDB Atlas certificate.\",\n ),\n ] = None\n\n @field_validator(\"mongo_uri\", mode=\"after\")\n @classmethod\n def replace_with_env_vars(cls, value: str) -> str:\n \"\"\"Replace string variables with environment variables, if possible\"\"\"\n res = value\n for match in re.finditer(r\"\\{[^{}]+\\}\", value):\n string_var = match.group()[1:-1]\n env_var = os.getenv(\n string_var, os.getenv(string_var.upper(), os.getenv(string_var.lower()))\n )\n if env_var is not None:\n res = res.replace(match.group(), env_var)\n else:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Could not find an environment variable for \"\n f\"{match.group()!r} from mongo_uri: {value}\"\n )\n )\n )\n return res\n\n @model_validator(mode=\"after\")\n def write_pem_content_to_file(self) -> ServerConfig:\n \"\"\"Write the MongoDB Atlas PEM content to a file\"\"\"\n if self.mongo_atlas_pem_content:\n self.mongo_certfile.parent.mkdir(parents=True, exist_ok=True)\n self.mongo_certfile.write_text(\n self.mongo_atlas_pem_content.get_secret_value()\n )\n\n return self\nreplace_with_env_vars(value) classmethod","text":"Replace string variables with environment variables, if possible
Source code inoptimade_gateway/common/config.py @field_validator(\"mongo_uri\", mode=\"after\")\n@classmethod\ndef replace_with_env_vars(cls, value: str) -> str:\n \"\"\"Replace string variables with environment variables, if possible\"\"\"\n res = value\n for match in re.finditer(r\"\\{[^{}]+\\}\", value):\n string_var = match.group()[1:-1]\n env_var = os.getenv(\n string_var, os.getenv(string_var.upper(), os.getenv(string_var.lower()))\n )\n if env_var is not None:\n res = res.replace(match.group(), env_var)\n else:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Could not find an environment variable for \"\n f\"{match.group()!r} from mongo_uri: {value}\"\n )\n )\n )\n return res\nwrite_pem_content_to_file()","text":"Write the MongoDB Atlas PEM content to a file
Source code inoptimade_gateway/common/config.py @model_validator(mode=\"after\")\ndef write_pem_content_to_file(self) -> ServerConfig:\n \"\"\"Write the MongoDB Atlas PEM content to a file\"\"\"\n if self.mongo_atlas_pem_content:\n self.mongo_certfile.parent.mkdir(parents=True, exist_ok=True)\n self.mongo_certfile.write_text(\n self.mongo_atlas_pem_content.get_secret_value()\n )\n\n return self\nSpecific OPTIMADE Gateway Python exceptions.
"},{"location":"api_reference/common/exceptions/#optimade_gateway.common.exceptions.OptimadeGatewayError","title":"OptimadeGatewayError","text":" Bases: Exception
General OPTIMADE Gateway exception.
Source code inoptimade_gateway/common/exceptions.py class OptimadeGatewayError(Exception):\n \"\"\"General OPTIMADE Gateway exception.\"\"\"\nLogging to both file and console
"},{"location":"api_reference/common/logger/#optimade_gateway.common.logger.disable_logging","title":"disable_logging()","text":"Temporarily disable logging.
Usage:
from optimade_gateway.common.logger import disable_logging\n\n# Do stuff, logging to all handlers.\n# ...\nwith disable_logging():\n # Do stuff, without logging to any handlers.\n # ...\n# Do stuff, logging to all handlers now re-enabled.\n# ...\noptimade_gateway/common/logger.py @contextmanager\ndef disable_logging():\n \"\"\"Temporarily disable logging.\n\n Usage:\n\n ```python\n from optimade_gateway.common.logger import disable_logging\n\n # Do stuff, logging to all handlers.\n # ...\n with disable_logging():\n # Do stuff, without logging to any handlers.\n # ...\n # Do stuff, logging to all handlers now re-enabled.\n # ...\n ```\n\n \"\"\"\n try:\n # Disable logging lower than CRITICAL level\n logging.disable(logging.CRITICAL)\n yield\n finally:\n # Re-enable logging to desired levels\n logging.disable(logging.NOTSET)\nCommon utility functions.
These functions may be used in general throughout the OPTIMADE Gateway Python code.
"},{"location":"api_reference/common/utils/#optimade_gateway.common.utils.clean_python_types","title":"clean_python_types(data, **dump_kwargs) async","text":"Turn any types into MongoDB-friendly Python types.
Use model_dump() method for Pydantic models. Use value property for Enums. Turn tuples and sets into lists.
optimade_gateway/common/utils.py async def clean_python_types(data: Any, **dump_kwargs: Any) -> Any:\n \"\"\"Turn any types into MongoDB-friendly Python types.\n\n Use `model_dump()` method for Pydantic models.\n Use `value` property for Enums.\n Turn tuples and sets into lists.\n \"\"\"\n if isinstance(data, (list, tuple, set)):\n res_list = []\n for datum in data:\n res_list.append(await clean_python_types(datum, **dump_kwargs))\n return res_list\n\n if isinstance(data, dict):\n res_dict = {}\n for key in list(data.keys()):\n res_dict[key] = await clean_python_types(data[key], **dump_kwargs)\n return res_dict\n\n if isinstance(data, BaseModel):\n # Pydantic model\n return await clean_python_types(data.model_dump(**dump_kwargs))\n\n if isinstance(data, Enum):\n return await clean_python_types(data.value, **dump_kwargs)\n\n if isinstance(data, type):\n return await clean_python_types(\n f\"{data.__module__}.{data.__name__}\", **dump_kwargs\n )\n\n if isinstance(data, AnyUrl):\n return await clean_python_types(str(data), **dump_kwargs)\n\n # Unknown or other basic type, e.g., str, int, etc.\n return data\nget_resource_attribute(resource, field, default=None, disambiguate=True)","text":"Return a resource's field's value
Get the field value no matter if the resource is a pydantic model or a Python dictionary.
Determine ambiguous field values and return them if desired (disambiguate). For example, if \"attributes.base_url\" is requested for a LinksResource it can be either a string, a Link model or a dictionary resembling the Link model.
Parameters:
Name Type Description Defaultresource BaseModel | dict[str, Any] | None The resource, from which to get the field value.
requiredfield str The resource field. This can be a dot-separated nested field, e.g., \"attributes.base_url\".
default Any The default value to return if field does not exist.
None disambiguate bool Whether or not to \"shortcut\" a field value. For example, for attributes.base_url, if True, this would return the string value or the value of it's \"href\" key.
True Returns:
Type DescriptionAny The resource's field's value.
Source code inoptimade_gateway/common/utils.py def get_resource_attribute(\n resource: BaseModel | dict[str, Any] | None,\n field: str,\n default: Any = None,\n disambiguate: bool = True,\n) -> Any:\n \"\"\"Return a resource's field's value\n\n Get the field value no matter if the resource is a pydantic model or a Python\n dictionary.\n\n Determine ambiguous field values and return them if desired (`disambiguate`).\n For example, if\n [`\"attributes.base_url\"`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes.base_url)\n is requested for a\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n it can be either a string, a\n [`Link`](https://www.optimade.org/optimade-python-tools/api_reference/models/jsonapi/#optimade.models.jsonapi.Link)\n model or a dictionary resembling the `Link` model.\n\n Parameters:\n resource: The resource, from which to get the field value.\n field: The resource field. This can be a dot-separated nested field, e.g.,\n `\"attributes.base_url\"`.\n default: The default value to return if `field` does not exist.\n disambiguate: Whether or not to \"shortcut\" a field value.\n For example, for `attributes.base_url`, if `True`, this would return the\n string value or the value of it's `\"href\"` key.\n\n Returns:\n The resource's field's value.\n\n \"\"\"\n if isinstance(resource, BaseModel):\n _get_attr = getattr\n elif isinstance(resource, dict):\n\n def _get_attr(mapping: dict, key: str, default: Any) -> Any: # type: ignore[misc]\n return mapping.get(key, default)\n\n elif resource is None:\n # Allow passing `None`, but simply return `default`\n return default\n else:\n raise TypeError(\n \"resource must be either a pydantic model or a Python dictionary, it was \"\n f\"of type {type(resource)!r}\"\n )\n\n fields = field.split(\".\")\n for _ in fields[:-1]:\n resource = _get_attr(resource, _, {})\n field = fields[-1]\n value = _get_attr(resource, field, default)\n\n if (\n disambiguate\n and field in (\"base_url\", \"next\", \"prev\", \"last\", \"first\")\n and not isinstance(value, (str, AnyUrl))\n ):\n value = _get_attr(value, \"href\", default)\n\n return value\nBase resource mapper.
Based on the BaseResourceMapper in OPTIMADE Python tools.
BaseResourceMapper","text":" Bases: BaseResourceMapper
Generic Resource Mapper that defines and performs the mapping between objects in the database and the resource objects defined by the specification.
NoteThis is a \"wrapped\" sub-class to make certain methods asynchronous.
Attributes:
Name Type DescriptionALIASES a tuple of aliases between OPTIMADE field names and the field names in the database , e.g. ((\"elements\", \"custom_elements_field\")).
LENGTH_ALIASES a tuple of aliases between a field name and another field that defines its length, to be used when querying, e.g. ((\"elements\", \"nelements\")). e.g. ((\"elements\", \"custom_elements_field\")).
ENTRY_RESOURCE_CLASS The entry type that this mapper corresponds to.
PROVIDER_FIELDS a tuple of extra field names that this mapper should support when querying with the database prefix.
TOP_LEVEL_NON_ATTRIBUTES_FIELDS the set of top-level field names common to all endpoints.
SUPPORTED_PREFIXES The set of prefixes registered by this mapper.
ALL_ATTRIBUTES The set of attributes defined across the entry resource class and the server configuration.
ENTRY_RESOURCE_ATTRIBUTES A dictionary of attributes and their definitions defined by the schema of the entry resource class.
ENDPOINT The expected endpoint name for this resource, as defined by the type in the schema of the entry resource class.
optimade_gateway/mappers/base.py class BaseResourceMapper(OptimadeBaseResourceMapper):\n \"\"\"\n Generic Resource Mapper that defines and performs the mapping\n between objects in the database and the resource objects defined by\n the specification.\n\n Note:\n This is a \"wrapped\" sub-class to make certain methods asynchronous.\n\n Attributes:\n ALIASES: a tuple of aliases between\n OPTIMADE field names and the field names in the database ,\n e.g. `((\"elements\", \"custom_elements_field\"))`.\n LENGTH_ALIASES: a tuple of aliases between\n a field name and another field that defines its length, to be used\n when querying, e.g. `((\"elements\", \"nelements\"))`.\n e.g. `((\"elements\", \"custom_elements_field\"))`.\n ENTRY_RESOURCE_CLASS: The entry type that this mapper corresponds to.\n PROVIDER_FIELDS: a tuple of extra field names that this\n mapper should support when querying with the database prefix.\n TOP_LEVEL_NON_ATTRIBUTES_FIELDS: the set of top-level\n field names common to all endpoints.\n SUPPORTED_PREFIXES: The set of prefixes registered by this mapper.\n ALL_ATTRIBUTES: The set of attributes defined across the entry\n resource class and the server configuration.\n ENTRY_RESOURCE_ATTRIBUTES: A dictionary of attributes and their definitions\n defined by the schema of the entry resource class.\n ENDPOINT: The expected endpoint name for this resource, as defined by\n the `type` in the schema of the entry resource class.\n\n \"\"\"\n\n @classmethod\n async def adeserialize(\n cls, results: dict | Iterable[dict]\n ) -> list[EntryResource] | EntryResource:\n \"\"\"Asynchronous version of the `deserialize()` class method.\n\n Parameters:\n results: A list of or a single dictionary, representing an entry-endpoint\n resource.\n\n Returns:\n The deserialized list of or single pydantic resource model for the input\n `results`.\n\n \"\"\"\n return super().deserialize(results)\n\n @classmethod\n def map_back(cls, doc: dict) -> dict:\n from optimade.server.routers.utils import BASE_URL_PREFIXES\n\n if \"_id\" in doc:\n _id = str(doc.pop(\"_id\"))\n if \"id\" not in doc:\n doc[\"id\"] = _id\n\n doc[\"links\"] = {\n \"self\": AnyUrl(\n url=(\n f\"{CONFIG.base_url.strip('/')}{BASE_URL_PREFIXES['major']}\"\n f\"/{cls.ENDPOINT}/{doc['id']}\"\n ),\n )\n }\n return super().map_back(doc)\nadeserialize(results) async classmethod","text":"Asynchronous version of the deserialize() class method.
Parameters:
Name Type Description Defaultresults dict | Iterable[dict] A list of or a single dictionary, representing an entry-endpoint resource.
requiredReturns:
Type Descriptionlist[EntryResource] | EntryResource The deserialized list of or single pydantic resource model for the input
list[EntryResource] | EntryResource results.
optimade_gateway/mappers/base.py @classmethod\nasync def adeserialize(\n cls, results: dict | Iterable[dict]\n) -> list[EntryResource] | EntryResource:\n \"\"\"Asynchronous version of the `deserialize()` class method.\n\n Parameters:\n results: A list of or a single dictionary, representing an entry-endpoint\n resource.\n\n Returns:\n The deserialized list of or single pydantic resource model for the input\n `results`.\n\n \"\"\"\n return super().deserialize(results)\nResource mapper for resources under /databases.
These resources are LinksResources.
DatabasesMapper","text":" Bases: LinksMapper
/databases-endpoint resources mapper.
optimade_gateway/mappers/databases.py class DatabasesMapper(LinksMapper):\n \"\"\"`/databases`-endpoint resources mapper.\"\"\"\n\n ENDPOINT = \"databases\"\nResource mapper for GatewayResource.
GatewaysMapper","text":" Bases: BaseResourceMapper
GatewayResource mapper.
optimade_gateway/mappers/gateways.py class GatewaysMapper(BaseResourceMapper):\n \"\"\"[`GatewayResource`][optimade_gateway.models.gateways.GatewayResource] mapper.\"\"\"\n\n ENDPOINT = \"gateways\"\n ENTRY_RESOURCE_CLASS = GatewayResource\nReplicate of LinksMapper in OPTIMADE Python tools.
LinksMapper","text":" Bases: BaseResourceMapper
Replicate of LinksMapper in OPTIMADE Python tools.
This is based on the OPTIMADE Gateway BaseResourceMapper however.
optimade_gateway/mappers/links.py class LinksMapper(BaseResourceMapper):\n \"\"\"Replicate of\n [`LinksMapper`](https://www.optimade.org/optimade-python-tools/api_reference/server/mappers/links/#optimade.server.mappers.links.LinksMapper)\n in OPTIMADE Python tools.\n\n This is based on the OPTIMADE Gateway\n [`BaseResourceMapper`][optimade_gateway.mappers.base.BaseResourceMapper] however.\n \"\"\"\n\n ENDPOINT = \"links\"\n ENTRY_RESOURCE_CLASS = LinksResource\n\n @classmethod\n def map_back(cls, doc: dict) -> dict:\n type_ = doc.get(\"type\") or \"links\"\n newdoc = super().map_back(doc)\n newdoc[\"type\"] = type_\n return newdoc\nResource mapper for QueryResource.
QueryMapper","text":" Bases: BaseResourceMapper
QueryResource mapper.
optimade_gateway/mappers/queries.py class QueryMapper(BaseResourceMapper):\n \"\"\"[`QueryResource`][optimade_gateway.models.queries.QueryResource] mapper.\"\"\"\n\n ENDPOINT = \"queries\"\n ENTRY_RESOURCE_CLASS = QueryResource\nPydantic models/schemas for the LinksResource used in /databases
"},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate","title":"DatabaseCreate","text":" Bases: EntryResourceCreate, LinksResourceAttributes
Model for creating new LinksResources representing /databases resources in the MongoDB.
Required fields:
namebase_urlOriginal required fields for a LinksResourceAttributes model:
namedescriptionlink_typeoptimade_gateway/models/databases.py class DatabaseCreate(EntryResourceCreate, LinksResourceAttributes):\n \"\"\"Model for creating new LinksResources representing `/databases` resources in the\n MongoDB.\n\n Required fields:\n\n - `name`\n - `base_url`\n\n Original required fields for a\n [`LinksResourceAttributes`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes)\n model:\n\n - `name`\n - `description`\n - `link_type`\n\n \"\"\"\n\n description: Annotated[\n str | None,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"description\"].description\n ),\n ] = None\n\n base_url: Annotated[\n JsonLinkType,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"base_url\"].description\n ),\n ]\n\n homepage: Annotated[\n JsonLinkType | None,\n StrictField(\n description=LinksResourceAttributes.model_fields[\"homepage\"].description,\n ),\n ] = None\n\n link_type: Annotated[\n LinkType | None,\n StrictField(\n title=LinksResourceAttributes.model_fields[\"link_type\"].title,\n description=LinksResourceAttributes.model_fields[\"link_type\"].description,\n ),\n ] = None\n\n @field_validator(\"link_type\", mode=\"after\")\n @classmethod\n def ensure_database_link_type(cls, value: LinkType) -> LinkType:\n \"\"\"Ensure databases are not index meta-database-only types\n\n I.e., ensure they're not of type `\"root\"` or `\"providers\"`.\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n if value in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway-usable database resources. Given link_type: {value}\"\n )\n return value\nbase_url: Annotated[JsonLinkType, StrictField(description=LinksResourceAttributes.model_fields['base_url'].description)] instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.description","title":"description: Annotated[str | None, StrictField(description=LinksResourceAttributes.model_fields['description'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.homepage","title":"homepage: Annotated[JsonLinkType | None, StrictField(description=LinksResourceAttributes.model_fields['homepage'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.link_type","title":"link_type: Annotated[LinkType | None, StrictField(title=LinksResourceAttributes.model_fields['link_type'].title, description=LinksResourceAttributes.model_fields['link_type'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/databases/#optimade_gateway.models.databases.DatabaseCreate.ensure_database_link_type","title":"ensure_database_link_type(value) classmethod","text":"Ensure databases are not index meta-database-only types
I.e., ensure they're not of type \"root\" or \"providers\".
Note
Both \"external\" and \"child\" can still represent index meta-dbs, but \"root\" and \"providers\" can not represent \"regular\" dbs.
optimade_gateway/models/databases.py @field_validator(\"link_type\", mode=\"after\")\n@classmethod\ndef ensure_database_link_type(cls, value: LinkType) -> LinkType:\n \"\"\"Ensure databases are not index meta-database-only types\n\n I.e., ensure they're not of type `\"root\"` or `\"providers\"`.\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n if value in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway-usable database resources. Given link_type: {value}\"\n )\n return value\nPydantic models/schemas for the Gateways resource.
"},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate","title":"GatewayCreate","text":" Bases: EntryResourceCreate, GatewayResourceAttributes
Model for creating new Gateway resources in the MongoDB
Source code inoptimade_gateway/models/gateways.py class GatewayCreate(EntryResourceCreate, GatewayResourceAttributes):\n \"\"\"Model for creating new Gateway resources in the MongoDB\"\"\"\n\n id: Annotated[\n str | None,\n OptimadeField(\n description=EntryResource.model_fields[\"id\"].description,\n support=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-support\"\n ],\n queryable=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-queryable\"\n ],\n pattern=r\"^[^/]*$\", # This pattern is the special addition\n ),\n ] = None\n\n database_ids: Annotated[\n set[str] | None,\n Field(description=\"A unique list of database IDs for registered databases.\"),\n ] = None\n\n databases: Annotated[\n list[LinksResource] | None,\n Field(\n description=GatewayResourceAttributes.model_fields[\"databases\"].description\n ),\n ] = None # type: ignore[assignment]\n\n @model_validator(mode=\"after\")\n def specify_databases(self) -> GatewayCreate:\n \"\"\"Either `database_ids` or `databases` must be non-empty.\n Both together is also fine.\n \"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"databases\")):\n raise ValueError(\"Either 'database_ids' or 'databases' MUST be specified\")\n return self\ndatabase_ids: Annotated[set[str] | None, Field(description='A unique list of database IDs for registered databases.')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.databases","title":"databases: Annotated[list[LinksResource] | None, Field(description=GatewayResourceAttributes.model_fields['databases'].description)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.id","title":"id: Annotated[str | None, OptimadeField(description=EntryResource.model_fields['id'].description, support=EntryResource.model_fields['id'].json_schema_extra['x-optimade-support'], queryable=EntryResource.model_fields['id'].json_schema_extra['x-optimade-queryable'], pattern='^[^/]*$')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayCreate.specify_databases","title":"specify_databases()","text":"Either database_ids or databases must be non-empty. Both together is also fine.
optimade_gateway/models/gateways.py @model_validator(mode=\"after\")\ndef specify_databases(self) -> GatewayCreate:\n \"\"\"Either `database_ids` or `databases` must be non-empty.\n Both together is also fine.\n \"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"databases\")):\n raise ValueError(\"Either 'database_ids' or 'databases' MUST be specified\")\n return self\nGatewayResource","text":" Bases: EntryResource
OPTIMADE gateway
A resource representing a dynamic collection of OPTIMADE databases. The gateway can be treated as any other OPTIMADE gateway, but the entries are an aggregate of multiple databases. The id of each aggregated resource will reflect the originating database.
optimade_gateway/models/gateways.py class GatewayResource(EntryResource):\n \"\"\"OPTIMADE gateway\n\n A resource representing a dynamic collection of OPTIMADE databases.\n The gateway can be treated as any other OPTIMADE gateway, but the entries are an\n aggregate of multiple databases. The `id` of each aggregated resource will reflect\n the originating database.\n \"\"\"\n\n id: Annotated[\n str,\n OptimadeField(\n description=EntryResource.model_fields[\"id\"].description,\n support=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-support\"\n ],\n queryable=EntryResource.model_fields[\"id\"].json_schema_extra[\n \"x-optimade-queryable\"\n ],\n pattern=r\"^[^/]*$\",\n ),\n ]\n\n type: Annotated[\n Literal[\"gateways\"],\n Field(description=\"The name of the type of an entry.\"),\n ] = \"gateways\"\n\n attributes: Annotated[\n GatewayResourceAttributes,\n Field(description=EntryResource.model_fields[\"attributes\"].description),\n ]\nattributes: Annotated[GatewayResourceAttributes, Field(description=EntryResource.model_fields['attributes'].description)] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResource.id","title":"id: Annotated[str, OptimadeField(description=EntryResource.model_fields['id'].description, support=EntryResource.model_fields['id'].json_schema_extra['x-optimade-support'], queryable=EntryResource.model_fields['id'].json_schema_extra['x-optimade-queryable'], pattern='^[^/]*$')] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResource.type","title":"type: Annotated[Literal['gateways'], Field(description='The name of the type of an entry.')] = 'gateways' class-attribute instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResourceAttributes","title":"GatewayResourceAttributes","text":" Bases: EntryResourceAttributes
Attributes for an OPTIMADE gateway
Source code inoptimade_gateway/models/gateways.py class GatewayResourceAttributes(EntryResourceAttributes):\n \"\"\"Attributes for an OPTIMADE gateway\"\"\"\n\n databases: Annotated[\n list[LinksResource],\n Field(\n description=(\n \"List of databases (OPTIMADE 'links') to be queried in this gateway.\"\n ),\n ),\n ]\n\n @field_validator(\"databases\", mode=\"after\")\n @classmethod\n def unique_base_urls(cls, value: list[LinksResource]) -> list[LinksResource]:\n \"\"\"Remove extra entries with repeated base_urls.\n\n Also, ensure databases are not of type `\"root\"` or `\"providers\"`\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n for resource in value:\n if resource.attributes.link_type in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway resources. Given database: {resource}\"\n )\n\n db_base_urls = [_.attributes.base_url for _ in value]\n unique_base_urls = set(db_base_urls)\n if len(db_base_urls) == len(unique_base_urls):\n return value\n\n repeated_base_urls = [_ for _ in unique_base_urls if db_base_urls.count(_) > 1]\n new_databases = [\n _ for _ in value if _.attributes.base_url not in repeated_base_urls\n ]\n for base_url in repeated_base_urls:\n new_databases.append(\n next(_ for _ in value if _.attributes.base_url == base_url)\n )\n warnings.warn(\n \"Removed extra database entries for a gateway, because the base_url was \"\n \"repeated. The first found database entry was kept, while the others were \"\n f\"removed. Original number of databases: {len(value)}. New number of \"\n f\"databases: {len(new_databases)} Repeated base_urls (number of repeats): \"\n \"{}\".format(\n [\n f\"{base_url} ({db_base_urls.count(base_url)})\"\n for base_url in repeated_base_urls\n ]\n ),\n OptimadeGatewayWarning,\n )\n return new_databases\ndatabases: Annotated[list[LinksResource], Field(description=\"List of databases (OPTIMADE 'links') to be queried in this gateway.\")] instance-attribute","text":""},{"location":"api_reference/models/gateways/#optimade_gateway.models.gateways.GatewayResourceAttributes.unique_base_urls","title":"unique_base_urls(value) classmethod","text":"Remove extra entries with repeated base_urls.
Also, ensure databases are not of type \"root\" or \"providers\"
Note
Both \"external\" and \"child\" can still represent index meta-dbs, but \"root\" and \"providers\" can not represent \"regular\" dbs.
optimade_gateway/models/gateways.py @field_validator(\"databases\", mode=\"after\")\n@classmethod\ndef unique_base_urls(cls, value: list[LinksResource]) -> list[LinksResource]:\n \"\"\"Remove extra entries with repeated base_urls.\n\n Also, ensure databases are not of type `\"root\"` or `\"providers\"`\n\n !!! note\n Both `\"external\"` and `\"child\"` can still represent index meta-dbs,\n but `\"root\"` and `\"providers\"` can not represent \"regular\" dbs.\n\n \"\"\"\n for resource in value:\n if resource.attributes.link_type in (LinkType.ROOT, LinkType.PROVIDERS):\n raise ValueError(\n \"Databases with 'root' or 'providers' link_type is not allowed for \"\n f\"gateway resources. Given database: {resource}\"\n )\n\n db_base_urls = [_.attributes.base_url for _ in value]\n unique_base_urls = set(db_base_urls)\n if len(db_base_urls) == len(unique_base_urls):\n return value\n\n repeated_base_urls = [_ for _ in unique_base_urls if db_base_urls.count(_) > 1]\n new_databases = [\n _ for _ in value if _.attributes.base_url not in repeated_base_urls\n ]\n for base_url in repeated_base_urls:\n new_databases.append(\n next(_ for _ in value if _.attributes.base_url == base_url)\n )\n warnings.warn(\n \"Removed extra database entries for a gateway, because the base_url was \"\n \"repeated. The first found database entry was kept, while the others were \"\n f\"removed. Original number of databases: {len(value)}. New number of \"\n f\"databases: {len(new_databases)} Repeated base_urls (number of repeats): \"\n \"{}\".format(\n [\n f\"{base_url} ({db_base_urls.count(base_url)})\"\n for base_url in repeated_base_urls\n ]\n ),\n OptimadeGatewayWarning,\n )\n return new_databases\nPydantic models/schemas for the Queries resource.
"},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QUERY_PARAMETERS","title":"QUERY_PARAMETERS: QueryParameters = {'annotations': {name: FieldInfo.from_annotation(parameter.annotation)for (name, parameter) in inspect.signature(EntryListingQueryParams).parameters.items()}, 'defaults': EntryListingQueryParams()} module-attribute","text":"Entry listing URL query parameters from the optimade package (EntryListingQueryParams).
EndpointEntryType","text":" Bases: Enum
Entry endpoint resource types, mapping to their pydantic models from the optimade package.
optimade_gateway/models/queries.py class EndpointEntryType(Enum):\n \"\"\"Entry endpoint resource types, mapping to their pydantic models from the\n `optimade` package.\"\"\"\n\n REFERENCES = \"references\"\n STRUCTURES = \"structures\"\n\n def get_resource_model(self) -> ReferenceResource | StructureResource:\n \"\"\"Get the matching pydantic model for a resource.\"\"\"\n return {\n \"references\": ReferenceResource,\n \"structures\": StructureResource,\n }[self.value]\n\n def get_response_model(\n self, single: bool = False\n ) -> (\n ReferenceResponseMany\n | ReferenceResponseOne\n | StructureResponseMany\n | StructureResponseOne\n ):\n \"\"\"Get the matching pydantic model for a successful response.\"\"\"\n if single:\n return {\n \"references\": ReferenceResponseOne,\n \"structures\": StructureResponseOne,\n }[self.value]\n return {\n \"references\": ReferenceResponseMany,\n \"structures\": StructureResponseMany,\n }[self.value]\nREFERENCES = 'references' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.EndpointEntryType.STRUCTURES","title":"STRUCTURES = 'structures' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.EndpointEntryType.get_resource_model","title":"get_resource_model()","text":"Get the matching pydantic model for a resource.
Source code inoptimade_gateway/models/queries.py def get_resource_model(self) -> ReferenceResource | StructureResource:\n \"\"\"Get the matching pydantic model for a resource.\"\"\"\n return {\n \"references\": ReferenceResource,\n \"structures\": StructureResource,\n }[self.value]\nget_response_model(single=False)","text":"Get the matching pydantic model for a successful response.
Source code inoptimade_gateway/models/queries.py def get_response_model(\n self, single: bool = False\n) -> (\n ReferenceResponseMany\n | ReferenceResponseOne\n | StructureResponseMany\n | StructureResponseOne\n):\n \"\"\"Get the matching pydantic model for a successful response.\"\"\"\n if single:\n return {\n \"references\": ReferenceResponseOne,\n \"structures\": StructureResponseOne,\n }[self.value]\n return {\n \"references\": ReferenceResponseMany,\n \"structures\": StructureResponseMany,\n }[self.value]\nEntryResource","text":" Bases: EntryResource
Entry Resource ensuring datetimes are not naive.
Source code inoptimade_gateway/models/queries.py class EntryResource(OptimadeEntryResource):\n \"\"\"Entry Resource ensuring datetimes are not naive.\"\"\"\n\n @field_validator(\"attributes\", mode=\"after\")\n @classmethod\n def ensure_non_naive_datetime(\n cls, value: EntryResourceAttributes\n ) -> EntryResourceAttributes:\n \"\"\"Set timezone to UTC if datetime is naive.\"\"\"\n if value.last_modified and value.last_modified.tzinfo is None:\n value.last_modified = value.last_modified.replace(tzinfo=timezone.utc)\n return value\nensure_non_naive_datetime(value) classmethod","text":"Set timezone to UTC if datetime is naive.
Source code inoptimade_gateway/models/queries.py @field_validator(\"attributes\", mode=\"after\")\n@classmethod\ndef ensure_non_naive_datetime(\n cls, value: EntryResourceAttributes\n) -> EntryResourceAttributes:\n \"\"\"Set timezone to UTC if datetime is naive.\"\"\"\n if value.last_modified and value.last_modified.tzinfo is None:\n value.last_modified = value.last_modified.replace(tzinfo=timezone.utc)\n return value\nGatewayQueryResponse","text":" Bases: Response
Response from a Gateway Query.
Source code inoptimade_gateway/models/queries.py class GatewayQueryResponse(Response):\n \"\"\"Response from a Gateway Query.\"\"\"\n\n data: Annotated[\n dict[str, list[EntryResource] | list[dict[str, Any]]],\n StrictField(uniqueItems=True, description=\"Outputted Data.\"),\n ]\n\n meta: Annotated[\n ResponseMeta,\n StrictField(description=\"A meta object containing non-standard information.\"),\n ]\n\n errors: Annotated[\n list[OptimadeError] | None,\n StrictField(\n description=(\n \"A list of OPTIMADE-specific JSON API error objects, where the field \"\n \"detail MUST be present.\"\n ),\n uniqueItems=True,\n ),\n ] = [] # noqa: RUF012\n\n included: Annotated[\n list[EntryResource] | list[dict[str, Any]] | None,\n StrictField(\n description=\"A list of unique included OPTIMADE entry resources.\",\n uniqueItems=True,\n union_mode=\"left_to_right\",\n ),\n ] = None\n\n @model_validator(mode=\"after\")\n def either_data_meta_or_errors_must_be_set(self) -> GatewayQueryResponse:\n \"\"\"Overwrite `either_data_meta_or_errors_must_be_set`.\n\n `errors` should be allowed to be present always for this special response.\n \"\"\"\n return self\ndata: Annotated[dict[str, list[EntryResource] | list[dict[str, Any]]], StrictField(uniqueItems=True, description='Outputted Data.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.errors","title":"errors: Annotated[list[OptimadeError] | None, StrictField(description='A list of OPTIMADE-specific JSON API error objects, where the field detail MUST be present.', uniqueItems=True)] = [] class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.included","title":"included: Annotated[list[EntryResource] | list[dict[str, Any]] | None, StrictField(description='A list of unique included OPTIMADE entry resources.', uniqueItems=True, union_mode='left_to_right')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.meta","title":"meta: Annotated[ResponseMeta, StrictField(description='A meta object containing non-standard information.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.GatewayQueryResponse.either_data_meta_or_errors_must_be_set","title":"either_data_meta_or_errors_must_be_set()","text":"Overwrite either_data_meta_or_errors_must_be_set.
errors should be allowed to be present always for this special response.
optimade_gateway/models/queries.py @model_validator(mode=\"after\")\ndef either_data_meta_or_errors_must_be_set(self) -> GatewayQueryResponse:\n \"\"\"Overwrite `either_data_meta_or_errors_must_be_set`.\n\n `errors` should be allowed to be present always for this special response.\n \"\"\"\n return self\nOptimadeQueryParameters","text":" Bases: BaseModel
Common OPTIMADE entry listing endpoint query parameters.
Source code inoptimade_gateway/models/queries.py class OptimadeQueryParameters(BaseModel):\n \"\"\"Common OPTIMADE entry listing endpoint query parameters.\"\"\"\n\n filter: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"filter\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].filter\n\n response_format: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"response_format\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].response_format\n\n email_address: Annotated[\n EmailStr | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"email_address\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].email_address\n\n response_fields: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"response_fields\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"response_fields\"]\n .metadata[0]\n .pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].response_fields\n\n sort: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"sort\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"sort\"].metadata[0].pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].sort\n\n page_limit: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_limit\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_limit\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_limit\n\n page_offset: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_offset\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_offset\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_offset\n\n page_number: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_number\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_number\n\n page_cursor: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_cursor\"].description,\n ge=QUERY_PARAMETERS[\"annotations\"][\"page_cursor\"].metadata[0].ge,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_cursor\n\n page_above: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_above\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_above\n\n page_below: Annotated[\n int | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"page_below\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].page_below\n\n include: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"include\"].description,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].include\n\n api_hint: Annotated[\n str | None,\n Field(\n description=QUERY_PARAMETERS[\"annotations\"][\"api_hint\"].description,\n pattern=QUERY_PARAMETERS[\"annotations\"][\"api_hint\"].metadata[0].pattern,\n ),\n ] = QUERY_PARAMETERS[\"defaults\"].api_hint\napi_hint: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['api_hint'].description, pattern=QUERY_PARAMETERS['annotations']['api_hint'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].api_hint class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.email_address","title":"email_address: Annotated[EmailStr | None, Field(description=QUERY_PARAMETERS['annotations']['email_address'].description)] = QUERY_PARAMETERS['defaults'].email_address class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.filter","title":"filter: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['filter'].description)] = QUERY_PARAMETERS['defaults'].filter class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.include","title":"include: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['include'].description)] = QUERY_PARAMETERS['defaults'].include class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_above","title":"page_above: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_above'].description)] = QUERY_PARAMETERS['defaults'].page_above class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_below","title":"page_below: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_below'].description)] = QUERY_PARAMETERS['defaults'].page_below class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_cursor","title":"page_cursor: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_cursor'].description, ge=QUERY_PARAMETERS['annotations']['page_cursor'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_cursor class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_limit","title":"page_limit: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_limit'].description, ge=QUERY_PARAMETERS['annotations']['page_limit'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_limit class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_number","title":"page_number: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_number'].description)] = QUERY_PARAMETERS['defaults'].page_number class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.page_offset","title":"page_offset: Annotated[int | None, Field(description=QUERY_PARAMETERS['annotations']['page_offset'].description, ge=QUERY_PARAMETERS['annotations']['page_offset'].metadata[0].ge)] = QUERY_PARAMETERS['defaults'].page_offset class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.response_fields","title":"response_fields: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['response_fields'].description, pattern=QUERY_PARAMETERS['annotations']['response_fields'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].response_fields class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.response_format","title":"response_format: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['response_format'].description)] = QUERY_PARAMETERS['defaults'].response_format class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.OptimadeQueryParameters.sort","title":"sort: Annotated[str | None, Field(description=QUERY_PARAMETERS['annotations']['sort'].description, pattern=QUERY_PARAMETERS['annotations']['sort'].metadata[0].pattern)] = QUERY_PARAMETERS['defaults'].sort class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate","title":"QueryCreate","text":" Bases: EntryResourceCreate, QueryResourceAttributes
Model for creating new Query resources in the MongoDB
Source code inoptimade_gateway/models/queries.py class QueryCreate(EntryResourceCreate, QueryResourceAttributes):\n \"\"\"Model for creating new Query resources in the MongoDB\"\"\"\n\n state: Annotated[\n QueryState | None,\n Field(\n title=QueryResourceAttributes.model_fields[\"state\"].title,\n description=QueryResourceAttributes.model_fields[\"state\"].description,\n json_schema_extra=QueryResourceAttributes.model_fields[\n \"state\"\n ].json_schema_extra,\n ),\n ] = None # type: ignore[assignment]\n endpoint: Annotated[\n EndpointEntryType | None,\n Field(\n title=QueryResourceAttributes.model_fields[\"endpoint\"].title,\n description=QueryResourceAttributes.model_fields[\"endpoint\"].description,\n json_schema_extra=QueryResourceAttributes.model_fields[\n \"endpoint\"\n ].json_schema_extra,\n ),\n ] = None # type: ignore[assignment]\n\n @field_validator(\"query_parameters\", mode=\"after\")\n @classmethod\n def sort_not_supported(\n cls, value: OptimadeQueryParameters\n ) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nendpoint: Annotated[EndpointEntryType | None, Field(title=QueryResourceAttributes.model_fields['endpoint'].title, description=QueryResourceAttributes.model_fields['endpoint'].description, json_schema_extra=QueryResourceAttributes.model_fields['endpoint'].json_schema_extra)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate.state","title":"state: Annotated[QueryState | None, Field(title=QueryResourceAttributes.model_fields['state'].title, description=QueryResourceAttributes.model_fields['state'].description, json_schema_extra=QueryResourceAttributes.model_fields['state'].json_schema_extra)] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryCreate.sort_not_supported","title":"sort_not_supported(value) classmethod","text":"Warn and reset value if sort is supplied.
optimade_gateway/models/queries.py @field_validator(\"query_parameters\", mode=\"after\")\n@classmethod\ndef sort_not_supported(\n cls, value: OptimadeQueryParameters\n) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nQueryParameters","text":" Bases: TypedDict
Type definition for QUERY_PARAMETERS.
optimade_gateway/models/queries.py class QueryParameters(TypedDict):\n \"\"\"Type definition for `QUERY_PARAMETERS`.\"\"\"\n\n annotations: dict[str, FieldInfo]\n defaults: EntryListingQueryParams\nannotations: dict[str, FieldInfo] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryParameters.defaults","title":"defaults: EntryListingQueryParams instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource","title":"QueryResource","text":" Bases: EntryResource
OPTIMADE query resource for a gateway
Source code inoptimade_gateway/models/queries.py class QueryResource(EntryResource):\n \"\"\"OPTIMADE query resource for a gateway\"\"\"\n\n type: Annotated[\n Literal[\"queries\"],\n Field(\n description=\"The name of the type of an entry.\",\n ),\n ] = \"queries\"\n\n attributes: QueryResourceAttributes\n\n async def response_as_optimade(\n self,\n url: None | (\n urllib.parse.ParseResult | urllib.parse.SplitResult | StarletteURL | str\n ) = None,\n ) -> EntryResponseMany | ErrorResponse:\n \"\"\"Return `attributes.response` as a valid OPTIMADE entry listing response.\n\n Note, this method disregards the state of the query and will simply return the\n query results as they currently are (if there are any at all).\n\n Parameters:\n url: Optionally, update the `meta.query.representation` value with this.\n\n Returns:\n A valid OPTIMADE entry-listing response according to the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints)\n or an error response, if errors were returned or occurred during the query.\n\n \"\"\"\n from optimade.server.routers.utils import (\n meta_values,\n )\n\n async def _update_id(\n entry_: EntryResource | dict[str, Any], database_provider_: str\n ) -> EntryResource | dict[str, Any]:\n \"\"\"Internal utility function to prepend the entries' `id` with\n `provider/database/`.\n\n Parameters:\n entry_: The entry as a model or a dictionary.\n database_provider_: `provider/database` string.\n\n Returns:\n The entry with an updated `id` value.\n\n \"\"\"\n if isinstance(entry_, dict):\n _entry = deepcopy(entry_)\n _entry[\"id\"] = f\"{database_provider_}/{entry_['id']}\"\n return _entry\n\n return entry_.model_copy(\n update={\"id\": f\"{database_provider_}/{entry_.id}\"},\n deep=True,\n ).model_dump(exclude_unset=True, exclude_none=True)\n\n if not self.attributes.response:\n # The query has not yet been initiated\n return ErrorResponse(\n errors=[\n {\n \"detail\": (\n \"Can not return as a valid OPTIMADE response as the query \"\n \"has not yet been initialized.\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_QUERY_NOT_INITIALIZED\",\n }\n ],\n meta=meta_values(\n url=url or f\"/queries/{self.id}?\",\n data_returned=0,\n data_available=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n meta_ = self.attributes.response.meta\n\n if url:\n meta_ = meta_.model_dump(exclude_unset=True)\n for repeated_key in (\n \"query\",\n \"api_version\",\n \"time_stamp\",\n \"provider\",\n \"implementation\",\n ):\n meta_.pop(repeated_key, None)\n meta_ = meta_values(url=url, **meta_)\n\n # Error response\n if self.attributes.response.errors:\n return ErrorResponse(\n errors=self.attributes.response.errors,\n meta=meta_,\n )\n\n # Data response\n results = []\n for database_provider, entries in self.attributes.response.data.items():\n results.extend(\n [await _update_id(entry, database_provider) for entry in entries]\n )\n\n return self.attributes.endpoint.get_response_model()(\n data=results,\n meta=meta_,\n links=self.attributes.response.links,\n )\nattributes: QueryResourceAttributes instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource.type","title":"type: Annotated[Literal['queries'], Field(description='The name of the type of an entry.')] = 'queries' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResource.response_as_optimade","title":"response_as_optimade(url=None) async","text":"Return attributes.response as a valid OPTIMADE entry listing response.
Note, this method disregards the state of the query and will simply return the query results as they currently are (if there are any at all).
Parameters:
Name Type Description Defaulturl None | ParseResult | SplitResult | URL | str Optionally, update the meta.query.representation value with this.
None Returns:
Type DescriptionEntryResponseMany | ErrorResponse A valid OPTIMADE entry-listing response according to the
EntryResponseMany | ErrorResponse OPTIMADE specification
EntryResponseMany | ErrorResponse or an error response, if errors were returned or occurred during the query.
Source code inoptimade_gateway/models/queries.py async def response_as_optimade(\n self,\n url: None | (\n urllib.parse.ParseResult | urllib.parse.SplitResult | StarletteURL | str\n ) = None,\n) -> EntryResponseMany | ErrorResponse:\n \"\"\"Return `attributes.response` as a valid OPTIMADE entry listing response.\n\n Note, this method disregards the state of the query and will simply return the\n query results as they currently are (if there are any at all).\n\n Parameters:\n url: Optionally, update the `meta.query.representation` value with this.\n\n Returns:\n A valid OPTIMADE entry-listing response according to the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints)\n or an error response, if errors were returned or occurred during the query.\n\n \"\"\"\n from optimade.server.routers.utils import (\n meta_values,\n )\n\n async def _update_id(\n entry_: EntryResource | dict[str, Any], database_provider_: str\n ) -> EntryResource | dict[str, Any]:\n \"\"\"Internal utility function to prepend the entries' `id` with\n `provider/database/`.\n\n Parameters:\n entry_: The entry as a model or a dictionary.\n database_provider_: `provider/database` string.\n\n Returns:\n The entry with an updated `id` value.\n\n \"\"\"\n if isinstance(entry_, dict):\n _entry = deepcopy(entry_)\n _entry[\"id\"] = f\"{database_provider_}/{entry_['id']}\"\n return _entry\n\n return entry_.model_copy(\n update={\"id\": f\"{database_provider_}/{entry_.id}\"},\n deep=True,\n ).model_dump(exclude_unset=True, exclude_none=True)\n\n if not self.attributes.response:\n # The query has not yet been initiated\n return ErrorResponse(\n errors=[\n {\n \"detail\": (\n \"Can not return as a valid OPTIMADE response as the query \"\n \"has not yet been initialized.\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_QUERY_NOT_INITIALIZED\",\n }\n ],\n meta=meta_values(\n url=url or f\"/queries/{self.id}?\",\n data_returned=0,\n data_available=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n meta_ = self.attributes.response.meta\n\n if url:\n meta_ = meta_.model_dump(exclude_unset=True)\n for repeated_key in (\n \"query\",\n \"api_version\",\n \"time_stamp\",\n \"provider\",\n \"implementation\",\n ):\n meta_.pop(repeated_key, None)\n meta_ = meta_values(url=url, **meta_)\n\n # Error response\n if self.attributes.response.errors:\n return ErrorResponse(\n errors=self.attributes.response.errors,\n meta=meta_,\n )\n\n # Data response\n results = []\n for database_provider, entries in self.attributes.response.data.items():\n results.extend(\n [await _update_id(entry, database_provider) for entry in entries]\n )\n\n return self.attributes.endpoint.get_response_model()(\n data=results,\n meta=meta_,\n links=self.attributes.response.links,\n )\nQueryResourceAttributes","text":" Bases: EntryResourceAttributes
Attributes for an OPTIMADE gateway query.
Source code inoptimade_gateway/models/queries.py class QueryResourceAttributes(EntryResourceAttributes):\n \"\"\"Attributes for an OPTIMADE gateway query.\"\"\"\n\n gateway_id: Annotated[\n str,\n Field(\n description=\"The OPTIMADE gateway ID for this query.\",\n ),\n ]\n\n query_parameters: Annotated[\n OptimadeQueryParameters,\n Field(\n description=(\n \"OPTIMADE query parameters for entry listing endpoints used for this \"\n \"query.\"\n ),\n json_schema_extra={\n \"type\": \"object\",\n },\n ),\n ]\n\n state: Annotated[\n QueryState,\n Field(\n description=\"Current state of Gateway Query.\",\n title=\"State\",\n json_schema_extra={\n \"type\": \"enum\",\n },\n ),\n ] = QueryState.CREATED\n\n response: Annotated[\n GatewayQueryResponse | None,\n Field(\n description=\"Response from gateway query.\",\n ),\n ] = None\n\n endpoint: Annotated[\n EndpointEntryType,\n Field(\n description=\"The entry endpoint queried, e.g., 'structures'.\",\n title=\"Endpoint\",\n json_schema_extra={\n \"type\": \"enum\",\n },\n ),\n ] = EndpointEntryType.STRUCTURES\n\n @field_validator(\"endpoint\", mode=\"after\")\n @classmethod\n def only_allow_structures(cls, value: EndpointEntryType) -> EndpointEntryType:\n \"\"\"Temporarily only allow queries to \"structures\" endpoints.\"\"\"\n if value != EndpointEntryType.STRUCTURES:\n raise NotImplementedError(\n 'OPTIMADE Gateway temporarily only supports queries to \"structures\" '\n 'endpoints, i.e.: endpoint=\"structures\"'\n )\n return value\nendpoint: Annotated[EndpointEntryType, Field(description=\"The entry endpoint queried, e.g., 'structures'.\", title='Endpoint', json_schema_extra={'type': 'enum'})] = EndpointEntryType.STRUCTURES class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.gateway_id","title":"gateway_id: Annotated[str, Field(description='The OPTIMADE gateway ID for this query.')] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.query_parameters","title":"query_parameters: Annotated[OptimadeQueryParameters, Field(description='OPTIMADE query parameters for entry listing endpoints used for this query.', json_schema_extra={'type': 'object'})] instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.response","title":"response: Annotated[GatewayQueryResponse | None, Field(description='Response from gateway query.')] = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.state","title":"state: Annotated[QueryState, Field(description='Current state of Gateway Query.', title='State', json_schema_extra={'type': 'enum'})] = QueryState.CREATED class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryResourceAttributes.only_allow_structures","title":"only_allow_structures(value) classmethod","text":"Temporarily only allow queries to \"structures\" endpoints.
Source code inoptimade_gateway/models/queries.py @field_validator(\"endpoint\", mode=\"after\")\n@classmethod\ndef only_allow_structures(cls, value: EndpointEntryType) -> EndpointEntryType:\n \"\"\"Temporarily only allow queries to \"structures\" endpoints.\"\"\"\n if value != EndpointEntryType.STRUCTURES:\n raise NotImplementedError(\n 'OPTIMADE Gateway temporarily only supports queries to \"structures\" '\n 'endpoints, i.e.: endpoint=\"structures\"'\n )\n return value\nQueryState","text":" Bases: Enum
Enumeration of possible states for a Gateway Query.
The states are enumerated here in the expected evolvement.
Source code inoptimade_gateway/models/queries.py class QueryState(Enum):\n \"\"\"Enumeration of possible states for a Gateway Query.\n\n The states are enumerated here in the expected evolvement.\n \"\"\"\n\n CREATED = \"created\"\n STARTED = \"started\"\n IN_PROGRESS = \"in progress\"\n FINISHED = \"finished\"\nCREATED = 'created' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.FINISHED","title":"FINISHED = 'finished' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.IN_PROGRESS","title":"IN_PROGRESS = 'in progress' class-attribute instance-attribute","text":""},{"location":"api_reference/models/queries/#optimade_gateway.models.queries.QueryState.STARTED","title":"STARTED = 'started' class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/","title":"resources","text":"Pydantic models/schemas for entry-endpoint resources.
This module is mainly used for a special pydantic base model, which can be used as a mix-in class when creating entry-endpoint resources.
"},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate","title":"EntryResourceCreate","text":" Bases: EntryResourceAttributes
Generic model for creating new entry resources in the MongoDB
Source code inoptimade_gateway/models/resources.py class EntryResourceCreate(EntryResourceAttributes):\n \"\"\"Generic model for creating new entry resources in the MongoDB\"\"\"\n\n model_config = ConfigDict(extra=\"ignore\")\n\n last_modified: datetime | None = None\n\n id: str | None = None\n\n @model_validator(mode=\"after\")\n def check_illegal_attributes_fields(self) -> EntryResourceCreate:\n \"\"\"Overwrite parental `check_illegal_attributes_fields` class validators.\"\"\"\n return self\nid: str | None = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.last_modified","title":"last_modified: datetime | None = None class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.model_config","title":"model_config = ConfigDict(extra='ignore') class-attribute instance-attribute","text":""},{"location":"api_reference/models/resources/#optimade_gateway.models.resources.EntryResourceCreate.check_illegal_attributes_fields","title":"check_illegal_attributes_fields()","text":"Overwrite parental check_illegal_attributes_fields class validators.
optimade_gateway/models/resources.py @model_validator(mode=\"after\")\ndef check_illegal_attributes_fields(self) -> EntryResourceCreate:\n \"\"\"Overwrite parental `check_illegal_attributes_fields` class validators.\"\"\"\n return self\nPydantic models/schemas for the API responses.
"},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.DatabasesResponse","title":"DatabasesResponse","text":" Bases: EntryResponseMany
Successful response for GET /databases
This model is essentially equal to LinksResponse with the exception of the data field's description.
optimade_gateway/models/responses.py class DatabasesResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /databases`\n\n This model is essentially equal to\n [`LinksResponse`](https://www.optimade.org/optimade-python-tools/api_reference/models/responses/#optimade.models.responses.LinksResponse)\n with the exception of the `data` field's description.\n \"\"\"\n\n data: Annotated[\n list[LinksResource],\n StrictField(\n description=(\n \"List of unique OPTIMADE links resource objects.\\nThese links resource \"\n \"objects represents OPTIMADE databases that can be used for queries in \"\n \"gateways.\"\n ),\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[LinksResource], StrictField(description='List of unique OPTIMADE links resource objects.\\nThese links resource objects represents OPTIMADE databases that can be used for queries in gateways.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.DatabasesResponseSingle","title":"DatabasesResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /databases and GET /databases/{database_id}
optimade_gateway/models/responses.py class DatabasesResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /databases` and `GET /databases/{database_id}`\"\"\"\n\n data: Annotated[\n LinksResource | None,\n Field(\n description=(\n \"A unique OPTIMADE links resource object.\\nThe OPTIMADE links resource \"\n \"object has just been created or found according to the specific query \"\n \"parameter(s) or URL id.\\nIt represents an OPTIMADE database that can \"\n \"be used for queries in gateways.\"\n ),\n ),\n ]\ndata: Annotated[LinksResource | None, Field(description='A unique OPTIMADE links resource object.\\nThe OPTIMADE links resource object has just been created or found according to the specific query parameter(s) or URL id.\\nIt represents an OPTIMADE database that can be used for queries in gateways.')] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.GatewaysResponse","title":"GatewaysResponse","text":" Bases: EntryResponseMany
Successful response for GET /gateways
optimade_gateway/models/responses.py class GatewaysResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /gateways`\"\"\"\n\n data: Annotated[\n list[GatewayResource],\n StrictField(\n description=\"List of unique OPTIMADE gateway resource objects.\",\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[GatewayResource], StrictField(description='List of unique OPTIMADE gateway resource objects.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.GatewaysResponseSingle","title":"GatewaysResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /gateways and GET /gateways/{gateway_id}.
optimade_gateway/models/responses.py class GatewaysResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /gateways` and `GET /gateways/{gateway_id}`.\"\"\"\n\n data: Annotated[\n GatewayResource | None,\n Field(\n description=(\n \"A unique OPTIMADE gateway resource object.\\nThe OPTIMADE gateway \"\n \"resource object has just been created or found according to the \"\n \"specific query parameter(s) or URL id.\"\n ),\n ),\n ]\ndata: Annotated[GatewayResource | None, Field(description='A unique OPTIMADE gateway resource object.\\nThe OPTIMADE gateway resource object has just been created or found according to the specific query parameter(s) or URL id.')] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.QueriesResponse","title":"QueriesResponse","text":" Bases: EntryResponseMany
Successful response for GET /gateways/{gateway_ID}/queries.
optimade_gateway/models/responses.py class QueriesResponse(EntryResponseMany):\n \"\"\"Successful response for `GET /gateways/{gateway_ID}/queries`.\"\"\"\n\n data: Annotated[\n list[QueryResource],\n StrictField(\n description=\"List of unique OPTIMADE gateway query resource objects.\",\n uniqueItems=True,\n ),\n ]\ndata: Annotated[list[QueryResource], StrictField(description='List of unique OPTIMADE gateway query resource objects.', uniqueItems=True)] instance-attribute","text":""},{"location":"api_reference/models/responses/#optimade_gateway.models.responses.QueriesResponseSingle","title":"QueriesResponseSingle","text":" Bases: EntryResponseOne
Successful response for POST /gateways/{gateway_ID}/queries and GET /gateways/{gateway_ID}/queries/{query_id}.
optimade_gateway/models/responses.py class QueriesResponseSingle(EntryResponseOne):\n \"\"\"Successful response for `POST /gateways/{gateway_ID}/queries`\n and `GET /gateways/{gateway_ID}/queries/{query_id}`.\"\"\"\n\n data: Annotated[\n QueryResource | None,\n Field(\n description=(\n \"A unique OPTIMADE gateway query resource object.\\nThe OPTIMADE \"\n \"gateway query resource object has just been created or found \"\n \"according to the specific query parameter(s) or URL id.\"\n ),\n ),\n ]\ndata: Annotated[QueryResource | None, Field(description='A unique OPTIMADE gateway query resource object.\\nThe OPTIMADE gateway query resource object has just been created or found according to the specific query parameter(s) or URL id.')] instance-attribute","text":""},{"location":"api_reference/models/search/","title":"search","text":"Pydantic models/schemas for the Search resource.
"},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search","title":"Search","text":" Bases: BaseModel
A general coordinated OPTIMADE search
Important
Either database_ids or optimade_urls MUST be specified.
optimade_gateway/models/search.py class Search(BaseModel):\n \"\"\"A general coordinated OPTIMADE search\n\n !!! important\n Either `database_ids` or `optimade_urls` MUST be specified.\n\n \"\"\"\n\n query_parameters: Annotated[\n OptimadeQueryParameters,\n Field(\n description=(\n \"OPTIMADE query parameters for entry listing endpoints used for this \"\n \"query.\"\n ),\n ),\n ] = OptimadeQueryParameters()\n\n database_ids: Annotated[\n set[str],\n Field(\n description=(\n \"A list of registered database IDs. Go to `/databases` to get all \"\n \"registered databases.\"\n ),\n ),\n ] = set()\n\n optimade_urls: Annotated[\n set[AnyUrl],\n Field(\n description=(\n \"A list of OPTIMADE base URLs. If a versioned base URL is supplied it \"\n \"will be used as is, as long as it represents a supported version. If \"\n \"an un-versioned base URL, standard version negotiation will be \"\n \"conducted to get the versioned base URL, which will be used as long \"\n \"as it represents a supported version. Note, a single URL can be \"\n \"supplied as well, and it will automatically be wrapped in a list in \"\n \"the server logic.\"\n ),\n ),\n ] = set()\n\n endpoint: Annotated[\n str,\n Field(\n description=(\n \"The entry endpoint queried. According to the OPTIMADE specification, \"\n \"this is the same as the resource's type.\"\n ),\n ),\n ] = \"structures\"\n\n @model_validator(mode=\"after\")\n def either_ids_or_urls(self) -> Search:\n \"\"\"Either `database_ids` or `optimade_urls` must be defined\"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"optimade_urls\")):\n raise ValueError(\n \"Either 'database_ids' or 'optimade_urls' MUST be specified.\"\n )\n return self\n\n @field_validator(\"query_parameters\", mode=\"after\")\n @classmethod\n def sort_not_supported(\n cls, value: OptimadeQueryParameters\n ) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\ndatabase_ids: Annotated[set[str], Field(description='A list of registered database IDs. Go to `/databases` to get all registered databases.')] = set() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.endpoint","title":"endpoint: Annotated[str, Field(description=\"The entry endpoint queried. According to the OPTIMADE specification, this is the same as the resource's type.\")] = 'structures' class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.optimade_urls","title":"optimade_urls: Annotated[set[AnyUrl], Field(description='A list of OPTIMADE base URLs. If a versioned base URL is supplied it will be used as is, as long as it represents a supported version. If an un-versioned base URL, standard version negotiation will be conducted to get the versioned base URL, which will be used as long as it represents a supported version. Note, a single URL can be supplied as well, and it will automatically be wrapped in a list in the server logic.')] = set() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.query_parameters","title":"query_parameters: Annotated[OptimadeQueryParameters, Field(description='OPTIMADE query parameters for entry listing endpoints used for this query.')] = OptimadeQueryParameters() class-attribute instance-attribute","text":""},{"location":"api_reference/models/search/#optimade_gateway.models.search.Search.either_ids_or_urls","title":"either_ids_or_urls()","text":"Either database_ids or optimade_urls must be defined
optimade_gateway/models/search.py @model_validator(mode=\"after\")\ndef either_ids_or_urls(self) -> Search:\n \"\"\"Either `database_ids` or `optimade_urls` must be defined\"\"\"\n if not any(getattr(self, field) for field in (\"database_ids\", \"optimade_urls\")):\n raise ValueError(\n \"Either 'database_ids' or 'optimade_urls' MUST be specified.\"\n )\n return self\nsort_not_supported(value) classmethod","text":"Warn and reset value if sort is supplied.
optimade_gateway/models/search.py @field_validator(\"query_parameters\", mode=\"after\")\n@classmethod\ndef sort_not_supported(\n cls, value: OptimadeQueryParameters\n) -> OptimadeQueryParameters:\n \"\"\"Warn and reset value if `sort` is supplied.\"\"\"\n if value.sort:\n warnings.warn(SortNotSupported())\n value.sort = None\n return value\nMongoDB collection for entry-endpoint resources.
The AsyncMongoCollection represents an asynchronous version of the equivalent MongoDB collection in optimade: MongoCollection.
AsyncMongoCollection","text":" Bases: EntryCollection
MongoDB Collection for use with asyncio
The asynchronicity is implemented using motor and asyncio.
optimade_gateway/mongo/collection.py class AsyncMongoCollection(EntryCollection):\n \"\"\"MongoDB Collection for use with `asyncio`\n\n The asynchronicity is implemented using [`motor`](https://motor.readthedocs.io) and\n [`asyncio`](https://asyncio.readthedocs.io/).\n \"\"\"\n\n def __init__(\n self,\n name: str,\n resource_cls: EntryResource,\n resource_mapper: BaseResourceMapper,\n ):\n \"\"\"Initialize the AsyncMongoCollection for the given parameters.\n\n Parameters:\n name: The name of the collection.\n resource_cls: The `EntryResource` model that is stored by the collection.\n resource_mapper: A resource mapper object that handles aliases and format\n changes between deserialization and response.\n\n \"\"\"\n from optimade_gateway.mongo.database import (\n MONGO_DB,\n )\n\n super().__init__(\n resource_cls=resource_cls,\n resource_mapper=resource_mapper,\n transformer=MongoTransformer(mapper=resource_mapper),\n )\n\n self.parser = LarkParser(version=(1, 0, 0), variant=\"default\")\n self.collection: MongoCollection = MONGO_DB[name]\n\n # Check aliases do not clash with mongo operators\n self._check_aliases(self.resource_mapper.all_aliases())\n self._check_aliases(self.resource_mapper.all_length_aliases())\n\n def __str__(self) -> str:\n \"\"\"Standard printing result for an instance.\"\"\"\n return (\n f\"<{self.__class__.__name__}: resource={self.resource_cls.__name__} \"\n f\"endpoint(mapper)={self.resource_mapper.ENDPOINT} \"\n f\"DB_collection={self.collection.name}>\"\n )\n\n def __repr__(self) -> str:\n \"\"\"Representation of instance.\"\"\"\n return (\n f\"{self.__class__.__name__}(name={self.collection.name!r}, \"\n f\"resource_cls={self.resource_cls!r}, \"\n f\"resource_mapper={self.resource_mapper!r})\"\n )\n\n def __len__(self) -> int:\n warn(\n OptimadeGatewayWarning(\n detail=(\n \"Cannot calculate length of collection using `len()`. Use \"\n \"`count()` instead.\"\n )\n )\n )\n return 0\n\n def insert(self, data: list[EntryResource]) -> None:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `ainsert(data: \"\n \"List[EntryResource])`.\"\n )\n\n async def ainsert(self, data: list[EntryResource]) -> None:\n \"\"\"Add the given entries to the underlying database.\n\n This is the asynchronous version of the parent class method named `insert()`.\n\n Arguments:\n data: The entry resource objects to add to the database.\n\n \"\"\"\n await self.collection.insert_many(await clean_python_types(data))\n\n def count(self, **kwargs) -> int:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `acount(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"**kwargs)`.\"\n )\n\n async def acount(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n **kwargs: Any,\n ) -> int:\n \"\"\"Count documents in Collection.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n **kwargs: Query parameters as keyword arguments. Valid keys will be passed\n to the\n [`AsyncIOMotorCollection.count_documents`](https://motor.readthedocs.io/en/stable/api-asyncio/asyncio_motor_collection.html#motor.motor_asyncio.AsyncIOMotorCollection.count_documents)\n method.\n\n Returns:\n int: The number of entries matching the query specified by the keyword\n arguments.\n\n \"\"\"\n if params is not None and kwargs:\n raise ValueError(\n \"When 'params' is supplied, no other parameters can be supplied.\"\n )\n\n if params is not None:\n kwargs = await self.ahandle_query_params(params)\n\n valid_method_keys = (\n \"filter\",\n \"skip\",\n \"limit\",\n \"hint\",\n \"maxTimeMS\",\n \"collation\",\n \"session\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n criteria[\"filter\"] = {}\n\n return await self.collection.count_documents(**criteria)\n\n def find(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n ]:\n \"\"\"\n Fetches results and indicates if more data is available.\n\n Also gives the total number of data available in the absence of `page_limit`.\n See\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n for more information.\n\n Parameters:\n params: Entry listing URL query params.\n\n Returns:\n A tuple of various relevant values:\n (`results`, `data_returned`, `more_data_available`, `exclude_fields`,\n `include_fields`).\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `afind(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"criteria: Optional[Dict[str, Any]])`.\"\n )\n\n async def afind(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n criteria: dict[str, Any] | None = None,\n ) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n ]:\n \"\"\"Perform the query on the underlying MongoDB Collection, handling projection\n and pagination of the output.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Either provide `params` or `criteria`. Not both, but at least one.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of entry resource objects, how much data was returned for the query,\n whether more data is available with pagination, and fields (excluded,\n included).\n\n \"\"\"\n if (params is None and criteria is None) or (\n params is not None and criteria is not None\n ):\n raise ValueError(\n \"Exacly one of either `params` and `criteria` must be specified.\"\n )\n\n # Set single_entry to False, this is done since if criteria is defined,\n # this is an unknown factor - better to then get a list of results.\n single_entry = False\n if criteria is None:\n criteria = await self.ahandle_query_params(params)\n else:\n single_entry = isinstance(params, SingleEntryQueryParams)\n\n response_fields = criteria.pop(\"fields\", self.all_fields)\n\n results, data_returned, more_data_available = await self._arun_db_query(\n criteria=criteria,\n single_entry=single_entry,\n )\n\n if single_entry:\n results = results[0] if results else None # type: ignore[assignment]\n\n if data_returned > 1:\n raise NotFound(\n detail=(\n f\"Instead of a single entry, {data_returned} entries were found\"\n ),\n )\n\n include_fields = (\n response_fields - self.resource_mapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS\n )\n bad_optimade_fields = set()\n bad_provider_fields = set()\n for field in include_fields:\n if field not in self.resource_mapper.ALL_ATTRIBUTES:\n if field.startswith(\"_\"):\n if any(\n field.startswith(f\"_{prefix}_\")\n for prefix in self.resource_mapper.SUPPORTED_PREFIXES\n ):\n bad_provider_fields.add(field)\n else:\n bad_optimade_fields.add(field)\n\n if bad_provider_fields:\n warn(\n UnknownProviderProperty(\n detail=(\n \"Unrecognised field(s) for this provider requested in \"\n f\"`response_fields`: {bad_provider_fields}.\"\n )\n )\n )\n\n if bad_optimade_fields:\n raise BadRequest(\n detail=(\n \"Unrecognised OPTIMADE field(s) in requested `response_fields`: \"\n f\"{bad_optimade_fields}.\"\n )\n )\n\n if results:\n results = await self.resource_mapper.adeserialize(results)\n\n return ( # type: ignore[return-value]\n results,\n data_returned,\n more_data_available,\n self.all_fields - response_fields,\n include_fields,\n )\n\n def handle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields\n or response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`ahandle_query_params(params: Union[EntryListingQueryParams, \"\n \"SingleEntryQueryParams])`.\"\n )\n\n async def ahandle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n ) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n This is the asynchronous version of the parent class method named\n `handle_query_params()`.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields or\n response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n return super().handle_query_params(params)\n\n def _run_db_query(\n self, criteria: dict[str, Any], single_entry: bool = False\n ) -> tuple[list[dict[str, Any]], int, bool]:\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`_arun_db_query(criteria: Dict[str, Any], single_entry: bool)`.\"\n )\n\n async def _arun_db_query(\n self, criteria: dict[str, Any], single_entry: bool = False\n ) -> tuple[list[dict[str, Any]], int, bool]:\n \"\"\"Run the query on the backend and collect the results.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Arguments:\n criteria: A dictionary representation of the query parameters.\n single_entry: Whether or not the caller is expecting a single entry\n response.\n\n Returns:\n The list of entries from the database (without any re-mapping), the total\n number of entries matching the query and a boolean for whether or not there\n is more data available.\n\n \"\"\"\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n if criteria.get(\"projection\", {}).get(\"_id\"):\n document[\"_id\"] = str(document[\"_id\"])\n results.append(document)\n\n if single_entry:\n data_returned = len(results)\n more_data_available = False\n else:\n criteria_nolimit = criteria.copy()\n criteria_nolimit.pop(\"limit\", None)\n data_returned = await self.acount(params=None, **criteria_nolimit)\n more_data_available = len(results) < data_returned\n\n return results, data_returned, more_data_available\n\n @staticmethod\n def _check_aliases(aliases: tuple[tuple[str, str]]) -> None:\n \"\"\"Check that aliases do not clash with mongo keywords.\n\n Parameters:\n aliases: Tuple of tuple of aliases to be checked.\n\n Raises:\n RuntimeError: If any alias starts with the dollar (`$`) character.\n\n \"\"\"\n if any(\n alias[0].startswith(\"$\") or alias[1].startswith(\"$\") for alias in aliases\n ):\n raise RuntimeError(f\"Cannot define an alias starting with a '$': {aliases}\")\n\n async def get_one(self, **criteria: Any) -> EntryResource:\n \"\"\"Get one resource based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a single resource.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A single resource from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one(**self._valid_find_keys(**criteria))\n )\n )\n\n async def get_multiple(self, **criteria: Any) -> list[EntryResource]:\n \"\"\"Get a list of resources based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a list of resources.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of resources from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n results.append(self.resource_cls(**self.resource_mapper.map_back(document)))\n\n return results\n\n async def create_one(self, resource: EntryResourceCreate) -> EntryResource:\n \"\"\"Create a new document in the MongoDB collection based on query parameters.\n\n Update the newly created document with an `\"id\"` field.\n The value will be the string representation of the `\"_id\"` field.\n This will only be done if `\"id\"` is not already present in `resource`.\n\n Parameters:\n resource: The resource to be created.\n\n Returns:\n The newly created document as a pydantic model entry resource.\n\n \"\"\"\n resource.last_modified = datetime.now(timezone.utc)\n result = await self.collection.insert_one(\n await clean_python_types(resource.model_dump(exclude_unset=True))\n )\n LOGGER.debug(\n \"Inserted resource %r in DB collection %s with ID %s\",\n resource,\n self.collection.name,\n result.inserted_id,\n )\n\n if not resource.id:\n LOGGER.debug(\"Updating resource with an `id` field equal to str(id_).\")\n await self.collection.update_one(\n {\"_id\": result.inserted_id}, {\"$set\": {\"id\": str(result.inserted_id)}}\n )\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one({\"_id\": result.inserted_id})\n )\n )\n\n async def exists(self, entry_id: str) -> bool:\n \"\"\"Assert whether entry_id exists in the collection (value of `\"id\"`)\n\n Parameters:\n entry_id: The `\"id\"` value of the entry.\n\n \"\"\"\n return bool(await self.collection.count_documents({\"id\": entry_id}))\n\n @staticmethod\n def _valid_find_keys(**kwargs: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Return valid MongoDB find() keys with values from kwargs\n\n Note, not including deprecated flags\n (see https://pymongo.readthedocs.io/en/3.11.0/api/pymongo/collection.html#pymongo.collection.Collection.find).\n \"\"\"\n valid_method_keys = (\n \"filter\",\n \"projection\",\n \"session\",\n \"skip\",\n \"limit\",\n \"no_cursor_timeout\",\n \"cursor_type\",\n \"sort\",\n \"allow_partial_results\",\n \"batch_size\",\n \"collation\",\n \"return_key\",\n \"show_record_id\",\n \"hint\",\n \"max_time_ms\",\n \"min\",\n \"max\",\n \"comment\",\n \"allow_disk_use\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n # Ensure documents are included in the result set\n criteria[\"filter\"] = {}\n\n return criteria\n__init__(name, resource_cls, resource_mapper)","text":"Initialize the AsyncMongoCollection for the given parameters.
Parameters:
Name Type Description Defaultname str The name of the collection.
requiredresource_cls EntryResource The EntryResource model that is stored by the collection.
resource_mapper BaseResourceMapper A resource mapper object that handles aliases and format changes between deserialization and response.
required Source code inoptimade_gateway/mongo/collection.py def __init__(\n self,\n name: str,\n resource_cls: EntryResource,\n resource_mapper: BaseResourceMapper,\n):\n \"\"\"Initialize the AsyncMongoCollection for the given parameters.\n\n Parameters:\n name: The name of the collection.\n resource_cls: The `EntryResource` model that is stored by the collection.\n resource_mapper: A resource mapper object that handles aliases and format\n changes between deserialization and response.\n\n \"\"\"\n from optimade_gateway.mongo.database import (\n MONGO_DB,\n )\n\n super().__init__(\n resource_cls=resource_cls,\n resource_mapper=resource_mapper,\n transformer=MongoTransformer(mapper=resource_mapper),\n )\n\n self.parser = LarkParser(version=(1, 0, 0), variant=\"default\")\n self.collection: MongoCollection = MONGO_DB[name]\n\n # Check aliases do not clash with mongo operators\n self._check_aliases(self.resource_mapper.all_aliases())\n self._check_aliases(self.resource_mapper.all_length_aliases())\nacount(params=None, **kwargs) async","text":"Count documents in Collection.
This is the asynchronous version of the parent class method named count().
Parameters:
Name Type Description Defaultparams None | EntryListingQueryParams | SingleEntryQueryParams URL query parameters, either from a general entry endpoint or a single-entry endpoint.
None **kwargs Any Query parameters as keyword arguments. Valid keys will be passed to the AsyncIOMotorCollection.count_documents method.
{} Returns:
Name Type Descriptionint int The number of entries matching the query specified by the keyword arguments.
Source code inoptimade_gateway/mongo/collection.py async def acount(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n **kwargs: Any,\n) -> int:\n \"\"\"Count documents in Collection.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n **kwargs: Query parameters as keyword arguments. Valid keys will be passed\n to the\n [`AsyncIOMotorCollection.count_documents`](https://motor.readthedocs.io/en/stable/api-asyncio/asyncio_motor_collection.html#motor.motor_asyncio.AsyncIOMotorCollection.count_documents)\n method.\n\n Returns:\n int: The number of entries matching the query specified by the keyword\n arguments.\n\n \"\"\"\n if params is not None and kwargs:\n raise ValueError(\n \"When 'params' is supplied, no other parameters can be supplied.\"\n )\n\n if params is not None:\n kwargs = await self.ahandle_query_params(params)\n\n valid_method_keys = (\n \"filter\",\n \"skip\",\n \"limit\",\n \"hint\",\n \"maxTimeMS\",\n \"collation\",\n \"session\",\n )\n criteria = {key: kwargs[key] for key in valid_method_keys if key in kwargs}\n\n if criteria.get(\"filter\") is None:\n criteria[\"filter\"] = {}\n\n return await self.collection.count_documents(**criteria)\nafind(params=None, criteria=None) async","text":"Perform the query on the underlying MongoDB Collection, handling projection and pagination of the output.
This is the asynchronous version of the parent class method named count().
Either provide params or criteria. Not both, but at least one.
Parameters:
Name Type Description Defaultparams None | EntryListingQueryParams | SingleEntryQueryParams URL query parameters, either from a general entry endpoint or a single-entry endpoint.
None criteria dict[str, Any] | None Already handled/parsed URL query parameters.
None Returns:
Type Descriptionlist[EntryResource] | EntryResource | None A list of entry resource objects, how much data was returned for the query,
int whether more data is available with pagination, and fields (excluded,
bool included).
Source code inoptimade_gateway/mongo/collection.py async def afind(\n self,\n params: None | (EntryListingQueryParams | SingleEntryQueryParams) = None,\n criteria: dict[str, Any] | None = None,\n) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n]:\n \"\"\"Perform the query on the underlying MongoDB Collection, handling projection\n and pagination of the output.\n\n This is the asynchronous version of the parent class method named `count()`.\n\n Either provide `params` or `criteria`. Not both, but at least one.\n\n Parameters:\n params: URL query parameters, either from a general entry endpoint or a\n single-entry endpoint.\n criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of entry resource objects, how much data was returned for the query,\n whether more data is available with pagination, and fields (excluded,\n included).\n\n \"\"\"\n if (params is None and criteria is None) or (\n params is not None and criteria is not None\n ):\n raise ValueError(\n \"Exacly one of either `params` and `criteria` must be specified.\"\n )\n\n # Set single_entry to False, this is done since if criteria is defined,\n # this is an unknown factor - better to then get a list of results.\n single_entry = False\n if criteria is None:\n criteria = await self.ahandle_query_params(params)\n else:\n single_entry = isinstance(params, SingleEntryQueryParams)\n\n response_fields = criteria.pop(\"fields\", self.all_fields)\n\n results, data_returned, more_data_available = await self._arun_db_query(\n criteria=criteria,\n single_entry=single_entry,\n )\n\n if single_entry:\n results = results[0] if results else None # type: ignore[assignment]\n\n if data_returned > 1:\n raise NotFound(\n detail=(\n f\"Instead of a single entry, {data_returned} entries were found\"\n ),\n )\n\n include_fields = (\n response_fields - self.resource_mapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS\n )\n bad_optimade_fields = set()\n bad_provider_fields = set()\n for field in include_fields:\n if field not in self.resource_mapper.ALL_ATTRIBUTES:\n if field.startswith(\"_\"):\n if any(\n field.startswith(f\"_{prefix}_\")\n for prefix in self.resource_mapper.SUPPORTED_PREFIXES\n ):\n bad_provider_fields.add(field)\n else:\n bad_optimade_fields.add(field)\n\n if bad_provider_fields:\n warn(\n UnknownProviderProperty(\n detail=(\n \"Unrecognised field(s) for this provider requested in \"\n f\"`response_fields`: {bad_provider_fields}.\"\n )\n )\n )\n\n if bad_optimade_fields:\n raise BadRequest(\n detail=(\n \"Unrecognised OPTIMADE field(s) in requested `response_fields`: \"\n f\"{bad_optimade_fields}.\"\n )\n )\n\n if results:\n results = await self.resource_mapper.adeserialize(results)\n\n return ( # type: ignore[return-value]\n results,\n data_returned,\n more_data_available,\n self.all_fields - response_fields,\n include_fields,\n )\nahandle_query_params(params) async","text":"Parse and interpret the backend-agnostic query parameter models into a dictionary that can be used by the specific backend.
This is the asynchronous version of the parent class method named handle_query_params().
Currently this method returns the pymongo interpretation of the parameters, which will need modification for modified for other backends.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams The initialized query parameter model from the server.
requiredRaises:
Type DescriptionForbidden If too large of a page limit is provided.
BadRequest If an invalid request is made, e.g., with incorrect fields or response format.
Returns:
Type Descriptiondict[str, Any] A dictionary representation of the query parameters.
Source code inoptimade_gateway/mongo/collection.py async def ahandle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n This is the asynchronous version of the parent class method named\n `handle_query_params()`.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields or\n response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n return super().handle_query_params(params)\nainsert(data) async","text":"Add the given entries to the underlying database.
This is the asynchronous version of the parent class method named insert().
Parameters:
Name Type Description Defaultdata list[EntryResource] The entry resource objects to add to the database.
required Source code inoptimade_gateway/mongo/collection.py async def ainsert(self, data: list[EntryResource]) -> None:\n \"\"\"Add the given entries to the underlying database.\n\n This is the asynchronous version of the parent class method named `insert()`.\n\n Arguments:\n data: The entry resource objects to add to the database.\n\n \"\"\"\n await self.collection.insert_many(await clean_python_types(data))\ncreate_one(resource) async","text":"Create a new document in the MongoDB collection based on query parameters.
Update the newly created document with an \"id\" field. The value will be the string representation of the \"_id\" field. This will only be done if \"id\" is not already present in resource.
Parameters:
Name Type Description Defaultresource EntryResourceCreate The resource to be created.
requiredReturns:
Type DescriptionEntryResource The newly created document as a pydantic model entry resource.
Source code inoptimade_gateway/mongo/collection.py async def create_one(self, resource: EntryResourceCreate) -> EntryResource:\n \"\"\"Create a new document in the MongoDB collection based on query parameters.\n\n Update the newly created document with an `\"id\"` field.\n The value will be the string representation of the `\"_id\"` field.\n This will only be done if `\"id\"` is not already present in `resource`.\n\n Parameters:\n resource: The resource to be created.\n\n Returns:\n The newly created document as a pydantic model entry resource.\n\n \"\"\"\n resource.last_modified = datetime.now(timezone.utc)\n result = await self.collection.insert_one(\n await clean_python_types(resource.model_dump(exclude_unset=True))\n )\n LOGGER.debug(\n \"Inserted resource %r in DB collection %s with ID %s\",\n resource,\n self.collection.name,\n result.inserted_id,\n )\n\n if not resource.id:\n LOGGER.debug(\"Updating resource with an `id` field equal to str(id_).\")\n await self.collection.update_one(\n {\"_id\": result.inserted_id}, {\"$set\": {\"id\": str(result.inserted_id)}}\n )\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one({\"_id\": result.inserted_id})\n )\n )\nexists(entry_id) async","text":"Assert whether entry_id exists in the collection (value of \"id\")
Parameters:
Name Type Description Defaultentry_id str The \"id\" value of the entry.
optimade_gateway/mongo/collection.py async def exists(self, entry_id: str) -> bool:\n \"\"\"Assert whether entry_id exists in the collection (value of `\"id\"`)\n\n Parameters:\n entry_id: The `\"id\"` value of the entry.\n\n \"\"\"\n return bool(await self.collection.count_documents({\"id\": entry_id}))\nfind(params)","text":"Fetches results and indicates if more data is available.
Also gives the total number of data available in the absence of page_limit. See EntryListingQueryParams for more information.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams Entry listing URL query params.
requiredReturns:
Type Descriptionlist[EntryResource] | EntryResource | None A tuple of various relevant values:
int (results, data_returned, more_data_available, exclude_fields,
bool include_fields).
optimade_gateway/mongo/collection.py def find(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> tuple[\n list[EntryResource] | EntryResource | None, int, bool, set[str], set[str]\n]:\n \"\"\"\n Fetches results and indicates if more data is available.\n\n Also gives the total number of data available in the absence of `page_limit`.\n See\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n for more information.\n\n Parameters:\n params: Entry listing URL query params.\n\n Returns:\n A tuple of various relevant values:\n (`results`, `data_returned`, `more_data_available`, `exclude_fields`,\n `include_fields`).\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method `afind(params: \"\n \"Optional[Union[EntryListingQueryParams, SingleEntryQueryParams]], \"\n \"criteria: Optional[Dict[str, Any]])`.\"\n )\nget_multiple(**criteria) async","text":"Get a list of resources based on criteria
WarningThis is not to be used for creating a REST API response, but is rather a utility function to easily retrieve a list of resources.
Parameters:
Name Type Description Default**criteria Any Already handled/parsed URL query parameters.
{} Returns:
Type Descriptionlist[EntryResource] A list of resources from the MongoDB (mapped to pydantic models).
Source code inoptimade_gateway/mongo/collection.py async def get_multiple(self, **criteria: Any) -> list[EntryResource]:\n \"\"\"Get a list of resources based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a list of resources.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A list of resources from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n results = []\n async for document in self.collection.find(**self._valid_find_keys(**criteria)):\n results.append(self.resource_cls(**self.resource_mapper.map_back(document)))\n\n return results\nget_one(**criteria) async","text":"Get one resource based on criteria
WarningThis is not to be used for creating a REST API response, but is rather a utility function to easily retrieve a single resource.
Parameters:
Name Type Description Default**criteria Any Already handled/parsed URL query parameters.
{} Returns:
Type DescriptionEntryResource A single resource from the MongoDB (mapped to pydantic models).
Source code inoptimade_gateway/mongo/collection.py async def get_one(self, **criteria: Any) -> EntryResource:\n \"\"\"Get one resource based on criteria\n\n Warning:\n This is not to be used for creating a REST API response,\n but is rather a utility function to easily retrieve a single resource.\n\n Parameters:\n **criteria: Already handled/parsed URL query parameters.\n\n Returns:\n A single resource from the MongoDB (mapped to pydantic models).\n\n \"\"\"\n criteria = criteria or {}\n\n return self.resource_cls(\n **self.resource_mapper.map_back(\n await self.collection.find_one(**self._valid_find_keys(**criteria))\n )\n )\nhandle_query_params(params)","text":"Parse and interpret the backend-agnostic query parameter models into a dictionary that can be used by the specific backend.
NoteCurrently this method returns the pymongo interpretation of the parameters, which will need modification for modified for other backends.
Parameters:
Name Type Description Defaultparams EntryListingQueryParams | SingleEntryQueryParams The initialized query parameter model from the server.
requiredRaises:
Type DescriptionForbidden If too large of a page limit is provided.
BadRequest If an invalid request is made, e.g., with incorrect fields or response format.
Returns:
Type Descriptiondict[str, Any] A dictionary representation of the query parameters.
Source code inoptimade_gateway/mongo/collection.py def handle_query_params(\n self, params: EntryListingQueryParams | SingleEntryQueryParams\n) -> dict[str, Any]:\n \"\"\"Parse and interpret the backend-agnostic query parameter models into a\n dictionary that can be used by the specific backend.\n\n Note:\n Currently this method returns the pymongo interpretation of the parameters,\n which will need modification for modified for other backends.\n\n Parameters:\n params: The initialized query parameter model from the server.\n\n Raises:\n Forbidden: If too large of a page limit is provided.\n BadRequest: If an invalid request is made, e.g., with incorrect fields\n or response format.\n\n Returns:\n A dictionary representation of the query parameters.\n\n \"\"\"\n raise NotImplementedError(\n \"This method cannot be used with this class and is a remnant from the \"\n \"parent class. Use instead the asynchronous method \"\n \"`ahandle_query_params(params: Union[EntryListingQueryParams, \"\n \"SingleEntryQueryParams])`.\"\n )\nInitialize the MongoDB database.
"},{"location":"api_reference/mongo/database/#optimade_gateway.mongo.database.MONGO_CLIENT","title":"MONGO_CLIENT = AsyncIOMotorClient(CONFIG.mongo_uri, **mongo_client_configuration) module-attribute","text":"The MongoDB motor client.
"},{"location":"api_reference/mongo/database/#optimade_gateway.mongo.database.MONGO_DB","title":"MONGO_DB = MONGO_CLIENT[CONFIG.mongo_database] module-attribute","text":"The MongoDB motor database. This is a representation of the database used for the gateway service.
"},{"location":"api_reference/queries/params/","title":"params","text":"URL query parameters.
"},{"location":"api_reference/queries/params/#optimade_gateway.queries.params.SearchQueryParams","title":"SearchQueryParams","text":"URL query parameters for GET /search
This is an extension of the EntryListingQueryParams class in optimade, which defines the standard entry listing endpoint query parameters.
The extra query parameters are as follows.
Attributes:
Name Type Descriptiondatabase_ids set[str] List of possible database IDs that are already known by the gateway. To be known they need to be registered with the gateway (currently not possible).
optimade_urls set[AnyUrl] A list of OPTIMADE base URLs. If a versioned base URL is supplied it will be used as is, as long as it represents a supported version. If an un-versioned base URL, standard version negotiation will be conducted to get the versioned base URL, which will be used as long as it represents a supported version.
Example: http://example.org/optimade/v1/search?optimade_urls=\"https://example.org/optimade_db/v1\",\"https://optimade.herokuapp.com\"
endpoint str The entry endpoint queried. According to the OPTIMADE specification, this is the same as the resource's type.
Example: structures
timeout int Timeout time (in seconds) to wait for a query to finish before redirecting (after starting the query). Note, if the query has not finished after the timeout time, a redirection will still be performed, but to a zero-results page, which can be refreshed to get the finished query (once it has finished).
as_optimade bool Return the response as a standard OPTIMADE entry listing endpoint response. Otherwise, the response will be based on the QueriesResponseSingle model.
optimade_gateway/queries/params.py class SearchQueryParams:\n \"\"\"URL query parameters for `GET /search`\n\n This is an extension of the\n [`EntryListingQueryParams`](https://www.optimade.org/optimade-python-tools/api_reference/server/query_params/#optimade.server.query_params.EntryListingQueryParams)\n class in `optimade`, which defines the standard entry listing endpoint query\n parameters.\n\n The extra query parameters are as follows.\n\n Attributes:\n database_ids (set[str]): List of possible database IDs that are already known by\n the gateway. To be known they need to be registered with the gateway\n (currently not possible).\n\n optimade_urls (set[AnyUrl]): A list of OPTIMADE base URLs. If a versioned base\n URL is supplied it will be used as is, as long as it represents a supported\n version. If an un-versioned base URL, standard version negotiation will be\n conducted to get the versioned base URL, which will be used as long as it\n represents a supported version.\n\n **Example**: `http://example.org/optimade/v1/search?optimade_urls=\"https://example.org/optimade_db/v1\",\"https://optimade.herokuapp.com\"`\n\n endpoint (str): The entry endpoint queried. According to the OPTIMADE\n specification, this is the same as the resource's type.\n\n **Example**: `structures`\n\n timeout (int): Timeout time (in seconds) to wait for a query to finish before\n redirecting (*after* starting the query). Note, if the query has not\n finished after the timeout time, a redirection will still be performed, but\n to a zero-results page, which can be refreshed to get the finished query\n (once it has finished).\n\n as_optimade (bool): Return the response as a standard OPTIMADE entry listing\n endpoint response. Otherwise, the response will be based on the\n [`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\n model.\n\n \"\"\"\n\n def __init__(\n self,\n *,\n database_ids: Annotated[\n set[str],\n Query(\n description=(\n \"Unique list of possible database IDs that are already known by \"\n \"the gateway. To be known they need to be registered with the \"\n \"gateway (currently not possible).\"\n ),\n ),\n ] = set(),\n optimade_urls: Annotated[\n set[AnyUrl],\n Query(\n description=(\n \"A unique list of OPTIMADE base URLs. If a versioned base URL is \"\n \"supplied it will be used as is, as long as it represents a \"\n \"supported version. If an un-versioned base URL, standard version \"\n \"negotiation will be conducted to get the versioned base URL, \"\n \"which will be used as long as it represents a supported version.\"\n ),\n ),\n ] = set(),\n endpoint: Annotated[\n str,\n Query(\n description=(\n \"The entry endpoint queried. According to the OPTIMADE \"\n \"specification, this is the same as the resource's type.\"\n ),\n ),\n ] = \"structures\",\n timeout: Annotated[\n int,\n Query(\n description=(\n \"Timeout time (in seconds) to wait for a query to finish before \"\n \"redirecting (*after* starting the query). Note, if the query has \"\n \"not finished after the timeout time, a redirection will still be \"\n \"performed, but to a zero-results page, which can be refreshed to \"\n \"get the finished query (once it has finished).\"\n ),\n ),\n ] = 15,\n as_optimade: Annotated[\n bool,\n Query(\n description=(\n \"Return the response as a standard OPTIMADE entry listing endpoint \"\n \"response. Otherwise, the response will be based on the \"\n \"[`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\"\n \" model.\"\n ),\n ),\n ] = False,\n ) -> None:\n self.database_ids = database_ids\n self.optimade_urls = optimade_urls\n self.endpoint = endpoint\n self.timeout = timeout\n self.as_optimade = as_optimade\nPerform OPTIMADE queries
"},{"location":"api_reference/queries/perform/#optimade_gateway.queries.perform.db_find","title":"db_find(database, endpoint, response_model, query_params='', raw_url=None)","text":"Imitate Collection.find() for any given database for entry-resource endpoints
Parameters:
Name Type Description Defaultdatabase LinksResource | dict[str, Any] The OPTIMADE implementation to be queried. It must have a valid base URL and id.
requiredendpoint str The entry-listing endpoint, e.g., \"structures\".
response_model EntryResponseMany | EntryResponseOne The expected OPTIMADE pydantic response model, e.g., optimade.models.StructureResponseMany.
query_params str URL query parameters to pass to the database.
'' raw_url AnyUrl | str | None A raw URL to use straight up instead of deriving a URL from database, endpoint, and query_params.
None Returns:
Type Descriptiontuple[ErrorResponse | EntryResponseMany | EntryResponseOne, str] Response as an optimade pydantic model and the database's ID.
optimade_gateway/queries/perform.py def db_find(\n database: LinksResource | dict[str, Any],\n endpoint: str,\n response_model: EntryResponseMany | EntryResponseOne,\n query_params: str = \"\",\n raw_url: AnyUrl | str | None = None,\n) -> tuple[ErrorResponse | EntryResponseMany | EntryResponseOne, str]:\n \"\"\"Imitate `Collection.find()` for any given database for entry-resource endpoints\n\n Parameters:\n database: The OPTIMADE implementation to be queried.\n It **must** have a valid base URL and id.\n endpoint: The entry-listing endpoint, e.g., `\"structures\"`.\n response_model: The expected OPTIMADE pydantic response model, e.g.,\n `optimade.models.StructureResponseMany`.\n query_params: URL query parameters to pass to the database.\n raw_url: A raw URL to use straight up instead of deriving a URL from `database`,\n `endpoint`, and `query_params`.\n\n Returns:\n Response as an `optimade` pydantic model and the `database`'s ID.\n\n \"\"\"\n if TYPE_CHECKING or bool(os.getenv(\"MKDOCS_BUILD\", \"\")): # pragma: no cover\n response: (\n httpx.Response\n | dict[str, Any]\n | EntryResponseMany\n | EntryResponseOne\n | ErrorResponse\n )\n\n if raw_url:\n url = str(raw_url)\n else:\n url = \"\"\n\n base_url = str(get_resource_attribute(database, \"attributes.base_url\")).rstrip(\n \"/\"\n )\n url += base_url\n\n # Check whether base_url is a versioned base URL\n if not any(base_url.endswith(_) for _ in BASE_URL_PREFIXES.values()):\n # Unversioned base URL - add the currently supported major version\n url += BASE_URL_PREFIXES[\"major\"]\n\n url += f\"/{endpoint.strip('/')}?{query_params}\"\n\n response = httpx.get(url, timeout=60)\n\n try:\n response = response.json()\n except json.JSONDecodeError:\n return (\n ErrorResponse(\n errors=[\n {\n \"detail\": f\"Could not JSONify response from {url}\",\n \"id\": \"OPTIMADE_GATEWAY_DB_FIND_MANY_JSONDECODEERROR\",\n }\n ],\n meta={\n \"query\": {\n \"representation\": f\"/{endpoint.strip('/')}?{query_params}\"\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n try:\n response = response_model(**response)\n except ValidationError:\n try:\n response = ErrorResponse(**response)\n except ValidationError as exc:\n # If it's an error and `meta` is missing, it is not a valid OPTIMADE\n # response, but this happens a lot, and is therefore worth having an\n # edge-case for.\n if \"errors\" in response:\n errors = list(response[\"errors\"])\n errors.append(\n {\n \"detail\": (\n f\"Could not pass response from {url} as either a \"\n f\"{response_model.__name__!r} or 'ErrorResponse'. \"\n f\"ValidationError: {exc}\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_DB_FINDS_MANY_VALIDATIONERRORS\",\n }\n )\n return (\n ErrorResponse(\n errors=errors,\n meta={\n \"query\": {\n \"representation\": (\n f\"/{endpoint.strip('/')}?{query_params}\"\n )\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n return (\n ErrorResponse(\n errors=[\n {\n \"detail\": (\n f\"Could not pass response from {url} as either a \"\n f\"{response_model.__name__!r} or 'ErrorResponse'. \"\n f\"ValidationError: {exc}\"\n ),\n \"id\": \"OPTIMADE_GATEWAY_DB_FINDS_MANY_VALIDATIONERRORS\",\n }\n ],\n meta={\n \"query\": {\n \"representation\": f\"/{endpoint.strip('/')}?{query_params}\"\n },\n \"api_version\": __api_version__,\n \"more_data_available\": False,\n },\n ),\n get_resource_attribute(database, \"id\"),\n )\n\n return response, get_resource_attribute(database, \"id\")\ndb_get_all_resources(database, endpoint, response_model, query_params='', raw_url=None) async","text":"Recursively retrieve all resources from an entry-listing endpoint
This function keeps pulling the links.next link if meta.more_data_available is True to ultimately retrieve all entries for endpoint.
Warning
This function can be dangerous if an endpoint with hundreds or thousands of entries is requested.
Parameters:
Name Type Description Defaultdatabase LinksResource | dict[str, Any] The OPTIMADE implementation to be queried. It must have a valid base URL and id.
requiredendpoint str The entry-listing endpoint, e.g., \"structures\".
response_model EntryResponseMany The expected OPTIMADE pydantic response model, e.g., optimade.models.StructureResponseMany.
query_params str URL query parameters to pass to the database.
'' raw_url AnyUrl | str | None A raw URL to use straight up instead of deriving a URL from database, endpoint, and query_params.
None Returns:
Type Descriptiontuple[list[EntryResource | dict[str, Any]], LinksResource | dict[str, Any]] A collected list of successful responses' data value and the database's ID.
optimade_gateway/queries/perform.py async def db_get_all_resources(\n database: LinksResource | dict[str, Any],\n endpoint: str,\n response_model: EntryResponseMany,\n query_params: str = \"\",\n raw_url: AnyUrl | str | None = None,\n) -> tuple[list[EntryResource | dict[str, Any]], LinksResource | dict[str, Any]]:\n \"\"\"Recursively retrieve all resources from an entry-listing endpoint\n\n This function keeps pulling the `links.next` link if `meta.more_data_available` is\n `True` to ultimately retrieve *all* entries for `endpoint`.\n\n !!! warning\n This function can be dangerous if an endpoint with hundreds or thousands of\n entries is requested.\n\n Parameters:\n database: The OPTIMADE implementation to be queried.\n It **must** have a valid base URL and id.\n endpoint: The entry-listing endpoint, e.g., `\"structures\"`.\n response_model: The expected OPTIMADE pydantic response model, e.g.,\n `optimade.models.StructureResponseMany`.\n query_params: URL query parameters to pass to the database.\n raw_url: A raw URL to use straight up instead of deriving a URL from `database`,\n `endpoint`, and `query_params`.\n\n Returns:\n A collected list of successful responses' `data` value and the `database`'s ID.\n\n \"\"\"\n resulting_resources = []\n\n response, _ = db_find(\n database=database,\n endpoint=endpoint,\n response_model=response_model,\n query_params=query_params,\n raw_url=raw_url,\n )\n\n if isinstance(response, ErrorResponse):\n # An errored response will result in no databases from a provider.\n LOGGER.error(\n \"Error while querying database (id=%r). Full response: %s\",\n get_resource_attribute(database, \"id\"),\n response.model_dump_json(indent=2),\n )\n return [], database\n\n resulting_resources.extend(response.data)\n\n if response.meta.more_data_available:\n next_page = get_resource_attribute(response, \"links.next\")\n if next_page is None:\n LOGGER.error(\n \"Could not find a 'next' link for an OPTIMADE query request to %r \"\n \"(id=%r). Cannot get all resources from /%s, even though this was \"\n \"asked and `more_data_available` is `True` in the response.\",\n get_resource_attribute(database, \"attributes.name\", \"N/A\"),\n get_resource_attribute(database, \"id\"),\n endpoint,\n )\n return resulting_resources, database\n\n more_resources, _ = await db_get_all_resources(\n database=database,\n endpoint=endpoint,\n response_model=response_model,\n query_params=query_params,\n raw_url=next_page,\n )\n resulting_resources.extend(more_resources)\n\n return resulting_resources, database\nperform_query(url, query) async","text":"Perform OPTIMADE query with gateway.
Parameters:
Name Type Description Defaulturl URL Original request URL.
requiredquery QueryResource The query to be performed.
requiredReturns:
Type DescriptionEntryResponseMany | ErrorResponse | GatewayQueryResponse This function returns the final response; a
EntryResponseMany | ErrorResponse | GatewayQueryResponse GatewayQueryResponse.
optimade_gateway/queries/perform.py async def perform_query(\n url: URL,\n query: QueryResource,\n) -> EntryResponseMany | ErrorResponse | GatewayQueryResponse:\n \"\"\"Perform OPTIMADE query with gateway.\n\n Parameters:\n url: Original request URL.\n query: The query to be performed.\n\n Returns:\n This function returns the final response; a\n [`GatewayQueryResponse`][optimade_gateway.models.queries.GatewayQueryResponse].\n\n \"\"\"\n await update_query(query, \"state\", QueryState.STARTED)\n\n gateway: GatewayResource = await get_valid_resource(\n await collection_factory(CONFIG.gateways_collection),\n query.attributes.gateway_id,\n )\n\n filter_queries = await prepare_query_filter(\n database_ids=[_.id for _ in gateway.attributes.databases],\n filter_query=query.attributes.query_parameters.filter,\n )\n\n url = url.replace(path=f\"{url.path.rstrip('/')}/{query.id}\")\n await update_query(\n query,\n \"response\",\n GatewayQueryResponse(\n data={},\n links=ToplevelLinks(next=None),\n meta=meta_values(\n url=url,\n data_available=0,\n data_returned=0,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n ),\n operator=None,\n **{\"$set\": {\"state\": QueryState.IN_PROGRESS}},\n )\n\n loop = asyncio.get_running_loop()\n with ThreadPoolExecutor(\n max_workers=min(\n 32, (os.cpu_count() or 0) + 4, len(gateway.attributes.databases)\n )\n ) as executor:\n # Run OPTIMADE DB queries in a thread pool, i.e., not using the main OS thread,\n # where the asyncio event loop is running.\n query_tasks = []\n for database in gateway.attributes.databases:\n query_params = await get_query_params(\n query_parameters=query.attributes.query_parameters,\n database_id=database.id,\n filter_mapping=filter_queries,\n )\n query_tasks.append(\n loop.run_in_executor(\n executor=executor,\n func=functools.partial(\n db_find,\n database=database,\n endpoint=query.attributes.endpoint.value,\n response_model=query.attributes.endpoint.get_response_model(),\n query_params=query_params,\n ),\n )\n )\n\n for query_task in query_tasks:\n (db_response, db_id) = await query_task\n\n await process_db_response(\n response=db_response,\n database_id=db_id,\n query=query,\n gateway=gateway,\n )\n\n # Pagination\n #\n # if isinstance(results, list) and get_resource_attribute(\n # query,\n # \"attributes.response.meta.more_data_available\",\n # False,\n # disambiguate=False, # Extremely minor speed-up\n # ):\n # # Deduce the `next` link from the current request\n # query_string = urllib.parse.parse_qs(url.query)\n # query_string[\"page_offset\"] = [\n # int(query_string.get(\"page_offset\", [0])[0]) # type: ignore[list-item]\n # + len(results[: query.attributes.query_parameters.page_limit])\n # ]\n # urlencoded = urllib.parse.urlencode(query_string, doseq=True)\n # base_url = get_base_url(url)\n\n # links = ToplevelLinks(next=f\"{base_url}{url.path}?{urlencoded}\")\n\n # await update_query(query, \"response.links\", links)\n\n await update_query(query, \"state\", QueryState.FINISHED)\n return query.attributes.response\nPrepare OPTIMADE queries.
"},{"location":"api_reference/queries/prepare/#optimade_gateway.queries.prepare.get_query_params","title":"get_query_params(query_parameters, database_id, filter_mapping) async","text":"Construct the parsed URL query parameters
Source code inoptimade_gateway/queries/prepare.py async def get_query_params(\n query_parameters: OptimadeQueryParameters,\n database_id: str,\n filter_mapping: Mapping[str, str | None],\n) -> str:\n \"\"\"Construct the parsed URL query parameters\"\"\"\n query_params = {\n param: value for param, value in query_parameters.model_dump().items() if value\n }\n if filter_mapping[database_id]:\n query_params.update({\"filter\": filter_mapping[database_id]})\n return urllib.parse.urlencode(query_params)\nprepare_query_filter(database_ids, filter_query) async","text":"Update the query parameter filter value to be database-specific
This is needed due to the served change of id values. If someone searches for a gateway-changed id, it needs to be reverted to be database-specific.
Parameters:
Name Type Description Defaultdatabase_ids list[str] List of the databases to create updated filter values for. These values are part of the gateway-changed id values and are essential.
filter_query str | None The submitted filter query parameter value. Can be None if not supplied.
Returns:
Type DescriptionMapping[str, str | None] A mapping for database IDs to database-specific filter query parameter values.
optimade_gateway/queries/prepare.py async def prepare_query_filter(\n database_ids: list[str], filter_query: str | None\n) -> Mapping[str, str | None]:\n \"\"\"Update the query parameter `filter` value to be database-specific\n\n This is needed due to the served change of `id` values.\n If someone searches for a gateway-changed `id`, it needs to be reverted to be\n database-specific.\n\n Parameters:\n database_ids: List of the databases to create updated filter values for.\n These values are part of the gateway-changed `id` values and are essential.\n filter_query: The submitted `filter` query parameter value. Can be `None` if not\n supplied.\n\n Returns:\n A mapping for database IDs to database-specific `filter` query parameter values.\n\n \"\"\"\n updated_filter = {}.fromkeys(database_ids, filter_query)\n\n if not filter_query:\n return updated_filter\n\n for id_match in re.finditer(\n r'\"(?P<id_value_l>[^\\s]*)\"[\\s]*'\n r\"(<|>|<=|>=|=|!=|CONTAINS|STARTS WITH|ENDS WITH|STARTS|ENDS)\"\n r\"[\\s]*id|[^_]+id[\\s]*\"\n r'(<|>|<=|>=|=|!=|CONTAINS|STARTS WITH|ENDS WITH|STARTS|ENDS)[\\s]*\"'\n r'(?P<id_value_r>[^\\s]*)\"',\n f\"={filter_query}\" if filter_query else \"\",\n ):\n matched_id: str = id_match.group(\"id_value_l\") or id_match.group(\"id_value_r\")\n for database_id in database_ids:\n if matched_id.startswith(f\"{database_id}/\"):\n updated_filter_query = updated_filter[database_id]\n if not updated_filter_query or not isinstance(\n updated_filter_query, str\n ):\n raise TypeError(\n \"Expected a string for filter query, got \"\n f\"{type(updated_filter_query)}\"\n )\n\n # Database found\n updated_filter[database_id] = updated_filter_query.replace(\n f\"{database_id}/\", \"\", 1\n )\n break\n else:\n warn(\n OptimadeGatewayWarning(\n title=\"Non-Unique Entry ID\",\n detail=(\n f\"The passed entry ID <id={matched_id}> may be ambiguous! To \"\n \"get a specific structures entry, one can prepend the ID with \"\n \"a database ID belonging to the gateway, followed by a forward\"\n f\" slash, e.g., '{database_ids[0]}/<local_database_ID>'. \"\n f\"Available databases for this gateway: {database_ids}\"\n ),\n )\n )\n return updated_filter\nProcess performed OPTIMADE queries.
"},{"location":"api_reference/queries/process/#optimade_gateway.queries.process.process_db_response","title":"process_db_response(response, database_id, query, gateway) async","text":"Process an OPTIMADE database response.
The passed query will be updated with the top-level meta information: data_available, data_returned, and more_data_available.
Since, only either data or errors should ever be present, one or the other will be either an empty list or None.
Parameters:
Name Type Description Defaultresponse ErrorResponse | EntryResponseMany | EntryResponseOne The OPTIMADE database response to be processed.
requireddatabase_id str The database's id under which the returned resources or errors will be delivered.
query QueryResource A resource representing the performed query.
requiredgateway GatewayResource A resource representing the gateway that was queried.
requiredReturns:
Type Descriptionlist[EntryResource] | list[dict[str, Any]] | EntryResource | dict[str, Any] | None The response's data.
optimade_gateway/queries/process.py async def process_db_response(\n response: ErrorResponse | EntryResponseMany | EntryResponseOne,\n database_id: str,\n query: QueryResource,\n gateway: GatewayResource,\n) -> list[EntryResource] | list[dict[str, Any]] | EntryResource | dict[str, Any] | None:\n \"\"\"Process an OPTIMADE database response.\n\n The passed `query` will be updated with the top-level `meta` information:\n `data_available`, `data_returned`, and `more_data_available`.\n\n Since, only either `data` or `errors` should ever be present, one or the other will\n be either an empty list or `None`.\n\n Parameters:\n response: The OPTIMADE database response to be processed.\n database_id: The database's `id` under which the returned resources or errors\n will be delivered.\n query: A resource representing the performed query.\n gateway: A resource representing the gateway that was queried.\n\n Returns:\n The response's `data`.\n\n \"\"\"\n results = []\n errors = []\n\n LOGGER.debug(\"Starting to process database_id: %s\", database_id)\n\n if isinstance(response, ErrorResponse):\n for error in response.errors:\n if isinstance(error.id, str) and error.id.startswith(\"OPTIMADE_GATEWAY\"):\n warn(error.detail, OptimadeGatewayWarning)\n else:\n # The model `ErrorResponse` does not allow the objects in the top-level\n # `errors` list to be parsed as dictionaries - they must be a pydantic\n # model.\n meta_error = {}\n if error.meta:\n meta_error = error.meta.model_dump()\n meta_error.update(\n {\n f\"_{CONFIG.provider.prefix}_source_gateway\": {\n \"id\": gateway.id,\n \"type\": gateway.type,\n \"links\": {\"self\": gateway.links.self},\n },\n f\"_{CONFIG.provider.prefix}_source_database\": {\n \"id\": database_id,\n \"type\": \"links\",\n \"links\": {\n \"self\": (\n str(gateway.links.self).split(\n \"gateways\", maxsplit=1\n )[0]\n + f\"databases/{database_id}\"\n )\n },\n },\n }\n )\n error.meta = Meta(**meta_error)\n errors.append(error)\n data_returned = 0\n more_data_available = False\n else:\n results = response.data\n\n if isinstance(results, list):\n data_returned = response.meta.data_returned or len(results)\n else:\n data_returned = response.meta.data_returned or (0 if not results else 1)\n\n more_data_available = response.meta.more_data_available or False\n\n data_available = response.meta.data_available or 0\n\n extra_updates = {\n \"$inc\": {\n \"response.meta.data_available\": data_available,\n \"response.meta.data_returned\": data_returned,\n }\n }\n if not get_resource_attribute(\n query,\n \"attributes.response.meta.more_data_available\",\n False,\n disambiguate=False, # Extremely minor speed-up\n ):\n # Keep it True, if set to True once.\n extra_updates.update(\n {\"$set\": {\"response.meta.more_data_available\": more_data_available}}\n )\n\n # This ensures an empty list under `response.data.{database_id}` is returned if the\n # case is simply that there are no results to return.\n if errors:\n extra_updates.update({\"$addToSet\": {\"response.errors\": {\"$each\": errors}}})\n await update_query(\n query,\n f\"response.data.{database_id}\",\n results,\n operator=None,\n **extra_updates,\n )\n\n return results\nUtility functions for the queries module.
update_query(query, field, value, operator=None, **mongo_kwargs) async","text":"Update a query's field attribute with value.
If field is a dot-separated value, then only the last field part may be a non-pre-existing field. Otherwise a KeyError or AttributeError will be raised.
Note
This can only update a field for a query's attributes, i.e., this function cannot update id, type or any other top-level resource field.
Important
mongo_kwargs will not be considered for updating the pydantic model instance.
Parameters:
Name Type Description Defaultquery QueryResource The query to be updated.
requiredfield str The attributes field (key) to be set. This can be a dot-separated key value to signify embedded fields.
Example: response.meta.
value Any The (possibly) new value for field.
operator str | None A MongoDB operator to be used for updating field with value.
None **mongo_kwargs Any Further MongoDB update filters.
{} Source code in optimade_gateway/queries/utils.py async def update_query(\n query: QueryResource,\n field: str,\n value: Any,\n operator: str | None = None,\n **mongo_kwargs: Any,\n) -> None:\n \"\"\"Update a query's `field` attribute with `value`.\n\n If `field` is a dot-separated value, then only the last field part may be a\n non-pre-existing field. Otherwise a `KeyError` or `AttributeError` will be raised.\n\n !!! note\n This can *only* update a field for a query's `attributes`, i.e., this function\n cannot update `id`, `type` or any other top-level resource field.\n\n !!! important\n `mongo_kwargs` will not be considered for updating the pydantic model instance.\n\n Parameters:\n query: The query to be updated.\n field: The `attributes` field (key) to be set.\n This can be a dot-separated key value to signify embedded fields.\n\n **Example**: `response.meta`.\n value: The (possibly) new value for `field`.\n operator: A MongoDB operator to be used for updating `field` with `value`.\n **mongo_kwargs: Further MongoDB update filters.\n\n \"\"\"\n operator = operator or \"$set\"\n\n if operator and not operator.startswith(\"$\"):\n operator = f\"${operator}\"\n\n update_time = datetime.now(timezone.utc)\n\n update_kwargs = {\"$set\": {\"last_modified\": update_time}}\n\n if mongo_kwargs:\n update_kwargs.update(mongo_kwargs)\n\n if operator and operator == \"$set\":\n update_kwargs[\"$set\"].update({field: value})\n elif operator:\n if operator in update_kwargs:\n update_kwargs[operator].update({field: value})\n else:\n update_kwargs.update({operator: {field: value}})\n\n # MongoDB\n collection = await collection_factory(CONFIG.queries_collection)\n result: UpdateResult = await collection.collection.update_one(\n filter={\"id\": {\"$eq\": query.id}},\n update=await clean_python_types(update_kwargs),\n )\n if result.matched_count != 1:\n LOGGER.error(\n (\n \"matched_count should have been exactly 1, it was: %s. \"\n \"Returned update_one result: %s\"\n ),\n result.matched_count,\n result.raw_result,\n )\n\n # Pydantic model instance\n query.attributes.last_modified = update_time\n if \".\" in field:\n field_list = field.split(\".\")\n sub_field: BaseModel | dict[str, Any] = getattr(query.attributes, field_list[0])\n for field_part in field_list[1:-1]:\n if isinstance(sub_field, dict):\n sub_field = sub_field.get(field_part, {})\n else:\n sub_field = getattr(sub_field, field_part)\n if isinstance(sub_field, dict):\n sub_field[field_list[-1]] = value\n else:\n setattr(sub_field, field_list[-1], value)\n else:\n setattr(query.attributes, field, value)\n/databases/*
This file describes the router for:
/databases/{id}\nwhere, id may be left out.
Database resources represent the available databases that may be used for the gateways.
One can register a new database (by using POST /databases) or look through the available databases (by using GET /databases) using standard OPTIMADE filtering.
get_database(request, database_id, params) async","text":"GET /databases/{database ID}
Return a single LinksResource representing the database resource object with id={database ID}.
optimade_gateway/routers/databases.py @ROUTER.get(\n \"/databases/{database_id:path}\",\n response_model=DatabasesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_database(\n request: Request,\n database_id: str,\n params: Annotated[SingleEntryQueryParams, Depends()],\n) -> DatabasesResponseSingle:\n \"\"\"`GET /databases/{database ID}`\n\n Return a single\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n representing the database resource object with `id={database ID}`.\n \"\"\"\n collection = await collection_factory(CONFIG.databases_collection)\n\n params.filter = f'id=\"{database_id}\"'\n (\n result,\n data_returned,\n more_data_available,\n fields,\n include_fields,\n ) = await collection.afind(params=params)\n\n if fields or include_fields and result is not None:\n result = handle_response_fields(result, fields, include_fields)\n\n result = result[0] if isinstance(result, list) and data_returned else None\n\n return DatabasesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=data_returned,\n data_available=await collection.acount(),\n more_data_available=more_data_available,\n schema=CONFIG.schema_url,\n ),\n )\nget_databases(request, params) async","text":"GET /databases
Return overview of all (active) databases.
Source code inoptimade_gateway/routers/databases.py @ROUTER.get(\n \"/databases\",\n response_model=DatabasesResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_databases(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> DatabasesResponse:\n \"\"\"`GET /databases`\n\n Return overview of all (active) databases.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.databases_collection),\n response_cls=DatabasesResponse,\n request=request,\n params=params,\n )\npost_databases(request, database) async","text":"POST /databases
Create/Register or return an existing LinksResource, representing a database resource object, according to database.
optimade_gateway/routers/databases.py @ROUTER.post(\n \"/databases\",\n response_model=DatabasesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Databases\"],\n responses=ERROR_RESPONSES,\n)\nasync def post_databases(\n request: Request, database: DatabaseCreate\n) -> DatabasesResponseSingle:\n \"\"\"`POST /databases`\n\n Create/Register or return an existing\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource),\n representing a database resource object, according to `database`.\n \"\"\"\n result, created = await resource_factory(database)\n collection = await collection_factory(CONFIG.databases_collection)\n\n return DatabasesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\n/gateways/*
This file describes the router for:
/gateways/{id}\nwhere, id may be left out.
get_gateway(request, gateway_id) async","text":"GET /gateways/{gateway ID}
Return a single GatewayResource.
optimade_gateway/routers/gateways.py @ROUTER.get(\n \"/gateways/{gateway_id}\",\n response_model=GatewaysResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_gateway(request: Request, gateway_id: str) -> GatewaysResponseSingle:\n \"\"\"`GET /gateways/{gateway ID}`\n\n Return a single\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource].\n \"\"\"\n collection = await collection_factory(CONFIG.gateways_collection)\n result = await get_valid_resource(collection, gateway_id)\n\n return GatewaysResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\nget_gateways(request, params) async","text":"GET /gateways
Return overview of all (active) gateways.
Source code inoptimade_gateway/routers/gateways.py @ROUTER.get(\n \"/gateways\",\n response_model=GatewaysResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_gateways(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> GatewaysResponse:\n \"\"\"`GET /gateways`\n\n Return overview of all (active) gateways.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.gateways_collection),\n response_cls=GatewaysResponse,\n request=request,\n params=params,\n )\npost_gateways(request, gateway) async","text":"POST /gateways
Create or return existing gateway according to gateway.
optimade_gateway/routers/gateways.py @ROUTER.post(\n \"/gateways\",\n response_model=GatewaysResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Gateways\"],\n responses=ERROR_RESPONSES,\n)\nasync def post_gateways(\n request: Request, gateway: GatewayCreate\n) -> GatewaysResponseSingle:\n \"\"\"`POST /gateways`\n\n Create or return existing gateway according to `gateway`.\n \"\"\"\n if gateway.database_ids:\n databases_collection = await collection_factory(CONFIG.databases_collection)\n\n databases = await databases_collection.get_multiple(\n filter={\"id\": {\"$in\": await clean_python_types(gateway.database_ids)}}\n )\n\n if not isinstance(gateway.databases, list):\n gateway.databases = []\n\n current_database_ids = [_.id for _ in gateway.databases]\n gateway.databases.extend(\n _ for _ in databases if _.id not in current_database_ids\n )\n\n result, created = await resource_factory(gateway)\n collection = await collection_factory(CONFIG.gateways_collection)\n\n return GatewaysResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\n/info/*
This file describes the router for:
/info/{entry}\nwhere, entry may be left out.
ENTRY_INFO_SCHEMAS: dict[str, type[EntryResource]] = {'databases': LinksResource, 'gateways': GatewayResource, 'queries': QueryResource} module-attribute","text":"This dictionary is used to define the /info/<entry_type> endpoints.
get_entry_info(request, entry) async","text":"GET /info/{entry}
Get information about the gateway service's entry-listing endpoints.
Source code inoptimade_gateway/routers/info.py @ROUTER.get(\n \"/info/{entry}\",\n response_model=EntryInfoResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Info\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_entry_info(request: Request, entry: str) -> EntryInfoResponse:\n \"\"\"`GET /info/{entry}`\n\n Get information about the gateway service's entry-listing endpoints.\n \"\"\"\n valid_entry_info_endpoints = ENTRY_INFO_SCHEMAS.keys()\n if entry not in valid_entry_info_endpoints:\n raise NotFound(\n detail=(\n f\"Entry info not found for {entry}, valid entry info endpoints are: \"\n f\"{', '.join(valid_entry_info_endpoints)}\"\n ),\n )\n\n schema = ENTRY_INFO_SCHEMAS[entry]\n queryable_properties = {\"id\", \"type\", \"attributes\"}\n properties = await aretrieve_queryable_properties(\n schema, queryable_properties, entry_type=entry\n )\n\n output_fields_by_format = {\"json\": list(properties)}\n\n return EntryInfoResponse(\n data=EntryInfoResource(\n formats=list(output_fields_by_format),\n description=getattr(schema, \"__doc__\", \"Entry Resources\"),\n properties=properties,\n output_fields_by_format=output_fields_by_format,\n ),\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=1,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\nget_info(request) async","text":"GET /info
An introspective endpoint for the gateway service.
Source code inoptimade_gateway/routers/info.py @ROUTER.get(\n \"/info\",\n response_model=InfoResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Info\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_info(request: Request) -> InfoResponse:\n \"\"\"`GET /info`\n\n An introspective endpoint for the gateway service.\n \"\"\"\n return InfoResponse(\n data=BaseInfoResource(\n id=BaseInfoResource.model_fields[\"id\"].default,\n type=BaseInfoResource.model_fields[\"type\"].default,\n attributes=BaseInfoAttributes(\n api_version=__api_version__,\n available_api_versions=[\n {\n \"url\": (\n f\"{get_base_url(request.url)}\"\n f\"/v{__api_version__.split('.', maxsplit=1)[0]}\"\n ),\n \"version\": __api_version__,\n }\n ],\n formats=[\"json\"],\n entry_types_by_format={\"json\": list(ENTRY_INFO_SCHEMAS.keys())},\n available_endpoints=sorted(\n [\n \"docs\",\n \"info\",\n \"links\",\n \"openapi.json\",\n \"redoc\",\n \"search\",\n *list(ENTRY_INFO_SCHEMAS.keys()),\n ]\n ),\n is_index=False,\n ),\n ),\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=1,\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n/links/*
This file describes the router for:
/links\nget_links(request, params) async","text":"GET /links
Return a regular /links response for an OPTIMADE implementation.
optimade_gateway/routers/links.py @ROUTER.get(\n \"/links\",\n response_model=LinksResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Links\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_links(\n request: Request, params: Annotated[EntryListingQueryParams, Depends()]\n) -> LinksResponse:\n \"\"\"`GET /links`\n\n Return a regular `/links` response for an OPTIMADE implementation.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.links_collection),\n response_cls=LinksResponse,\n request=request,\n params=params,\n )\nGeneral /queries endpoint to handle gateway queries
This file describes the router for:
/queries/{id}\nwhere, id may be left out.
get_queries(request, params) async","text":"GET /queries
Return overview of all (active) queries.
Source code inoptimade_gateway/routers/queries.py @ROUTER.get(\n \"/queries\",\n response_model=QueriesResponse,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_queries(\n request: Request,\n params: Annotated[EntryListingQueryParams, Depends()],\n) -> QueriesResponse:\n \"\"\"`GET /queries`\n\n Return overview of all (active) queries.\n \"\"\"\n return await get_entries(\n collection=await collection_factory(CONFIG.queries_collection),\n response_cls=QueriesResponse,\n request=request,\n params=params,\n )\nget_query(request, query_id, response) async","text":"GET /queries/{query_id}
Return a single QueryResource.
optimade_gateway/routers/queries.py @ROUTER.get(\n \"/queries/{query_id}\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_query(\n request: Request,\n query_id: str,\n response: Response,\n) -> QueriesResponseSingle:\n \"\"\"`GET /queries/{query_id}`\n\n Return a single [`QueryResource`][optimade_gateway.models.queries.QueryResource].\n \"\"\"\n collection = await collection_factory(CONFIG.queries_collection)\n query: QueryResource = await get_valid_resource(collection, query_id)\n\n if query.attributes.response and query.attributes.response.errors:\n for error in query.attributes.response.errors:\n if error.status:\n for part in error.status.split(\" \"):\n try:\n response.status_code = int(part)\n break\n except ValueError:\n pass\n if response.status_code and response.status_code >= 300:\n break\n else:\n response.status_code = 500\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\npost_queries(request, query) async","text":"POST /queries
Create or return existing gateway query according to query.
optimade_gateway/routers/queries.py @ROUTER.post(\n \"/queries\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Queries\"],\n status_code=status.HTTP_202_ACCEPTED,\n responses=ERROR_RESPONSES,\n)\nasync def post_queries(\n request: Request,\n query: QueryCreate,\n) -> QueriesResponseSingle:\n \"\"\"`POST /queries`\n\n Create or return existing gateway query according to `query`.\n \"\"\"\n await validate_resource(\n await collection_factory(CONFIG.gateways_collection), query.gateway_id\n )\n\n result, created = await resource_factory(query)\n\n background_tasks: set[asyncio.Task] = set()\n\n if created:\n task = asyncio.create_task(perform_query(url=request.url, query=result))\n\n # Add task to the set. This creates a strong reference.\n background_tasks.add(task)\n\n # To prevent keeping references to finished tasks forever,\n # make each task remove its own reference from the set after\n # completion:\n task.add_done_callback(background_tasks.discard)\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=result,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\nGeneral /search endpoint to completely coordinate an OPTIMADE gateway query
This file describes the router for:
/search\nget_search(request, response, search_params, entry_params) async","text":"GET /search
Coordinate a new OPTIMADE query in multiple databases through a gateway:
Search POST data - calling POST /search.search_params.timeout seconds before returning the query, if it has not finished before.GET /queries/{query_id}.This endpoint works similarly to GET /queries/{query_id}, where one passes the query parameters directly in the URL, instead of first POSTing a query and then going to its URL. Hence, a QueryResponseSingle is the standard response model for this endpoint.
If the timeout time is reached and the query has not yet finished, the user is redirected to the specific URL for the query.
If the as_optimade query parameter is True, the response will be parseable as a standard OPTIMADE entry listing endpoint like, e.g., /structures. For more information see the OPTIMADE specification.
optimade_gateway/routers/search.py @ROUTER.get(\n \"/search\",\n response_model=Union[QueriesResponseSingle, ErrorResponse, EntryResponseMany],\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Search\"],\n responses=ERROR_RESPONSES,\n)\nasync def get_search(\n request: Request,\n response: Response,\n search_params: Annotated[SearchQueryParams, Depends()],\n entry_params: Annotated[EntryListingQueryParams, Depends()],\n) -> QueriesResponseSingle | EntryResponseMany | ErrorResponse | RedirectResponse:\n \"\"\"`GET /search`\n\n Coordinate a new OPTIMADE query in multiple databases through a gateway:\n\n 1. Create a [`Search`][optimade_gateway.models.search.Search] `POST` data - calling\n `POST /search`.\n 1. Wait [`search_params.timeout`][optimade_gateway.queries.params.SearchQueryParams]\n seconds before returning the query, if it has not finished before.\n 1. Return query - similar to `GET /queries/{query_id}`.\n\n This endpoint works similarly to `GET /queries/{query_id}`, where one passes the\n query parameters directly in the URL, instead of first POSTing a query and then\n going to its URL. Hence, a\n [`QueryResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle] is\n the standard response model for this endpoint.\n\n If the timeout time is reached and the query has not yet finished, the user is\n redirected to the specific URL for the query.\n\n If the `as_optimade` query parameter is `True`, the response will be parseable as a\n standard OPTIMADE entry listing endpoint like, e.g., `/structures`.\n For more information see the\n [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#entry-listing-endpoints).\n\n \"\"\"\n try:\n search = Search(\n query_parameters=OptimadeQueryParameters(\n **{\n field: getattr(entry_params, field)\n for field in OptimadeQueryParameters.model_fields\n if getattr(entry_params, field)\n }\n ),\n optimade_urls=search_params.optimade_urls,\n endpoint=search_params.endpoint,\n database_ids=search_params.database_ids,\n )\n except ValidationError as exc:\n raise BadRequest(\n detail=(\n \"A Search object could not be created from the given URL query \"\n f\"parameters. Error(s): {exc.errors}\"\n )\n ) from exc\n\n queries_response = await post_search(request, search=search)\n\n if not queries_response.data:\n LOGGER.error(\n \"QueryResource not found in POST /search response:\\n%s\", queries_response\n )\n raise RuntimeError(\n \"Expected the response from POST /search to return a QueryResource, it did \"\n \"not\"\n )\n\n once = True\n start_time = time()\n while time() < (start_time + search_params.timeout) or once:\n # Make sure to run this at least once (e.g., if timeout=0)\n once = False\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n query: QueryResource = await collection.get_one(\n filter={\"id\": queries_response.data.id}\n )\n\n if query.attributes.state == QueryState.FINISHED:\n if query.attributes.response and query.attributes.response.errors:\n for error in query.attributes.response.errors:\n if error.status:\n for part in error.status.split(\" \"):\n try:\n response.status_code = int(part)\n break\n except ValueError:\n pass\n if response.status_code and response.status_code >= 300:\n break\n else:\n response.status_code = 500\n\n if search_params.as_optimade:\n response = await query.response_as_optimade(url=request.url)\n LOGGER.debug(\n \"Returning response as OPTIMADE entry listing:\\n%s\", response\n )\n return response\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n ),\n )\n\n await asyncio.sleep(0.1)\n\n # The query has not yet succeeded and we're past the timeout time -> Redirect to\n # /queries/<id>\n return RedirectResponse(query.links.self)\npost_search(request, search) async","text":"POST /search
Coordinate a new OPTIMADE query in multiple databases through a gateway:
optimade_urls and database_idsGatewayCreate modelPOST gateway resource to get ID - using functionality of POST /gatewaysPOST Query resource - using functionality of POST /queriesPOST /queries response - QueriesResponseSingleoptimade_gateway/routers/search.py @ROUTER.post(\n \"/search\",\n response_model=QueriesResponseSingle,\n response_model_exclude_defaults=False,\n response_model_exclude_none=False,\n response_model_exclude_unset=True,\n tags=[\"Search\"],\n status_code=status.HTTP_202_ACCEPTED,\n responses=ERROR_RESPONSES,\n)\nasync def post_search(request: Request, search: Search) -> QueriesResponseSingle:\n \"\"\"`POST /search`\n\n Coordinate a new OPTIMADE query in multiple databases through a gateway:\n\n 1. Search for gateway in DB using `optimade_urls` and `database_ids`\n 1. Create [`GatewayCreate`][optimade_gateway.models.gateways.GatewayCreate] model\n 1. `POST` gateway resource to get ID - using functionality of `POST /gateways`\n 1. Create new [Query][optimade_gateway.models.queries.QueryCreate] resource\n 1. `POST` Query resource - using functionality of `POST /queries`\n 1. Return `POST /queries` response -\n [`QueriesResponseSingle`][optimade_gateway.models.responses.QueriesResponseSingle]\n\n \"\"\"\n databases_collection = await collection_factory(CONFIG.databases_collection)\n # NOTE: It may be that the final list of base URLs (`base_urls`) contains the same\n # provider(s), but with differring base URLS, if, for example, a versioned base URL\n # is supplied.\n base_urls: set[AnyUrl] = set()\n\n if search.database_ids:\n databases = await databases_collection.get_multiple(\n filter={\"id\": {\"$in\": await clean_python_types(search.database_ids)}}\n )\n base_urls |= {\n get_resource_attribute(database, \"attributes.base_url\")\n for database in databases\n if get_resource_attribute(database, \"attributes.base_url\") is not None\n }\n\n if search.optimade_urls:\n base_urls |= {_ for _ in search.optimade_urls if _ is not None}\n\n if not base_urls:\n msg = \"No (valid) OPTIMADE URLs with:\"\n if search.database_ids:\n msg += (\n f\"\\n Database IDs: {search.database_ids} and corresponding found \"\n \"URLs: \"\n f\"{[get_resource_attribute(database, 'attributes.base_url') for database in databases]}\" # noqa: E501\n )\n if search.optimade_urls:\n msg += f\"\\n Passed OPTIMADE URLs: {search.optimade_urls}\"\n raise BadRequest(detail=msg)\n\n # Ensure all URLs are `pydantic.AnyUrl`s\n if not all(isinstance(_, AnyUrl) for _ in base_urls):\n raise InternalServerError(\n \"Could unexpectedly not validate all base URLs as proper URLs.\"\n )\n\n databases = await databases_collection.get_multiple(\n filter={\"base_url\": {\"$in\": await clean_python_types(base_urls)}}\n )\n\n if len(databases) == len(base_urls):\n # At this point it is expected that the list of databases in `databases`\n # is a complete set of databases requested.\n pass\n\n elif len(databases) < len(base_urls):\n # There are unregistered databases, i.e., databases not in the local collection\n current_base_urls: set[AnyUrl] = {\n get_resource_attribute(database, \"attributes.base_url\")\n for database in databases\n }\n databases.extend(\n [\n LinksResource(\n id=str(url)\n .replace(\".\", \"__\")[len(url.scheme) + 3 :]\n .split(\"?\", maxsplit=1)[0]\n .split(\"#\", maxsplit=1)[0],\n type=\"links\",\n attributes=LinksResourceAttributes(\n name=str(url)[len(url.scheme) + 3 :]\n .split(\"?\", maxsplit=1)[0]\n .split(\"#\", maxsplit=1)[0],\n description=\"\",\n base_url=url,\n link_type=LinkType.CHILD,\n homepage=None,\n ),\n )\n for url in base_urls - current_base_urls\n ]\n )\n else:\n LOGGER.error(\n \"Found more database entries in MongoDB than then number of passed base \"\n \"URLs. This suggests ambiguity in the base URLs of databases stored in \"\n \"MongoDB.\\n base_urls: %s\\n databases %s\",\n base_urls,\n databases,\n )\n raise InternalServerError(\"Unambiguous base URLs. See logs for more details.\")\n\n gateway = GatewayCreate(databases=databases)\n gateway, created = await resource_factory(gateway)\n\n if created:\n LOGGER.debug(\"A new gateway was created for a query (id=%r)\", gateway.id)\n else:\n LOGGER.debug(\"A gateway was found and reused for a query (id=%r)\", gateway.id)\n\n query = QueryCreate(\n endpoint=search.endpoint,\n gateway_id=gateway.id,\n query_parameters=search.query_parameters,\n )\n query, created = await resource_factory(query)\n\n background_tasks: set[asyncio.Task] = set()\n\n if created:\n task = asyncio.create_task(perform_query(url=request.url, query=query))\n\n # Add task to the set. This creates a strong reference.\n background_tasks.add(task)\n\n # To prevent keeping references to finished tasks forever,\n # make each task remove its own reference from the set after\n # completion:\n task.add_done_callback(background_tasks.discard)\n\n collection = await collection_factory(CONFIG.queries_collection)\n\n return QueriesResponseSingle(\n links=ToplevelLinks(next=None),\n data=query,\n meta=meta_values(\n url=request.url,\n data_returned=1,\n data_available=await collection.acount(),\n more_data_available=False,\n schema=CONFIG.schema_url,\n **{f\"_{CONFIG.provider.prefix}_created\": created},\n ),\n )\nUtility functions for all routers.
"},{"location":"api_reference/routers/utils/#optimade_gateway.routers.utils.COLLECTIONS","title":"COLLECTIONS: dict[str, AsyncMongoCollection] = {} module-attribute","text":"A lazy-loaded dictionary of asynchronous MongoDB entry-endpoint collections.
"},{"location":"api_reference/routers/utils/#optimade_gateway.routers.utils.aretrieve_queryable_properties","title":"aretrieve_queryable_properties(schema, queryable_properties, entry_type=None) async","text":"Asynchronous implementation of retrieve_queryable_properties() from optimade
Reference to the function in the optimade API documentation: retrieve_queryable_properties().
Recursively loops through the schema of a pydantic model and resolves all references, returning a dictionary of all the OPTIMADE-queryable properties of that model.
Parameters:
Name Type Description Defaultschema type[EntryResource] The schema of the pydantic model.
requiredqueryable_properties Iterable[str] The list of properties to find in the schema.
requiredentry_type str | None The entry type of the model, if any.
None Returns:
Type DescriptionQueryableProperties A flat dictionary with properties as keys, containing the field description,
QueryableProperties unit, sortability, support level, queryability and type, where provided.
Source code inoptimade_gateway/routers/utils.py async def aretrieve_queryable_properties(\n schema: type[EntryResource],\n queryable_properties: Iterable[str],\n entry_type: str | None = None,\n) -> QueryableProperties:\n \"\"\"Asynchronous implementation of `retrieve_queryable_properties()` from `optimade`\n\n Reference to the function in the `optimade` API documentation:\n [`retrieve_queryable_properties()`](https://www.optimade.org/optimade-python-tools/api_reference/server/schemas/#optimade.server.schemas.retrieve_queryable_properties).\n\n Recursively loops through the schema of a pydantic model and resolves all\n references, returning a dictionary of all the OPTIMADE-queryable properties of that\n model.\n\n Parameters:\n schema: The schema of the pydantic model.\n queryable_properties: The list of properties to find in the schema.\n entry_type: The entry type of the model, if any.\n\n Returns:\n A flat dictionary with properties as keys, containing the field description,\n unit, sortability, support level, queryability and type, where provided.\n\n \"\"\"\n return retrieve_queryable_properties(\n schema=schema,\n queryable_properties=queryable_properties,\n entry_type=entry_type,\n )\ncollection_factory(name) async","text":"Get or initiate an entry-endpoint resource collection.
This factory utilizes the global dictionary COLLECTIONS. It lazily instantiates the collections and then caches them in the dictionary.
Parameters:
Name Type Description Defaultname str The configured name for the entry-endpoint resource collection.
requiredReturns:
Type DescriptionAsyncMongoCollection The OPTIMADE Gateway asynchronous implementation of the
AsyncMongoCollection MongoCollection.
Raises:
Type DescriptionValueError If the supplied name is not one of the configured valid collection names.
optimade_gateway/routers/utils.py async def collection_factory(name: str) -> AsyncMongoCollection:\n \"\"\"Get or initiate an entry-endpoint resource collection.\n\n This factory utilizes the global dictionary\n [`COLLECTIONS`][optimade_gateway.routers.utils.COLLECTIONS].\n It lazily instantiates the collections and then caches them in the dictionary.\n\n Parameters:\n name: The configured name for the entry-endpoint resource collection.\n\n Returns:\n The OPTIMADE Gateway asynchronous implementation of the\n [`MongoCollection`](https://www.optimade.org/optimade-python-tools/api_reference/server/entry_collections/mongo/#optimade.server.entry_collections.mongo.MongoCollection).\n\n Raises:\n ValueError: If the supplied `name` is not one of the configured valid collection\n names.\n\n \"\"\"\n if name in COLLECTIONS:\n return COLLECTIONS[name]\n\n if name == CONFIG.databases_collection:\n from optimade_gateway.mappers.databases import DatabasesMapper as ResourceMapper\n elif name == CONFIG.gateways_collection:\n from optimade_gateway.mappers.gateways import ( # type: ignore[no-redef]\n GatewaysMapper as ResourceMapper,\n )\n elif name == CONFIG.queries_collection:\n from optimade_gateway.mappers.queries import ( # type: ignore[no-redef]\n QueryMapper as ResourceMapper,\n )\n elif name == CONFIG.links_collection:\n from optimade_gateway.mappers.links import ( # type: ignore[no-redef]\n LinksMapper as ResourceMapper,\n )\n else:\n raise ValueError(\n f\"{name!r} is not a valid entry-endpoint resource collection name. \"\n \"Configured valid names: \"\n f\"{(CONFIG.databases_collection, CONFIG.gateways_collection, CONFIG.queries_collection, CONFIG.links_collection)}\" # noqa: E501\n )\n\n COLLECTIONS[name] = AsyncMongoCollection(\n name=name,\n resource_cls=ResourceMapper.ENTRY_RESOURCE_CLASS,\n resource_mapper=ResourceMapper,\n )\n\n return COLLECTIONS[name]\nget_entries(collection, response_cls, request, params) async","text":"Generalized /{entries} endpoint getter
optimade_gateway/routers/utils.py async def get_entries(\n collection: AsyncMongoCollection,\n response_cls: EntryResponseMany,\n request: Request,\n params: EntryListingQueryParams,\n) -> EntryResponseMany:\n \"\"\"Generalized `/{entries}` endpoint getter\"\"\"\n (\n results,\n data_returned,\n more_data_available,\n fields,\n include_fields,\n ) = await collection.afind(params=params)\n\n if more_data_available:\n # Deduce the `next` link from the current request\n query = urllib.parse.parse_qs(request.url.query)\n query[\"page_offset\"] = [int(query.get(\"page_offset\", [0])[0]) + len(results)] # type: ignore[list-item, arg-type]\n urlencoded = urllib.parse.urlencode(query, doseq=True)\n base_url = get_base_url(request.url)\n\n links = ToplevelLinks(next=f\"{base_url}{request.url.path}?{urlencoded}\")\n else:\n links = ToplevelLinks(next=None)\n\n if fields or include_fields:\n results = handle_response_fields(results, fields, include_fields)\n\n return response_cls(\n links=links,\n data=results,\n meta=meta_values(\n url=request.url,\n data_returned=data_returned,\n data_available=await collection.acount(),\n more_data_available=more_data_available,\n schema=CONFIG.schema_url,\n ),\n )\nget_valid_resource(collection, entry_id) async","text":"Validate and retrieve a resource
Source code inoptimade_gateway/routers/utils.py async def get_valid_resource(\n collection: AsyncMongoCollection, entry_id: str\n) -> EntryResource:\n \"\"\"Validate and retrieve a resource\"\"\"\n await validate_resource(collection, entry_id)\n return await collection.get_one(filter={\"id\": entry_id})\nresource_factory(create_resource) async","text":"Get or create a resource
Currently supported resources:
\"databases\" (DatabaseCreate -> LinksResource)\"gateways\" (GatewayCreate -> GatewayResource)\"queries\" (QueryCreate -> QueryResource)For each of the resources, \"uniqueness\" is determined in the following way:
DatabasesThe base_url field is considered unique across all databases.
If a base_url is provided via a Link model, the base_url.href value is used to query the MongoDB.
The collected list of databases.attributes.base_url values is considered unique across all gateways.
In the database, the search is done as a combination of the length/size of the databases' Python list/MongoDB array and a match on all (using the MongoDB $all operator) of the databases.attributes.base_url element values, when compared with the create_resource.
Important
The database_ids attribute must not contain values that are not also included in the databases attribute, in the form of the IDs for the individual databases. If this should be the case an OptimadeGatewayError will be thrown.
The gateway_id, query_parameters, and endpoint fields are collectively considered to define uniqueness for a QueryResource in the MongoDB collection.
Attention
Only the /structures entry endpoint can be queried with multiple expected responses.
This means the endpoint field defaults to \"structures\", i.e., the StructureResource resource model.
Parameters:
Name Type Description Defaultcreate_resource DatabaseCreate | GatewayCreate | QueryCreate The resource to be retrieved or created anew.
requiredReturns:
Type Descriptiontuple[LinksResource | GatewayResource | QueryResource, bool] Two things in a tuple:
GatewayResource; a QueryResource; or a LinksResource andoptimade_gateway/routers/utils.py async def resource_factory(\n create_resource: DatabaseCreate | GatewayCreate | QueryCreate,\n) -> tuple[LinksResource | GatewayResource | QueryResource, bool]:\n \"\"\"Get or create a resource\n\n Currently supported resources:\n\n - `\"databases\"`\n ([`DatabaseCreate`][optimade_gateway.models.databases.DatabaseCreate]\n ->\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource))\n - `\"gateways\"` ([`GatewayCreate`][optimade_gateway.models.gateways.GatewayCreate] ->\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource])\n - `\"queries\"` ([`QueryCreate`][optimade_gateway.models.queries.QueryCreate] ->\n [`QueryResource`][optimade_gateway.models.queries.QueryResource])\n\n For each of the resources, \"uniqueness\" is determined in the following way:\n\n === \"Databases\"\n The `base_url` field is considered unique across all databases.\n\n If a `base_url` is provided via a\n [`Link`](https://www.optimade.org/optimade-python-tools/api_reference/models/jsonapi/#optimade.models.jsonapi.Link)\n model, the `base_url.href` value is used to query the MongoDB.\n\n === \"Gateways\"\n The collected list of `databases.attributes.base_url` values is considered\n unique across all gateways.\n\n In the database, the search is done as a combination of the length/size of the\n `databases`' Python list/MongoDB array and a match on all (using the MongoDB\n `$all` operator) of the\n [`databases.attributes.base_url`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResourceAttributes.base_url)\n element values, when compared with the `create_resource`.\n\n !!! important\n The `database_ids` attribute **must not** contain values that are not also\n included in the `databases` attribute, in the form of the IDs for the\n individual databases. If this should be the case an\n [`OptimadeGatewayError`][optimade_gateway.common.exceptions.OptimadeGatewayError]\n will be thrown.\n\n === \"Queries\"\n The `gateway_id`, `query_parameters`, and `endpoint` fields are collectively\n considered to define uniqueness for a\n [`QueryResource`][optimade_gateway.models.queries.QueryResource] in the MongoDB\n collection.\n\n !!! attention\n Only the `/structures` entry endpoint can be queried with multiple expected\n responses.\n\n This means the `endpoint` field defaults to `\"structures\"`, i.e., the\n [`StructureResource`](https://www.optimade.org/optimade-python-tools/all_models/#optimade.models.structures.StructureResource)\n resource model.\n\n Parameters:\n create_resource: The resource to be retrieved or created anew.\n\n Returns:\n Two things in a tuple:\n\n - Either a\n [`GatewayResource`][optimade_gateway.models.gateways.GatewayResource];\n a [`QueryResource`][optimade_gateway.models.queries.QueryResource]; or a\n [`LinksResource`](https://www.optimade.org/optimade-python-tools/api_reference/models/links/#optimade.models.links.LinksResource)\n and\n - whether or not the resource was newly created.\n\n \"\"\"\n created = False\n\n if isinstance(create_resource, DatabaseCreate):\n collection_name = CONFIG.databases_collection\n\n base_url = get_resource_attribute(create_resource, \"base_url\")\n\n mongo_query: dict[str, Any] = {\n \"$or\": [\n {\"base_url\": {\"$eq\": base_url}},\n {\"base_url.href\": {\"$eq\": base_url}},\n ]\n }\n elif isinstance(create_resource, GatewayCreate):\n collection_name = CONFIG.gateways_collection\n\n # One MUST have taken care of database_ids prior to calling `resource_factory()`\n database_attr_ids = {_.id for _ in create_resource.databases or []}\n unknown_ids = {\n database_id\n for database_id in create_resource.database_ids or []\n if database_id not in database_attr_ids\n }\n if unknown_ids:\n raise OptimadeGatewayError(\n \"When using `resource_factory()` for `GatewayCreate`, `database_ids` \"\n f\"MUST not include unknown IDs. Passed unknown IDs: {unknown_ids}\"\n )\n\n mongo_query = {\n \"databases\": {\"$size\": len(create_resource.databases or [])},\n \"databases.attributes.base_url\": {\n \"$all\": [_.attributes.base_url for _ in create_resource.databases or []]\n },\n }\n elif isinstance(create_resource, QueryCreate):\n collection_name = CONFIG.queries_collection\n\n # Currently only /structures entry endpoints can be queried with multiple\n # expected responses.\n create_resource.endpoint = (\n create_resource.endpoint\n if create_resource.endpoint is not None\n else EndpointEntryType(\"structures\")\n )\n\n mongo_query = {\n \"gateway_id\": {\"$eq\": create_resource.gateway_id},\n \"query_parameters\": {\"$eq\": create_resource.query_parameters},\n \"endpoint\": {\"$eq\": create_resource.endpoint},\n }\n else:\n raise TypeError(\n \"create_resource must be either a DatabaseCreate, GatewayCreate, or \"\n f\"QueryCreate object, not {type(create_resource)!r}\"\n )\n\n collection = await collection_factory(collection_name)\n result, data_returned, more_data_available, _, _ = await collection.afind(\n criteria={\"filter\": await clean_python_types(mongo_query)}\n )\n\n if more_data_available:\n raise OptimadeGatewayError(\n \"more_data_available MUST be False for a single entry response, however it \"\n f\"is {more_data_available}\"\n )\n\n if result:\n if data_returned > 1:\n raise OptimadeGatewayError(\n f\"More than one {result[0].type} were found. IDs of found \"\n f\"{result[0].type}: {[_.id for _ in result]}\"\n )\n if isinstance(result, list):\n result = result[0]\n else:\n if isinstance(create_resource, DatabaseCreate):\n # Set required `LinksResourceAttributes` values if not set\n if not create_resource.description:\n create_resource.description = (\n f\"{create_resource.name} created by OPTIMADE gateway database \"\n \"registration.\"\n )\n\n if not create_resource.link_type:\n create_resource.link_type = LinkType.EXTERNAL\n\n if not create_resource.homepage:\n create_resource.homepage = None\n\n elif (\n isinstance(create_resource, GatewayCreate)\n and \"database_ids\" in create_resource.model_fields_set\n ):\n # Do not store `database_ids`\n del create_resource.database_ids\n create_resource.model_fields_set.remove(\"database_ids\")\n\n elif isinstance(create_resource, QueryCreate):\n create_resource.state = QueryState.CREATED\n\n result = await collection.create_one(create_resource)\n LOGGER.debug(\"Created new %s: %r\", result.type, result)\n created = True\n\n return result, created\nvalidate_resource(collection, entry_id) async","text":"Validate whether a resource exists in a collection
Source code inoptimade_gateway/routers/utils.py async def validate_resource(collection: AsyncMongoCollection, entry_id: str) -> None:\n \"\"\"Validate whether a resource exists in a collection\"\"\"\n if not await collection.exists(entry_id):\n raise NotFound(\n detail=f\"Resource <id={entry_id}> not found in {collection}.\",\n )\n