Add architecture documentation (#599)

* Add docs on architecture.

* Add opening remark.

* Add to index

* Add headings, and copyedits

Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

* MyST cannot resolve reference to target that is later in document?

* Explicit targets for architecture docs links

* https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#creating-explicit-targets
* Hint in sphinx config for adding Myst config options

* Strip -target for friendlier archor fragments.

* Add -arch to scope the anchor to this document.

* Add links to further reading

---------

Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>
Co-authored-by: Padraic Shafer <pshafer@bnl.gov>

docs/source/conf.py

@@ -138,6 +138,11 @@
 }
 
 
+# -- Options for MyST -----------------------------------------------------
+
+# myst_heading_anchors = 2
+
+
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.

docs/source/explanations/architecture.md

@@ -0,0 +1,126 @@
+# Architecture
+
+This is an overview of the major components of Tiled.
+
+(client-arch)=
+## Client
+
+Tiled ships with a Python client. This is separable from the rest of the
+project, and could potentially someday be split off as a separate package. The
+client enables the user to navigate and access the data in a Tiled server using
+familiar Python item-lookup and slicing idioms, while it generates HTTP
+requests to transfer metadata and data.
+
+(connection-context-arch)=
+### Connection _Context_
+When the user connects, a **Context** object is created. The Context is
+shared by all client-side objects that use this connection. It wraps an
+[httpx HTTP client][httpx], which in turn wraps an HTTP connection pool and (if
+applicable) authentication-related state---either an API key or a pair of
+OAuth2 access and refresh tokens.
+
+Further Reading:
+* [`Context` reference](#context-ref)
+
+(client-side-caching-arch)=
+### Client-side _Caching_
+The Context also may hold an HTTP response **Cache**, similar to a web
+browser's cache. This is currently not enabled by default because it is
+experimental.
+
+Further Reading:
+* [Client HTTP Response Cache overview](#client-http-response-cache)
+* [`Cache` reference](#client-http-response-cache-ref)
+
+(server-arch)=
+## Server
+
+The Tiled HTTP server is built using the framework [FastAPI][], which is built
+on [Starlette][]. A key feature of FastAPI is auto-generated [OpenAPI][]
+documentation, which Tiled serves at the `GET /docs` endpoint. FastAPI
+works with [Pydantic][] to parse and validate requests.
+
+(authentication-arch)=
+### Authentication
+Most endpoints require authentication, unless the server is configured to be
+public. For single-user deployments, a single API key is specified or randomly
+generated at server startup. For multi-user deployments, an **Authentication
+Database** (PostgreSQL or SQLite) is used to store session information and to
+validate API keys.
+
+Further Reading:
+* [Security](#security)
+* [Login Tutorial](#login-tutorial)
+* [Authentication Details](#auth-details)
+* [Tiled Authentication Database](#tiled-authn-database)
+
+(accessing-data-and-metadata-arch)=
+### Accessing Data and Metadata
+Endpoints that serve metadata or data resolve the URL path to identify the
+relevant [**Adapter**](#adapter-arch), which returns the data
+as a scientific data structure. It may be a "lazy" data structure,
+representing data that will be loaded later -- on demand and piecemeal.
+
+(content-negotiation-and-serializers-arch)=
+### Content Negotiation and _Serializers_
+The endpoint implements [content negotiation][], comparing the list of requested formats
+that are accepted by the client to those supported by the server for this particular
+dataset. It dispatches to a registry of **Serializers**, which convert the data
+structure into bytes which can be sent in a response by FastAPI. Custom
+serializers may be registered during server configuration.
+
+Further Reading:
+* [Custom Export Formats](#custom-export-formats)
+* [Media Type Format Registry reference](#media-type-registry-ref)
+
+(compression-arch)=
+### Compression
+On its way, the response may be compressed, again using content negotiation
+to compare the list of compression schemes supported by the client (if any)
+to those the server deems appropriate to the dataset.
+
+(adapter-arch)=
+## Adapter
+
+In Tiled, an **Adapter** provides a standard interface to data, regardless of
+how it is stored. Adapters that wrap different structure have different
+interfaces. For example, an array adapter implements `read_block` whereas a
+table adapter implements `read_partition`. But all array adapters are alike,
+and all table adapters are alike. They enable the server to abstract over
+the details of _how_ to get the data.
+
+Typically Adapters take a filepath or URI and, perhaps, some additional
+configuration. For development, test, or demonstration purposes, Adapters can wrap
+in-memory data, such as a numpy array. Several Adapters are included in the Tiled
+codebase, in the spirit of "batteries included," but Adapters can be defined in
+external modules, too, and operate on the same footing as the built-in ones.
+
+Further Reading:
+* [Adapters](#adapters-ref)
+
+(catalog-arch)=
+## Catalog
+
+The Catalog is an Adapter that stores the metadata and structure for a
+potentially large number of datasets in a SQL database (PostgreSQL or SQLite).
+This enables it to efficiently respond to _metadata_ requests and _search_
+requests without opening any data files. Requests for _data_ are then
+dispatched down to the appropriate [Adapter](#adapter-arch) which can load the data
+from the given storage medium and format.
+
+Not all Tiled servers are configured to use the Catalog:
+
+* Demo deployments wrapping a handful of datasets use data Adapters directly,
+  with no need for a database.
+* Specialized deployments with custom Adapters may use a custom database or
+  external service.
+
+But for most standard applications, including serving a directory of files or
+providing a writable data store, the Catalog is used.
+
+[FastAPI]: https://fastapi.tiangolo.com/
+[httpx]: https://www.python-httpx.org/
+[Starlette]: https://www.starlette.io/
+[OpenAPI]: https://www.openapis.org/
+[Pydantic]: https://docs.pydantic.dev/
+[content negotiation]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation

docs/source/explanations/caching.md

@@ -23,6 +23,7 @@ Tiled has two kinds of caching:
    resources differently for different requests. The object cache will not provide
    quite the same speed boost as a response cache, but it has a broader impact.
 
+(client-http-response-cache)=
 ## Client-side HTTP Response Cache
 
 The client response cache is an LRU response cache backed by a SQLite file.

docs/source/how-to/custom-export-formats.md

@@ -1,3 +1,4 @@
+(custom-export-formats)=
 # Add Custom Export Formats
 
 The Tiled server can provide data---in whole or in part---in a variety of

docs/source/how-to/tiled-authn-database.md

@@ -1,3 +1,4 @@
+(tiled-authn-database)=
 # Set up a database for a scaled authenticated deloyment
 
 When Tiled is configured to use authentication provider(s), it employs a SQL

docs/source/index.md

@@ -35,6 +35,7 @@ how-to/tiled-authn-database
 ```{toctree}
 :caption: Explanations
 
+explanations/architecture
 explanations/standards
 explanations/structures
 explanations/metadata

docs/source/reference/authentication.md

@@ -1,3 +1,4 @@
+(auth-details)=
 # Authentication Details
 
 See {doc}`../explanations/security` for an overview.

docs/source/reference/python-client.md

@@ -252,6 +252,7 @@ Tiled currently includes two clients for each structure family:
    tiled.client.xarray.DatasetClient.read
 ```
 
+(context-ref)=
 ## Context
 
 ```{eval-rst}
@@ -269,6 +270,7 @@ Tiled currently includes two clients for each structure family:
    tiled.client.context.Context.tokens
 ```
 
+(client-http-response-cache-ref)=
 ## Cache
 
 ```{eval-rst}

docs/source/reference/service.md

@@ -1,5 +1,6 @@
 # Service-side Components
 
+(adapters-ref)=
 ## Adapters
 
 ### Python Object Adapters
@@ -66,6 +67,7 @@ assumption about your metadata / data and its meaning.
    tiled.query_registration.register
 ```
 
+(media-type-registry-ref)=
 ## Media Type (Format) Registry
 
 This is a registry of formats that the service can *write* upon a client's request.

docs/source/tutorials/login.md

@@ -1,3 +1,4 @@
+(login-tutorial)=
 # Log into an Authenticated Tiled Server
 
 For this tutorial, we will log in to the demo Tiled server at