Skip to content

Gateway

Jump to bottom
Joe Kralicky edited this page Mar 1, 2023 · 13 revisions

Summary

The Gateway is the central point of entry for opni agent and metric communication.

Table of contents

Architecture

The Opni Gateway is a multi-faceted API server that manages connections and communication with agents. It comprises several servers, each with a specific role.

API Servers

  • Public gRPC API Server: This is the only publicly accessible server, offering a minimal set of APIs necessary for agents to authenticate and connect to the gateway. Most other interactions with the gateway occur through a long-lived bidirectional stream, which agents initiate by connecting to a service on this endpoint.

  • Internal Management Server: This server exposes RESTful APIs for managing core internal resources such as clusters, bootstrap tokens, RBAC objects, and capabilities. Additionally, it enables API extensions, allowing plugins to expose custom gRPC services at the same endpoint as the core management API. These endpoints aren't accessible outside the cluster.

  • Internal HTTP Server: This server handles the /metrics endpoint and the admin dashboard. The dashboard is a single page app served from static web assets embedded into the binary at build time. The HTTP server also supports API extensions, allowing plugins to register custom routes. Similar to the management server, these endpoints aren't accessible outside the cluster.

  • Local HTTP Server: This server, only accessible within the gateway pod, handles the /debug/pprof endpoint for diagnostics and the /healthz endpoint for Kubelet health checks.

Plugins

The gateway uses the hashicorp/go-plugin library to manage plugins. A fixed set of interfaces are available for plugins to implement. These interfaces let plugins interact with different systems within the gateway. A single plugin binary can have implementations for many interfaces, and there are no required interfaces.

Plugins contain the majority of the implementation details and logic for the "capabilities" of Opni, such as Monitoring and Logging. They may also contain other APIs that aren't part of the core gateway.

Plugin Loader

The agent uses a Plugin Loader to load plugin binaries from disk. The agent loads the plugins once on startup, and it doesn't unload or restart them. The plugin loader loads all plugins at the same time in parallel. Plugins can't have dependencies on other plugins, but they can use the management API to query for the existence of specific API extensions, for example.

Compiling Plugins

The plugins/ directory stores all plugin code. Each subdirectory has a main module, which compiles to a separate plugin binary, named plugin_<subdirectory>. For example, the plugins/example directory contains a main module that compiles to plugin_example. The plugin loader will only load binaries prefixed with plugin_.

API Extensions

API Extensions are an important mechanism that enables a plugin to communicate with other systems in the central Opni cluster, with agents, other plugins, and with the admin dashboard and CLI. The gateway facilitates transparently routing relevant API requests to the plugins, making the process of implementing API extensions easy.

  • Management API Extensions

    Management API extensions are GRPC services served by plugins, which are transparently forwarded to them by the gateway. The gateway is the only public entrypoint into Opni, so it has the responsibility for serving the aggregate of all necessary API endpoints and routing requests to the various backends and plugins.

    Management API extension services can have unary, server-streaming, and client-streaming RPCs. bidirectional-streaming RPCs are not implemented.

  • HTTP API Extensions

    HTTP API extensions are simpler - plugins register route prefixes they want to serve, and the gateway serves them at its internal HTTP endpoint, then round-trips requests to be handled by the plugin matching the request's URL path.

  • Stream API Extensions

    Stream API extensions are different from other types of API extensions. They allow plugins to register GRPC services to their respective side of a long-lived stream that connects each agent to the gateway. The raw stream is abstracted away, and instead both sides of the connection register GRPC services and interact with their peer via standard GRPC clients. These services can only contain unary RPCs.

    In a similar way as with management API extensions, when plugins register GRPC services, the plugin host will route incoming requests to the appropriate plugin. Because both sides of the connection each have their own set of plugins, any plugin from either side will be able to interact with services from any other plugin on the other side of the connection.

    When implementing a plugin that uses stream API extensions, it's important to consider the location of the plugin within the system topology. A single gateway will serve many agents, and each agent will have its own stream. The implementation of gateway-side stream API extension services should ensure method handlers are thread-safe and reentrant, as they will be invoked concurrently by multiple clients. The gateway also injects the peer's agent ID into the context of each RPC call. On the other hand, the agent-side services will only have a single client (the gateway), which simplifies the implementation of the service.

    The plugin hosts (gateway or agent) can also register services to the same stream. By default, services registered by the plugin host (gateway/agent) itself aren't visible from the host's own plugins. This is enabled for specific services only, such as the Identity service for agents.

Scale and performance

Scaling the gateway is currently not supported, since there are a couple stateful components that need to be upgraded to support it. This is a work in progress. The majority of the components of the gateway are stateless.

The gateway should be able to handle a large number of clusters before scaling is necessary. Load testing is still in progress, tracked here: https://github.com/rancher/opni/issues/275

Security

Being the only component of Opni that is designed to be exposed to the internet, security is a critical aspect of the gateway. There are two major areas of importance: communication between the gateway and agents, and communication between the gateway and browsers, such as through Grafana or the admin dashboard.

  • Communication between the gateway and agents

    The agent bootstrap process is the first step in securely connecting agents to the gateway.

    The bootstrap process is as follows:

    1. The server generates a self-signed keypair, and a bootstrap token. The bootstrap tokens are modeled after Kubernetes bootstrap tokens.
    2. The client is given the bootstrap token and one or more fingerprints of public keys in the server's certificate chain ("pinned" public keys). It first sends an empty request to the server's /bootstrap/join endpoint. The client cannot yet trust the server's self-signed certificate, so it does not send any important data in the request.
    3. During the TLS handshake, the client computes the fingerprints of the public keys in the server's offered certificates, and compares them to its pinned fingerprints. If any of the fingerprints match, and the server's certificate chain is valid (i.e. each certificate is signed by the next certificate in the chain), the client trusts the server and completes the TLS handshake. The client will save the server's root certificate in its keyring for later on.
    4. The server responds with several JWS messages with detached payloads (one for each active bootstrap token).
    5. The client finds the JWS with the matching bootstrap token ID, fills in the detached payload (the bootstrap token), and sends it back to the server's /bootstrap/join endpoint along with the client's own unique identifier it wishes to use (typically the client's kube-system namespace resource UID) and an ephemeral X25519 public key. This step requires the client to trust the server, because the complete JWS (which includes the bootstrap token) is a secret.
    6. The server verifies the reconstructed JWS. If it is correct, the server can now trust the client. The server responds with its own ephemeral X25519 public key.
    7. Both the client and server use their ephemeral private key and their peer's public key to generate a shared secret using ECDH. Then, this secret is passed through a KDF to create two static 32-byte keys. One is used to generate and verify MACs for client->server messages, and the other is used to generate and verify MACs for server->client messages. The KDF is the same as in the libsodium key exchange api.
    8. These keys are added to identical keyrings on both the client and server, and saved to persistent storage.

    In subsequent connections to the gateway, the agent must use this keyring to authenticate itself as follows:

    1. The client sends a connection request to the gateway to attempt to establish a long-lived bidirectional stream. The same TLS configuration is used, as before. In its initial request, the client sends its ID and a random string of 32 bytes.
    2. If the ID doesn't exist, the server rejects the request. Otherwise, the server replies to the client with a random 32-byte challenge.
    3. The client computes a MAC over clientId || clientRandom || serverChallenge and sends the result back to the server.
    4. The server validates the response, and if successful, will send a final message to the client containing the authenticated ID, as well as a MAC over all previous messages exchanged. The server challenge validation step happens in constant time.
    5. The client validates this MAC using its own record of the exchanged messages.

  • Communication between the gateway and browsers

    • Admin dashboard

      The admin dashboard is served by the gateway directly and has access to the management APIs. This endpoint is not protected by any authentication and is currently not meant to be exposed outside the central Opni cluster - rather, accessed via port-forwarding, rancher service proxy, or similar access methods. However, this will likely change in the future as we introduce a more comprehensive user authentication system.

    • Grafana dashboards

      Grafana is designed to be publicly accessible, and as such communications between Grafana and Opni are secure. The Grafana dashboards deployed by Opni Monitoring are configured to use OIDC for user authentication and authorization, and to forward an Authentication header to Opni APIs when making HTTP requests to the Opni Prometheus data source (which routes through the gateway to the metrics plugin). The gateway uses this header to authenticate and authorize the user who is signed into Grafana by validating the bearer token with the OpenID Provider's JWKS. This JWKS contains a set of public keys and is served at a standard endpoint by OpenID compatible servers.

High availability

HA gateway is not yet supported, but will be in the near future. Tracking here: https://github.com/rancher/opni/issues/275

Testing

Most of the Gateway's tests are integration-style located in test/integration. There are also tests for most packages in pkg/ used by the gateway.

Tracking here: https://github.com/rancher/opni/issues/813

There are also basic e2e tests in test/e2e, tracking additional work on these here: https://github.com/rancher/opni/issues/814

Test coverage details are available on codecov: https://app.codecov.io/gh/rancher/opni

Architecture
How Tos
Releases
Clone this wiki locally