-
Notifications
You must be signed in to change notification settings - Fork 0
Apply
The Apply family of Commands implement Kubernetes-style declarative resource definitions for Rapyuta.io resources. It also implements Helm-style templating with values.yaml file.
The original Rapyuta.io API resources is unclear and inconsistent in places. This makes it very hard to implement consistent resources for the Client. To implement the Kubernetes-style resources, the CLI defines its own version of Rapyuta.io Resources dubbed V2.
But, because it implements its own version of Resources that the current API does not support, Client-side translation is required. It brings in significant amount of Technical Debt into the project.
We have the plan to bring V2 resources to different parts of the Rapyuta.io system (Backend, Frontend, etc). Because of this, the resources are defined in JSON Schema.
The project uses fastjsonschema library to auto-generate the validation code using the JSON Schema definitions. The generate-validation.py script is available in the scripts/ directory of the Project. To generate the validations, run this script from the root of the repository without any arguments.
The model module defines an Abstract Class that requires Child classes to implement a set of CRUD methods. It also implements the top-level apply and delete method in terms of the CRUD methods that can be shared by all the Child classes.
Each resource module then defines a model.py file that inherits the Base Model and implement the resource specific CRUD methods.
One design decision is to refer to dependencies by name everywhere. It helps with the reproducibility because resources can be re-created any number of times across the projects with stable names and the manifests will still work. It is one of the great features of the Apply command, but it also brings in some Technical Debt.
The current API does not support this, instead everything works based on the GUID. So, the CLI needs to do a lot of heavy lifting to translate the Name to GUID at runtime. This may look trivial at first but it gets complex when we are dealing with multiple resources being applied at the same time. One resource in the sequence might be referring to the previous one that doesn’t exist on the server. So, pre-fetching the List of resources is out of question. Also, listing resources every single time is also not feasible because it slows down the process significantly.
The CLI implements the ResolverCache. It tries to cache the GUID of the resources it has already seen, and falls back to ListAPIs otherwise. It speeds up things a bit but still requires a lot of additional API calls and Client-side processing. This can be completely avoided in future by adding support for Name-based references on Server-side.
The CLI can apply a whole bunch of resources at once. This requires the understanding of dependencies between the resources. To implement this, we’ve used a consistent format for defining dependencies in JSON Schema. The Applier does a pre-processing of all the submitted resources to look for the dependency definitions, and construct an in-memory directed Dependency Graph.
The Applier performs Topological Sort on the Dependency Graph to figure out the order. The Topological Sort implementation allows us to Apply unrelated branches parallely as well. So, Applier can run parallel workers to apply branches parallelly.