Create an easy way to define images inline in manifests

[wokalski]

Problem

Defining images separately from the gitops configuration just introduces extra cruft in typical setups where you don't share your image publicly. We have a monorepo to which we're adding nix-based kubernetes manifest definition and thus would like to have two simple commands:

1. Manifest generation (current functionality of nixidy)
2. Building images

Solution

Im thinking of adding an opinionated lib.mkImage that would expect identical input to nix2container.buildImage. Now, when environmentPackages are evaluated, the injected version of this function would simply take the image name and tag and concatenate them to form a string.

However, I would also provide a target analogous to copyToRegistry in nix2container that would recursively go through the definition and run a build of the image for every invocation of lib.mkImage. Now - to be frank I'm not entirely sure how to implement the latter yet since my nix foo is weak but I believe in principle it can be implemented.

Let me know what you think, I believe this can make the DX and overall devops story remarkable.

[arnarg]

Thanks for your interest in the project :)

I have been briefly thinking about how this could be implemented and I think it would need to be a separate attribute set passed to the modules. Similarly how in nixos pkgs is passed to every module.

You could then build and push all images recursively like is done with all packages in nixpkgs and in nixidy it would be used something like:

{registry, ...}: {
  applications.test = {
    namespace = "test";
    createNamespace = true;

    resources."apps/v1".Deployment.test = {
      # ...
      image = registry.test;
    };
  };
}

If you want to have the mkImage function call in the same file as the kubernetes definition, this might be a bit difficult to implement.

What are your thoughts on that?

[wokalski]

I like your idea better, I think.

I'll get to it eventually. I have to get a better feel of how I would like this to be used.

I'm really excited for it. So far nixidy is perfect for my needs.

[arnarg]

I'm glad to hear that. What does your ideal usage of nixidy look like?

I found it a bit clunky to define whole manifests in nix because many manifests use lists that don't merge well in nix. This is nicer in kubenix as it knows the schemas for resources and can therefore transform attributesets into lists as a final step.

I have just been using helm charts and kustomize for most things and nixidy as a final patching tool but I'd like to make this nicer.

[wokalski]

What do you mean by:

because many manifests use lists that don't merge well in nix

[arnarg]

For example ports on various resources is always a list/array in kubernetes manifests.

This is clunky to write in nix:

ports = [
  {
    name = "http";
    containerPort = 8080;
    protocol = "TCP";
  }
];

While this would be nicer:

ports.http = {
  containerPort = 8080;
  protocol = "TCP";
};

As the port can be set with lib.mkDefault and then overwritten in another module. The current workaround is to pull all overwritable stuff into user defined options.

In Essence nixidy just merges the manifests the best it can and then passes it to toYAML.

[wokalski]

I'll keep this in mind. As I move more of stuff over maybe I'll recognize some patterns and suggest APIs or at least communicate pain-points. Potentially to strike the balance between flexibility and ergonomics we could transform the resources field to use an ergonomic API similar to kubenix and then have another field such as rawManifests or something that provides the flexibility.

I can see this already exists 😄 in some form. I'll keep in mind what you said. If kubenix has a perfect API for declaring common manifests (I don't know about that) then we could probably add a kubenix field to application.

[arnarg]

@wokalski I finally had some time again to give this project any love and I'm using it now for my own "homelab" cluster.

Is there still interest from your side to use this?

I have not progressed on the topic of this issue but I have managed to use kubenix' generated resource options in nixidy.

I have an example of an application that is defined entirely in nix: https://github.com/arnarg/cluster/blob/03c73e87407c3b4895503483d0b9378ec75650f9/modules/miniflux/default.nix

I plan on creating re-usable generators to generate resource options from CRDs and then add common ones like CiliumNetworkPolicy.

[wokalski]

I am still using it indeed! I don't know of a better option for nix k8s manifests. It's just that I'm only using it for my test cluster (for now anyway).