New nix-based build workflow

[urob]

I had some fun re-organizing my ZMK workspace, leveraging west to set everything up. While at it, I also moved from a docker-based workflow to a nix-based workflow. I updated the readme with technical details. Let me add a bit more motivation and background here.

Does it affect past or future forks of this repo?

The short answer is: No! The new workflow is completely optional. The Github Actions workflow and traditional local development workflows all continue to work.

The only caveat is that I deleted my old docker-based build-scripts from this repository since I will no longer maintain them. If you still need them, you can download them from the latest commit before I deleted them.

What do I mean by ZMK workspace?

After fully embracing ZMK's module system, I decided to make my zmk-config (i.e., this repository) the entry point for everything ZMK. So instead of having multiple ZMK-related repositories scattered across my filesystem, they now all live inside a single zmk-workspace:

zmk-workspace
├── config
├── firmware (created after building)
├── modules
│   ├── auto-layer
│   ├── helpers
│   └── tri-state
└── zmk
    └── ...

Technically, zmk-workspace is the same as this zmk-config (I decided to rename it on my computer to reflect it's new bigger role). This may seem odd at first. After all, neither zmk nor any of the zmk-modules live in this repository. What ties everything together is my west.yml, which defines the structure of the workspace. In my opinion, using west to organize the workspace has several advantages:

• It mirrors the current Github Actions workflow, eliminating the scope for build discrepancies
• All ZMK modules are automatically registered to the workspace - no need to manually point to them when building locally!
• This is subjective, but I like having everything ZMK related in a single workspace with minimal hirarchies
• All west modules (i.e., zmk, zephyr and all zmk-modules) are fully functional git repos - stepping inside any of these subdirectories, one can switch zmk branches as usual, commit changes, merge PRs , etc. It's sort of like git-submodules, just better :)

Why a nix-based workflow?

Honestly, because it's awesome! Nix is a package manager that can be installed with a single command in macOS and Linux or WSL (not to be confused with nixOS, which is a operation system based on Nix). The major advantage is that it allows creating virtual, fully reproducible shells that isolate arbitrary development environments from the rest of the system.

Think about it as a Python venv but for the entire toolchain (in our case that's the zephyr-sdk, cmake, dtc, a python environment with west and various other packages, and even keymap-drawer). Like venv, the nix environment is completely isolated from the rest of the system. In that sense, it serves the same purpose as a docker container. But unlike docker, it spawns a native shell that we can interact with, without any added friction.

Sounds interesting but is it really convenient?

Here's where two additional tools come in: direnv and just. The former completely automates the whole setup and activation of the nix-environment. The first time we cd into our zmk-workspace, it automatically creates and activates the nix-environment. After that, the environment gets automatically activated and deactivated every time I enter and leave the workspace. (The environment is cached, so there is no waiting after it has been created the first time).

The second tool that I find really useful is just. Think about it as a Makefile on steroids. I have it set up to make various zmk-related recipes readily available from anywhere within the workspace.

For instance, typing just build all will parse build.yaml and build the firmware for all board and shield combinations, just like the Github Actions workflow does. It also supports matching of build targets. For example, just build zen will build any board and shield combination matching zen - in my case that's corneish_zen_v2_left and corneish_zen_v2_right.

On what platforms does it work?

It should run on all Linux and macOS systems (both Intel and Arm). Windows is indirectly supported via WSL. Currently, there is one known issue with keymap-drawer on arm-based Linux-systems, which is currently disabled.

For the current status of any system, see the latest run of the test suite.

Where do I get all the goodies?

Glad you're convinced :) I added step-by-step instructions to the readme. It's really not that much, most of the setup is automated (mainly through flake.nix, which sets up west, the zephyr-sdk and build tools, and through the just init recipe, which sets up the workspace).

[infused-kim]

This is fantastic. I can’t wait to try it out this weekend. Thank you for sharing and documenting it.

It’s going to be awesome on macOS to have a local toolchain that doesn’t require docker.

[urob]

There's one tiny change needed in the flake.nix file for this to run on macOS. You'll want to change system to either "x86_64-darwin" or "aarch64-darwin" depending on whether you have a Intel or Arm CPU

EDIT: As of c52a249 this is now automated.

I don't have a mac myself so I couldn't try it but hopefully everything should work. Let me know how it goes!

[urob]

Also I don't think so but there's a small chance I forgot to add some dependencies to the environment which I already had globally installed. If anything is missing, you can add it to the flake.nix here (and let me know so I can fix it):

zmk-config/flake.nix

Lines 28 to 31 in 09be0ba

	pkgs.cmake
	pkgs.dtc
	pkgs.ninja
	pkgs.yq

E.g., if you want a code formatter without installing it globally, you could add pkgs.prettierd, etc

(Readily available packages can be searched here: https://search.nixos.org/packages, other's like zephyr-sdk in our case can be packaged with a bit of extra work.)

[infused-kim]

Thank you. I didn’t get a chance to try it this weekend, but I will report back when I do.

[urob]

I have one report for from someone for whom it worked on macOS silicon. I made one change in the meantime trying to figure something out. In case I ended up breaking things, let me know & I'll undo it.

[k3d3]

That's awesome! I've been running my ZMK configs through zmk-nix - I assume you've looked at that already?

[urob]

Thanks for sharing the link, I hadn't seen this one before. From what I can tell the purpose it a bit different though. I'm using nix to create an isolated & automated dev environment (similar to how one would use docker). But utilize the standard zephy-sdk and west build for building.

I only had a cursory look, so I'd be curious to hear your thoughts. From what I can tell, the repo you shared, re-implements the entire build chain in nix, including the zephyr-sdk. To be honest, I don't fully understand the benefits of doing so. Is it about speed / caching? Or simply to use a more familiar configuration environment for people who already know nix? How well is it maintained? I see a lot of requirements being hard-coded (ex) - are they kept up to date when the requirements in ZMK or Zephyr change?

[urob]

I added some automated tests to validate the local build environment for all major nix systems (x86_64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin).

As of today the tests are passing for all tested systems. So this should work on all Linux and macOS systems. (Windows is indirectly supported via WSL.)

One small caveat is that I had to disable keymap-drawer on aarch64-linux due to an upstream issue (#95, NixOS/nixpkgs#372375).