WIP

Author: pktpls

inventory/README.md

@@ -1,3 +1,19 @@
+
+# About our inventory
+
+- Differences from usual Ansible projects
+- How bbb-configs uses Ansible
+- How hosts are constructed
+  - base_inventory
+  - keyed groups
+  - merge_vars
+- Working with the inventory
+
+
+
+
+---
+
 ### .. magic inside
 
 this inventory consists of three inventories.
@@ -15,3 +31,72 @@ this is a configuration for the "constructed" inventory plugin. it dynamically c
 
 # keyed_groups_stage_2.config
 this is yet another configuration for the "constructed" inventory plugin. in contrast to the first stage its purpose is to dynamically construct even more group memberships based on inherited vars from the previous stage.
+
+---
+
+Ansible inventory script for bbb-configs location files
+
+By default, Ansible isn't well suited for how we organize hosts and locations.
+We want one complete file for every location, which consists of one or more
+hosts with various roles as well as shared and individual config data.
+Ansible on the other hand starts at the "host vars" and then
+uses the "keyed groups" plugin to merge multiple stages of "group vars".
+For our case, this results in an amount of files that can feel overwhelming.
+
+This inventory script allows for complete single location files while
+at the same time being compatible with Ansible's host vars and group vars.
+Note that certain shared and common config data is still loaded using
+keyed groups and group vars where it makes sense (e.g. role and model).
+
+Ansible expects this script to respond to `--host <hostname>` and `--list`.
+For more information on inventory scripts, see:
+https://docs.ansible.com/ansible/latest/collections/ansible/builtin/script_inventory.html
+
+With `--host <hostname>`, if no matching host is found in `locations/*.yml`,
+we then look in Ansible's default `host_vars/` directory.
+
+With `--list`, we scan both locations/ and host_vars/.
+
+Data for hosts in host_vars/ is constructed by Ansible:
+1. Combine the data from all files in host_vars/<hostname>/
+2. Add keyed_groups as specified in inventory/
+3. Merge the data from all related groups in group_vars/<hostname>/
+
+Data for a specific host in locations/ is constructed as follows:
+1. Copy the location data into a new object, ignore the `.hosts` array
+2. Merge the array entry from `.hosts` with a matching `.hostname` value,
+   with shallow merge only at the moment
+3. Add keyed_groups as specified in inventory/
+4. Merge the data from all related groups in group_vars/<hostname>/
+
+For example:
+
+```yml
+location: pktpls
+hosts:
+  - hostname: pktpls-core
+    role: corerouter
+    string: host-var-has-precedence
+    object: { two: 456 }
+    array: [ bar ]
+string: will-be-overridden
+object: { one: 123 }
+array: [ foo ]
+```
+
+results in:
+
+```json
+{
+  "location": "pktpls",
+  "hostname": "pktpls-core",
+  "role": "corerouter",
+  "string": "host-var-has-precedence",
+  "object": {
+    "two": 456
+  },
+  "array": [
+    "bar"
+  ]
+}
+```

inventory/base_inventory

@@ -1,12 +1,67 @@
 #!/usr/bin/env bash
 
+#
+# This inventory script translates our data model to Ansible's data model.
+# We have one file per network location, and one or more hosts per location.
+# Ansible has groups and hosts, with one file per host, which wasn't practical.
+#
+# This script's output is a JSON object containing an array of all host names,
+# and an array with the initial hostvars for each host. See
+# https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html
+# for more general information on inventory construction.
+#
+# The queries and conversions are done with the help of the `jq` and `yq` tools.
+#
+# We first grab the JSON representation of every location YAML definition,
+# then read `.hosts[].hostname` for each location to create the list of all hosts.
+# Previously we also scanned the `host_vars/` directory for backward compatibility.
+#
+# Usually Ansible then calls this script with `--host <name>` for every host.
+# That would obviously get very slow with hundreds of hosts,
+# which is why Ansible lets us construct all hostvars objects in advance.
+#
+# To construct the hostvars object for a host,
+# we take as a base the full location object (without `.hosts`),
+# then merge the object from `.hosts[]` with a matching `hostname` value.
+# This way, host values overwrite location values.
+# (We actually merge host<-location<-host to preserve JSON ordering.)
+#
+# In total there are five stages to construct inventory and hostvars:
+#
+# 1. This inventory script
+#
+# 2. `host_vars/`
+#     Old hosts in `host_vars/` don't need the conversion in this script,
+#     they're still loaded by Ansible automatically.
+#
+# 3. Keyed groups
+#     The next two stages after inventory construction handle keyed groups,
+#     which load additional group data based on certain property values.
+#     This is handled outside of this script by an Ansible plugin.
+#     There are two stages so that new data from stage 1 can set properties
+#     that result in more new data in stage 2 (e.g. `model` and OpenWrt version).
+#     The first stage handles `location` (old), `target`, `model`, and `role`,
+#     while the second handles `target` and `openwrt_version`.
+#
+# 4. Merge vars
+#     The last stage is the `merge_vars` task.
+#     All hostvars construction so far was only able to overwrite properties.
+#     In some cases we need to merge with the existing property.
+#     For specific properties, a "merge var" can be set:
+#     `packages: ["some-pkg"]` and `xxx__packages__to_merge: ["another-pkg"]`
+#     where `xxx` is an arbitrary name to allow for multiple merge vars.
+#     Before any templates or tasks make use of hostvars,
+#     these merge vars are merged together into one.
+#     Applies to `ssh_keys`, `packages`, `sysctl`, `rclocal`,
+#     `disabled_services`, `wireless_profiles`, `channel_assignments_*`
+#
+
 set -e
 # set -x
 
 case "$1" in
 --host)
-    # No op - won't be called by Ansible anymore because --list contains all data.
-    # See https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html#tuning-the-external-inventory-script
+    # No op
     echo "{}"
     exit 0
     ;;