Merge pull request #1686 from bgilbert/devdocs

docs/development.md

@@ -18,6 +18,80 @@ sudo apt-get install libblkid-dev
 sudo dnf install libblkid-devel
 ```
 
+## Development notes
+
+See also the [Ignition rationale](rationale.md).
+
+### Code structure
+
+The [frontend](https://github.com/coreos/ignition/tree/main/config) handles config parsing and validation which need not run on the target system.  The [backend](https://github.com/coreos/ignition/tree/main/internal) performs the configuration of the target system.  The frontend is a stable library API that is used by other programs, so existing frontend API cannot be changed without bumping the Ignition major version.
+
+### Adding functionality
+
+New config directives should only be added if the desired behavior cannot reasonably be achieved with existing directives.  User-friendly wrappers for existing syntax ("sugar") should be handled by [Butane](https://github.com/coreos/butane).
+
+New behavior should only be added in the current experimental spec.  If new functionality is backported to older specs, and a config using an older spec comes to depend on that functionality, then it won't be obvious that the config will not work on all Ignition versions supporting that spec.  It's not always possible to follow this restriction, since the backend doesn't know what config version the user specified.  Where possible, use config validation to prevent the backend from seeing config directives that a spec version doesn't support (for example, values of the filesystem `format` field).
+
+New functionality added to a config spec must be declarative: it must describe what should exist, not what Ignition should do.  In particular, field names should not include verbs.  When adding functionality, carefully think through the interactions between features, which can be non-trivial.  Features should be orthogonal, minimal, and low-level; making them user-friendly is the responsibility of Butane sugar.
+
+When reprovisioning an existing node, the config may want to reuse existing disks and filesystems without reformatting them.  Config directives should support detecting and reusing an existing object (RAID volume, filesystem, etc.) if its properties match those specified in the config.
+
+Ignition specs should not include distro-specific functionality such as package management.  Features may require support from the distro (for example, setting kernel arguments), but such features should be broadly applicable.  Distro-specific options such as support for SELinux, or paths to external binaries, can be configured at build time in the [`distro`](https://github.com/coreos/ignition/blob/main/internal/distro/distro.go) package.  Distro-specific glue (e.g. support for reformatting the root filesystem) should be implemented outside the Ignition codebase, in Dracut modules that run between Ignition stages (see below).
+
+Ideally, functionality should not be added to an experimental spec in the same Ignition release that the spec is stabilized.  Doing so prevents users from trying out the functionality before we commit to maintaining it.
+
+### Modifying existing functionality
+
+Bugfixes can be backported to older specs if working configs will not be affected and the current behavior is unintended.  New config validations can (and should) be backported if the prohibited behavior always would have failed anyway.
+
+Existing config semantics can only be changed (without adding a flag field) if the spec major version is bumped.  This might be appropriate e.g. to clean up some awkward syntax or change some default behavior.  However, altogether removing functionality is very costly.  The spec 2.x to 3.x transition was difficult because some 2.x configs cannot be represented in spec 3.x, so users were required to manually update their configs.  If a major version bump only rearranges functionality but doesn't remove any, Ignition can automatically translate the previous major version to the current one, and no [flag day](https://en.wikipedia.org/wiki/Flag_day_(computing)) will be required.
+
+### Validation and failures
+
+Ignition is a low-level tool and does not attempt to prevent configs from doing unreasonable things.
+
+- Config validation should fail when it's clear from inspection that Ignition will be unable to perform the requested provisioning.  This must occur when the config is contradictory (e.g. specifies the same filesystem object as both a file and a directory) and thus would break the declarativeness of the spec, and may occur where the problem will produce a routine error at runtime (e.g. invalid arguments to `mkfs`).
+- Ignition must fail at runtime when it is dynamically unable to perform the requested provisioning.
+- If a config is implementable but will render the system non-functional, Ignition should execute the config anyway.  Any user-friendly detection of unreasonable configs should happen in Butane.
+
+### Platforms
+
+Platform providers should continue retrying config fetch until it's clear whether the user has provided an Ignition config.  If Ignition eventually timed out when fetching a config, then a slow network device or block device (on a very large machine or a heavily-loaded VM host) could cause Ignition to prematurely fail, or to continue booting without applying the specified config.
+
+Platform providers must allow the user not to provide a config, e.g. to boot an exploratory OS instance and use Afterburn to inject SSH keys.
+
+Ignition must never read from config providers that aren't under the control of the platform, since this could allow config injection from unintended sources.  For example, `169.254.169.254` is a link-local address and could easily be spoofed on platforms that don't specially handle that address.  As a corollary, the platform ID must always be explicitly set by the OS image, never guessed.
+
+### Network access
+
+All config fields that cause network accesses, directly or indirectly, should be added to the `fetch-offline` needs-net detector.
+
+Any network accesses performed by subprocesses should be mimicked by Ignition before starting the subprocess.  This ensures that Ignition's retry logic is used, so Ignition doesn't improperly fail if the network is still coming up.
+
+### Execution stages
+
+Ignition execution is divided into stages to allow other OS functionality to run in the middle of provisioning:
+
+1. `fetch-offline` stage
+1. OS enables networking if required
+1. `fetch` stage
+1. OS examines fetched config and e.g. copies root filesystem contents to RAM
+1. `disks` stage
+1. OS mounts root filesystem
+1. `mount` stage
+1. OS does any preprocessing of configured filesystems, including copying root filesystem contents back
+1. `files` stage
+1. OS does any postprocessing of provisioned system
+1. `umount` stage
+
+New stages should only be created when OS hooks would otherwise need to run in the middle of a stage.  Note that various external projects hardcode the list of Ignition stages.
+
+### Security
+
+Ignition must always provide secure defaults, and does not provide config directives that support or encourage unsafe behavior.  For example, Ignition does not support disabling HTTPS certificate checks, nor seeding the system entropy pool from potentially deterministic sources.  Similarly, the LUKS `discard` option exists because users may legitimately want to trade off some security for hardware longevity, but Ignition defaults to the secure option.
+
+Users might put secrets in Ignition configs.  On many platforms, userdata is accessible to unprivileged programs at runtime, potentially leaking those secrets.  When the userdata is accessible via a network service, users can configure firewall rules to prevent such access, but this may not be possible for hypervisors that expose userdata through a kernel interface.  Where possible, platform providers should provide a `DelConfig` method allowing Ignition to delete the userdata from the platform after provisioning is complete.
+
 ## Modifying the config spec
 
 Install [schematyper](https://github.com/idubinskiy/schematyper) to generate Go structs from JSON schema definitions.
@@ -49,7 +123,15 @@ After modifying `go.mod` run `make vendor` to update the vendor directory.
 
 Group changes to `go.mod`, `go.sum` and `vendor/` in their own commit; do not make code changes and vendoring changes in the same commit.
 
-## Running Blackbox Tests
+## Testing
+
+Ignition uses three different test frameworks:
+
+- Unit tests (`./test`) validate functionality that only affects internal program state.
+- Blackbox tests validate config directives that affect the target disk.
+- Fedora CoreOS [kola tests](https://coreos.github.io/coreos-assembler/kola/) validate functionality that interacts with platforms (e.g. config fetching) or the rest of the OS.  kola tests may be [internal](https://github.com/coreos/coreos-assembler/tree/main/mantle/kola/tests/ignition) or [external](https://github.com/coreos/fedora-coreos-config/tree/testing-devel/tests/kola/ignition).
+
+### Running blackbox tests
 
 ```sh
 ./build_blackbox_tests
@@ -68,9 +150,9 @@ You can get a list of available tests to run by passing the `-list` option, like
 sudo sh -c 'PATH=$PWD/bin/amd64:$PATH ./tests.test -list'
 ```
 
-## Test Host System Dependencies
+### Blackbox test host system dependencies
 
-The following packages are required by the Blackbox Test:
+The following packages are required by the blackbox tests:
 
 * `util-linux`
 * `dosfstools`
@@ -82,27 +164,21 @@ The following packages are required by the Blackbox Test:
 * `mdadm`
 * `libblkid-devel`
 
-## Writing Blackbox Tests
+### Writing blackbox tests
 
 To add a blackbox test create a function which yields a `Test` object. A `Test` object consists of the following fields:
 
-Name: `string`
-
-In: `[]Disk` object, which describes the Disks that should be created before Ignition is run.
-
-Out: `[]Disk` object, which describes the Disks that should be present after Ignition is run.
-
-MntDevices: `MntDevice` object, which describes any disk related variable replacements that need to be done to the Ignition config before Ignition is run. This is done so that disks which are created during the test run can be referenced inside of an Ignition config.
-
-SystemDirFiles: `[]File` object which describes the Files that should be written into Ignition's system config directory before Ignition is run.
-
-Config: `string` type where the specific config version should be replaced by `$version` and will be updated before Ignition is run.
-
-ConfigMinVersion: `string` type which describes the minimum config version the test should be run with. Copies of the test will be generated for every version, inside the same major version, that is equal to or greater than the specified ConfigMinVersion. If the test should run only once with a specfic config version, leave this field empty and replace $version in the `Config` field with the desired version.
+- `Name`: `string`
+- `In`: `[]Disk` object, which describes the Disks that should be created before Ignition is run.
+- `Out`: `[]Disk` object, which describes the Disks that should be present after Ignition is run.
+- `MntDevices`: `MntDevice` object, which describes any disk related variable replacements that need to be done to the Ignition config before Ignition is run. This is done so that disks which are created during the test run can be referenced inside of an Ignition config.
+- `SystemDirFiles`: `[]File` object which describes the Files that should be written into Ignition's system config directory before Ignition is run.
+- `Config`: `string` type where the specific config version should be replaced by `$version` and will be updated before Ignition is run.
+- `ConfigMinVersion`: `string` type which describes the minimum config version the test should be run with. Copies of the test will be generated for every version, inside the same major version, that is equal to or greater than the specified ConfigMinVersion. If the test should run only once with a specfic config version, leave this field empty and replace $version in the `Config` field with the desired version.
 
 The test should be added to the init function inside of the test file. If the test module is being created then an `init` function should be created which registers the tests and the package must be imported inside of `tests/registry/registry.go` to allow for discovery.
 
-UUIDs may be required in the following fields of a Test object: In, Out, and Config. Replace all GUIDs with GUID varaibles which take on the format `$uuid<num>` (e.g. $uuid123). Where `<num>` must be a positive integer. GUID variables with identical `<num>` fields will be replaced with identical GUIDs. For example, look at [tests/positive/partitions/zeros.go](https://github.com/coreos/ignition/blob/main/tests/positive/partitions/zeros.go).
+UUIDs may be required in the following fields of a `Test` object: `In`, `Out`, and `Config`. Replace all GUIDs with GUID varaibles which take on the format `$uuid<num>` (e.g. $uuid123). Where `<num>` must be a positive integer. GUID variables with identical `<num>` fields will be replaced with identical GUIDs. For example, look at [tests/positive/partitions/zeros.go](https://github.com/coreos/ignition/blob/main/tests/positive/partitions/zeros.go).
 
 ## Releasing Ignition
 

docs/rationale.md

@@ -28,8 +28,14 @@ Ignition configs describe the state of a system. Ignition configs do _not_ list
 
 Ignition configs do not allow users to provide arbitrary logic (including scripts for Ignition to run). Users describe which filesystems must exist, which files must be created, which users must exist, and etc. Any further customization must use systemd services, created by Ignition.
 
+Ignition does not guess how the system should be configured.  For example, it does not automatically select a disk to hold any requested partitions.  Users must specify the configuration explicitly.
+
 ## Ignition configs should not be written by hand
 
-Ignition configs were designed to be human readable, but difficult to write, to discourage users from attempting to write configs by hand.
+Ignition configs were designed to be human readable, but difficult to write, to discourage users from attempting to write configs by hand.  Ignition configs support basic primitives such as files, directories, filesystems, and partitions, but do not provide higher-level functionality for configuring specific system services.
 
 Use [Butane](https://coreos.github.io/butane/), or a similar tool, to generate Ignition configs. Butane reads [Butane configs](https://coreos.github.io/butane/specs/), which are YAML files containing distribution-independent Ignition directives and optional distribution-specific directives that provide an easier way to perform certain configuration tasks. Butane validates a Butane config for common errors and produces an Ignition config ready to be passed to a machine.
+
+## Ignition is distro-independent
+
+Ignition provides functionality that is broadly applicable to image-based Linux distributions.  It does not provide distro-specific functionality such as package management.