docs: updated readme and added spell checking

Author: Morriz

.cspell.json

@@ -0,0 +1,26 @@
+{
+  "version": "0.2",
+  "ignorePaths": [ ],
+  "dictionaryDefinitions": [ ],
+  "dictionaries": [ ],
+  "words": [
+    "blocklists",
+    "crowdsec",
+    "cscli",
+    "dcpx",
+    "dcux",
+    "dida",
+    "healthchecks",
+    "howto",
+    "itsup",
+    "letsencrypt",
+    "openapi",
+    "portforwarding",
+    "traefik",
+    "upstreams",
+    "uvicorn",
+    "venv"
+  ],
+  "ignoreWords": [ ],
+  "import": [ ]
+}

.env.sample

@@ -1,8 +1,8 @@
-# uncomment in your prod env to make the repo update itself:
+# uncomment in your prod env to make the repo update itself via webhook:
 # PYTHON_ENV=production
 
 # Trusted ips for proxy protocol, usually the ip of your router (sending to traefik)
-TRUSTED_IPS_CIDRS=192.168.0.0/24
+TRUSTED_IPS_CIDRS=192.168.0.1,192.168.1.1,10.0.0.1
 
 # Traefik dashboard domain name
 TRAEFIK_DOMAIN=traefik.example.com

.gitignore

@@ -2,10 +2,12 @@ __pycache__
 .aider*
 .env
 .history
+.mypy_cache
 .venv
 certs/*
 db.yml
 proxy/docker-compose.yml
 proxy/nginx/*.conf
 proxy/traefik/*.yml
 upstream
+acme.json

.vscode/extensions.json

@@ -2,6 +2,7 @@
   "recommendations": [
     "esbenp.prettier-vscode",
     "ms-python.pylint",
-    "ms-python.mypy-type-checker"
+    "ms-python.mypy-type-checker",
+    "streetsidesoftware.code-spell-checker",
   ]
 }

.vscode/settings.json

@@ -10,6 +10,7 @@
     "editor.tabSize": 4
   },
   "black-formatter.importStrategy": "fromEnvironment",
+  "cSpell.enabled": true,
   "editor.defaultFormatter": "esbenp.prettier-vscode",
   "isort.check": true,
   "isort.importStrategy": "fromEnvironment",
@@ -44,5 +45,5 @@
   "python.testing.unittestEnabled": true,
   "[dotenv]": {
     "editor.defaultFormatter": "foxundermoon.shell-format"
-  }
+  },
 }

README.md

@@ -1,84 +1,156 @@
-# itsUP
+# itsUP <!-- omit in toc -->
 
-**Lean, automated, poor man's infra for lightweight services running in docker.**
+_Lean, secure, automated, zero downtime<sup>\*</sup>, poor man's infra for services running in docker._
 
-## Fully managed docker compose infra
+<img align="center" src="assets/freight-conductor.png">
+<p></p>
+<p>
+Running a home network? Then you may already have a custom setup, probably using docker compose. You might enjoy all the maintenance and tinkering, but you are surely aware of the pitfalls and potential downtime. If you think that is ok, or if you don't want automation, then this stack is probably not for you.
+Still interested? Then read on...
+</p>
 
-Single machine multi docker compose architecture with a low cpu/storage footprint and near-zero<sup>\*</sup> downtime.
+**Table of contents:**
 
-It runs two nginx proxies in series (proxy -> terminate) to be able to:
+- [Key concepts](#key-concepts)
+  - [Single source of truth](#single-source-of-truth)
+  - [Managed proxy setup](#managed-proxy-setup)
+  - [Managed service deployments \& updates](#managed-service-deployments--updates)
+  - [\*Zero downtime?](#zero-downtime)
+- [Prerequisites](#prerequisites)
+- [Howto](#howto)
+  - [Install \& run](#install--run)
+  - [Configure services](#configure-services)
+  - [Configure plugins](#configure-plugins)
+    - [CrowdSec](#crowdsec)
+  - [Using the Api \& OpenApi spec](#using-the-api--openapi-spec)
+    - [Webhooks](#webhooks)
+- [Dev/ops tools](#devops-tools)
+  - [utility functions for dev workflow](#utility-functions-for-dev-workflow)
+  - [Utility scripts](#utility-scripts)
+- [Questions one might have](#questions-one-might-have)
+  - [What about Nginx?](#what-about-nginx)
+  - [Does this scale to more machines?](#does-this-scale-to-more-machines)
+- [Disclaimer](#disclaimer)
 
-- terminate SSL/TLS
-- do SSL/TLS passthrough
-- target many upstream endpoints
+## Key concepts
 
-**Advantages:**
+### Single source of truth
 
-- shared docker network: encrypted intra-node communication (over a shared network named `proxynet`)
-- near-zero-downtime\*
+One file (`db.yml`) is used for all the infra and workloads it creates and manages, to ensure a predictable and reliable automated workflow.
+This means abstractions are used which means a trade off between flexibility and reliability, but the stack is easily modified and enhanced to meet your needs. We strive to mirror docker compose functionality, which means no concessions are necessary from a docker compose enthusiast's perspective.
 
-_<sup>_</sup>Near-zero-downtime?\*
+### Managed proxy setup
 
-Well, all (stateless) nodes that get rotated do not incur downtime, yet nginx neads a reload signal. During low traffic that will allow for a graceful handling of outstanding http sessions and no downtime, but may be problematic if nginx needs to wait for a magnitude of open sessions. In that case a timeout will force the last open sessions to be terminated.
-This approach is a very reliable and cheap approach to achieve zero downtime.
+itsUP generates and manages `proxy/docker-compose.yml` that runs two proxies in series (tcp -> web), with only the first exposing ports, to be able to:
 
-_But what about stateful services?_
+1. do TLS passthrough to existing endpoints (most people have secure Home Assistant setups already)
+2. terminate TLS and forward securely to managed endpoints over an encrypted `proxynet` docker network
 
-It is surely possible to deploy stateful services but those MUST NOT be targeted with the `entrypoint: xxx` prop, as those services are the entrypoints which MUST be stateless, as those are rolled up with the `docker rollout` by the automation. In order to update those services you are on your own, but it's a breeze compared to local installs, as you can just docker compose commands.
+### Managed service deployments & updates
 
-**Prerequisites:**
+itsUP generates and manages `upstream/{project}/docker-compose.yml` files to deploy container workloads as defined as a service in `db.yml`.
+This centralizes and abstracts away the plethora of custom docker compose setups that are mostly uniform in their approach anyway, so controlling their artifacts from one source of truth makes a lot of sense.
 
-- [docker](https://www.docker.com)
+### <sup>\*</sup>Zero downtime?
+
+Like with all docker orchestration platforms (even Kubernetes) this is dependent on the containers:
+
+- are healthchecks correctly implemented?
+- Are SIGHUP signals respected to shutdown within an acceptable time frame?
+- Are the containers stateless?
+
+itsUP will rollout changes by:
+
+1. bringing up a new container and wait till it is healthy (if it has a health check then max 60s, otherwise assumes it is healthy after 10s)
+2. kill the old container and wait for it to drain, then removes it
+
+_What about stateful services?_
+
+It is surely possible to deploy stateful services but beware that those might not be good candidates for the `docker rollout` automation. In order to update those services it is strongly advised to first read the upgrade documentation for the newer version and follow the prescribed steps. More mature databases might have integrated these steps in the runtime, but expect that to be an exception. So, to garner correct results you are on your own and will have to read up on your chosen solutions.
+
+## Prerequisites
+
+**Tools:**
+
+- [docker](https://www.docker.com) daemon and client
 - docker [rollout](https://github.com/Wowu/docker-rollout) plugin
-- Portforwarding of port `80` and `443` to the machine running this stack.
 
-## Howto
+**Infra:**
 
-### Configure
+- Portforwarding of port `80` and `443` to the machine running this stack. This stack MUST overtake whatever routing you now have, but don't worry, as it supports your home assistant setup and forwards any traffic it expects to it (if you finish the pre-configured `home-assistant` project in `db.yml`)
+- A wildcard dns domain like `*.itsup.example.com` that points to your home ip. This allows to choose whatever subdomain for your services. You may of course choose and manage any domain in a similar fashion for a public service, but I suggest not going through such trouble for anything private.
 
-1. Copy `db.yml.sample` to `db.yml` and edit your project and their services (see explanations below).
-2. Copy `.env.sample` to `.env` and set the correct info.
-3. [OPTIONAL] In case you want to run the api create an `API_KEY` (`openssl rand -hex 16`) and put in `.env`.
+## Howto
 
 ### Install & run
 
-Install everything and start the proxy and api so that we can receive incoming challenge webhooks.
+These are the scripts to install everything and start the proxy and api so that we can receive incoming challenge webhooks:
 
-1. `bin/install.sh`: installs all project deps.
-2. `bin/start-all.sh`: starts the proxy and the api server.
+1. `bin/install.sh`: creates a local `.venv` and installs all python project deps.
+2. `bin/start-all.sh`: starts the proxy (docker compose) and the api server (uvicorn).
 3. `bin/apply.py`: applies all of `db.yml`.
-4. `bin/api-logs.sh`: tail the output of the api server.
+4. `bin/api-logs.sh`: tails the output of the api server. (The
+
+But before doing so please configure your stuff:
+
+### Configure services
+
+1. Copy `.env.sample` to `.env` and set the correct info (comments should be self explanatory).
+2. Copy `db.yml.sample` to `db.yml` and edit your project and their services (see explanations below).
 
-### Adding an upstream service
+Project and service configuration is explained below with the following scenarios:
 
-1. Edit `db.yml` and add your projects with their service(s), and make sure the project has `entrypoint: {your_entrypoint_svc}`.
-2. Run `bin/apply.py` to get certs, write needed artifacts, update relevant docker stacks and reload nginx.
+**Adding an upstream service that will be deployed and managed:**
 
-### Adding a passthrough endpoint
+1. Edit `db.yml` and add your projects with their service(s), and make sure the project has `entrypoint: {the name of your entrypoint svc}`.
+2. Run `bin/apply.py` to write all artifacts and deploy/update relevant docker stacks.
+
+**Adding a passthrough endpoint:**
 
 1. Add a project without `entrypoint:` and one service, which now need `name`, `domain` and `passthrough: true`.
 2. Run `bin/apply.py` to roll out the changes.
 
-### Adding a local (host) endpoint
+**Adding a local (host) endpoint:**
 
 1. Add a project without `entrypoint:` and one service, which only need `name` and `domain`.
 2. Run `bin/apply.py` to roll out the changes.
 
-### Plugins
+### Configure plugins
 
 You can enable and configure plugins in `db.yml`. Right now we support the following:
 
 #### CrowdSec
 
-[CrowdSec](https://www.crowdsec.net) can run as a container via plugin [crowdsec-bouncer-traefik-plugin](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin). First set `enable: true`, run `bin/write-artifacts.py`, and bring up the stack (or just the `crowdsec` container:
+[CrowdSec](https://www.crowdsec.net) can run as a container via plugin [crowdsec-bouncer-traefik-plugin](https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin).
+
+**Step 1: generate api key**
+
+First set `enable: true`, run `bin/write-artifacts.py`, and bring up the `crowdsec` container:
+
+```
+docker compose up -d crowdsec
+```
+
+Now we can execute the command to get the key:
 
 ```
 docker compose exec crowdsec cscli bouncers add crowdsecBouncer
 ```
 
 Put the resulting api key in the plugin configuration in `db.yml` and apply with `bin/apply.py`.
+Crowdsec is now running and wired up, but does not use any blocklists yet. Those can be managed manually, but preferable is to become part of the community by creating an account with CrowdSec to get access and contribute to the community blocklists, as well as view results in your account's dashboards.
+
+**Step 2: connect your instance with the CrowdSec console**
+
+After creating an account create a machine instance in the console, and register the enrollment key in your stack:
+
+```
+docker compose exec crowdsec cscli console enroll ${enrollment key}
+```
+
+**Step 3: subscribe to 3rd party blocklists**
 
-### Api & OpenApi spec
+### Using the Api & OpenApi spec
 
 The API allows openapi compatible clients to do management on this stack (ChatGPT works wonders).
 
@@ -110,13 +182,31 @@ Source `lib/functions.sh` to get:
 - `dcp`: run a `docker compose` command targeting the proxy stack (`proxy` + `terminate` services): `dcp logs -f`
 - `dcu`: run a `docker compose` command targeting a specific upstream: `dcu test up`
 - `dca`: run a `docker compose` command targeting all upstreams: `dca ps`
+- `dcpx`: execute a command in one of the proxy containers: `dcpx traefik-web 'rm -rf /etc/acme/acme.json && shutdown' && dcp up`
+- `dcux`: execute a command in one of the upstream containers: `dcux test test-informant env`
 
-In effect these wrapper commands achieve the same as when going into an `upstream/*` folder and running `docker compose` there.
-I don't want to switch folders/terminals all the time and want to keep history of my commands so I choose this approach.
+In effect these wrapper commands achieve the same as when going into an `upstream/\*`folder and running`docker compose` there.
+I don't want to switch folders/terminals all the time and want to keep a "project root" history of my commands so I choose this approach.
 
-### Scripts
+### Utility scripts
 
 - `bin/update-certs.py`: pull certs and reload the proxy if any certs were created or updated. You could run this in a crontab every week if you want to stay up to date.
 - `bin/write-artifacts.py`: after updating `db.yml` you can run this script to generate new artifacts.
-- `bin/validate-db.py`: after manually editing `db.yml` please run this (also ran from `bin/write-artifacts.py`)
+- `bin/validate-db.py`: also ran from `bin/write-artifacts.py`
 - `bin/requirements-update.sh`: You may want to update requirements once in a while ;)
+
+## Questions one might have
+
+### What about Nginx?
+
+As you may have noted there is a lot of functionality based on Nginx in this repo. I started out using their proxy, but later on ran into the problem of their engine not picking up upstream changes, learning that only the paid Nginx+ does that. I heavily relied on kubernetes in the past years and such was not an issue in their `ingress-NGINX` controller. When I found that Traefik does not suffer this, AND manages letsencrypt certs gracefully, AND gives us label based L7 functionality (like in Kubernetes), I decided to integrate that instead. Weary about its performance though, I intended to keep both approaches side by side. The Nginx part is not working anymore, but I left the code for others to see how one can overcome certain problems in that ecosystem. If one would like to use Nginx for some reason (it is about 40% faster), it is very easy to switch back. But be aware it implies hooking up the hacky `bin/update-certs.py` script to a cron tab for automatic cert rotation.
+
+### Does this scale to more machines?
+
+In the future we might consider expanding this setup to use docker swarm, as it should be easy to do. For now we like to keep it simple.
+
+## Disclaimer
+
+**Don't blame this infra automation tooling for anything going wrong inside your containers!**
+
+I suggest you repeat that mantra now and then and question yourself when things go wrong: where lies the problem?

api/main.py

@@ -1,5 +1,6 @@
 #!.venv/bin/python
 import os
+from functools import cache
 from logging import info
 from typing import Any, Dict, List
 
@@ -94,6 +95,7 @@ async def github_workflow_job_handler(
 
 @app.get("/projects", response_model=List[Project])
 @app.get("/projects/{project}", response_model=Project)
+@cache
 def get_projects_handler(project: str = None, _: None = Depends(verify_apikey)) -> List[Project] | Project:
     """Get the list of all or one project"""
     if project:

assets/freight-conductor.png

@@ -10,7 +10,7 @@ drun() {
     $image sh -c "'$@'"
 }
 
-dcp() {
+dc_() {
   dir=$1
   project=$2
   part=$3
@@ -31,44 +31,43 @@ dcp() {
   eval "export HOST_GID=$(id -g daemon) && docker compose --project-directory $dir -p $project -f $dir/docker-compose.yml $cmd $@"
 }
 
-dcpx() {
+dcx_() {
   dir=$1
   project=$2
   svc=$3
   shift
   shift
   shift
-  dcp $dir $project exec $svc sh -c "'$@'"
+  dc_ $dir $project exec $svc sh -c "'$@'"
 }
 
 # Run docker compose in the proxy
-dcpp() {
-  dcp proxy proxy $@
+dcp() {
+  dc_ proxy proxy $@
 }
-dcppx() {
-  dcpx proxy proxy $@
+dcpx() {
+  dcx_ proxy proxy $@
 }
 
 # Run docker compose in an upstream
-dcpu() {
-  upstream=$1
-  project=$(basename $upstream)
+dcu() {
+  project=$1
   shift
-  [ -z "$upstream" ] && echo "No upstream project given!" && return 1
-  dcp $upstream $project $@
+  [ -z "$project" ] && echo "No upstream project given!" && return 1
+  dc_ upstream/$project $project $@
 }
-dcpux() {
-  upstream=$1
-  project=$(basename $upstream)
+dcux() {
+  project=$1
   shift
-  dcpx $upstream $project $@
+  dcx_ upstream/$project $project $@
 }
 
 # Run docker compose command in all upstreams
-dcpa() {
+dca() {
   [ -z "$@" ] && echo "No arguments given!" && return 1
   for upstream in $(ls -d upstream/*); do
-    dcpu $upstream $@
+    project=$(basename $upstream)
+    dcu $project $@
   done
 }
 

lib/functions.sh

@@ -11,6 +11,7 @@ class Env(BaseModel):
 class Plugin(BaseModel):
     """Plugin model"""
 
+    version: str
     enabled: bool = False
     """Wether or not the plugin is enabled"""
     apikey: str = None