From 1cd75832e02f62dd3cf65cb7facb5d2d4302fd1d Mon Sep 17 00:00:00 2001 From: Tyler Witlin Date: Sun, 31 Dec 2023 17:52:29 -0500 Subject: [PATCH] fix: update readme Signed-off-by: Tyler Witlin --- README.md | 142 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 83 insertions(+), 59 deletions(-) diff --git a/README.md b/README.md index b2bda45fda..501fc215f2 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@
- + -### My home operations repository :octocat: +### My Home Operations Repository :octocat: -_... managed with Flux, Renovate and GitHub Actions_ 🤖 +_... managed with Flux, Renovate, and GitHub Actions_ 🤖
+
[![Discord](https://img.shields.io/discord/673534664354430999?style=for-the-badge&label&logo=discord&logoColor=white&color=blue)](https://discord.gg/k8s-at-home) @@ -16,11 +17,22 @@ _... managed with Flux, Renovate and GitHub Actions_ 🤖
+
+ +[![Age-Days](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_age_days&style=flat-square&label=Age)](https://github.com/kashalls/kromgo/)   +[![Uptime-Days](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_uptime_days&style=flat-square&label=Uptime)](https://github.com/kashalls/kromgo/)   +[![Node-Count](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_node_count&style=flat-square&label=Nodes)](https://github.com/kashalls/kromgo/)   +[![Pod-Count](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_pod_count&style=flat-square&label=Pods)](https://github.com/kashalls/kromgo/)   +[![CPU-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_cpu_usage&style=flat-square&label=CPU)](https://github.com/kashalls/kromgo/)   +[![Memory-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_memory_usage&style=flat-square&label=Memory)](https://github.com/kashalls/kromgo/)   +[![Power-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.witl.xyz%2Fquery%3Fformat%3Dendpoint%26metric%3Dcluster_power_usage&style=flat-square&label=Power)](https://github.com/kashalls/kromgo/) + +
--- ## 📖 Overview -This is a mono repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using the tools like [Ansible](https://www.ansible.com/), [Terraform](https://www.terraform.io/), [Kubernetes](https://kubernetes.io/), [Flux](https://github.com/fluxcd/flux2), [Renovate](https://github.com/renovatebot/renovate) and [GitHub Actions](https://github.com/features/actions). +This is a mono repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using the tools like [Ansible](https://www.ansible.com/), [Pulumi](https://pulumi.com), [Kubernetes](https://kubernetes.io/), [Flux](https://github.com/fluxcd/flux2), [Renovate](https://github.com/renovatebot/renovate) and [GitHub Actions](https://github.com/features/actions). --- @@ -35,58 +47,62 @@ My cluster is [talos](https://talos.dev) provisioned overtop bare-metal. This is ### Core Components -- [actions-runner-controller](https://github.com/actions/actions-runner-controller): Self-hosted Github runners. -- [cilium/cilium](https://github.com/cilium/cilium): Internal Kubernetes networking plugin. -- [cert-manager](https://cert-manager.io/docs/): Creates SSL certificates for services in my Kubernetes cluster. -- [external-dns](https://github.com/kubernetes-sigs/external-dns): Automatically manages DNS records from my cluster in a cloud DNS provider. -- [external-secrets](https://github.com/external-secrets/external-secrets/): Managed Kubernetes secrets using [1Password Connect](https://github.com/1Password/connect). -- [ingress-nginx](https://github.com/kubernetes/ingress-nginx/): Ingress controller to expose HTTP traffic to pods over DNS. -- [rook](https://github.com/rook/rook): Distributed block storage for peristent storage. -- [sops](https://toolkit.fluxcd.io/guides/mozilla-sops/): Managed secrets for Kubernetes, Ansible and Terraform which are commited to Git. -- [tf-controller](https://github.com/weaveworks/tf-controller): Additional Flux component used to run Terraform from within a Kubernetes cluster. -- [volsync](https://github.com/backube/volsync) and [snapscheduler](https://github.com/backube/snapscheduler): Backup and recovery of persistent volume claims. +- [actions-runner-controller](https://github.com/actions/actions-runner-controller): self-hosted Github runners +- [cilium](https://github.com/cilium/cilium): internal Kubernetes networking plugin +- [cert-manager](https://cert-manager.io/docs/): creates SSL certificates for services in my cluster +- [external-dns](https://github.com/kubernetes-sigs/external-dns): automatically syncs DNS records from my cluster ingresses to a DNS provider +- [external-secrets](https://github.com/external-secrets/external-secrets/): managed Kubernetes secrets using [1Password Connect](https://github.com/1Password/connect). +- [ingress-nginx](https://github.com/kubernetes/ingress-nginx/): ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer +- [rook](https://github.com/rook/rook): distributed block storage for persistent storage +- [sops](https://toolkit.fluxcd.io/guides/mozilla-sops/): managed secrets for Kubernetes, Ansible, and Terraform which are committed to Git +- [spegel](https://github.com/XenitAB/spegel): stateless cluster local OCI registry mirror +- [tf-controller](https://github.com/weaveworks/tf-controller): additional Flux component used to run Terraform from within a Kubernetes cluster. +- [volsync](https://github.com/backube/volsync): backup and recovery of persistent volume claims ### GitOps -[Flux](https://github.com/fluxcd/flux2) watches my [kubernetes](./kubernetes/) folder (see Directories below) and makes the changes to my cluster based on the YAML manifests. +[Flux](https://github.com/fluxcd/flux2) watches the clusters in my [kubernetes](./kubernetes/) folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository. -The way Flux works for me here is it will recursively search the [kubernetes/apps](./kubernetes/apps) folder until it finds the most top level `kustomization.yaml` per directory and then apply all the resources listed in it. That aforementioned `kustomization.yaml` will generally only have a namespace resource and one or many Flux kustomizations. Those Flux kustomizations will generally have a `HelmRelease` or other resources related to the application underneath it which will be applied. +The way Flux works for me here is it will recursively search the `kubernetes/${cluster}/apps` folder until it finds the most top level `kustomization.yaml` per directory and then apply all the resources listed in it. That aforementioned `kustomization.yaml` will generally only have a namespace resource and one or many Flux kustomizations (`ks.yaml`). Under the control of those Flux kustomizations there will be a `HelmRelease` or other resources related to the application which will be applied. -[Renovate](https://github.com/renovatebot/renovate) watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged [Flux](https://github.com/fluxcd/flux2) applies the changes to my cluster. +[Renovate](https://github.com/renovatebot/renovate) watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster. ### Directories -This Git repository contains the following directories under [kubernetes](./kubernetes/). +This Git repository contains the following directories under [Kubernetes](./kubernetes/). ```sh -📁 kubernetes # Kubernetes cluster defined as code -├─📁 bootstrap # Flux installation -├─📁 flux # Main Flux configuration of repository -└─📁 apps # Apps deployed into my cluster grouped by namespace (see below) +📁 kubernetes +├── 📁 kyak # kyak cluster +│ ├── 📁 apps # applications +│ ├── 📁 bootstrap # bootstrap procedures +│ ├── 📁 flux # core flux configuration +│ └── 📁 templates # re-useable components +└── 📁 sol # sol cluster + ├── 📁 apps # applications + ├── 📁 bootstrap # bootstrap procedures + └── 📁 flux # core flux configuration ``` -### Cluster layout - -Below is a a high level look at the layout of how my directory structure with Flux works. In this brief example you are able to see that `authelia` will not be able to run until `glauth` and `cloudnative-pg` are running. It also shows that the `Cluster` custom resource depends on the `cloudnative-pg` Helm chart. This is needed because `cloudnative-pg` installs the `Cluster` custom resource definition in the Helm chart. - -```python -# Key: :: -GitRepository :: home-kubernetes - Kustomization :: cluster - Kustomization :: cluster-apps - Kustomization :: cluster-apps-authelia - DependsOn: - Kustomization :: cluster-apps-glauth - Kustomization :: cluster-apps-cloudnative-pg-cluster - HelmRelease :: authelia - Kustomization :: cluster-apps-glauth - HelmRelease :: glauth - Kustomization :: cluster-apps-cloudnative-pg - HelmRelease :: cloudnative-pg - Kustomization :: cluster-apps-cloudnative-pg-cluster - DependsOn: - Kustomization :: cluster-apps-cloudnative-pg - Cluster :: postgres +### Flux Workflow + +This is a high-level look how Flux deploys my applications with dependencies. Below there are 3 apps `postgres`, `lldap` and `authelia`. `postgres` is the first app that needs to be running and healthy before `lldap` and `authelia`. Once `postgres` is healthy `lldap` will be deployed and after that is healthy `authelia` will be deployed. + +```mermaid +graph TD; + id1>Kustomization: cluster] -->|Creates| id2>Kustomization: cluster-apps]; + id2>Kustomization: cluster-apps] -->|Creates| id3>Kustomization: postgres]; + id2>Kustomization: cluster-apps] -->|Creates| id6>Kustomization: lldap] + id2>Kustomization: cluster-apps] -->|Creates| id8>Kustomization: authelia] + id2>Kustomization: cluster-apps] -->|Creates| id5>Kustomization: postgres-cluster] + id3>Kustomization: postgres] -->|Creates| id4[HelmRelease: postgres]; + id5>Kustomization: postgres-cluster] -->|Depends on| id3>Kustomization: postgres]; + id5>Kustomization: postgres-cluster] -->|Creates| id10[Postgres Cluster]; + id6>Kustomization: lldap] -->|Creates| id7(HelmRelease: lldap); + id6>Kustomization: lldap] -->|Depends on| id5>Kustomization: postgres-cluster]; + id8>Kustomization: authelia] -->|Creates| id9(HelmRelease: authelia); + id8>Kustomization: authelia] -->|Depends on| id5>Kustomization: postgres-cluster]; + id9(HelmRelease: authelia) -->|Depends on| id7(HelmRelease: lldap); ``` ### Networking @@ -116,11 +132,11 @@ The alternative solution to these two problems would be to host a Kubernetes clu | [GitHub](https://github.com/) | Hosting this repository and continuous integration/deployments | Free | | [Cloudflare](https://www.cloudflare.com/) | Domain, DNS and proxy management | ~$30/yr | | [1Password](https://1password.com/) | Secrets with [External Secrets](https://external-secrets.io/) | ~$65/yr | -| [Terraform Cloud](https://www.terraform.io/) | Storing Terraform state | Free | | [B2 Storage](https://www.backblaze.com/b2) | Offsite application backups | ~$5/mo | | [Pushover](https://pushover.net/) | Kubernetes Alerts and application notifications | Free | | [NextDNS](https://nextdns.io/) | My routers DNS server which includes AdBlocking | ~20/yr | -| | | Total: ~$18/mo | +| [Frugal](https://frugalusenet.com/) | Usenet access | ~$35/yr | +| | | Total: ~$20/mo | --- @@ -130,25 +146,33 @@ The alternative solution to these two problems would be to host a Kubernetes clu On my Vyos router I have [Bind9](https://github.com/isc-projects/bind9) and [dnsdist](https://dnsdist.org/) deployed as containers. In my cluster `external-dns` is deployed with the `RFC2136` provider which syncs DNS records to `bind9`. -Downstream DNS servers configured in `dnsdist` such as `bind9` (above) and [NextDNS](https://nextdns.io/). All my clients use `dnsdist` as the upstream DNS server, this allows for more granularity with configuring DNS across my networks. These could be things like giving each of my VLANs a specific `nextdns` profile, or having all requests for my domain forward to `bind9` on certain networks, or only using `1.1.1.1` instead of `nextdns` on certain networks where adblocking isn't needed. +Downstream DNS servers configured in `dnsdist` such as `bind9` (above) and [NextDNS](https://nextdns.io/). All my clients use `dnsdist` as the upstream DNS server, this allows for more granularity with configuring DNS across my networks. These could be things like giving each of my VLANs a specific `nextdns` profile, or having all requests for my domain forward to `bind9` on certain networks, or only using `1.1.1.1` instead of `nextdns` on certain networks where adblocking isn't required. ### Public DNS -Outside the `external-dns` instance mentioned above another instance is deployed in my cluster and configure to sync DNS records to [Cloudflare](https://www.cloudflare.com/). The only ingresses this `external-dns` instance looks at to gather DNS records to put in `Cloudflare` are ones that have an ingress class name of `external` and an ingress annotation of `external-dns.alpha.kubernetes.io/target`. +Outside the `external-dns` instance mentioned above another instance is deployed in my cluster and configured to sync DNS records to [Cloudflare](https://www.cloudflare.com/). The only ingress this `external-dns` instance looks at to gather DNS records to put in `Cloudflare` are ones that have an ingress class name of `external` and contain an ingress annotation `external-dns.alpha.kubernetes.io/target`. +--- ## 🔧 Hardware -| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose | -| ------------------------ | ----- | ------------ | ----------------------- | ---- | ---------------- | ------------------------ | -| Supermicro SYS-510T-ML | 1 | 256GB NVMe | N/A | 16GB | Vyos | Router | -| Dell Optiplex 3060 Micro | 1 | 240GB SSD | N/A | 32GB | Talos | Kubernetes Master | -| Dell Optiplex 3080 Micro | 2 | 256GB SSD | N/A | 16GB | Talos | Kubernetes Master | -| Lenovo M910q Tiny | 2 | 512GB NVMe | 500GB SSD (rook-ceph) | 16GB | Talos | Kubernetes Worker | -| Lenovo M720q Tiny | 2 | 480GB NVMe | N/A | 16GB | Talos | Kubernetes Worker | -| HP EliteDesk 800 G4 SFF | 2 | 240GB NVMe | 500GB SSD (rook-ceph) | 16GB | Talos | Kubernetes Worker | -| HPE DL160 G10 | 1 | 512GB SSD | 2x6TB HDD (rook-ceph) | 32GB | Talos | Kubernetes Worker | -| HPE DL160 G10 | 1 | 500GB SSD | 16TB zfs mirror | 64GB | Ubuntu 22.04 | Shared file storage | +| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose | +|----------------------------|-------|--------------|-------------------------|-------|------------------|------------------------| +| Supermicro SYS-510T-ML | 1 | 256GB NVMe | N/A | 16GB | Vyos | Router | +| Dell Optiplex 3060 Micro | 1 | 240GB SSD | N/A | 32GB | Talos | Kubernetes master | +| Dell Optiplex 3080 Micro | 2 | 256GB SSD | N/A | 16GB | Talos | Kubernetes master | +| Lenovo M910q Tiny | 2 | 512GB NVMe | 500GB SSD (rook-ceph) | 16GB | Talos | Kubernetes worker | +| Lenovo M720q Tiny | 2 | 480GB NVMe | N/A | 16GB | Talos | Kubernetes worker | +| HP EliteDesk 800 G4 SFF | 2 | 240GB NVMe | 500GB SSD (rook-ceph) | 16GB | Talos | Kubernetes worker | +| HPE DL160 G10 | 1 | 512GB SSD | 2x6TB HDD (rook-ceph) | 32GB | Talos | Kubernetes worker | +| HPE DL160 G10 | 1 | 500GB SSD | 16TB zfs mirror | 128GB | Ubuntu 23.10 | Shared file storage | +| Dell R630 | 1 | SSD | 3x1.9TB HDD (rook-ceph) | 192GB | Fedora 39 | Single node k3s cluter | +| TESmart 8 Port KVM Switch | 1 | - | - | - | - | Network KVM (PiKVM) | +| PiKVM v4 plus | 1 | - | - | - | PiKVM (Arch) | Network KVM | +| Tripplite SMART3000RMXLN | 1 | - | - | - | - | UPS | +| Aruba Instant on 1930 24G | 1 | - | - | - | - | Switch | +| Cisco Nexus 9372PX | 1 | - | - | - | - | Switch | +| DELL EMC PowerSwitch N2048 | 1 | - | - | - | - | Switch | --- @@ -164,7 +188,7 @@ Outside the `external-dns` instance mentioned above another instance is deployed ## 🤝 Gratitude and Thanks -Thanks to all the people who donate their time to the [Kubernetes @Home](https://discord.gg/k8s-at-home) Discord community. A lot of inspiration for my cluster comes from the people that have shared their clusters using the [k8s-at-home](https://github.com/topics/k8s-at-home) GitHub topic. Be sure to check out the [Kubernetes @Home search](https://nanne.dev/k8s-at-home-search/) for ideas on how to deploy applications or get ideas on what you can deploy. +Thanks to all the people who donate their time to the [Home Operations](https://discord.gg/home-operations) Discord community. Be sure to check out [kubesearch.dev](https://kubesearch.dev/) for ideas on how to deploy applications or get ideas on what you may deploy. ---