RFC: Pulp Content Management for GloriousFlywheel CI

[Jesssullivan]

RFC 0001: Pulp Content Management

RFC Document: docs/rfcs/0001-pulp-content-management.md (in tinyland.dev repo)
Status: Draft
Author: xoxd

Summary

Deploy a Pulp instance as a unified content management and caching proxy for the GloriousFlywheel CI runner fleet. Pulp fronts npm, PyPI, container registries, and Nix binary caches behind a single self-hosted service.

Motivation

• Docker Hub rate limits (429) hit CI runners and K8s pod pulls
• npm publishing rate limits block bulk publishing runs
• Nix binary cache (Attic) is single-purpose — no shared infra
• GitLab Dependency Proxy is container-only with limited visibility
• Build reproducibility depends on external registries being available

Content Types

Type	Plugin	Replaces
Container images	pulp_container	GitLab Dependency Proxy, manual skopeo mirrors
npm packages	pulp_npm	Direct npmjs.com access
Python packages	pulp_python	Direct PyPI access
RPM/DNF	pulp_rpm	Direct Rocky Linux repos
Nix cache	pulp_file	Attic

Implementation Phases

1. Phase 1 (Week 1): Container cache — Docker Hub pull-through, replace manual mirrors
2. Phase 2 (Week 2): npm cache — npmjs.com pull-through, absorb 429 rate limits
3. Phase 3 (Week 3): Python + RPM caches
4. Phase 4 (Week 4): Nix cache evaluation (Pulp vs Attic)

Resource Estimate

~650m CPU, 1.7Gi RAM, 12Gi storage on existing Civo K8s cluster.

Open Questions

1. Should Pulp replace Attic for Nix binary caching, or coexist?
2. What retention policy for pull-through caches?
3. Should developer workstations also use Pulp, or only CI?

See the full RFC in docs/rfcs/0001-pulp-content-management.md.

[Jesssullivan]

Bzlmod Overlay Deployment Option

Rather than embedding Pulp config in the monorepo CI, Pulp could be structured as a bzlmod overlay module peering with the existing GloriousFlywheel org overlay. This keeps infra versioned and composable.

Registry Structure

graph TD
    subgraph "Org Bzlmod Registry (overlay)"
        A[bazel_registry.json]
        B[modules/gloriousflywheel/] 
        C[modules/tinyland_pulp/]
        D[modules/tinyland_cache/]
    end

    subgraph "tinyland_pulp module"
        C --> E[MODULE.bazel<br/>deps: rules_pkg, rules_oci]
        C --> F[BUILD.bazel<br/>pkg_tar, kustomize targets]
        C --> G[K8s manifests<br/>pulp-api, content, worker,<br/>redis, postgresql]
    end

    subgraph "Consumers"
        H[GloriousFlywheel runners]
        I[Civo K8s cluster]
        J[Developer workstations]
    end

    B -->|existing| H
    C -->|bazel_dep| H
    F -->|bazel build<br/>@tinyland_pulp//:deploy_bundle| I
    D -->|shared cache config| J

Loading

Module Dependency Graph

graph LR
    subgraph "BCR (public)"
        RP[rules_pkg 1.1.0]
        RO[rules_oci]
        RK[rules_kustomize]
    end

    subgraph "Org Overlay (private)"
        GF[gloriousflywheel]
        TP[tinyland_pulp]
    end

    subgraph "Pulp Targets"
        T1["//pulp:manifests<br/>(filegroup)"]
        T2["//pulp:deploy_bundle<br/>(pkg_tar)"]
        T3["//pulp:kustomize_overlay<br/>(kustomization)"]
        T4["//pulp:pulp_images<br/>(oci_pull pinned digests)"]
    end

    TP --> RP
    TP --> RO
    TP --> RK
    GF --> TP
    TP --> T1 & T2 & T3 & T4

Loading

Content Type Plugin Mapping

flowchart TB
    subgraph "Pulp Plugins (deployed via overlay)"
        PC[pulp_container<br/>Docker Hub, GHCR, GitLab]
        PN[pulp_npm<br/>npmjs.com cache]
        PP[pulp_python<br/>PyPI cache]
        PR[pulp_rpm<br/>Rocky Linux repos]
        PF[pulp_file<br/>Nix binary cache]
        PA[pulp_ansible<br/>Galaxy collections]
    end

    subgraph "CI Consumers"
        C1[container-build jobs]
        C2[pnpm install]
        C3[uv / pip install]
        C4[dnf install]
        C5[nix build]
        C6[ansible-galaxy install]
    end

    PC --> C1
    PN --> C2
    PP --> C3
    PR --> C4
    PF --> C5
    PA --> C6

Loading

Advantages over monorepo-embedded approach

• Versioned: Pulp config changes tracked as registry module versions
• Composable: bazel_dep(name = "tinyland_pulp") from any repo
• Reproducible: bazel build @tinyland_pulp//:deploy_bundle produces identical output
• Peered: Lives alongside GloriousFlywheel runner config in the same org overlay
• OCI-pinned: Container image digests locked in BUILD.bazel, not floating tags

Note: Also added pulp_ansible to the plugin list — supports Ansible collection/role distribution (relevant for crush-dots Galaxy caching).