From 96e738c3c2584a0908601ab6e3f2a9c8d3ca3587 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Thu, 8 Jan 2026 14:43:36 +0100 Subject: [PATCH 01/17] Added initial version of DR-005-infra.md --- docs/design_decisions/DR-005-infra.md | 292 ++++++++++++++++++++++++++ 1 file changed, 292 insertions(+) create mode 100644 docs/design_decisions/DR-005-infra.md diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md new file mode 100644 index 00000000000..22c457c3c34 --- /dev/null +++ b/docs/design_decisions/DR-005-infra.md @@ -0,0 +1,292 @@ + + +# DR-005-Infra: Polyrepo Release Process with SemVer, Manifest Repository, and Continuous Integration Tracking + +- **Status:** Proposed +- **Owner:** Infrastructure Community +- **Date:** 2026-01-08 + +--- + +## 1. Context + +This project consists of multiple independently developed modules stored in separate repositories (polyrepo setup). +Each module evolves independently and is versioned using Semantic Versioning (SemVer). + +This ADR **builds on DR-002 (Infrastructure Architecture)**, which establishes: +- the polyrepo structure, +- centralized responsibility for cross-repository integration, +- and infrastructure-owned integration tooling and processes. + +Within that context, this ADR defines **how coordinated product releases are produced** from independently developed and versioned modules, while allowing continuous integration to reduce the gap between development and release. + +The project delivers a **coordinated product release** that integrates a specific, tested combination of module states. This requires: + +- reproducible and auditable release snapshots, +- explicit stabilization phases, +- continuous integration before formal releases, +- independent module lifecycles without blocking development, +- and a process that scales across many repositories and teams. + +Commonly referenced workflows do not fully address this problem space: +- **Trunk-Based Development** assumes continuous deployment and does not define coordinated product releases. +- **Gitflow** introduces coordinated release branching across repositories and lacks a central integration manifest. + +This ADR defines a release process explicitly designed for **polyrepo systems with coordinated integration releases and continuous verification**. + +--- + +## 2. Goals and Requirements + +- Reproducible and auditable release snapshots +- Explicit stabilization phases +- Continuous integration before formal releases +- Independent module lifecycles without blocking development +- A process that scales across many repositories and teams + +--- + +## 3. Options Considered + +### 3.1 Trunk-Based Development Only + +Assumes continuous deployment and does not define coordinated product releases. + +**Pros**: +- Simplifies development workflow. +- Encourages frequent integration. + +**Cons**: +- Does not address the need for coordinated product releases. +- Lacks explicit stabilization phases. +- Not suitable for polyrepo systems requiring integration snapshots. + +### 3.2 Gitflow Across Repositories + +Introduces coordinated release branching across repositories but lacks a central integration manifest. + +**Pros**: +- Well-known branching model. +- Provides release branches for stabilization. + +**Cons**: +- Requires coordination of release branches across all repositories. +- Lacks a single source of truth for integration state. +- Does not scale well with increasing module count. + +### 3.3 SemVer-Based Polyrepo Release Process with Manifest Repository + +A dedicated manifest repository defines integration state, with independent module versioning and two integration modes (tracking and pinned). + +**Pros**: +- Single source of truth for product integration. +- Supports continuous verification via tracking mode. +- Provides reproducible releases via pinned mode. +- Scales with module count and team autonomy. +- Clear separation between development, integration, and stabilization. + +**Cons**: +- Requires explicit integration governance. +- Introduces additional coordination effort compared to single-repo workflows. +- Relies on strict SemVer discipline in modules. + +--- + +## 4. Decision + +We adopt a **SemVer-based polyrepo release process with a dedicated manifest repository, continuous integration tracking, and release trains identified by tags**. + +The solution is based on four core principles: + +1. **Independent module versioning with strict SemVer guarantees** +2. **A manifest repository as the single source of truth for integration** +3. **Two integration modes: tracking (continuous verification) and pinned (release/stabilization)** +4. **Release stabilization via immutable tags (and an optional manifest release branch)** + +### 1. Module Versioning (SemVer) + +- Each module repository publishes releases following Semantic Versioning: + - **PATCH**: backward-compatible bug fixes + - **MINOR**: backward-compatible feature additions + - **MAJOR**: backward-incompatible changes +- Version numbers are **global, linear, and unique per module**. +- Once a version is released, it is immutable and must never be reused. +- There is no automatic or periodic major version bump tied to product releases. + +**Compatibility rule:** +- A PATCH release must be produced from a code line that is compatible with the previous release. +- Modules must not include incompatible changes in PATCH releases. + +> **Explicitly out of scope:** +> Module-internal development workflows (e.g. trunk-based development, feature branches, rebasing strategies) are intentionally not prescribed by this ADR. + +### 2. Manifest Repository + +A dedicated **manifest (integration) repository** defines which exact module states form a product snapshot. + +Responsibilities of the manifest repository: +- Define the complete product composition. +- Pin module states either as: + - released module versions, or + - explicit commit hashes (for tracking mode). +- Contain integration-specific artifacts such as: + - end-to-end tests, + - integration configuration, + - packaging or distribution logic. +- Act as the **single source of truth** for product integration. + +Any change to module references in the manifest repository is made via a pull request and reviewed as an integration change. + +### 3. Integration Modes + +The manifest repository operates in two distinct modes. + +#### 3.1 Tracking Mode (Continuous Verification) + +- Used on manifest `main`. +- The manifest may track module branches or “next release” branches by automatically updating pinned commit hashes. +- Goal: + - continuously verify that the *current development state* of modules can be integrated, + - reduce the gap between development and release, + - detect integration issues early. +- Tracking references are **not considered release-ready** and are not used for formal release stages. + +#### 3.2 Pinned Mode (Release and Stabilization) + +- Used for release preparation and formal stages. +- The manifest pins **immutable identifiers only**: + - released module versions, or + - fixed commit hashes. +- Pinned mode guarantees full reproducibility and auditability. +- All release stages and final releases must use pinned mode. + +### 4. Product Release Identification and Structure + +Product releases are identified using **train-style tags**: + +- `vX.Y.0-alpha.N` +- `vX.Y.0-beta.N` +- `vX.Y.0-rc.N` +- `vX.Y.0` (final) + +Product versions reuse SemVer-compatible notation for clarity and tooling compatibility, but they represent **integration snapshots**, not API stability guarantees for individual modules. + +### 5. Release Stabilization Line + +To allow iteration without disturbing ongoing integration: + +- A stabilization line may be introduced **in the manifest repository only**, e.g.: + - an optional branch `release/vX.Y`, or + - an equivalent agreed stabilization mechanism. +- Stabilization happens **only** in the manifest repository. +- Module repositories are not required to coordinate release branches. + +This allows: +- continuous tracking on manifest `main`, +- controlled stabilization for a given product release. + +### 6. Release Stages + +#### Alpha +- Goal: early system-level integration. +- Allowed changes: + - compatible bug fixes (PATCH), + - compatible features (MINOR) if strictly required. + +#### Beta +- Goal: feature freeze and stabilization. +- Allowed changes: + - bug fixes only (PATCH). + +#### Release Candidate (RC) +- Goal: final validation. +- Allowed changes: + - critical bug or security fixes only (PATCH). + +#### Final +- Tag `vX.Y.0` marks the final release. +- Optional product patch releases (`vX.Y.1`, `vX.Y.2`, …) follow the same rules. + +### 7. Backports + +- Fixes are applied first on the appropriate module code line. +- Compatible PATCH releases are created if required. +- The manifest is updated to reference the new immutable version or commit. +- A new stage or patch tag is created. + +### 8. Breaking Changes During Stabilization + +- Breaking changes during stabilization are strongly discouraged. +- If required (e.g. priority or organizational decisions), they must: + - be explicitly declared, + - include a documented mitigation strategy (compatibility layer, coordinated upgrades, scope reduction, or deferral), + - and be recorded in the manifest repository. + +### 9. Module Release-Line Strategies (Non-Normative Examples) + +Modules may choose any internal strategy that preserves SemVer guarantees. Common compliant patterns include: + +1. **Disciplined mainline releases** + PATCH releases are cut before incompatible changes are merged. +2. **Temporary release branches** + Short-lived branches cut from the last release tag to produce compatible PATCH releases. +3. **Persistent release branches** + Long-lived branches for maintained version lines or LTS support. + +The choice is module-local and does not affect the integration model. + +### 10. Ownership and Governance + +- The manifest repository is owned by the infrastructure/integration team, as defined in DR-002. +- As release stages progress (beta, rc), changes to the manifest require increasing scrutiny and justification. +- Governance applies only to the manifest repository and does not constrain module-internal workflows. + +--- + +## 5. Rationale + +This approach reflects established industry practice for large-scale polyrepo systems using +manifest-based integration and release trains (e.g. Android/AOSP, Chromium-style roll-ups), +while remaining explicit, flexible, and compatible with Semantic Versioning. + +Option 3.1 (Trunk-Based Development Only) has been rejected because it does not address the need for coordinated product releases or explicit stabilization phases. + +Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. + +Option 3.3 (SemVer-Based Polyrepo Release Process with Manifest Repository) has been selected as it provides a single source of truth for integration, supports both continuous verification and reproducible releases, and scales with module count and team autonomy. + +--- + +## 6. Consequences & Challenges + +### Positive +- Reproducible, auditable product releases. +- Continuous integration without blocking module development. +- Clear separation between development, integration, and stabilization. +- Scales with increasing module count and team autonomy. + +### Negative +- Requires explicit integration governance. +- Introduces additional coordination effort compared to single-repo workflows. +- Relies on strict SemVer discipline in modules. + +## 7. Explicit Non-Goals +- This ADR does **not** prescribe: + - module-internal branching models, + - commit workflows, + - or team-specific development practices. +- This ADR is **not Gitflow**: + - stabilization is centralized in the manifest repository, + - modules are not required to align release branches. +- This ADR does **not** assume a monorepo or continuous deployment model. From 38f2a00698abb5baacae3fcd2553773e68818a23 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Thu, 8 Jan 2026 15:08:04 +0100 Subject: [PATCH 02/17] Added chapter numbers --- docs/design_decisions/DR-005-infra.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index 22c457c3c34..e2aaf201500 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -114,7 +114,7 @@ The solution is based on four core principles: 3. **Two integration modes: tracking (continuous verification) and pinned (release/stabilization)** 4. **Release stabilization via immutable tags (and an optional manifest release branch)** -### 1. Module Versioning (SemVer) +### 4.1 Module Versioning (SemVer) - Each module repository publishes releases following Semantic Versioning: - **PATCH**: backward-compatible bug fixes @@ -131,7 +131,7 @@ The solution is based on four core principles: > **Explicitly out of scope:** > Module-internal development workflows (e.g. trunk-based development, feature branches, rebasing strategies) are intentionally not prescribed by this ADR. -### 2. Manifest Repository +### 4.2 Manifest Repository A dedicated **manifest (integration) repository** defines which exact module states form a product snapshot. @@ -148,11 +148,11 @@ Responsibilities of the manifest repository: Any change to module references in the manifest repository is made via a pull request and reviewed as an integration change. -### 3. Integration Modes +### 4.3 Integration Modes The manifest repository operates in two distinct modes. -#### 3.1 Tracking Mode (Continuous Verification) +#### 4.3.1 Tracking Mode (Continuous Verification) - Used on manifest `main`. - The manifest may track module branches or “next release” branches by automatically updating pinned commit hashes. @@ -162,7 +162,7 @@ The manifest repository operates in two distinct modes. - detect integration issues early. - Tracking references are **not considered release-ready** and are not used for formal release stages. -#### 3.2 Pinned Mode (Release and Stabilization) +#### 4.3.2 Pinned Mode (Release and Stabilization) - Used for release preparation and formal stages. - The manifest pins **immutable identifiers only**: @@ -171,7 +171,7 @@ The manifest repository operates in two distinct modes. - Pinned mode guarantees full reproducibility and auditability. - All release stages and final releases must use pinned mode. -### 4. Product Release Identification and Structure +### 4.4 Product Release Identification and Structure Product releases are identified using **train-style tags**: @@ -182,7 +182,7 @@ Product releases are identified using **train-style tags**: Product versions reuse SemVer-compatible notation for clarity and tooling compatibility, but they represent **integration snapshots**, not API stability guarantees for individual modules. -### 5. Release Stabilization Line +### 4.5 Release Stabilization Line To allow iteration without disturbing ongoing integration: @@ -196,7 +196,7 @@ This allows: - continuous tracking on manifest `main`, - controlled stabilization for a given product release. -### 6. Release Stages +### 4.6 Release Stages #### Alpha - Goal: early system-level integration. @@ -218,14 +218,14 @@ This allows: - Tag `vX.Y.0` marks the final release. - Optional product patch releases (`vX.Y.1`, `vX.Y.2`, …) follow the same rules. -### 7. Backports +### 4.7 Backports - Fixes are applied first on the appropriate module code line. - Compatible PATCH releases are created if required. - The manifest is updated to reference the new immutable version or commit. - A new stage or patch tag is created. -### 8. Breaking Changes During Stabilization +### 4.8 Breaking Changes During Stabilization - Breaking changes during stabilization are strongly discouraged. - If required (e.g. priority or organizational decisions), they must: @@ -233,7 +233,7 @@ This allows: - include a documented mitigation strategy (compatibility layer, coordinated upgrades, scope reduction, or deferral), - and be recorded in the manifest repository. -### 9. Module Release-Line Strategies (Non-Normative Examples) +### 4.9 Module Release-Line Strategies (Non-Normative Examples) Modules may choose any internal strategy that preserves SemVer guarantees. Common compliant patterns include: @@ -246,7 +246,7 @@ Modules may choose any internal strategy that preserves SemVer guarantees. Commo The choice is module-local and does not affect the integration model. -### 10. Ownership and Governance +### 4.10 Ownership and Governance - The manifest repository is owned by the infrastructure/integration team, as defined in DR-002. - As release stages progress (beta, rc), changes to the manifest require increasing scrutiny and justification. From a5f3aff8757a8b423cad22a04318558f8fa3cd92 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Thu, 8 Jan 2026 15:09:53 +0100 Subject: [PATCH 03/17] Shortened semver chapter --- docs/design_decisions/DR-005-infra.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index e2aaf201500..5fb628487c0 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -116,10 +116,7 @@ The solution is based on four core principles: ### 4.1 Module Versioning (SemVer) -- Each module repository publishes releases following Semantic Versioning: - - **PATCH**: backward-compatible bug fixes - - **MINOR**: backward-compatible feature additions - - **MAJOR**: backward-incompatible changes +- Each module repository publishes releases following [Semantic Versioning 2.0.0](https://semver.org/). - Version numbers are **global, linear, and unique per module**. - Once a version is released, it is immutable and must never be reused. - There is no automatic or periodic major version bump tied to product releases. From 7942367e085c61b45f7a311ab65399239eaee972 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Fri, 9 Jan 2026 15:21:41 +0100 Subject: [PATCH 04/17] Moved out of scope section and removed a section --- docs/design_decisions/DR-005-infra.md | 33 ++++++++------------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index 5fb628487c0..b4905ade1a5 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -55,6 +55,16 @@ This ADR defines a release process explicitly designed for **polyrepo systems wi - Independent module lifecycles without blocking development - A process that scales across many repositories and teams +**Out of scope** +- This ADR does **not** prescribe: + - module-internal branching models, + - commit workflows, + - or team-specific development practices. +- This ADR is **not Gitflow**: + - stabilization is centralized in the manifest repository, + - modules are not required to align release branches. +- This ADR does **not** assume a monorepo or continuous deployment model. + --- ## 3. Options Considered @@ -264,26 +274,3 @@ Option 3.2 (Gitflow Across Repositories) has been rejected because it requires c Option 3.3 (SemVer-Based Polyrepo Release Process with Manifest Repository) has been selected as it provides a single source of truth for integration, supports both continuous verification and reproducible releases, and scales with module count and team autonomy. --- - -## 6. Consequences & Challenges - -### Positive -- Reproducible, auditable product releases. -- Continuous integration without blocking module development. -- Clear separation between development, integration, and stabilization. -- Scales with increasing module count and team autonomy. - -### Negative -- Requires explicit integration governance. -- Introduces additional coordination effort compared to single-repo workflows. -- Relies on strict SemVer discipline in modules. - -## 7. Explicit Non-Goals -- This ADR does **not** prescribe: - - module-internal branching models, - - commit workflows, - - or team-specific development practices. -- This ADR is **not Gitflow**: - - stabilization is centralized in the manifest repository, - - modules are not required to align release branches. -- This ADR does **not** assume a monorepo or continuous deployment model. From f1bdf9a562dcb13dd05a4cc058a1c7ca6dfa60c6 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Fri, 9 Jan 2026 16:33:35 +0100 Subject: [PATCH 05/17] Incorporated review comments and shrinked down the document --- docs/design_decisions/DR-005-infra.md | 222 ++++++-------------------- 1 file changed, 52 insertions(+), 170 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index b4905ade1a5..dd07a1b78b9 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -11,7 +11,7 @@ https://www.apache.org/licenses/LICENSE-2.0 SPDX-License-Identifier: Apache-2.0 --> -# DR-005-Infra: Polyrepo Release Process with SemVer, Manifest Repository, and Continuous Integration Tracking +# DR-005-Infra: Development, Release and Bugfix workflows - **Status:** Proposed - **Owner:** Infrastructure Community @@ -22,9 +22,8 @@ SPDX-License-Identifier: Apache-2.0 ## 1. Context This project consists of multiple independently developed modules stored in separate repositories (polyrepo setup). -Each module evolves independently and is versioned using Semantic Versioning (SemVer). -This ADR **builds on DR-002 (Infrastructure Architecture)**, which establishes: +This ADR builds on [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md), which establishes: - the polyrepo structure, - centralized responsibility for cross-repository integration, - and infrastructure-owned integration tooling and processes. @@ -54,24 +53,19 @@ This ADR defines a release process explicitly designed for **polyrepo systems wi - Continuous integration before formal releases - Independent module lifecycles without blocking development - A process that scales across many repositories and teams - -**Out of scope** -- This ADR does **not** prescribe: - - module-internal branching models, - - commit workflows, - - or team-specific development practices. -- This ADR is **not Gitflow**: - - stabilization is centralized in the manifest repository, - - modules are not required to align release branches. -- This ADR does **not** assume a monorepo or continuous deployment model. +- Working on the main branch should be always possible +- Working on a release branch should be always possible and not harm the development on the main branch (resp. Vice versa) +- Working on a bugfix should always be possible (for any old release) +- Module developers must know how to name their released versions +- It must be clear how to do the integration, means what to reference (e.g. „extended“ semver as 1.2.3-etas-r1.0) --- ## 3. Options Considered -### 3.1 Trunk-Based Development Only +### 3.1 Trunk-Based Development with SemVer -Assumes continuous deployment and does not define coordinated product releases. +Assumes continuous deployment on a longlived branch, e.g. `main` and does not define coordinated product releases. **Pros**: - Simplifies development workflow. @@ -81,6 +75,37 @@ Assumes continuous deployment and does not define coordinated product releases. - Does not address the need for coordinated product releases. - Lacks explicit stabilization phases. - Not suitable for polyrepo systems requiring integration snapshots. +- SemVer is not capable to mimic tree structures (see the following diagram) + +```{mermaid} +gitGraph + commit id: "1.2.3" + + branch release + checkout release + commit id: "1.2.3-r1.0" + + checkout main + branch bugfix + checkout main + + commit + commit id: "1.2.4" + commit + commit id: "1.2.6" + + checkout bugfix + commit id: "bugfix-work" + + checkout release + merge bugfix + commit id: "1.2.5" + + checkout main + merge bugfix +``` +> *Explaination:* Using SemVer quickly reaches its limits. +If branching off from a version e.g. 1.2.3 and needing a bugfix while the development on main goes on where other versions e.g. 1.2.4 are created, now on a release branch you would need to use the version 1.2.5. The development on main for the next version needs to use 1.2.6 although the logical predecessor is 1.2.4. This violates SemVer since backward compatibility in that case is (or could be) broken. ### 3.2 Gitflow Across Repositories @@ -95,182 +120,39 @@ Introduces coordinated release branching across repositories but lacks a central - Lacks a single source of truth for integration state. - Does not scale well with increasing module count. -### 3.3 SemVer-Based Polyrepo Release Process with Manifest Repository +### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer + +As described in [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for release(d) versions. Because of the earlier described limitations of SemVer, the correct module versions should not be referenced in the `MODULE.bazel` (in the manifest repository) as e.g., `1.2.3` but either by referencing the git commit hash directly or using a relaxed SemVer string, e.g. as `1.2.3-v1.0` where the string after the hyphen represents the respective S-CORE release. -A dedicated manifest repository defines integration state, with independent module versioning and two integration modes (tracking and pinned). +With that approach releases are possible, e.g. by creating a release branch in the integration repository as well as in the affected module repositories. Also bugfixes of "old" releases are possible by checking out the repective release branch in the reference integration and if necessary to also created bugfixes in the affected modules. **Pros**: - Single source of truth for product integration. -- Supports continuous verification via tracking mode. -- Provides reproducible releases via pinned mode. +- Supports continuous verification. +- Provides reproducible releases. - Scales with module count and team autonomy. - Clear separation between development, integration, and stabilization. **Cons**: - Requires explicit integration governance. - Introduces additional coordination effort compared to single-repo workflows. -- Relies on strict SemVer discipline in modules. --- ## 4. Decision -We adopt a **SemVer-based polyrepo release process with a dedicated manifest repository, continuous integration tracking, and release trains identified by tags**. - -The solution is based on four core principles: - -1. **Independent module versioning with strict SemVer guarantees** -2. **A manifest repository as the single source of truth for integration** -3. **Two integration modes: tracking (continuous verification) and pinned (release/stabilization)** -4. **Release stabilization via immutable tags (and an optional manifest release branch)** - -### 4.1 Module Versioning (SemVer) - -- Each module repository publishes releases following [Semantic Versioning 2.0.0](https://semver.org/). -- Version numbers are **global, linear, and unique per module**. -- Once a version is released, it is immutable and must never be reused. -- There is no automatic or periodic major version bump tied to product releases. - -**Compatibility rule:** -- A PATCH release must be produced from a code line that is compatible with the previous release. -- Modules must not include incompatible changes in PATCH releases. - -> **Explicitly out of scope:** -> Module-internal development workflows (e.g. trunk-based development, feature branches, rebasing strategies) are intentionally not prescribed by this ADR. - -### 4.2 Manifest Repository - -A dedicated **manifest (integration) repository** defines which exact module states form a product snapshot. - -Responsibilities of the manifest repository: -- Define the complete product composition. -- Pin module states either as: - - released module versions, or - - explicit commit hashes (for tracking mode). -- Contain integration-specific artifacts such as: - - end-to-end tests, - - integration configuration, - - packaging or distribution logic. -- Act as the **single source of truth** for product integration. - -Any change to module references in the manifest repository is made via a pull request and reviewed as an integration change. - -### 4.3 Integration Modes - -The manifest repository operates in two distinct modes. - -#### 4.3.1 Tracking Mode (Continuous Verification) - -- Used on manifest `main`. -- The manifest may track module branches or “next release” branches by automatically updating pinned commit hashes. -- Goal: - - continuously verify that the *current development state* of modules can be integrated, - - reduce the gap between development and release, - - detect integration issues early. -- Tracking references are **not considered release-ready** and are not used for formal release stages. - -#### 4.3.2 Pinned Mode (Release and Stabilization) - -- Used for release preparation and formal stages. -- The manifest pins **immutable identifiers only**: - - released module versions, or - - fixed commit hashes. -- Pinned mode guarantees full reproducibility and auditability. -- All release stages and final releases must use pinned mode. - -### 4.4 Product Release Identification and Structure +We decided for **Option 3.3**. -Product releases are identified using **train-style tags**: - -- `vX.Y.0-alpha.N` -- `vX.Y.0-beta.N` -- `vX.Y.0-rc.N` -- `vX.Y.0` (final) - -Product versions reuse SemVer-compatible notation for clarity and tooling compatibility, but they represent **integration snapshots**, not API stability guarantees for individual modules. - -### 4.5 Release Stabilization Line - -To allow iteration without disturbing ongoing integration: - -- A stabilization line may be introduced **in the manifest repository only**, e.g.: - - an optional branch `release/vX.Y`, or - - an equivalent agreed stabilization mechanism. -- Stabilization happens **only** in the manifest repository. -- Module repositories are not required to coordinate release branches. - -This allows: -- continuous tracking on manifest `main`, -- controlled stabilization for a given product release. - -### 4.6 Release Stages - -#### Alpha -- Goal: early system-level integration. -- Allowed changes: - - compatible bug fixes (PATCH), - - compatible features (MINOR) if strictly required. - -#### Beta -- Goal: feature freeze and stabilization. -- Allowed changes: - - bug fixes only (PATCH). - -#### Release Candidate (RC) -- Goal: final validation. -- Allowed changes: - - critical bug or security fixes only (PATCH). - -#### Final -- Tag `vX.Y.0` marks the final release. -- Optional product patch releases (`vX.Y.1`, `vX.Y.2`, …) follow the same rules. - -### 4.7 Backports - -- Fixes are applied first on the appropriate module code line. -- Compatible PATCH releases are created if required. -- The manifest is updated to reference the new immutable version or commit. -- A new stage or patch tag is created. - -### 4.8 Breaking Changes During Stabilization - -- Breaking changes during stabilization are strongly discouraged. -- If required (e.g. priority or organizational decisions), they must: - - be explicitly declared, - - include a documented mitigation strategy (compatibility layer, coordinated upgrades, scope reduction, or deferral), - - and be recorded in the manifest repository. - -### 4.9 Module Release-Line Strategies (Non-Normative Examples) - -Modules may choose any internal strategy that preserves SemVer guarantees. Common compliant patterns include: - -1. **Disciplined mainline releases** - PATCH releases are cut before incompatible changes are merged. -2. **Temporary release branches** - Short-lived branches cut from the last release tag to produce compatible PATCH releases. -3. **Persistent release branches** - Long-lived branches for maintained version lines or LTS support. - -The choice is module-local and does not affect the integration model. - -### 4.10 Ownership and Governance - -- The manifest repository is owned by the infrastructure/integration team, as defined in DR-002. -- As release stages progress (beta, rc), changes to the manifest require increasing scrutiny and justification. -- Governance applies only to the manifest repository and does not constrain module-internal workflows. - ---- - -## 5. Rationale +**Rationale** This approach reflects established industry practice for large-scale polyrepo systems using manifest-based integration and release trains (e.g. Android/AOSP, Chromium-style roll-ups), -while remaining explicit, flexible, and compatible with Semantic Versioning. +while remaining explicit, and flexible. -Option 3.1 (Trunk-Based Development Only) has been rejected because it does not address the need for coordinated product releases or explicit stabilization phases. +It provides a single source of truth for integration, supports both continuous verification and reproducible releases, and scales with module count and team autonomy. -Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. +Option 3.1 (Trunk-Based Development Only) has been rejected because it does not address the need for coordinated product releases or explicit stabilization phases in a poly repo environment. -Option 3.3 (SemVer-Based Polyrepo Release Process with Manifest Repository) has been selected as it provides a single source of truth for integration, supports both continuous verification and reproducible releases, and scales with module count and team autonomy. +Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. --- From d920ed9b84e1085dab139f70e9b01cf8d22f2227 Mon Sep 17 00:00:00 2001 From: Oliver Emrich Date: Mon, 12 Jan 2026 09:21:40 +0100 Subject: [PATCH 06/17] Update docs/design_decisions/DR-005-infra.md Co-authored-by: Andreas Zwinkau <95761648+a-zw@users.noreply.github.com> Signed-off-by: Oliver Emrich --- docs/design_decisions/DR-005-infra.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index dd07a1b78b9..d310a8aa014 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -155,4 +155,3 @@ Option 3.1 (Trunk-Based Development Only) has been rejected because it does not Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. ---- From 5557a135b04ac6cd5edbb886538bc6e5d1604c32 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Mon, 12 Jan 2026 09:31:05 +0100 Subject: [PATCH 07/17] Adressed discussion_r2681167607 --- docs/design_decisions/DR-005-infra.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index d310a8aa014..e700f0823b1 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -65,17 +65,16 @@ This ADR defines a release process explicitly designed for **polyrepo systems wi ### 3.1 Trunk-Based Development with SemVer -Assumes continuous deployment on a longlived branch, e.g. `main` and does not define coordinated product releases. +Uses trunk-based development where modules develop on `main` branches and release with standard SemVer tags. The manifest repository (per DR-002) continuously tracks the latest module versions. When a product release is needed, release branches are created in the manifest and affected module repositories. Module bugfixes on release branches are tagged with standard SemVer, and the manifest's release branch references these versions. **Pros**: - Simplifies development workflow. - Encourages frequent integration. +- Well-understood versioning convention. **Cons**: -- Does not address the need for coordinated product releases. -- Lacks explicit stabilization phases. -- Not suitable for polyrepo systems requiring integration snapshots. -- SemVer is not capable to mimic tree structures (see the following diagram) +- Standard SemVer cannot represent parallel release streams. When a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous (see diagram below). +- Versioning conflicts arise when merging bugfixes back to `main`. ```{mermaid} gitGraph @@ -104,20 +103,21 @@ gitGraph checkout main merge bugfix ``` -> *Explaination:* Using SemVer quickly reaches its limits. -If branching off from a version e.g. 1.2.3 and needing a bugfix while the development on main goes on where other versions e.g. 1.2.4 are created, now on a release branch you would need to use the version 1.2.5. The development on main for the next version needs to use 1.2.6 although the logical predecessor is 1.2.4. This violates SemVer since backward compatibility in that case is (or could be) broken. +> *Explanation:* Using SemVer quickly reaches its limits. +If branching off from version 1.2.3 and needing a bugfix while development on `main` continues (creating 1.2.4), the release branch would need version 1.2.5 for the bugfix. The next version on `main` must then be 1.2.6, even though its logical predecessor is 1.2.4. This violates SemVer since backward compatibility could be broken. ### 3.2 Gitflow Across Repositories -Introduces coordinated release branching across repositories but lacks a central integration manifest. +The manifest repository creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Since Bazel requires either version tags or commit hashes in `MODULE.bazel`, the manifest must be updated manually each time a module's release branch is tagged or advanced. This creates a workflow where teams coordinate to stabilize their module release branches, tag them, and then update the manifest's release branch to reference those tags or commits. **Pros**: - Well-known branching model. - Provides release branches for stabilization. **Cons**: -- Requires coordination of release branches across all repositories. -- Lacks a single source of truth for integration state. +- Requires manual updates to the manifest whenever module release branches are tagged or advanced. +- Manual coordination across all module repositories to create, maintain, and tag release branches. +- Frequent manual manifest updates during stabilization increase coordination overhead. - Does not scale well with increasing module count. ### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer @@ -154,4 +154,3 @@ It provides a single source of truth for integration, supports both continuous v Option 3.1 (Trunk-Based Development Only) has been rejected because it does not address the need for coordinated product releases or explicit stabilization phases in a poly repo environment. Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. - From 98f4e8c98abde27357c72cc04d164aa589a247f4 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Mon, 12 Jan 2026 09:50:01 +0100 Subject: [PATCH 08/17] Addresses review comment discussion_r2681174185 --- docs/design_decisions/DR-005-infra.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index e700f0823b1..5bedcff6a2e 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -108,16 +108,17 @@ If branching off from version 1.2.3 and needing a bugfix while development on `m ### 3.2 Gitflow Across Repositories -The manifest repository creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Since Bazel requires either version tags or commit hashes in `MODULE.bazel`, the manifest must be updated manually each time a module's release branch is tagged or advanced. This creates a workflow where teams coordinate to stabilize their module release branches, tag them, and then update the manifest's release branch to reference those tags or commits. +Uses the Gitflow branching model where modules maintain both `main` and `develop` branches, with release branches created for stabilization. The manifest repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Modules tag bugfixes on release branches with standard SemVer, and the manifest's release branch references these versions. **Pros**: - Well-known branching model. -- Provides release branches for stabilization. +- Explicit `develop` branch separates ongoing work from release stabilization. +- Release branches provide clear stabilization phases. **Cons**: -- Requires manual updates to the manifest whenever module release branches are tagged or advanced. -- Manual coordination across all module repositories to create, maintain, and tag release branches. -- Frequent manual manifest updates during stabilization increase coordination overhead. +- Standard SemVer suffers from the same parallel release stream problem as Option 3.1 (version numbering conflicts). +- Additional overhead of maintaining separate `develop` branches across all repositories. +- More complex branching model increases coordination complexity in a polyrepo setup. - Does not scale well with increasing module count. ### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer From a5399bb6fb54c79d9ad0a75dfb1d280c3b455abd Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Mon, 12 Jan 2026 09:53:22 +0100 Subject: [PATCH 09/17] Addresses review comment discussion_r2681184227 --- docs/design_decisions/DR-005-infra.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index 5bedcff6a2e..ebda453790a 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -53,11 +53,12 @@ This ADR defines a release process explicitly designed for **polyrepo systems wi - Continuous integration before formal releases - Independent module lifecycles without blocking development - A process that scales across many repositories and teams +- Minimal integration governance to coordinate releases across repositories - Working on the main branch should be always possible - Working on a release branch should be always possible and not harm the development on the main branch (resp. Vice versa) - Working on a bugfix should always be possible (for any old release) - Module developers must know how to name their released versions -- It must be clear how to do the integration, means what to reference (e.g. „extended“ semver as 1.2.3-etas-r1.0) +- It must be clear how to do the integration, means what to reference (e.g. „extended" semver as 1.2.3-etas-r1.0) --- @@ -135,7 +136,6 @@ With that approach releases are possible, e.g. by creating a release branch in t - Clear separation between development, integration, and stabilization. **Cons**: -- Requires explicit integration governance. - Introduces additional coordination effort compared to single-repo workflows. --- From 1d96828fe17c0d67a93ead2a3eccef49d9ec1161 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Mon, 12 Jan 2026 10:00:33 +0100 Subject: [PATCH 10/17] Restructured chatper 2 --- docs/design_decisions/DR-005-infra.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index ebda453790a..2c73495a3ac 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -46,20 +46,29 @@ This ADR defines a release process explicitly designed for **polyrepo systems wi --- -## 2. Goals and Requirements +## 2. Requirements and Goals + +### 2.1 Requirements + +Options that do not satisfy these requirements are not viable and will be rejected: - Reproducible and auditable release snapshots -- Explicit stabilization phases -- Continuous integration before formal releases -- Independent module lifecycles without blocking development -- A process that scales across many repositories and teams -- Minimal integration governance to coordinate releases across repositories - Working on the main branch should be always possible - Working on a release branch should be always possible and not harm the development on the main branch (resp. Vice versa) - Working on a bugfix should always be possible (for any old release) - Module developers must know how to name their released versions - It must be clear how to do the integration, means what to reference (e.g. „extended" semver as 1.2.3-etas-r1.0) +### 2.2 Optimization Goals + +Among viable options, we optimize for: + +- Explicit stabilization phases +- Continuous integration before formal releases +- Independent module lifecycles without blocking development +- A process that scales across many repositories and teams +- Minimal integration governance to coordinate releases across repositories + --- ## 3. Options Considered From 531251e4635675fa28f306c39943210d031bd2dd Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Mon, 12 Jan 2026 10:10:13 +0100 Subject: [PATCH 11/17] Minor changes --- docs/design_decisions/DR-005-infra.md | 30 ++++++++++----------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index 2c73495a3ac..c8676c749c0 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -30,19 +30,11 @@ This ADR builds on [DR-002 (Integration Testing in a Distributed Monolith)](./DR Within that context, this ADR defines **how coordinated product releases are produced** from independently developed and versioned modules, while allowing continuous integration to reduce the gap between development and release. -The project delivers a **coordinated product release** that integrates a specific, tested combination of module states. This requires: - -- reproducible and auditable release snapshots, -- explicit stabilization phases, -- continuous integration before formal releases, -- independent module lifecycles without blocking development, -- and a process that scales across many repositories and teams. - Commonly referenced workflows do not fully address this problem space: -- **Trunk-Based Development** assumes continuous deployment and does not define coordinated product releases. -- **Gitflow** introduces coordinated release branching across repositories and lacks a central integration manifest. +- **Trunk-Based Development** with standard SemVer cannot represent parallel release streams needed for supporting multiple product versions simultaneously. +- **Gitflow** adds branching complexity without solving the versioning problem for parallel releases in a polyrepo setup. -This ADR defines a release process explicitly designed for **polyrepo systems with coordinated integration releases and continuous verification**. +This ADR evaluates branching and versioning strategies explicitly designed for **polyrepo systems with coordinated integration releases and continuous verification**. --- @@ -57,7 +49,7 @@ Options that do not satisfy these requirements are not viable and will be reject - Working on a release branch should be always possible and not harm the development on the main branch (resp. Vice versa) - Working on a bugfix should always be possible (for any old release) - Module developers must know how to name their released versions -- It must be clear how to do the integration, means what to reference (e.g. „extended" semver as 1.2.3-etas-r1.0) +- The versioning scheme must clearly indicate which product release a module version belongs to, enabling parallel release streams ### 2.2 Optimization Goals @@ -155,12 +147,12 @@ We decided for **Option 3.3**. **Rationale** -This approach reflects established industry practice for large-scale polyrepo systems using -manifest-based integration and release trains (e.g. Android/AOSP, Chromium-style roll-ups), -while remaining explicit, and flexible. - -It provides a single source of truth for integration, supports both continuous verification and reproducible releases, and scales with module count and team autonomy. +Options 3.1 and 3.2 both violate the requirement for a versioning scheme that clearly indicates which product release a module version belongs to. Standard SemVer cannot represent parallel release streams - when a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous and conflicts arise (as demonstrated in the diagram in Option 3.1). This makes both options non-viable. -Option 3.1 (Trunk-Based Development Only) has been rejected because it does not address the need for coordinated product releases or explicit stabilization phases in a poly repo environment. +Option 3.3 is the only viable option as it satisfies all requirements through relaxed SemVer (e.g., `1.2.3-v1.0`) where the suffix indicates the product release. Additionally, it optimizes for our goals by: +- Providing a single source of truth for product integration +- Supporting continuous verification through the manifest repository +- Scaling with module count and team autonomy +- Enabling clear separation between development, integration, and stabilization -Option 3.2 (Gitflow Across Repositories) has been rejected because it requires coordinating release branches across all repositories and lacks a central integration manifest, which does not scale well. +This approach reflects established industry practice for large-scale polyrepo systems using manifest-based integration and release trains (e.g., Android/AOSP, Chromium-style roll-ups). From 34403a62e0b2bb9e83f1eea8675735f2bd121418 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Tue, 13 Jan 2026 16:46:31 +0100 Subject: [PATCH 12/17] Fixed typos and inconsistencies --- docs/design_decisions/DR-005-infra.md | 158 -------------------------- 1 file changed, 158 deletions(-) delete mode 100644 docs/design_decisions/DR-005-infra.md diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md deleted file mode 100644 index c8676c749c0..00000000000 --- a/docs/design_decisions/DR-005-infra.md +++ /dev/null @@ -1,158 +0,0 @@ - - -# DR-005-Infra: Development, Release and Bugfix workflows - -- **Status:** Proposed -- **Owner:** Infrastructure Community -- **Date:** 2026-01-08 - ---- - -## 1. Context - -This project consists of multiple independently developed modules stored in separate repositories (polyrepo setup). - -This ADR builds on [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md), which establishes: -- the polyrepo structure, -- centralized responsibility for cross-repository integration, -- and infrastructure-owned integration tooling and processes. - -Within that context, this ADR defines **how coordinated product releases are produced** from independently developed and versioned modules, while allowing continuous integration to reduce the gap between development and release. - -Commonly referenced workflows do not fully address this problem space: -- **Trunk-Based Development** with standard SemVer cannot represent parallel release streams needed for supporting multiple product versions simultaneously. -- **Gitflow** adds branching complexity without solving the versioning problem for parallel releases in a polyrepo setup. - -This ADR evaluates branching and versioning strategies explicitly designed for **polyrepo systems with coordinated integration releases and continuous verification**. - ---- - -## 2. Requirements and Goals - -### 2.1 Requirements - -Options that do not satisfy these requirements are not viable and will be rejected: - -- Reproducible and auditable release snapshots -- Working on the main branch should be always possible -- Working on a release branch should be always possible and not harm the development on the main branch (resp. Vice versa) -- Working on a bugfix should always be possible (for any old release) -- Module developers must know how to name their released versions -- The versioning scheme must clearly indicate which product release a module version belongs to, enabling parallel release streams - -### 2.2 Optimization Goals - -Among viable options, we optimize for: - -- Explicit stabilization phases -- Continuous integration before formal releases -- Independent module lifecycles without blocking development -- A process that scales across many repositories and teams -- Minimal integration governance to coordinate releases across repositories - ---- - -## 3. Options Considered - -### 3.1 Trunk-Based Development with SemVer - -Uses trunk-based development where modules develop on `main` branches and release with standard SemVer tags. The manifest repository (per DR-002) continuously tracks the latest module versions. When a product release is needed, release branches are created in the manifest and affected module repositories. Module bugfixes on release branches are tagged with standard SemVer, and the manifest's release branch references these versions. - -**Pros**: -- Simplifies development workflow. -- Encourages frequent integration. -- Well-understood versioning convention. - -**Cons**: -- Standard SemVer cannot represent parallel release streams. When a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous (see diagram below). -- Versioning conflicts arise when merging bugfixes back to `main`. - -```{mermaid} -gitGraph - commit id: "1.2.3" - - branch release - checkout release - commit id: "1.2.3-r1.0" - - checkout main - branch bugfix - checkout main - - commit - commit id: "1.2.4" - commit - commit id: "1.2.6" - - checkout bugfix - commit id: "bugfix-work" - - checkout release - merge bugfix - commit id: "1.2.5" - - checkout main - merge bugfix -``` -> *Explanation:* Using SemVer quickly reaches its limits. -If branching off from version 1.2.3 and needing a bugfix while development on `main` continues (creating 1.2.4), the release branch would need version 1.2.5 for the bugfix. The next version on `main` must then be 1.2.6, even though its logical predecessor is 1.2.4. This violates SemVer since backward compatibility could be broken. - -### 3.2 Gitflow Across Repositories - -Uses the Gitflow branching model where modules maintain both `main` and `develop` branches, with release branches created for stabilization. The manifest repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Modules tag bugfixes on release branches with standard SemVer, and the manifest's release branch references these versions. - -**Pros**: -- Well-known branching model. -- Explicit `develop` branch separates ongoing work from release stabilization. -- Release branches provide clear stabilization phases. - -**Cons**: -- Standard SemVer suffers from the same parallel release stream problem as Option 3.1 (version numbering conflicts). -- Additional overhead of maintaining separate `develop` branches across all repositories. -- More complex branching model increases coordination complexity in a polyrepo setup. -- Does not scale well with increasing module count. - -### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer - -As described in [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for release(d) versions. Because of the earlier described limitations of SemVer, the correct module versions should not be referenced in the `MODULE.bazel` (in the manifest repository) as e.g., `1.2.3` but either by referencing the git commit hash directly or using a relaxed SemVer string, e.g. as `1.2.3-v1.0` where the string after the hyphen represents the respective S-CORE release. - -With that approach releases are possible, e.g. by creating a release branch in the integration repository as well as in the affected module repositories. Also bugfixes of "old" releases are possible by checking out the repective release branch in the reference integration and if necessary to also created bugfixes in the affected modules. - -**Pros**: -- Single source of truth for product integration. -- Supports continuous verification. -- Provides reproducible releases. -- Scales with module count and team autonomy. -- Clear separation between development, integration, and stabilization. - -**Cons**: -- Introduces additional coordination effort compared to single-repo workflows. - ---- - -## 4. Decision - -We decided for **Option 3.3**. - -**Rationale** - -Options 3.1 and 3.2 both violate the requirement for a versioning scheme that clearly indicates which product release a module version belongs to. Standard SemVer cannot represent parallel release streams - when a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous and conflicts arise (as demonstrated in the diagram in Option 3.1). This makes both options non-viable. - -Option 3.3 is the only viable option as it satisfies all requirements through relaxed SemVer (e.g., `1.2.3-v1.0`) where the suffix indicates the product release. Additionally, it optimizes for our goals by: -- Providing a single source of truth for product integration -- Supporting continuous verification through the manifest repository -- Scaling with module count and team autonomy -- Enabling clear separation between development, integration, and stabilization - -This approach reflects established industry practice for large-scale polyrepo systems using manifest-based integration and release trains (e.g., Android/AOSP, Chromium-style roll-ups). From 26458df34f7371258a310f3d5b7a592f60754331 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Tue, 13 Jan 2026 17:12:25 +0100 Subject: [PATCH 13/17] Reworked chapter 3.1 --- docs/design_decisions/DR-005-infra.md | 177 ++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/design_decisions/DR-005-infra.md diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md new file mode 100644 index 00000000000..ed19d775b80 --- /dev/null +++ b/docs/design_decisions/DR-005-infra.md @@ -0,0 +1,177 @@ + + +# DR-005-Infra: Development, Release and Bugfix workflows + +- **Status:** Proposed +- **Owner:** Infrastructure Community +- **Date:** 2026-01-08 + +--- + +## 1. Context + +This project consists of multiple independently developed modules stored in separate +repositories (polyrepo setup). + +This ADR builds on +[DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md), which +establishes: +- the polyrepo structure, +- centralized responsibility for cross-repository integration, +- and infrastructure-owned integration tooling and processes. + +Within that context, this ADR defines **how coordinated product releases are produced** +from independently developed and versioned modules, while allowing continuous +integration to reduce the gap between development and release. + +Commonly referenced workflows do not fully address this problem space: +- **Trunk-Based Development** with standard SemVer cannot represent parallel +release streams needed for supporting multiple product versions simultaneously. +- **Gitflow** adds branching complexity without solving the versioning problem +for parallel releases in a polyrepo setup. + +This ADR evaluates branching and versioning strategies explicitly designed for +**polyrepo systems with coordinated integration releases and continuous verification**. + +--- + +## 2. Requirements and Goals + +### 2.1 Requirements + +Options that do not satisfy these requirements are not viable and will be rejected: + +- Reproducible and auditable release snapshots +- During working on a release, working on the main branch should still be possible +- Working on a release branch should be always possible and not harm the development on the main branch (resp. vice versa) +- Working on a bugfix should always be possible (for any old release) +- Module developers must know how to name their released versions +- The versioning scheme must clearly indicate which product release a module version belongs to, enabling parallel release, means essentially the ability to maintain old releases +- Explicit stabilization phases +- Continuous integration before formal releases + +### 2.2 Optimization Goals + +Among viable options, we optimize for: + +- Independent module lifecycles without blocking development +- A process that scales across many repositories and teams +- Minimal integration governance to coordinate releases across repositories + +--- + +## 3. Options Considered + +### 3.1 Trunk-Based Development + +Trunk-based development is a branching model where all developers work on a single main +branch ("trunk"), integrating changes frequently—ideally daily. Feature and bugfix +development happens directly on the trunk or in very short-lived branches that are +merged back quickly. Long-lived release branches and parallel release streams are +explicitly avoided. Releases are created directly from the trunk, and if a release +branch is needed for final stabilization or a hotfix, it is kept as short-lived +as possible. Bugfixes are made on the trunk and only cherry-picked to a release +branch in rare cases where a hotfix for an older release is required. Versioning +is linear, and the model does not support parallel maintenance of multiple +product versions. + +**Pros**: +- Very simple and lightweight branch structure. +- Maximizes integration and fast feedback through continuous integration. +- Minimizes merge conflicts and versioning issues. + +**Cons**: +- Parallel maintenance of multiple product versions (e.g., for long-term support) is +not feasible. +- Not suitable for polyrepo setups that require parallel support of several releases, +as parallel release streams are not supported by design. + +> Note: The versioning and merge conflicts described in the diagram below typically +arise when deviating from strict trunk-based development and introducing long-lived +release branches. This is no longer considered state-of-the-art trunk-based development. + +```{mermaid} +gitGraph + commit id: "1.2.3" + + branch release + checkout release + commit id: "1.2.3-v1.0" + + checkout main + branch bugfix + checkout main + + commit + commit id: "1.2.4" + commit + commit id: "1.2.6" + + checkout bugfix + commit id: "bugfix-work" + + checkout release + merge bugfix + commit id: "1.2.5" + + checkout main + merge bugfix +``` + +### 3.2 Gitflow Across Repositories + +Uses the Gitflow branching model where modules maintain both `main` and `develop` branches, with release branches created for stabilization. The manifest repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Modules tag bugfixes on release branches with standard SemVer, and the manifest's release branch references these versions. + +**Pros**: +- Well-known branching model. +- Explicit `develop` branch separates ongoing work from release stabilization. +- Release branches provide clear stabilization phases. + +**Cons**: +- Standard SemVer suffers from the same parallel release stream problem as Option 3.1 (version numbering conflicts). +- Additional overhead of maintaining separate `develop` branches across all repositories. +- More complex branching model increases coordination complexity in a polyrepo setup. +- Does not scale well with increasing module count. + +### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer + +As described in [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for released versions. Because of the earlier described limitations of SemVer, the correct module versions should not be referenced in the `MODULE.bazel` (in the manifest repository) as e.g., `1.2.3` but either by referencing the git commit hash directly or using a relaxed SemVer string, e.g. as `1.2.3-v1.0` where the string after the hyphen represents the respective S-CORE release. + +With that approach releases are possible, e.g. by creating a release branch in the manifest repository as well as in the affected module repositories. Also bugfixes of "old" releases are possible by checking out the respective release branch in the reference integration and if necessary to also create bugfixes in the affected modules. + +**Pros**: +- Single source of truth for product integration. +- Supports continuous verification. +- Provides reproducible releases. +- Scales with module count and team autonomy. +- Clear separation between development, integration, and stabilization. + +**Cons**: +- Requires explicit integration governance. + +## 4. Decision + +We decided for **Option 3.3**. + +**Rationale** + +Options 3.1 and 3.2 both violate the requirement for a versioning scheme that clearly indicates which product release a module version belongs to. Standard SemVer cannot represent parallel release streams - when a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous and conflicts arise (as demonstrated in the diagram in Option 3.1). This makes both options non-viable. + +Option 3.3 is the only viable option as it satisfies all requirements through relaxed SemVer (e.g., `1.2.3-v1.0`) where the suffix indicates the product release. Additionally, it optimizes for our goals by: +- Providing a single source of truth for product integration +- Supporting continuous verification through the manifest repository +- Scaling with module count and team autonomy +- Enabling clear separation between development, integration, and stabilization + +This approach reflects established industry practice for large-scale polyrepo systems using manifest-based integration and release trains (e.g., Android/AOSP, Chromium-style roll-ups). From 04d653dca58b5a8ec5a61b461955918c09be2547 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Tue, 13 Jan 2026 17:23:00 +0100 Subject: [PATCH 14/17] Reworked some chapters and fixed grammar --- docs/design_decisions/DR-005-infra.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index ed19d775b80..0de6fc71599 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -57,7 +57,9 @@ Options that do not satisfy these requirements are not viable and will be reject - Working on a release branch should be always possible and not harm the development on the main branch (resp. vice versa) - Working on a bugfix should always be possible (for any old release) - Module developers must know how to name their released versions -- The versioning scheme must clearly indicate which product release a module version belongs to, enabling parallel release, means essentially the ability to maintain old releases +- The versioning scheme must clearly indicate which product release a module version +belongs to, enabling parallel release, which essentially means the ability to +maintain previous releases. - Explicit stabilization phases - Continuous integration before formal releases @@ -148,7 +150,11 @@ Uses the Gitflow branching model where modules maintain both `main` and `develop As described in [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for released versions. Because of the earlier described limitations of SemVer, the correct module versions should not be referenced in the `MODULE.bazel` (in the manifest repository) as e.g., `1.2.3` but either by referencing the git commit hash directly or using a relaxed SemVer string, e.g. as `1.2.3-v1.0` where the string after the hyphen represents the respective S-CORE release. -With that approach releases are possible, e.g. by creating a release branch in the manifest repository as well as in the affected module repositories. Also bugfixes of "old" releases are possible by checking out the respective release branch in the reference integration and if necessary to also create bugfixes in the affected modules. +With that approach releases are possible, e.g. by creating a release branch in the +manifest repository as well as in the affected module repositories. Bugfixes for +previous releases are handled by checking out the corresponding release branch +in the manifest and affected modules, applying the fix, and updating the manifest +to reference the new module version. **Pros**: - Single source of truth for product integration. From fd4cdd74a190d3ef73ee7083dc4d40c66300e943 Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Wed, 14 Jan 2026 15:36:02 +0100 Subject: [PATCH 15/17] Finalized DR-005 --- docs/design_decisions/DR-005-infra.md | 35 +++++++++++++++------------ 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.md index 0de6fc71599..bf90631bb5e 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.md @@ -98,6 +98,7 @@ product versions. not feasible. - Not suitable for polyrepo setups that require parallel support of several releases, as parallel release streams are not supported by design. +- Requires explicit integration governance. > Note: The versioning and merge conflicts described in the diagram below typically arise when deviating from strict trunk-based development and introducing long-lived @@ -133,7 +134,10 @@ gitGraph ### 3.2 Gitflow Across Repositories -Uses the Gitflow branching model where modules maintain both `main` and `develop` branches, with release branches created for stabilization. The manifest repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each participating module repository creates corresponding release branches. Modules tag bugfixes on release branches with standard SemVer, and the manifest's release branch references these versions. +Uses the Gitflow branching model where modules maintain both `main` and `develop` +branches, with release branches created for stabilization. The manifest +repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each +participating module repository creates corresponding release branches. **Pros**: - Well-known branching model. @@ -141,20 +145,26 @@ Uses the Gitflow branching model where modules maintain both `main` and `develop - Release branches provide clear stabilization phases. **Cons**: +- Parallel maintenance of multiple product versions (e.g., for long-term support) is +not feasible. - Standard SemVer suffers from the same parallel release stream problem as Option 3.1 (version numbering conflicts). - Additional overhead of maintaining separate `develop` branches across all repositories. - More complex branching model increases coordination complexity in a polyrepo setup. - Does not scale well with increasing module count. +- Requires explicit integration governance. ### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer -As described in [DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for released versions. Because of the earlier described limitations of SemVer, the correct module versions should not be referenced in the `MODULE.bazel` (in the manifest repository) as e.g., `1.2.3` but either by referencing the git commit hash directly or using a relaxed SemVer string, e.g. as `1.2.3-v1.0` where the string after the hyphen represents the respective S-CORE release. +As described in +[DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is +a dedicated manifest repository containing the "known goods sets". There is a known +good set for the latest version, but also known good sets for released versions. +The known good set specifies the exact version of each module. Additionally, we store +the hash to ensure the integrity of the version. -With that approach releases are possible, e.g. by creating a release branch in the -manifest repository as well as in the affected module repositories. Bugfixes for -previous releases are handled by checking out the corresponding release branch -in the manifest and affected modules, applying the fix, and updating the manifest -to reference the new module version. +Bugfixes for previous releases are handled by checking out the corresponding commits +in the manifest and the affected modules, applying the fix, and updating the +manifest to reference the new module version. **Pros**: - Single source of truth for product integration. @@ -172,12 +182,7 @@ We decided for **Option 3.3**. **Rationale** -Options 3.1 and 3.2 both violate the requirement for a versioning scheme that clearly indicates which product release a module version belongs to. Standard SemVer cannot represent parallel release streams - when a bugfix is needed on a release branch while `main` has advanced, version numbering becomes ambiguous and conflicts arise (as demonstrated in the diagram in Option 3.1). This makes both options non-viable. - -Option 3.3 is the only viable option as it satisfies all requirements through relaxed SemVer (e.g., `1.2.3-v1.0`) where the suffix indicates the product release. Additionally, it optimizes for our goals by: -- Providing a single source of truth for product integration -- Supporting continuous verification through the manifest repository -- Scaling with module count and team autonomy -- Enabling clear separation between development, integration, and stabilization +Options 3.1 and 3.2 are not feasable for parallel maintenance of multiple product +versions. -This approach reflects established industry practice for large-scale polyrepo systems using manifest-based integration and release trains (e.g., Android/AOSP, Chromium-style roll-ups). +Option 3.3 is the only viable option as it satisfies all requirements. From 60446619068e03d72fe4a1da9351161992998aaf Mon Sep 17 00:00:00 2001 From: "Emrich Oliver (ETAS)" Date: Wed, 14 Jan 2026 15:52:11 +0100 Subject: [PATCH 16/17] Converted md to rst --- .../{DR-005-infra.md => DR-005-infra.rst} | 162 ++++++++++-------- 1 file changed, 93 insertions(+), 69 deletions(-) rename docs/design_decisions/{DR-005-infra.md => DR-005-infra.rst} (60%) diff --git a/docs/design_decisions/DR-005-infra.md b/docs/design_decisions/DR-005-infra.rst similarity index 60% rename from docs/design_decisions/DR-005-infra.md rename to docs/design_decisions/DR-005-infra.rst index bf90631bb5e..51a0240bc65 100644 --- a/docs/design_decisions/DR-005-infra.md +++ b/docs/design_decisions/DR-005-infra.rst @@ -1,32 +1,38 @@ - + SPDX-License-Identifier: Apache-2.0 -# DR-005-Infra: Development, Release and Bugfix workflows -- **Status:** Proposed -- **Owner:** Infrastructure Community -- **Date:** 2026-01-08 +DR-005-Infra: Development, Release and Bugfix workflows +======================================================= ---- +.. dec_rec:: Development, Release and Bugfix workflows + :id: dec_rec__infra__dev_release_bugfix + :status: accepted + :context: Infrastructure + :decision: Polyrepo release process with manifest repository -## 1. Context + Date: 2026-01-08 + + +Context +------- This project consists of multiple independently developed modules stored in separate repositories (polyrepo setup). This ADR builds on -[DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md), which +`DR-002 (Integration Testing in a Distributed Monolith) <./DR-002-infra.md>`_, which establishes: + - the polyrepo structure, - centralized responsibility for cross-repository integration, - and infrastructure-owned integration tooling and processes. @@ -36,19 +42,21 @@ from independently developed and versioned modules, while allowing continuous integration to reduce the gap between development and release. Commonly referenced workflows do not fully address this problem space: + - **Trunk-Based Development** with standard SemVer cannot represent parallel -release streams needed for supporting multiple product versions simultaneously. + release streams needed for supporting multiple product versions simultaneously. - **Gitflow** adds branching complexity without solving the versioning problem -for parallel releases in a polyrepo setup. + for parallel releases in a polyrepo setup. This ADR evaluates branching and versioning strategies explicitly designed for **polyrepo systems with coordinated integration releases and continuous verification**. ---- -## 2. Requirements and Goals +Requirements and Goals +---------------------- -### 2.1 Requirements +Requirements +^^^^^^^^^^^^ Options that do not satisfy these requirements are not viable and will be rejected: @@ -58,12 +66,13 @@ Options that do not satisfy these requirements are not viable and will be reject - Working on a bugfix should always be possible (for any old release) - Module developers must know how to name their released versions - The versioning scheme must clearly indicate which product release a module version -belongs to, enabling parallel release, which essentially means the ability to -maintain previous releases. + belongs to, enabling parallel release, which essentially means the ability to + maintain previous releases. - Explicit stabilization phases - Continuous integration before formal releases -### 2.2 Optimization Goals +Optimization Goals +^^^^^^^^^^^^^^^^^^ Among viable options, we optimize for: @@ -71,11 +80,12 @@ Among viable options, we optimize for: - A process that scales across many repositories and teams - Minimal integration governance to coordinate releases across repositories ---- -## 3. Options Considered +Options Considered +------------------ -### 3.1 Trunk-Based Development +Option 3.1: Trunk-Based Development +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Trunk-based development is a branching model where all developers work on a single main branch ("trunk"), integrating changes frequently—ideally daily. Feature and bugfix @@ -88,75 +98,84 @@ branch in rare cases where a hotfix for an older release is required. Versioning is linear, and the model does not support parallel maintenance of multiple product versions. -**Pros**: +**Pros** + - Very simple and lightweight branch structure. - Maximizes integration and fast feedback through continuous integration. - Minimizes merge conflicts and versioning issues. -**Cons**: +**Cons** + - Parallel maintenance of multiple product versions (e.g., for long-term support) is -not feasible. + not feasible. - Not suitable for polyrepo setups that require parallel support of several releases, -as parallel release streams are not supported by design. + as parallel release streams are not supported by design. - Requires explicit integration governance. -> Note: The versioning and merge conflicts described in the diagram below typically -arise when deviating from strict trunk-based development and introducing long-lived -release branches. This is no longer considered state-of-the-art trunk-based development. +.. note:: + The versioning and merge conflicts described in the diagram below typically + arise when deviating from strict trunk-based development and introducing long-lived + release branches. This is no longer considered state-of-the-art trunk-based development. + +.. code-block:: none -```{mermaid} -gitGraph - commit id: "1.2.3" + gitGraph + commit id: "1.2.3" - branch release - checkout release - commit id: "1.2.3-v1.0" + branch release + checkout release + commit id: "1.2.3-v1.0" - checkout main - branch bugfix - checkout main + checkout main + branch bugfix + checkout main - commit - commit id: "1.2.4" - commit - commit id: "1.2.6" + commit + commit id: "1.2.4" + commit + commit id: "1.2.6" - checkout bugfix - commit id: "bugfix-work" + checkout bugfix + commit id: "bugfix-work" - checkout release - merge bugfix - commit id: "1.2.5" + checkout release + merge bugfix + commit id: "1.2.5" - checkout main - merge bugfix -``` + checkout main + merge bugfix -### 3.2 Gitflow Across Repositories -Uses the Gitflow branching model where modules maintain both `main` and `develop` +Option 3.2: Gitflow Across Repositories +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Uses the Gitflow branching model where modules maintain both ``main`` and ``develop`` branches, with release branches created for stabilization. The manifest -repository (per DR-002) creates release branches (e.g., `release/v1.0`), and each +repository (per DR-002) creates release branches (e.g., ``release/v1.0``), and each participating module repository creates corresponding release branches. -**Pros**: +**Pros** + - Well-known branching model. -- Explicit `develop` branch separates ongoing work from release stabilization. +- Explicit ``develop`` branch separates ongoing work from release stabilization. - Release branches provide clear stabilization phases. -**Cons**: +**Cons** + - Parallel maintenance of multiple product versions (e.g., for long-term support) is -not feasible. + not feasible. - Standard SemVer suffers from the same parallel release stream problem as Option 3.1 (version numbering conflicts). -- Additional overhead of maintaining separate `develop` branches across all repositories. +- Additional overhead of maintaining separate ``develop`` branches across all repositories. - More complex branching model increases coordination complexity in a polyrepo setup. - Does not scale well with increasing module count. - Requires explicit integration governance. -### 3.3 Polyrepo Release Process with Manifest Repository and relaxed version of SemVer + +Option 3.3: Polyrepo Release Process with Manifest Repository and relaxed version of SemVer +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As described in -[DR-002 (Integration Testing in a Distributed Monolith)](./DR-002-infra.md) there is +`DR-002 (Integration Testing in a Distributed Monolith) <./DR-002-infra.md>`_ there is a dedicated manifest repository containing the "known goods sets". There is a known good set for the latest version, but also known good sets for released versions. The known good set specifies the exact version of each module. Additionally, we store @@ -166,21 +185,26 @@ Bugfixes for previous releases are handled by checking out the corresponding com in the manifest and the affected modules, applying the fix, and updating the manifest to reference the new module version. -**Pros**: +**Pros** + - Single source of truth for product integration. - Supports continuous verification. - Provides reproducible releases. - Scales with module count and team autonomy. - Clear separation between development, integration, and stabilization. -**Cons**: +**Cons** + - Requires explicit integration governance. -## 4. Decision + +Decision +-------- We decided for **Option 3.3**. -**Rationale** +Rationale +^^^^^^^^^ Options 3.1 and 3.2 are not feasable for parallel maintenance of multiple product versions. From ae6224d1894ee82a8ea0ae9143b9efaecbf55d8c Mon Sep 17 00:00:00 2001 From: Oliver Emrich Date: Fri, 16 Jan 2026 07:48:02 +0100 Subject: [PATCH 17/17] Update docs/design_decisions/DR-005-infra.rst Co-authored-by: Alexander Lanin Signed-off-by: Oliver Emrich --- docs/design_decisions/DR-005-infra.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_decisions/DR-005-infra.rst b/docs/design_decisions/DR-005-infra.rst index 51a0240bc65..d259ab5347d 100644 --- a/docs/design_decisions/DR-005-infra.rst +++ b/docs/design_decisions/DR-005-infra.rst @@ -117,7 +117,7 @@ product versions. arise when deviating from strict trunk-based development and introducing long-lived release branches. This is no longer considered state-of-the-art trunk-based development. -.. code-block:: none +.. mermaid:: gitGraph commit id: "1.2.3"