[Meta] Update Prebuilt Detection Rules Release Process

[terrancedejesus]

Parent Epic (If Applicable)

Meta Summary

Importance: Reduce time-to-release by fixing tools and processes for the release workflow that do not currently work. A list of these has been included below, along with a summary of what needs to be resolved.

• Release Fleet Workflow (release-fleet.yml) - Needs to be reviewed for reliability. Does not handle exceptions for branches such as 8.10. Needs arguments to target backport branches in integrations.
• Release Docs Workflow (release-docs.yml) - Needs to be reviewed for reliability. Code logic should take into account ASCII doc URL formatting or broken links, deprecated rule handling and new minor release pages with main as base.
• Updated release docs

Estimated Time to Complete

~2 weeks

Potential Blockers

No potential blockers for this meta.

Tasklist

Another tasklist will be created as each bug is identified and analyzed.

Meta Tasks

Preview Give feedback

1. Provide Week 1 Update Comment
2. Provide Week 2 Update or Closeout Comment
3. [Bug] Update Prebuilt Detection Rules Release Process #3403
   6 of 6
   backport: auto bug fleet-release python +4
   
4. Address any issues with updated workflow AFTER new lock versions
5. [Bug] Adjust build-release CLI and fix links when generating security docs #3434
   backport: auto documentation fleet-release python +4
   

[terrancedejesus]

Review release-fleet workflow for updates and bugs

The release fleet workflow needs to be reviewed. There were changes to how we push package updates to the Integrations repository that may need to be addressed in this workflow. We may also be able to simplify this process by automatically obtaining the last locked version commit hash for each branch, rather than relying on the workflow input.

An extract-commit-hash job can be added to this workflow in place of the check-commit job. The original job relied on the engineer to find the SHA commit hash from the lock versions workflow that commits to each protected branch when version lock changes are merged into main. Since our release-fleet workflow runs from each branch, it should be able to use git log --grep to filter for that commit and choose the latest, then carry this forward in the next job. We then need to make adjustments in the fleet-pr job to reconcile whats carried from the previous.

    extract-commit-hash:
      name: Extract Commit Hash
      runs-on: ubuntu-latest
      outputs:
        commit_hash: ${{ steps.extract_commit_hash.outputs.commit_hash }}
      steps:
        - name: Checkout detection-rules
          uses: actions/checkout@v3
          with:
            path: detection-rules
            fetch-depth: 0

        - name: Extract commit hash
          id: extract_commit_hash
          run: |
            cd detection-rules
            COMMIT_HASH=$(git log --grep="Lock versions for releases" -1 --format="%H")
            echo "COMMIT_HASH=$COMMIT_HASH" >> $GITHUB_ENV
          shell: bash

Target branch of Integrations repository
When we build a package from each respective branch, we used to create a PR to main with these changes for the security_detection_engine package. However, recently we now use the backporting strategy and now push a package to each backport-security_detection_engine-x.x branch that mirrors where we are releasing from. Since the branches are pre-determined and entered as input, the process does not break.

bump-pkg-versions CLI command
This CLI command needed to be revisited as some bugs were introduced when we added a RegistryPackageManifestV3 schema. This schema reflects how registry_data is suppose to be formatted in the packages.yml. Since bump-pkg-versions is responsible for finding the next package version that should be used and auto-updates the packages.yml file, the code assumed some fields would always exist from the RegistryPackageManifestV1 schema that was not true anymore.

As a result, the logic was adjusted a bit to only add release if its found in the loaded package. If not, it skips this step.

@dev_group.command("bump-pkg-versions")
@click.option("--major-release", is_flag=True, help="bump the major version")
@click.option("--minor-release", is_flag=True, help="bump the minor version")
@click.option("--patch-release", is_flag=True, help="bump the patch version")
@click.option("--new-package", type=click.Choice(['true', 'false']), help="indicates new package")
@click.option("--maturity", type=click.Choice(['beta', 'ga'], case_sensitive=False),
              required=True, help="beta or production versions")
def bump_versions(major_release: bool, minor_release: bool, patch_release: bool, new_package: str, maturity: str):
    """Bump the versions"""

    pkg_data = load_etc_dump('packages.yml')['package']
    kibana_ver = Version.parse(pkg_data['name'], optional_minor_and_patch=True)
    pkg_ver = Version.parse(pkg_data["registry_data"]["version"])
    pkg_kibana_ver = Version.parse(pkg_data["registry_data"]["conditions"]["kibana.version"].lstrip("^"))
    if major_release:
        major_bump = kibana_ver.bump_major()
        pkg_data["name"] = f"{major_bump.major}.{major_bump.minor}"
        pkg_data["registry_data"]["conditions"]["kibana.version"] = f"^{pkg_kibana_ver.bump_major()}"
        pkg_data["registry_data"]["version"] = str(pkg_ver.bump_major().bump_prerelease("beta"))
    if minor_release:
        minor_bump = kibana_ver.bump_minor()
        pkg_data["name"] = f"{minor_bump.major}.{minor_bump.minor}"
        pkg_data["registry_data"]["conditions"]["kibana.version"] = f"^{pkg_kibana_ver.bump_minor()}"
        pkg_data["registry_data"]["version"] = str(pkg_ver.bump_minor().bump_prerelease("beta"))
    if patch_release:
        latest_patch_release_ver = find_latest_integration_version("security_detection_engine",
                                                                   maturity, pkg_kibana_ver)

        # if an existing minor or major does not have a package, bump from the last
        # example is 8.10.0-beta.1 is last, but on 9.0.0 major
        # example is 8.10.0-beta.1 is last, but on 8.11.0 minor
        if latest_patch_release_ver.minor != pkg_kibana_ver.minor:
            latest_patch_release_ver = latest_patch_release_ver.bump_minor()
        if latest_patch_release_ver.major != pkg_kibana_ver.major:
            latest_patch_release_ver = latest_patch_release_ver.bump_major()

        if maturity == "ga":
            pkg_data["registry_data"]["version"] = str(latest_patch_release_ver.bump_patch())
        else:
            # passing in true or false from GH actions; not using eval() for security purposes
            if new_package == "true":
                latest_patch_release_ver = latest_patch_release_ver.bump_patch()
            pkg_data["registry_data"]["version"] = str(latest_patch_release_ver.bump_prerelease("beta"))

        if 'release' in pkg_data['registry_data']:
            pkg_data['registry_data']['release'] = maturity

    click.echo(f"Kibana version: {pkg_data['name']}")
    click.echo(f"Package Kibana version: {pkg_data['registry_data']['conditions']['kibana.version']}")
    click.echo(f"Package version: {pkg_data['registry_data']['version']}")

    save_etc_dump({"package": pkg_data}, "packages.yml")

Adjust fleet package building to add historical rules by default
We are only releasing packages for current-3. At the time of this comment, that is 8.9-8.12. We introduced adding historical rules to the package in 8.7 and thus had to set a flag for this so avoid issues with backports and when releasing from, for example, 8.7-8.10 where historical rules were required for some packages and not required for others.

Since we are past this window, there is not longer a need to have this flag set to "no" by default and rather it can be set to "yes". Optionally, we could remove it entirely and have the default always add historical rules. Note that I tested the build on 8.3 branch with this flag set to "yes" and the build package was fine.

Update bump-pkg-versions command to address schema failures*
With 8.12, our security_detection_engine package will be on elastic package version 3.0. As a result we added a RegistryPackageManifestV3 which is used to load the packages.yml registry_data data through to verify it is valid. With the changes from version 1 to version 3, the release key is no longer valid, however, we were still assigning this in the bump-pkg-versions logic. This has been adjusted to only add this if the field exists when we load packages.yml. Remember that this file is frozen across each protected branch and thus would reflect which branch we use version 1 or version 3 for package registry

After these changes were applied, a workflow run was done from the working branch for these changes which was successful and pushed a PR to integrations as expected. Reference: elastic/integrations#8987

[terrancedejesus]

Adjust packages.yml in 8.10 branch so kibana.version is 8.10.1
Elastic stack version 8.10.0 had a severe vulnerability which caused package registry team to only allow 8.10.1 and up deployments to run for 8.10 integration packages. As a result, when we push 8.10.x rule packages to integrations for updated, the manifest (built from the registry_data in packages.yml) needs to be 8.10.1. This is specifically the conditions > kibana.version field value. By default, this is how Fleet determines which integration versions are compatible with which stack versions. We always make the patch .0 when we create a new branch. However, the 8.10 branch needs to be changed from 8.10.0 to 8.10.1. If not, the build will fail upstream in the integrations repository when built.

  registry_data:
    categories:
    - security
    conditions:
      kibana.version: ^8.13.0
      elastic:
        subscription: basic
    description: Prebuilt detection rules for Elastic Security
    format_version: 3.0.0
    icons:
    - size: 16x16
      src: /img/security-logo-color-64px.svg
      type: image/svg+xml
    source:
      license: Elastic-2.0
    name: security_detection_engine
    owner:
      github: elastic/protections
      type: elastic
    title: Prebuilt Security Detection Rules
    type: integration
    version: 8.13.0-beta.0

Additionally, an important consideration is when we add historical rules, we used to rely on name from packages.yml to help query the EPR API for historical releases. The name in each packages.yml for each branch reflects the branch stack version. If we use 8.10 to query EPR, it will not return packages pushed that are compatible with 8.10.1. Thus when doing build-release CLI command now, we pass the value of kibana.version from packages.yml instead. Once we update the branch so this value is 8.10.1 we will not need to make any adjustments in the other methods that query the EPR API.

    if add_historical:
        previous_pkg_version = find_latest_integration_version("security_detection_engine", "ga",
                                                               registry_data['conditions']['kibana.version'].strip("^"))
        sde = SecurityDetectionEngine()
        historical_rules = sde.load_integration_assets(previous_pkg_version)
        historical_rules = sde.transform_legacy_assets(historical_rules)

        docs = IntegrationSecurityDocsMDX(config['registry_data']['version'], Path(f'releases/{config["name"]}-docs'),
                                          True, historical_rules, package, note=update_message)
        docs.generate()

        click.echo(f'[+] Adding historical rules from {previous_pkg_version} package')
        package.add_historical_rules(historical_rules, config['registry_data']['version'])

To resolve this we need to checkout 8.10 branch, update the kibana version and push manually.

[terrancedejesus]

Update 01-26-2024

We have made progress with updating the Detection Rules release process, ultimately addressing bugs and exceptions that were introduced from changes upstream. Please review to the tasklist of this issue to view which has been addressed so far.

Moving forward, we should have the bugs and exceptions addressed in the linked pull request. We will need to still review #2807, make updates and merge accordingly, once done, this issue can be marked as complete.

[terrancedejesus]

Update 02-06-2024

This issue has been re-opened. It was closed due to the only linked pull request being merged, however, changes still need to occur.

• Update build-release CLI command to exclude doc generation unless explicitly flagged
• Update security doc generation to create correct ASCII doc links
• Test changes after merging in lock versions PR

The detection rules release has been started with @shashank-elastic.