tracker: Rebase onto Fedora 38

[dustymabe]

Rebase to a new version of Fedora (N=38)

At previous Fedora major release

Open tickets to track related work for this release

• Fedora Changes Considerations (example): tracker: Fedora 38 changes considerations #1357
• Package Additions/Removals (example) F38: tracker for package additions/removals #1435
• Test Week (template) tracker: Fedora 38 Test Week on March 28  #1440

At Branching

Branching is when a new stream is "branched" off of rawhide. This eventually becomes the next major Fedora (N).

Release engineering changes

• Verify that a few tags were created when branching occurred:

• f${N+1}-coreos-signing-pending

• f${N+1}-coreos-continuous

• Add and tag a package (any package) which is in the stable repos into the continuous tag. This will create the initial yum repo that's used as input for building the COSA container.

• koji add-pkg --owner ${FAS_USERNAME} f${N+1}-coreos-continuous $PKG

  • example: koji add-pkg --owner dustymabe f36-coreos-continuous fedora-release
  • This example uses the fedora-release RPM, but it could be any other.

• koji tag-build f${N+1}-coreos-continuous $BUILD

  • example: koji tag-build f36-coreos-continuous fedora-release-36-0.16

• Add the N+1 signing key short hash (usually found here) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="12c944d0 9570ff31 45719a39 9867c58f"

coreos-installer changes

• Update coreos-installer to know about the signing key used for the future new major version of Fedora (N+1).
• Drop the signing key for the obsolete stable release (N-2).

Update rawhide stream

• Update manifest.yaml to list N+1 as the releasever.

Enable branched stream

• Update manifest.yaml to list N as the releasever.
• Update config.yaml to un-comment out the branched stream definition.

At Fedora (N) Beta

Update fedora-coreos-config next-devel

• Bump releasever in manifest.yaml

• Update the repos in manifest.yaml if needed

• Run cosa fetch --dry-run --update-lockfile

  • this updates the x86_64 lockfile - the others will get updated when bump-lockfile runs.
  • in the future we may support this in cosa fetch directly

• PR the result

• Re-enable next-devel if needed (docs)

• Disable branched stream since it is no longer needed.

  • Update config.yaml to comment out the branched stream definition.

Ship rebased next

• Ship next
• Set a new update barrier for the final release of N-1 on next. In the barrier entry set a link to the docs. See discussion

Preparing for Fedora (N) GA

Update fedora-coreos-config testing-devel

• Bump releasever in manifest.yaml
• Update the repos in manifest.yaml if needed
• Sync the lockfiles for all arches from next-devel
• Bump the base Fedora version in ci/buildroot/Dockerfile
• PR the result

At Fedora (N) GA

Ship rebased testing

• Ship testing
• Set a new update barrier for the final release of N-1 on testing. In the barrier entry set a link to the docs. See discussion

Disable branched stream

• Update streams.groovy to remove the branched stream in the list of mechanical refs.

Untag old packages

koji untag N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:

• Find the key short hash. Usually found here. Then:

f32key=12c944d0
key=$f32key
echo > untaglist # create or empty out file
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
    if koji buildinfo $build | grep $key 1>/dev/null; then
        echo "Adding $build to untag list"
        echo "${build}" >> untaglist
    fi
done

Now we have a list of builds to untag. But we need a few more sanity checks.

• Make sure none of the builds are used in N based FCOS. Check by running:

f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -B 9 $key
podman rmi quay.io/fedora/fedora-coreos:testing-devel

If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed.

• For any RPMS still used by N-1 based FCOS let's remove them from the untaglist. Check by running:

f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:stable rpm -qai | grep -B 9 $key
podman rmi quay.io/fedora/fedora-coreos:stable

NOTE: This assumes stable is still on N-1.

Remove any entries from the untaglist file that are still being used.

• After verifying the list looks good, untag:

# use xargs so we don't exhaust bash string limit
cat untaglist | xargs -L50 koji untag-build coreos-pool

• Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

• Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"

Disable next-devel stream

We prefer to disable next-devel when there is no difference between testing-devel and next-devel. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N if next-devel and testing-devel are in lockstep, then disable next-devel.

• Follow the instructions here to disable next-devel

Switch upstream packages to shipping release binaries from Fedora (N)

• Update repo-templates config.yaml with the version number and GPG key ID for Fedora (N).

After Fedora (N) GA

Ship rebased stable

• Ship stable
• Set a new update barrier for the final release of N-1 on stable. In the barrier entry set a link to the docs. See discussion

Open ticket for the next Fedora rebase

• Create a new ticket from the rebase template
  • label with FN label where N is the Fedora version.

Miscellaneous container updates

These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment.

• Update coreos-assembler or open ticket to update:
  • Dockerfile
• Update coreos-installer - Dockerfile: update to Fedora 38 coreos-installer#1167
  • Dockerfile
• Update Ignition - Dockerfile.validate: update to Fedora 38 ignition#1611
  • Dockerfile.validate
• Update Butane - Dockerfile: update to Fedora 38 butane#459
  • Dockerfile
• Update fedora-coreos-cincinnati
  • Dockerfile
• Update config-bot
  • Dockerfile
• Update coreos-koji-tagger
  • Dockerfile
  • ImageStream
  • BuildConfig
• Update coreos-ostree-importer
  • Dockerfile
  • ImageStream
  • BuildConfig

[dustymabe]

• tracker: Fedora 38 changes considerations #1357

[jlebon]

When updating testing-devel to f38, remember to drop these entries from kola-denylist.yaml: https://github.com/coreos/fedora-coreos-config/blob/1a37a6ec05e7f611b5d4acf343c35ea51bc4d35a/kola-denylist.yaml#L20-L39

[aaradhak]

• add Fedora 39 signing key to coreos-pool tag

[aaradhak]

• signing-keys: add Fedora 39 key coreos-installer#1113

[aaradhak]

• [rawhide] manifest: rawhide is now F39 fedora-coreos-config#2246

[aaradhak]

• config: enable branched stream fedora-coreos-pipeline#817
• [branched] manifest: bump to F38 fedora-coreos-config#2251

[aaradhak]

• [next-devel] Move to Fedora Linux 38 fedora-coreos-config#2254

[aaradhak]

• config: re-enable next-devel fedora-coreos-pipeline#830

[dustymabe]

In general you may see us fast-tracking packages in next-devel before they reach bodhi stable. This is to prevent downgrades (newer packages in f37 than in f38, due to the freeze).

[jlebon]

coreos-status email: https://hackmd.io/aw7c1xNLRNSxcomW63Carw

[dustymabe]

The fix for this went into next stream release 38.20230322.1.0. Please try out the new release and report issues.

[dustymabe]

• Set a new update barrier for the final release of N-1 on next. In the barrier entry set a link to the docs. See discussion

The last release of f37 is a barrier, but the link won't be to the signing key issue because there was another reason to do it that way.

[dustymabe]

Fedora 38 is a Go for Tuesday April 18th.

The only package that we need that isn't currently in next-devel is fuse3-3.13.1-2.fc38 (part of this push) so I'll fast-track that to get a next-devel build sooner.

[dustymabe]

• Move to Fedora Linux 38 fedora-coreos-config#2376

[dustymabe]

• config.yaml: disable next-devel fedora-coreos-pipeline#858

[dustymabe]

The fix for this went into testing stream release 38.20230414.2.0. Please try out the new release and report issues.

[dustymabe]

For the "Set a new update barrier for the final release of N-1 on $STREAM" we'll let the update barriers that we put in place for #1441 suffice here.

• updates: make 37.20230303.1.1 a barrier release fedora-coreos-streams#671
• updates: make 37.20230322.2.0 a barrier release fedora-coreos-streams#676
• Roll out stable 37.20230322.3.0, testing 37.20230401.2.0, next 38.20230402.1.0 fedora-coreos-streams#682

[dustymabe]

stable was switched over in: coreos/fedora-coreos-config@4524c7d

[dustymabe]

• config.yaml: bump current fedora to Fedora 38 repo-templates#110

[dustymabe]

The fix for this went into stable stream release 38.20230414.3.0.

[dustymabe]

• Bump to Fedora 38 coreos-assembler#3451

[dustymabe]

f36 based packages have been untagged from the koji pool and the f36 key has been removed using koji edit-tag coreos-pool -x tag2distrepo.keys="5323552a eb10b464 18B8e74c".

[jlebon]

The only steps remaining in this one are the container updates, which have now been superseded by Fedora 39. Tracked at #1490.