Container Registry Search Order and "Org Squatting"

Why am I here?

If you ran a container that displayed the following:

 _   _  ___  _   _ _   ___ _
| | | |/ _ \| \ | | | / / | |
| |_| | | | |  \| | |/ /| | |
|  _  | | | |     |   < |_|_|
| | | | |_| | |\  | |\ \ _ _
|_| |_|\___/|_| \_|_| \_(_|_)

Not the container image you were expecting?
Relax, you have only been h0nX0r3d, not hacked.
For more information on why you are seeing this:
https://github.com/bgeesaman/registry-search-order

and you followed that link here, you might be wondering how and why that happened.

First things first: you have not been hacked, but you came very close. Your system either directly or indirectly ran this container, but I assure you, it's a benign image and only prints the message above.

What Happened?

One of a few things:

1. You searched an image on docker.io or quay.io that looked official but wasn't and used it.

2. You run RHEL/Centos/Fedora or a system with Podman installed that enables a "registry search order" capability, and you specified a shortened path like repo/image:tag . Your configuration then landed on this image first instead of the desired image in a registry later in your search order. The containers-common package on your system installed a file named /etc/containers/registries.conf which specified a configuration with something like:

   # Libpod/Podman/Fedora
   [registries.search]
   registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com', 'registry.centos.org']

   # Centos 7
   [registries.search]
   registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']

   # Centos 8/RHEL 8
   [registries.search]
   registries = ['registry.redhat.io', 'quay.io', 'docker.io']

Org-Squatting

The registry search order feature is useful, but a horrible goose might prevent you from having a beautiful day in the village if they decide to "squat" on your organization name in another registry and "mirror" images of identical names and tags to run an image of their choosing and not the one you were expecting.

And that's how you might accidentally fall into this form of watering-hole attack and have untrusted code run on your machine or inside your cluster.

Oops. Not good.

After some inspiration from @raesene here, I decided to see how practical this form of attack would be and to use this mechanism to raise awareness of the issue and now this issue. This took the form of:

1. Registering identically named organizations on quay.io that were available for common/popular organizations on docker.io/DockerHub in case quay.io comes first in your search order.
2. Registering identically named organizations that live on registry.redhat.io/registry.access.redhat.com on docker.io/DockerHub in case docker.io comes first.
3. Creating a benign image with the sole purpose of raising awareness by having it link to an informative repository README.
4. Finally, the benign image was mirrored to all the squatted orgs with identical names and tags of the images from their official locations.

Hence the horrible goose reference: HONK HONK HONK

Note: Neither House House nor Panic are in any way affiliated with this project, nor do they endorse it.

What Can I Do to Prevent This From Happening Again?

A few things:

1. If you hold an organization in docker.io but not in quay.io or vice versa, go ahead and register it with a reference back to your official registry location.
2. If your organization name embeds a version number in it like org3, be sure to register org4-org100 in both docker.io and quay.io to prevent user confusion and trust issues.
3. Configure your /etc/containers/registries.conf registries.search list to place your trusted registries first and consider removing all public registries from this list.
4. Use the fully-qualified image path when referencing images to run on the command line or in kubernetes pod specs to always ensure you pull from the correct registry. e.g. docker.io/myorg/myimage:mytag instead of myorg/myimage:mytag
5. Instead of referencing a tag like mytag, consider using the digest format: quay.io/myorg/myimage@sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 as it prevents both org-squatting and registry compromise situations.
6. Cryptographically sign all container images as they are built and validate the signature before running by configuring containers-policy.json. If using Kubernetes, run an admission controller or similar to ensure only images being run come from approved registries and/or have a trusted signature.
7. Optionally, submit bugs/feedback to your upstream distribution and politely ask them to adjust the search order.
8. If your organization was already squatted on, contact the registry provider for guidance. If you are the owner of one of the organizations where this image is currently being mirrored, open an issue in this repo and I will gladly "toss it back".

Resources

1. Untitled Goose Game - Note: Neither House House nor Panic are in any way affiliated with this project, nor do they endorse it. That said, it's a great game, and you should consider owning it.
2. Registries.conf
3. Containers-common RPM
4. @Raesene's Blog - Typosquatting in a Multi-Registry World
5. Reported Issue to Red Hat
6. Issue created by Red Hat in libpod
7. https://access.redhat.com/articles/3116561 and https://bugzilla.redhat.com/show_bug.cgi?id=1700140
8. Kubernetes Dynamic Admission Control