New compose stack on dev/compose with signing and reload enabled

[rochacbruno]

Galaxy Simplified Compose Stack

Profiles:

• aap - Run galaxy_ng for integration with Ansible Automation Platform and Resource Server
• TBD community - Run galaxy_ng for galaxy.ansible.com development
• TDB cloud - Run galaxy_ng for console.redhat.com development

Requirements

• docker compose version >=2

Usage

Pick a profile as needed and on the root of galaxy_ng repository.

Examples assumes aap as the profile, change as needed.

Build images

docker compose -f dev/compose/aap.yaml build

Run the stack

docker compose -f dev/compose/aap.yaml up
# optionally pass `-d` to release the terminal

Exec commands on the manager service

Bash

$ docker compose -f dev/compose/aap.yaml exec manager /bin/bash
bash-4.4#

Django Admin

$  docker compose -f dev/compose/aap.yaml exec manager pulpcore-manager
Type 'pulpcore-manager help <subcommand>' for help on a specific subcommand.

Available subcommands:

[app]
    add-signing-service
    analyze-publication
...

Settings

$ docker compose -f dev/compose/aap.yaml exec manager dynaconf get DATABASES | python -m json.tool
{
  "default": {
    "ENGINE": "django.db.backends.postgresql",
    "HOST": "postgres",
    "NAME": "galaxy_ng",
    "PASSWORD": "galaxy_ng",
    "PORT": 5432,
    "USER": "galaxy_ng"
  }
}

$ docker compose -f dev/compose/aap.yaml exec manager dynaconf list
CONTENT_ORIGIN<str> 'https://localhost'
CACHE_ENABLED<bool> False
CACHE_SETTINGS<dict> {'EXPIRES_TTL': 600}
ALLOWED_CONTENT_CHECKSUMS<list> ['sha224', 'sha256', 'sha384', 'sha512']
...

Reload

Changing .py and .yaml files on any of the DEV_SOURCE_PATH directories will trigger reload of api, worker, and content services.

---

This PR

• All compose related files on dev/compose folder
• Each folder on dev/compose has its own README.md explaining the files
• Enabled hot reload when source changes

Signing:

• add GPG config
• Run all containers as root (for editable installs)
• add signing scripts
• add signing keys
• add signing-service
• add repo publickey to staging and published

Extra:

• Schedule Resource Sync Task
• Fix set-repo-keyring to accept a path to a keyfile instead of a keyring

[rochacbruno]

This is kept for backwards compatibility only, keyring is no more required, repo now has gpgkey as a plain text instead of a reference to a keyring.

[cutwater]

I'd move large multiline bash scripts into standalone files that are mounted to a container.

[rochacbruno]

@cutwater We are trying to avoid Over-Abstraction and multiple layers of indirection.

The initial version of this PR had scripts encoded as base64 inlined in the yaml file, then on those cases we agreed to move to external script.

I think the advantage is that reading the YAML would be enough to understand at least the higher level step-by-step that is being executed to prepare the environment.

A scenario where I see this being helpful is when hacking/debugging, and I want to hook some other calls or experiment a different parameter for example on a management command.

I can just copy/paste the yaml file as aap_hacking.yaml and do my changes with little changes to sparse bash scripts that are nested in a way we can´t easily tell or explain.

We had this experience before on the old ./compose environment, on pplugin_template and again on oci-env

The goal is that reading the aap.yaml only, anyone can understand the step-by-step and manipulate in the way it is needed to develop and debug.

I think we can encapsulate some atomic actions as scripts, but reading the yaml needs to make it obvious what is happening.

Also, I see advantage on moving some things to a dev Dockerfile, but we will discuss it further if needed, we are doing baby-step improvements to avoid over-engineering.

[rochacbruno]

I think the rule of thumb here would be to stick to the SOLID S principle.

An external script will do One Single thing only, never write scripts that performs nested calls to other scripts.

Multiple scripts must be chained explicitly on the compose yaml and with meaningful name that describes what the script is doing.

A refactor that would work but still keep the visibility would be:

bash -c "
   /src/galaxy_ng/dev/compose/bin/wait /etc/pulp/certs/database_fields.symmetric.key
   /src/galaxy_ng/dev/compose/bin/wait /etc/pulp/certs/signing-public.key
   /src/galaxy_ng/dev/compose/bin/run_migrations
"

What we want to avoid is:

command: magic_entrypoint.sh

[jctanner]

I agree ... let's not give someone a target to over-engineer and push us towards re-creating oci-env all over again.