Add optimize mode for sync tasks

[hstct]

Required PR: pulp/pulpcore#2999

closes #564

Co-Authored-By: Quirin Pamp pamp@atix.de

This will change the default behaviour of the sync to use optimize
unless it is explicitly set to False.

[quba42]

I would adjust the structure of last_sync_details somewhat as follows:

last_sync_details:
  remote_options:
    distributions: "<value>"
    components: "<value>"
    architectures: "<value>"
    download_policy: "<value>"
  <distribution>:
    artifact_set_sha256: "<value>"
    <component>:
      <architecture>:
        artifact_set_sha256: "<value>"

Remote options belong on the top level, since they are the same for all distributions.
We also need to make sure download_policy is one of the remote options we check.
Rather than having a list of (relative_path, package_index_artifact_set_sha256) tuples we could just make the comonents and architectures nested dict keys just like the distribution. This is mostly a preference thing, and should not ultimately make much of a difference. (The relative_path, can be generated from the component and the architecture, so this contains essentially the same information).

[mdellweg]

Let me just add another thought here. Can we not add information to the repository-version itself how it was made? This would be immutable and the repository can always look at it's latest version to retrieve the thing you are implementing here.

[quba42]

Can we not add information to the repository-version itself how it was made?

It is an interesting thought, which might have other benefits as well (like being able to debug later how really old repo versions came about). If we go this route, I kind of would like some buy in from pulp_rpm or even pulpcore though. The current implementation is analogous to the one in pulp_rpm and I do like keeping plugins in sync on similar things. I will bring this up in open floor on Tuesday.

[quba42]

I created a discourse discussion and posted it in #pulp-rpm for discussing the possibility of using repo versions instead of repos: https://discourse.pulpproject.org/t/optimize-sync-and-last-sync-details-in-pulp-rpm/531

[quba42]

Most of my suggestions just reflect small style changes.
Overall this looks pretty good to me.
There are still some details where I added a reminder to double check things.

The big question is whether we want to go ahead with the repository.last_sync_details design, or if we want to rework to use something like repo_version.creation_details instead. Unfortunately it does not make sense to punt down the road by accepting the implementation we have and then "maybe adjusting it later", since we should get these model changes right from the get go.

[hstct]

The current version relies on pulp/pulpcore#2999 to be merged before the tests can pass.

[quba42]

The latest version of this PR, exclusively stores remote options on the new RepositoryVersion field from pulpcore.
The needed artifact_set_sha256 comparison values are instead queried from the content in the previous repository version (this information has always been present there).

We have successfully tested skipping entire Release files as well as individual package indices. As such this change is now mostly ready for final review.

[quba42]

I now approve these changes, but since I have been more involved in the latest version as an author, I will look for another secondary review amongst my colleagues. 😉

[quba42]

After final internal review/debate, we have decided to merge the current state of this new feature (once the tests turn green).

[quba42]

This ended up revealing a previously masked problem with our sanity checks.
We fixed it again using: #633