Incorporating reproducibility metadata (package & dependency checksums)

[dgkf]

Motivation

Given a repository of quality metrics and a package repository, how do we guarantee that the metrics would be reproducible given packages from that repository?

User Experience

From an end-user perspective (administrator, analyst, etc), a user may want to include assertions about checksum consistency as part of their filter. Assuming CRAN is a moving target, this may mean that packages that were permitted a week ago may be filtered out today. To ensure that metrics continue to reflect a repository, a snapshot of the repository would be needed.

A user may provide a filter such as:

options(available_package_filters = risk_filter(...))

Where assertions about checksum matches (either for the package source code, hard dependencies or soft dependencies) can be enforced and used as part of the filtering criteria.

Repository structure

This will mean updating the PACKAGES format to include this metadata

Package: A3
Version: 1.0.0
Depends: R (>= 2.15.0), xtable, pbapply
Suggests: randomForest, e1071
# ... additional stats ...
MD5sum: 027ebdd8affce8f0effaecfcd5f5ade2
MD5sumReqDeps: 0a1b2c3d4f5g6h
MD5sumAllDeps: 0a1b2c3d4f5g6h
# ... metrics ...

Or alternatively, we can store hashes for the dependencies used during evaluation

Package: A3
Version: 1.0.0
Depends: R (>= 2.15.0), xtable, pbapply
Suggests: randomForest, e1071
MD5sum: 027ebdd8affce8f0effaecfcd5f5ade2
MD5sum/xtable: 0a1b2c3d4f5g6h
MD5sum/pbapply: 0a1b2c3d4f5g6h
MD5sum/randomForest: 0a1b2c3d4f5g6h
MD5sum/e1071: 0a1b2c3d4f5g6h

This would have the benefit of allowing us to ignore situations where Suggests dependencies are not available to an end user or were not available during evaluation and is a bit more interpretable at the expense of file size.

Implementation

I think the tools for deriving these checksums should live with the filtering tools because it will need to be re-derived for the package repository to apply a filter. But that function should be used in these pipelines to derive the same checksums during metric derivation.

[dgkf]

Trying to carry this forward there are a couple decisions we need to make:

On which approach we want to use for metadata

1. Using only MD5sum/Required and MD5sum/All
   • we reduce the amount of information in the packages file. This would keep the amount of additional information minimal. Assuming each package has roughly 10-30 dependencies, this would cut down on about 20 lines of text per package, likely cutting the PACKAGES file size down by ~20-30% compared to the other option.
   • however, this would mean that we would need to do more work to re-derive these aggregate MD5 sums reproducibly to perform checks and it would be less transparent how they're derived or what is mismatched.
2. Having a record per MD5sum/<dependency>
   • would add more text to to the PACKAGES file size
   • would be far easier to cross-reference and determine exactly which dependencies are out-of-sync

I propose that we not try to prematurely optimize for file size and just include every dependency's MD5sum that was used during evaluation. If the filesize becomes prohibitive we can think about other options, but I think this use case warrants the added filesize.

On how we derive MD5sums

Then there's the matter of how we obtain these MD5sums. These MD5sums describe the source code tarballs themselves, which are only available at install. If we wanted to be 100% rigorous, we would re-install all dependencies from source, recording the tarballs for all of them.

This might be okay in our repository pipelines, but would be painful for desktop users who probably don't want to install all recursive dependencies for each run.

If we don't mind being a bit more lenient, we could look at the Repository: field in the installed package DESCRIPTION file, which would contain either CRAN or a repository url. With this, we could try to recover the MD5 easily in cases where the dependency is up-to-date with the version available in the repository. In cases where the dependency is outdated, we'd still have to download the .tar.gz from the /archive and then build the MD5sum.

I propose that we:

1. Permit assuming MD5sums from the repository for the latest version of packages (for users that want older versions, the assumption would be that they have a snapshot)
2. Perhaps file a feature request to pak to embed the MD5sum in the installed package DESCRIPTION, which would then allow us to recover the MD5sum for outdated packages.

---

Wishlist: it would be wonderful if installation from a tarball added the MD5sum into the installed package DESCRIPTION file. This would let us circumvent the re-discovery of the MD5sum by querying the source repository. Maybe something we can propose to pak/R core?