Handling multi repo projects

[cmeessen]

I would like to start a discussion about how to handle multi-repo projects. Let's assume we have a framework that is organised in a GitHub/GitLab group. It contains multiple repositories, where each repo has it's own DOI and may even have different licenses. How would this be presented?

Would we need a new category "Software Frameworks", or can this be integrated into the regular Software entries?

Will there be an overview page for the framework and subpages for the individual components?

How would the commit graph be displayed (one summary graph, one graph with multiple lines for each repo)?

[jmaassen]

We also have quite a few examples of multi-repo software. Currently we tend to release / version / archive these independently, which sometimes make sense, but sometimes not. Combining the commit graph should be doable (summing the activity, or showing different lines for each repo). Tracking the versions and showing the citation info may be a bit more complicated.

A braindump of some examples that come to mind:

esmvaltool + esmvalcore: this is a framework to analyse and compare the results of climate models. esmvalcore contains the python framework used to implement basic things such as I/O, preprocessing, regridding, etc. esmvaltool adds a toolbox of analysis tools and scripts implemented in various languages and with varying goals (and of varying quality). This allows researchers to select the combination of analysis tools they need, or even add their own without having to change anything in the esmvalcore code base. As you cannot use esmvaltool without esmvalcore (and vice-versa does not make sense either), they are typically released in lock-step. The contributors list of both are different (but partly/largely overlapping). The software quality requirements for esmvalcore are higher than for esmvaltool. Currently, there is a lot of overlap in the RSD pages for both. Having a single page could be more logical, as would be merging the development activity in some way.

amuse and omuse: a similar example, amuse is a python framework use to combine existing astrophysics simulations (in various languages), omuse does the same but for ocean simulations. Other muse variants are on the drawing board, for hydrology, water management, fusion, etc. In this case it makes sense to have separate releases of muse + all its variants, as they target completely different communities. A single "framework" page for muse pointing to the others makes sense, but that can also be done with the current setup (using related software).

knime nodes: knime is an existing workflow system that it often used in life sciences. A user can add functionality by adding 'nodes' to their workflow which perform some simulation or analysis step. Often these nodes are just a bit of glue code to link knime to some external tool. We've created many such nodes, which are currently separately mentioned in the RSD. These pages are often quite minimal, as there is not much to tell. While it does make sense to independently develop and release them (they do different things), it could make sense to have a single page describing them. Merging the development graphs is okay, but I'm not sure how to handle the different DOIs in such a setup.

xenon: a java library designed to simplify job submission and data movement to remote machines. The library itself can be used directly, or combined with plug-ins that extend the functionality (such as cloud support) which are developed in separate repos. In addition, there are wrappers for the library for other languages (PyXenon), communication frameworks (xenon-grpc), a command line interface (xenon-cli), a workflow system (xenon-flow), etc., each in a different repo and with different contributors. The different repos are released mostly independently, although an update of one sometimes triggers an update of several others. Versions are typically not in sync. Some of the RSD pages are quite minimal, or have large overlap with others. In this case presenting it as a single "ecosystem" could make more sense.