-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Executable path in configuration branches #133
Comments
Would this mean that there will be no way to determine which exe was used for a specific commit of a configuration? |
If you are simply looking at the configuration git repo, no, not directly. You would still know it was a version of OM3 installed in the COSIMA Spack instance in the |
As always ... I think this is a key step that I don't want us to be skipping! :) I also have the same questions as Dougie ... how do you know which executable works with which configuration when new options are added (or if we move to a different major release one day). Especially when we start keeping and analysing longer model runs etc. |
Sorry, maybe this was not very clear, but this would only be for development. When doing a release of a configuration one would update the OM3 exec path and payu manifest to point to a released version of OM3. Therefore, from the point of view of the users, it would always be clear what version of OM3 is required to run a given configuration. Otherwise, the assumption would always be that the tip of configuration branches should work with the tip of the OM3 main branch. If it does not work, then it needs to be fixed ASAP anyway. |
PS:
I would say one should never use the development version of OM3 or the configurations for production, only properly released versions. |
Relevant: ACCESS-NRI/model-config-tests#10 |
@dougiesquire Thanks for linking the issues. That actually helps in clarifying things. In particular, there is a key difference between what is discussed here and what is discussed in ACCESS-NRI/model-config-tests#10 Each release goes to its own spack environment and these are (hopefully) never removed/changed, while there is only one development spack environment (currently called That means that in the released configurations, the path to the environment where OM3 is installed must be specified somehow, either through the modules, through the exec path, or both. What I suggested some time ago and @aidanheerdegen is now pushing forward is to only specify it through the modules. On the other hand, when using the spack development environment, one cannot use the modules, as these do not change when updating OM3, only when updating spack itself. This can still be done with the exec path, as that currently contains a unique spack generated hash that changes whenever changes are done to the spack environment. What I'm suggesting in this issue here is to actually never bother distinguishing the OM3 executables used in the development versions of the configurations and keep the exec path fixed. |
This seems a key capability to have - otherwise an exe update could break development configs without us realising it, right? To avoid that we'd still need to check every config branch somehow, so it doesn't seem any less onerous to me. Should we prioritise setting up automated cherry-picking (#119) instead? That would help automate checking that the configs all work with the new exe. |
Summary of recent meeting where this issue was discussed: There are basically 4 options for the exec path:
Option 2. was the favoured one, in order to ensure that executables being used by developers never change silently. |
Today I spent some time to figure out how to get the git commit in the exec path, so we can have multiple development versions installed simultaneously, and the simplest is to provide spack with the specific commit to be used when installing OM3. For example:
This works fine (after solving a couple of issues with submodules) and we end up with the following exec path:
A bit long, as it includes the full git hash plus a truncated spack hash, but does what we want. |
This is great - thanks @micaeljtoliveira |
LGTM, thanks @micaeljtoliveira - can this issue be closed now? |
I haven't checked if this has now been implemented in all the configs. If that's the case, then yes, this issue should be closed. |
I'll do the missing one now |
With the current setup, the configurations' executable path includes the corresponding spack hash. Although this is fine and desirable for releases (albeit not necessary, as it would be better to replace the hash by a more meaningful version number) this can be a major hurdle during development, as a PR needs to be made for all configurations to update the hash whenever we update the OM3 executables.
The current approach does have some important advantages:
On the other hand, the number of "official" configurations is going to increase in the short/medium term, making it quite cumbersome to update the executable over a dozen configurations, greatly increasing the amount of work required from the developers to keep things up-to-date. This can be mitigated once we have an automated cherry-pick mechanism in place, but it will still requite some work.
Instead, one could change the exec path to be always the same for the development branches. I did not propose this before, as initially I thought some changes in payu would be required, but in the meantime I've realised this can also be done using spack environment views. I think this approach would be fine for now, specially because frequent updates are to be expected at this stage in development. Any releases would still require the path to be updated. Later on, once development slows down, we can always revisit this choice.
@COSIMA/access-om3-developers Happy to get some feedback on this before I start creating some PRs.
The text was updated successfully, but these errors were encountered: