Reenable incremental builds #162

james-bruten-mo · 2026-01-20T14:50:27Z

PR Summary

Sci/Tech Reviewer: @MatthewHambley
Code Reviewer: @t00sa

At the git migration incremental builds on the command line no longer worked as timestamps in github are based on clone date, not commit date. This reenables this functionality by fetching/pulling changes if the build is being rerun. This has been tested on the command line for both a run with physics code (lfric_atm) and without (gungho_model). The 2nd build time is much quicker as expected.

This also enables building using the local github mirrors which will be useful once these are also available on the HPC. Again, these have been tested on the cli and with incremental builds.

This change adds get_git_sources which is mostly a copy of the same file from SimSys_Scripts. Having 2 copies isn't ideal, but I can't think of another way to make that SimSys_Scripts file available when building locally (short of installing it as a library). Hopefully this is something that'll be improved by fab!

Code Quality Checklist

I have performed a self-review of my own code
My code follows the project's style guidelines
Comments have been included that aid understanding and enhance the readability of the code
My changes generate no new warnings
All automated checks in the CI pipeline have completed successfully

Testing

I have tested this change locally, using the LFRic Core rose-stem suite
If required (e.g. API changes) I have also run the LFRic Apps test suite using this branch
If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - test_incremental_builds_change/run2

Suite Information

Item	Value
Suite Name	test_incremental_builds_change/run2
Suite User	james.bruten
Workflow Start	2026-01-20T15:03:56
Groups Run	developer

Dependency	Reference	Main Like
casim	MetOffice/casim@2025.12.1	True
jules	MetOffice/jules@2025.12.1	True
lfric_apps	james-bruten-mo/lfric_apps@improve_local_builds	False
lfric_core	MetOffice/lfric_core@aa32824	True
moci	MetOffice/moci@2025.12.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2025.12.1	True
socrates	MetOffice/socrates@2025.12.1	True
socrates-spectral	MetOffice/socrates-spectral@2025.12.1	True
ukca	MetOffice/ukca@2025.12.1	True

Task Information

✅ succeeded tasks - 1106

Security Considerations

I have reviewed my changes for potential security issues
Sensitive data is properly handled (if applicable)
Authentication and authorisation are properly implemented (if applicable)

Performance Impact

Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

I understand this area of code and the changes being added
The proposed changes correspond to the pull request description
Documentation is sufficient (do documentation papers need updating)
Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

All dependencies have been resolved
Related Issues have been properly linked and addressed
CLA compliance has been confirmed
Code quality standards have been met
Tests are adequate and have passed
Documentation is complete and accurate
Security considerations have been addressed
Performance impact is acceptable

MatthewHambley

Some improvements possible to structure and code. Also some lack of clarity regarding why certain things are done in certain ways.

MatthewHambley · 2026-01-21T13:24:37Z

build/extract/extract_science.py

    """

-    tempdir = Path(tempfile.mkdtemp())
+    use_mirrors: bool = (os.getenv('LOCAL_BUILD_MIRRORS', 'False') == 'True')


For something like this it might be best to test merely on the presence of the variable. This avoids the game of guess the magic value.

Changed to use presence of the path

MatthewHambley · 2026-01-21T13:25:50Z

build/extract/extract_science.py


-    tempdir = Path(tempfile.mkdtemp())
+    use_mirrors: bool = (os.getenv('LOCAL_BUILD_MIRRORS', 'False') == 'True')
+    mirror_loc: Path = os.getenv("MIRROR_LOC", "")


Or, in fact, the presence of the variable could signal the requirement while it's content is the location. And how about whole words while we're at it? MIRROR_LOCATION

Sure, happy to use that here

MatthewHambley · 2026-01-21T13:26:21Z

build/extract/extract_science.py

+    if not mirror_loc and use_mirrors:
+        raise KeyError(
+            "Use Mirrors is set true, but the MIRROR_LOC environment variable hasn't"
+            "been set"
+        )
+


Given the above, there is no longer a need for this test.

Rep, removed

MatthewHambley · 2026-01-21T13:28:27Z

build/extract/get_git_sources.py

+# which you should have received as part of this distribution.
+# *****************************COPYRIGHT*******************************
+"""
+Clone sources for a rose-stem run for use with git bdiff module in scripts


Does it not also clone them for interactive builds? More generally I don't see any explanation as to why this complicated thing needs to be done.

A copy and paste error, changed

MatthewHambley · 2026-01-21T14:31:49Z

build/extract/get_git_sources.py

+) -> None:
+
+    if ".git" in source:
+        if use_mirrors:


For a future change, it may be possible to use a class to wrap repository access. That way the test can be performed once, the correct object instantiated and no further concerns had about which one it is.

MatthewHambley · 2026-01-21T15:47:51Z

build/extract/get_git_sources.py

+        commands = (
+            f"git -C {loc} init",
+            f"git -C {loc} remote add origin {repo_source}",
+            f"git -C {loc} fetch origin {repo_ref}",
+            f"git -C {loc} checkout FETCH_HEAD",
+            f"git -C {loc} fetch origin main:main",
+        )


This seems a lot of effort. Doesn't it just recreate a clone?

Yes, it was initially written to see if we could save space while cloning sources for use in rose-stem. It turns out the difference is very small, but as this was marginally smaller, I left it in. Given it's copied from SimSys_Scripts I don't think it's worth modifying here

MatthewHambley · 2026-01-21T15:51:10Z

build/extract/get_git_sources.py

+    # Fetch the main branch from origin
+    # Ignore errors - these are likely because the main branch already exists
+    # Instead write them as warnings
+    command = f"git -C {loc} fetch origin main:main"


If you need to treat the copy as a repository, why not clone it. That's what distributed repositories are all about. You might take advantage of a sparse clone to gain the advantage of filtering you are getting from the rsync exclusion list.

MatthewHambley · 2026-01-21T15:53:48Z

build/local_build.py

    """

-    return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    return Path(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))


From memory:

Suggested change

return Path(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

Path(__file__).absolute.parent.parent

Ta - absolute is a function but otherwise good

MatthewHambley · 2026-01-21T16:00:00Z

build/local_build.py

+        args.working_dir = Path(project_path) / "working"
    else:
        # If the working dir doesn't end in working, set that here
        if not args.working_dir.strip("/").endswith("working"):


This test is easier to do when working with a Path object. working_dir.name == 'working'. You can use type-Path with add_argument() to have argparse give it the correct type and reject anything which wont parse as a path.

Thanks, done

MatthewHambley · 2026-01-21T16:03:25Z

documentation/source/developer_guide/local_builds.rst

+| ``-m --mirrors``     | False                       | If True, this will attempt  |
+| ``store_true``       |                             | to extract using local      |
+|                      |                             | github mirrors              |
+----------------------+-----------------------------+-----------------------------+
+| ``--mirror-loc``     | MetOffice Mirror Location   | The path to the github      |
+|                      |                             | mirror location             |


As previously, why not collapse these into a single argument for mirror location? The need to use mirrors determined by the prevision of a mirror location to use.

For the environment variables passed to extract_science.py then yes I agree, we can use the presence of the mirror location variable to show whether we're using the mirrors.

For local_build.py (which these docs are referring to) I think we want the --mirrors store_true argument as otherwise we'll need to always pass the location of the mirrors as an argument when using them, which isn't ideal

…mo/lfric_apps into improve_local_builds

james-bruten-mo

Thanks Matthew - those comments are addressed, and for the most part changes done

james-bruten-mo · 2026-01-21T16:17:21Z

build/extract/extract_science.py

    """

-    tempdir = Path(tempfile.mkdtemp())
+    use_mirrors: bool = (os.getenv('LOCAL_BUILD_MIRRORS', 'False') == 'True')


Changed to use presence of the path

james-bruten-mo · 2026-01-21T16:17:46Z

build/extract/extract_science.py


-    tempdir = Path(tempfile.mkdtemp())
+    use_mirrors: bool = (os.getenv('LOCAL_BUILD_MIRRORS', 'False') == 'True')
+    mirror_loc: Path = os.getenv("MIRROR_LOC", "")


Sure, happy to use that here

james-bruten-mo · 2026-01-21T16:17:59Z

build/extract/extract_science.py

+    if not mirror_loc and use_mirrors:
+        raise KeyError(
+            "Use Mirrors is set true, but the MIRROR_LOC environment variable hasn't"
+            "been set"
+        )
+


Rep, removed

james-bruten-mo · 2026-01-21T16:19:20Z

build/extract/get_git_sources.py

+# which you should have received as part of this distribution.
+# *****************************COPYRIGHT*******************************
+"""
+Clone sources for a rose-stem run for use with git bdiff module in scripts


A copy and paste error, changed

james-bruten-mo · 2026-01-21T16:23:36Z

build/extract/get_git_sources.py

+        sync_repo(source, ref, dest)
+
+
+def run_command(


Potentially not - it's a function I tend to copy and paste around to let me set the defaults as I want them, eg. I don't always have to set the text or timeout entries. Given it matches the version on SimSys_Scripts I'm happy to keep it here

james-bruten-mo · 2026-01-21T16:43:35Z

build/extract/get_git_sources.py

+def clone_repo(repo_source: str, repo_ref: str, loc: Path) -> None:
+    """
+    Clone the repo and checkout the provided ref
+    Only if a remote source
+    """


james-bruten-mo · 2026-01-21T16:45:51Z

build/extract/get_git_sources.py

+        commands = (
+            f"git -C {loc} init",
+            f"git -C {loc} remote add origin {repo_source}",
+            f"git -C {loc} fetch origin {repo_ref}",
+            f"git -C {loc} checkout FETCH_HEAD",
+            f"git -C {loc} fetch origin main:main",
+        )


Yes, it was initially written to see if we could save space while cloning sources for use in rose-stem. It turns out the difference is very small, but as this was marginally smaller, I left it in. Given it's copied from SimSys_Scripts I don't think it's worth modifying here

james-bruten-mo · 2026-01-21T16:51:43Z

build/local_build.py

    """

-    return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    return Path(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))


Ta - absolute is a function but otherwise good

james-bruten-mo · 2026-01-21T17:00:53Z

documentation/source/developer_guide/local_builds.rst

+| ``-m --mirrors``     | False                       | If True, this will attempt  |
+| ``store_true``       |                             | to extract using local      |
+|                      |                             | github mirrors              |
+----------------------+-----------------------------+-----------------------------+
+| ``--mirror-loc``     | MetOffice Mirror Location   | The path to the github      |
+|                      |                             | mirror location             |


For the environment variables passed to extract_science.py then yes I agree, we can use the presence of the mirror location variable to show whether we're using the mirrors.

For local_build.py (which these docs are referring to) I think we want the --mirrors store_true argument as otherwise we'll need to always pass the location of the mirrors as an argument when using them, which isn't ideal

james-bruten-mo · 2026-01-21T17:05:48Z

build/local_build.py

+        args.working_dir = Path(project_path) / "working"
    else:
        # If the working dir doesn't end in working, set that here
        if not args.working_dir.strip("/").endswith("working"):


Thanks, done

james-bruten-mo added 4 commits January 19, 2026 15:57

incremenental with core working

1c35728

incremental physics builds

cc1d00d

Merge remote-tracking branch 'upstream/main' into improve_local_builds

137ba5f

incremental builds

132ccf8

github-actions bot added the cla-signed This contributor has signed the CLA. label Jan 20, 2026

github-actions bot assigned james-bruten-mo Jan 20, 2026

james-bruten-mo added 3 commits January 20, 2026 14:57

Merge remote-tracking branch 'upstream/main' into improve_local_builds

90d113d

minor changes

82f3795

catch missing env var

35903fb

james-bruten-mo removed the cla-signed This contributor has signed the CLA. label Jan 20, 2026

james-bruten-mo marked this pull request as ready for review January 20, 2026 20:59

github-actions bot requested a review from MatthewHambley January 20, 2026 21:00

james-bruten-mo mentioned this pull request Jan 21, 2026

extract_science.py rebuild fix #95

Closed

28 tasks

update local build docs

b6137e1

github-actions bot added the cla-modified The CLA has been modified as part of this PR - added by GA label Jan 21, 2026

Merge branch 'main' into improve_local_builds

8e9c249

james-bruten-mo removed the cla-modified The CLA has been modified as part of this PR - added by GA label Jan 21, 2026

MatthewHambley requested changes Jan 21, 2026

View reviewed changes

james-bruten-mo added 2 commits January 21, 2026 17:06

review comments

7500ed6

Merge branch 'improve_local_builds' of ssh://github.com/james-bruten-…

c59a673

…mo/lfric_apps into improve_local_builds

james-bruten-mo commented Jan 21, 2026

View reviewed changes

james-bruten-mo requested a review from MatthewHambley January 21, 2026 17:07

	return Path(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
	Path(__file__).absolute.parent.parent

Reenable incremental builds #162

Are you sure you want to change the base?

Reenable incremental builds #162

Uh oh!

Conversation

james-bruten-mo commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Code Quality Checklist

Testing

trac.log

Test Suite Results - lfric_apps - test_incremental_builds_change/run2

Suite Information

Task Information

Security Considerations

Performance Impact

AI Assistance and Attribution

Documentation

PSyclone Approval

Sci/Tech Review

Code Review

Uh oh!

MatthewHambley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-bruten-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-bruten-mo commented Jan 20, 2026 •

edited

Loading