Feature #2863 initial conda recipe

[georgemccabe]

This recipe builds METplus 6.0.0 using MET 12.0.2.

Pull Request Testing

• Describe testing already performed for these changes:
  

Build recipe on Linux (seneca) and MacOS arm64 (ral-dimeola)
Ran MET unit tests to confirm that installation was successful

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

Review recipe files for accuracy

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Do these changes introduce new SonarQube findings? [No]
  
  If yes, please describe:

• Please complete this pull request review by 2/21/2025.
  

Pull Request Checklist

See the METplus Workflow for details.

• Add any new Python packages to the METplus Components Python Requirements table.
• For any new datasets, an entry to the METplus Verification Datasets Guide.
• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the version that will include these changes
  Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases
• After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

I looked through the 2 new files in internal/scripts/conda/recipe and don't see any obvious problems. To test, I ran the following commands on my newer model Mac laptop, as instructed by @georgemccabe over Slack. Note that this requires that cmake be installed:

# install conda
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

# create conda environment with packages needed to build, index, and verify conda recipe
conda create --name build_env
conda activate build_env
conda install -c conda-forge conda-build conda-index conda-verify

# get conda recipe from METplus branch and build it
git clone https://github.com/dtcenter/METplus
cd METplus
git pull
git checkout feature_2863_conda
cd internal/scripts/conda
conda build recipe -c conda-forge -k --error-overlinking

After upgrading MacPorts on my on older Platform: osx-64 Mac laptop, the build succeeds:

Total time: 0:26:56.2
CPU usage: sys=0:06:47.5, user=0:56:26.9
Maximum memory usage observed: 2.6G
Total disk usage observed (not including envs): 1.3G

Additional commands:

# note path to .tar.gz or .conda file that was created,
# e.g. /Users/mccabe/conda-bld/osx-arm64/metplus-6.0.0-py310h921d19a_0.tar.bz2

# create directory to use as a local channel and
# create subdirectory based on OS name,
# e.g. for the path above, create the osx-arm64 subdirectory

mkdir -p /Users/${USER}/my-test-channel/osx-64

# copy the recipe file that was created from conda build, e.g.
# /Users/johnhg/miniforge3/envs/build_env/conda-bld/osx-64/metplus-6.0.0-py310hfaed013_0.conda

cp /Users/johnhg/miniforge3/envs/build_env/conda-bld/osx-64/metplus-6.0.0-py310hfaed013_0.conda /Users/${USER}/my-test-channel/osx-64/

# index channel to create other necessary files

conda index /Users/${USER}/my-test-channel

# create a new conda environment using python 3.10
# and install metplus from the local channel

conda create --name test_build_met python=3.10
conda activate test_build_met
conda install -c file:///Users/${USER}/my-test-channel metplus

# test that you can run a MET tool from the path, e.g. grid_stat

Note that here, grid_stat fails to run:

(test_build_met) musial6:METplus johnhg$ grid_stat                                                                                                          
dyld[53272]: Library not loaded: @rpath/libnetcdf-cxx4.1.dylib                                                                                              
  Referenced from: <91E3E561-08C2-37E5-A50C-2BD11F5C0897> /Users/johnhg/miniforge3/envs/test_build_met/bin/grid_stat                                        
  Reason: tried: '/Users/johnhg/miniforge3/envs/test_build_met/bin/../lib/libnetcdf-cxx4.1.dylib' (no such file), '/Users/johnhg/miniforge3/envs/test_build_met/bin/../lib/libnetcdf-cxx4.1.dylib' (no such file)                                                                                                       
Abort trap: 6  

Here's the NetCDF library files I see:

(test_build_met) musial6:METplus johnhg$ ls /Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf*
/Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf.19.dylib     /Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf_c++4.a                   
/Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf.dylib        /Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf_c++4.dylib               
/Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf.settings     /Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf_c++4.la                  
/Users/johnhg/miniforge3/envs/test_build_met/lib/libnetcdf_c++4.1.dylib   

Final steps:

# consider running the unit tests to make sure everything works properly
# here are some example env vars I used to set it up (assuming you have MET in your home dir):

TEST_ENV=test_build_met
export MET_TEST_INPUT=/Users/${USER}/data/METplus/met_unit_test_input
export MET_TEST_OUTPUT=/Users/${USER}/data/METplus/met_test_output
export MET_TEST_BASE=/Users/${USER}/MET/internal/test_unit
export MET_BASE=/Users/${USER}/.conda/envs/${TEST_ENV}/share/met
export MET_TEST_MET_PYTHON_EXE=/Users/${USER}/.conda/envs/${TEST_ENV}/bin/python3

cd ${MET_TEST_BASE}
./bin/unit_test.sh

[georgemccabe]

@JohnHalleyGotway , I added instructions to build/test a conda recipe locally in the Contributor's Guide: https://metplus--2900.org.readthedocs.build/en/2900/Contributors_Guide/conda_recipe.html

[JohnHalleyGotway]

@georgemccabe I continue to have the same failure I had late last week.

While the conda build recipe -c conda-forge -k --error-overlinking step runs fine, the indexing of the new channel fails:

(build_env) ral-brock20:conda johnhg$ conda index /Users/johnhg/my-test-channel
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'index' (choose from activate, clean, commands, compare, config, create, deactivate, env, export, info, init, install, list, notices, package, doctor, repoquery, debug, metapackage, build, convert, inspect, develop, skeleton, render, remove, uninstall, rename, run, search, update, upgrade)

I'll note this warning:

==> WARNING: A newer version of conda exists. <==
    current version: 24.11.3
    latest version: 25.1.1

Please update conda by running

    $ conda update -n base -c conda-forge conda

But running the recommended conda update command does not actually update the version.

[JohnHalleyGotway]

@georgemccabe I continue to have the same failure I had late last week.

While the conda build recipe -c conda-forge -k --error-overlinking step runs fine, the indexing of the new channel fails:

(build_env) ral-brock20:conda johnhg$ conda index /Users/johnhg/my-test-channel
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'index' (choose from activate, clean, commands, compare, config, create, deactivate, env, export, info, init, install, list, notices, package, doctor, repoquery, debug, metapackage, build, convert, inspect, develop, skeleton, render, remove, uninstall, rename, run, search, update, upgrade)

I'll note this warning:

==> WARNING: A newer version of conda exists. <==
    current version: 24.11.3
    latest version: 25.1.1

Please update conda by running

    $ conda update -n base -c conda-forge conda

But running the recommended conda update command does not actually update the version.

[georgemccabe]

@JohnHalleyGotway, can you run the following command while the build_env environment is activate and see if that helps?

conda install -c conda-forge conda-index

[jprestop]

@georgemccabe Will there also be documentation in the User's Guide for users who want to take advantage of the conda functionality? For the Contributor's Guide documentation, I suggest moving this chapter up to be Chapter 2 or Chapter 3 (pushing the others down).

[georgemccabe]

Not yet. This recipe is for METplus 6.0.0, which was already released. We still want to beta test this a little more and then provide instructions for the next release.

I moved this chapter to be chater 3.

I also made the other in-line suggestions above.

@georgemccabe Will there also be documentation in the User's Guide for users who want to take advantage of the conda functionality? For the Contributor's Guide documentation, I suggest moving this chapter up to be Chapter 2 or Chapter 3 (pushing the others down).

[John-Sharples]

@georgemccabe Great to see progress on this. As you know, we've been hoping for an official conda build for some time now.

I tried to follow the build instructions on Fedora 41, but consistently hit issues with eckit:
/home/jsharple/.conda/envs/build_env/conda-bld/metplus_1740636086326/_build_env/lib/gcc/x86_64-conda-linux-gnu/12.4.0/include/c++/ctime:80:11: error: 'timespec_get' has not been declared in '::' 80 | using ::timespec_get;

This looks to be some issue with the conda env vs. the native system. I tried a number of fixes, mostly around installing gcc, make, cmake directly into the conda env, but no luck yet. If I have any time tomorrow I'll try again.

[georgemccabe]

Hi @John-Sharples, thanks for testing this. I had also run into issues with my local environment conflicting with the eckit install. I had success using a clean conda environment to build the recipe, but I realize there may be other conflicting tools/libraries on the system. I have also had some luck changing my path, or rather changing my active conda env and calling conda build using the full path to the conda exe. I wish the conda build would only rely on what is provided in the package to avoid these types of conflicts.

According to the conda-forge Knowledge Base re: Using CMake, there are some CMake arguments that can be used:

-DCMAKE_FIND_FRAMEWORK=NEVER and -DCMAKE_FIND_APPBUNDLE=NEVER Prevent CMake from using system-wide macOS packages.

I know it mentions macOS specifically, but I wonder if this or another argument could be used to prevent system tools from being used. I have not yet tried using these options because I was able to get a successful build and the automated builds from the conda-forge PR also passed. This shouldn't be an issue installing the package from conda-forge, but it would be nice to resolve these types of issues to prevent problems testing new versions in the future.

Let me know if you are able to get this to build locally.

[John-Sharples]

Thanks @georgemccabe
No luck with those flags, or by calling the full path to conda executable.

I did manage to get eckit to build by installing make, cmake and gcc directly into my conda env (e.g. conda install cmake) and removing them from the build reqs in meta.yaml. However, I then got a subsequent error in atlas, which I haven't been able to get past.

Unfortunatley I don't have much time to devote to this, but i'll let you know if I do get a build working.