Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

conda-build script lines containing # break if selectors are also present on line #5618

Open
2 tasks done
matthewfeickert opened this issue Feb 11, 2025 · 3 comments
Open
2 tasks done

conda-build script lines containing # break if selectors are also present on line #5618

matthewfeickert opened this issue Feb 11, 2025 · 3 comments
Labels
type::bug describes erroneous operation, use severity::* to classify the type

Comments

@matthewfeickert
Copy link

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

In conda-build v25.1.2, if a command is given in an output[].build.script that contains a # as part of the syntax and there is a selector on that line, then the first # is treated as a terminating comment and all characters following it are ignored in the render of the YAML.

Example:

    - sed -i '/ninja =/s/^# //g' $PREFIX/MG5_aMC/input/mg5_configuration.txt  # [linux and (x86_64 or ppc64le)]

gets parsed and rendered as

    - sed -i '/ninja =/s/^

Attempting to escape the entire command by placing it in quotes

    # Quote the whole command to guard the first pound sign
    - "sed -i '/ninja =/s/^# //g' $PREFIX/MG5_aMC/input/mg5_configuration.txt"  # [linux and (x86_64 or ppc64le)]

results in a yaml.parser.ParserError

...
Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/conda_build/metadata.py", line 360, in yamlize
    return yaml.load(data, Loader=StringifyNumbersLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "yaml/_yaml.pyx", line 673, in yaml._yaml.CParser.get_single_node
  File "yaml/_yaml.pyx", line 687, in yaml._yaml.CParser._compose_document
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 731, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 845, in yaml._yaml.CParser._compose_mapping_node
  File "yaml/_yaml.pyx", line 729, in yaml._yaml.CParser._compose_node
  File "yaml/_yaml.pyx", line 808, in yaml._yaml.CParser._compose_sequence_node
  File "yaml/_yaml.pyx", line 860, in yaml._yaml.CParser._parse_next_event
yaml.parser.ParserError: while parsing a block collection
  in "<unicode string>", line 18, column 5
did not find expected '-' indicator
  in "<unicode string>", line 91, column 13

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/conda-build", line 11, in <module>
    sys.exit(execute())
             ^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/cli/main_build.py", line 622, in execute
    api.build(
  File "/opt/conda/lib/python3.12/site-packages/conda_build/api.py", line 211, in build
    return build_tree(
           ^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/build.py", line 3630, in build_tree
    metadata_tuples = render_recipe(
                      ^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/render.py", line 974, in render_recipe
    m = MetaData(str(recipe), config=config)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/metadata.py", line 1189, in __init__
    self.parse_again(permit_undefined_jinja=True, allow_no_other_outputs=True)
  File "/opt/conda/lib/python3.12/site-packages/conda_build/metadata.py", line 1277, in parse_again
    self.meta = parse(
                ^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/metadata.py", line 521, in parse
    res = yamlize(data)
          ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/metadata.py", line 364, in yamlize
    raise UnableToParse(original=e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/conda_build/exceptions.py", line 22, in __init__
    super().__init__(*args, **kwargs)
TypeError: CondaError.__init__() missing 1 required positional argument: 'message'
Traceback (most recent call last):
  File "/home/feickert/Code/GitHub/conda-forge/hep-packaging-coordination/mg5amcnlo-feedstock/build-locally.py", line 116, in <module>
    main()
  File "/home/feickert/Code/GitHub/conda-forge/hep-packaging-coordination/mg5amcnlo-feedstock/build-locally.py", line 104, in main
    run_docker_build(ns)
  File "/home/feickert/Code/GitHub/conda-forge/hep-packaging-coordination/mg5amcnlo-feedstock/build-locally.py", line 39, in run_docker_build
    subprocess.check_call([script])
  File "/home/feickert/.pyenv/versions/3.11.7/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.scripts/run_docker_build.sh']' returned non-zero exit status 1.

It was pointed out by @jaimergp that this is due to the selector regex of

conda-build/conda_build/metadata.py

Lines 251 to 260 in e5b060e

# Selectors must be either:
# - at end of the line
# - embedded (anywhere) within a comment
#
# Notes:
# - [([^\[\]]+)\] means "find a pair of brackets containing any
# NON-bracket chars, and capture the contents"
# - (?(2)[^\(\)]*)$ means "allow trailing characters iff group 2 (#.*) was found."
# Skip markdown link syntax.
sel_pat = re.compile(r"(.+?)\s*(#.*)?\[([^\[\]]+)\](?(2)[^\(\)]*)$")

Context: I've only used conda-build from the context of running conda-forge's build-locally.py scripts in feedstocks.

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/conda/.condarc
 populated config files : /opt/conda/.condarc
                          /home/conda/.condarc
          conda version : 25.1.1
    conda-build version : 25.1.2
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=25.1.1=0
                          __glibc=2.34=0
                          __linux=6.8.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/conda/feedstock_root/build_artifacts/pkg_cache
                          /opt/conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/25.1.1 requests/2.32.3 CPython/3.12.8 Linux/6.8.0-51-generic almalinux/9.5 glibc/2.34 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Conda Config

==> /opt/conda/.condarc <==
add_pip_as_python_dependency: False
auto_update_conda: False
aggressive_update_packages:
  - ca-certificates
  - certifi
channel_priority: strict
channels:
  - conda-forge
show_channel_urls: True
conda_build:
  error_overlinking: True
  pkg_format: 2
  zstd_compression_level: 19

==> /home/conda/.condarc <==
pkgs_dirs:
  - /home/conda/feedstock_root/build_artifacts/pkg_cache
  - /opt/conda/pkgs
solver: libmamba
conda-build:
  root-dir: /home/conda/feedstock_root/build_artifacts

==> envvars <==
allow_softlinks: False
bld_path: /home/conda/feedstock_root/build_artifacts

Conda list

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
anaconda-client           1.12.3             pyhd8ed1ab_1    conda-forge
archspec                  0.2.5              pyhd8ed1ab_0    conda-forge
attrs                     25.1.0             pyh71513ae_0    conda-forge
beautifulsoup4            4.13.3             pyha770c72_0    conda-forge
boltons                   24.0.0             pyhd8ed1ab_1    conda-forge
brotli-python             1.1.0           py312h2ec8cdc_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.4               hb9d3cd8_0    conda-forge
ca-certificates           2025.1.31            hbcca054_0    conda-forge
certifi                   2024.12.14         pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py312h06ac9bb_0    conda-forge
chardet                   5.2.0           py312h7900ff3_2    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
click                     8.1.8              pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
conda                     25.1.1          py312h7900ff3_0    conda-forge
conda-build               25.1.2          py312h7900ff3_0    conda-forge
conda-env                 2.6.0                         1    conda-forge
conda-forge-ci-setup      4.14.4          py312hb3d6910_100    conda-forge
conda-forge-metadata      0.11.0             pyhd8ed1ab_1    conda-forge
conda-index               0.5.0              pyhd8ed1ab_0    conda-forge
conda-libmamba-solver     25.1.1             pyhd8ed1ab_0    conda-forge
conda-oci-mirror          0.2.1              pyhd8ed1ab_1    conda-forge
conda-package-handling    2.4.0              pyh7900ff3_2    conda-forge
conda-package-streaming   0.11.0             pyhd8ed1ab_0    conda-forge
cpp-expected              1.1.0                hf52228f_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecated                1.2.18             pyhd8ed1ab_0    conda-forge
distro                    1.9.0              pyhd8ed1ab_1    conda-forge
filelock                  3.17.0             pyhd8ed1ab_0    conda-forge
fmt                       11.0.2               h434a139_0    conda-forge
frozendict                2.4.6           py312h66e93f0_0    conda-forge
git                       2.47.1          pl5321h59d505e_0    conda-forge
h2                        4.2.0              pyhd8ed1ab_0    conda-forge
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
importlib_resources       6.5.2              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.5              pyhd8ed1ab_0    conda-forge
joblib                    1.4.2              pyhd8ed1ab_1    conda-forge
jq                        1.7.1                hd590300_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_1    conda-forge
jsonpointer               3.0.0           py312h7900ff3_1    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_1    conda-forge
jsonschema-specifications 2024.10.1          pyhd8ed1ab_1    conda-forge
jupyter_core              5.7.2              pyh31011fe_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
libarchive                3.7.7                h4585015_3    conda-forge
libcurl                   8.11.1               h332b0f4_0    conda-forge
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblief                   0.14.1               h5888daf_2    conda-forge
liblzma                   5.6.4                hb9d3cd8_0    conda-forge
liblzma-devel             5.6.4                hb9d3cd8_0    conda-forge
libmamba                  2.0.5                h49b8a8d_1    conda-forge
libmambapy                2.0.5           py312hbaee817_1    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsolv                   0.7.30               h3509ff9_0    conda-forge
libsqlite                 3.48.0               hee588c1_1    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.13.5               h8d12d68_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
lz4-c                     1.10.0               h5888daf_1    conda-forge
lzo                       2.10              hd590300_1001    conda-forge
mamba                     2.0.5                h8871ed4_1    conda-forge
markupsafe                3.0.2           py312h178313f_1    conda-forge
menuinst                  2.2.0           py312h7900ff3_0    conda-forge
more-itertools            10.6.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_1    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
nlohmann_json             3.11.3               he02047a_1    conda-forge
oniguruma                 6.9.10               hb9d3cd8_0    conda-forge
openssl                   3.4.0                h7b32b05_1    conda-forge
oras-py                   0.1.14             pyhd8ed1ab_0    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
patch                     2.7.6             h7f98852_1002    conda-forge
patchelf                  0.17.2               h58526e2_0    conda-forge
pcre2                     10.44                hba22ea6_2    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pip                       25.0               pyh8b19718_0    conda-forge
pkginfo                   1.12.0             pyhd8ed1ab_1    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_2    conda-forge
platformdirs              4.3.6              pyhd8ed1ab_1    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_1    conda-forge
psutil                    6.1.1           py312h66e93f0_0    conda-forge
py-lief                   0.14.1          py312h2ec8cdc_2    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py312h66e93f0_2    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.12.8          h9e4cc4f_1_cpython    conda-forge
python-dateutil           2.9.0.post0        pyhff2d567_1    conda-forge
python-fastjsonschema     2.21.1             pyhd8ed1ab_0    conda-forge
python-libarchive-c       5.1             py312h7900ff3_1    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2025.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.2           py312h178313f_2    conda-forge
rattler-build             0.35.9               hff40e2b_0    conda-forge
rattler-build-conda-compat 1.3.3              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.36.2             pyh29332c3_0    conda-forge
reproc                    14.2.5.post0         hb9d3cd8_0    conda-forge
reproc-cpp                14.2.5.post0         h5888daf_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_1    conda-forge
ripgrep                   14.1.1               h8fae777_0    conda-forge
rpds-py                   0.22.3          py312h12e396e_0    conda-forge
ruamel.yaml               0.18.10         py312h66e93f0_0    conda-forge
ruamel.yaml.clib          0.2.8           py312h66e93f0_1    conda-forge
setuptools                75.8.0             pyhff2d567_0    conda-forge
shyaml                    0.6.2              pyhd3deb0d_0    conda-forge
simdjson                  3.11.6               h84d6215_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
spdlog                    1.15.1               hb29a8c4_0    conda-forge
su-exec                   0.2               h166bdaf_1003    conda-forge
tini                      0.19.0               h166bdaf_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.2.1              pyhd8ed1ab_1    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_1    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_1    conda-forge
truststore                0.10.1             pyh29332c3_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2025a                h78e105d_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
wrapt                     1.17.2          py312h66e93f0_0    conda-forge
xz                        5.6.4                hbcc6ac9_0    conda-forge
xz-gpl-tools              5.6.4                hbcc6ac9_0    conda-forge
xz-tools                  5.6.4                hb9d3cd8_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zstandard                 0.23.0          py312hef9b889_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Additional Context

This was first encounted while working on conda-forge/mg5amcnlo-feedstock#6.
@matthewfeickert matthewfeickert added the type::bug describes erroneous operation, use severity::* to classify the type label Feb 11, 2025
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Feb 11, 2025
@beckermr
Copy link
Contributor

That's a fun one. As a workaround you can use a jinja2 if block.

@matthewfeickert
Copy link
Author

As a workaround you can use a jinja2 if block.

Are the conda-build selectors already valid jinja2, such that I could do

  {% if (linux and (x86_64 or ppc64le)) %}
    - sed -i '/ninja =/s/^# //g' $PREFIX/MG5_aMC/input/mg5_configuration.txt
  {% endif %}

? I think so from initial poking.

@beckermr
Copy link
Contributor

Yes they should be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
2 participants
@matthewfeickert @beckermr