RHEL 8 & Spack v1.1.1: Concretization failure with spack-packages commit e6d347de0cecc04e9837ae1e90a42f99354501b7 (gcc: default to +binutils (#3106))

[harshula]

Some HPCs are still using RHEL 8.x. Spack concretization behaviour with GCC has changed since commit e6d347d from PR #3106

spack debug report

• Spack: 1.1.1 (spack/spack@2e2169d)
• Builtin repo: 353d270
• Python: 3.6.8
• Platform: linux-rocky8-zen5

Reproducer

$ docker pull rockylinux/rockylinux:8.10

$ docker run -it --rm rockylinux/rockylinux:8.10

$ dnf update -y  && dnf -y install     autoconf     automake     binutils     bison     bzip2     csh     flex     curl     file     findutils     gcc     gcc-c++     gcc-gfortran     git     glibc-devel     jq     libtool     gnupg2     hg     hostname     iproute     make     patch     patchutils     perl     pkgconf     pkgconf-m4     pkgconf-pkg-config     python3     python3-pip     python3-setuptools     svn     unzip     vim     which     xz     zstd  && rm -rf /var/cache/dnf  && dnf clean all

$ cd /opt/

$ git clone https://github.com/spack/spack.git --branch releases/v1.1

$ cd spack/

$ . share/spack/setup-env.sh 

$ spack compiler find

remote: Enumerating objects: 19701, done.
remote: Counting objects: 100% (19701/19701), done.
remote: Compressing objects: 100% (10553/10553), done.
remote: Total 19701 (delta 1338), reused 13774 (delta 1224), pack-reused 0 (from 0)
==> Added 1 new compiler to /root/.spack/packages.yaml
    gcc@8.5.0
==> Compilers are defined in the following files:
    /root/.spack/packages.yaml


$ spack compiler info gcc

[e]  gcc@=8.5.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky8 target=x86_64

  prefix: /usr
  compilers:
    cc: /usr/bin/gcc
    cxx: /usr/bin/g++
    fortran: /usr/bin/gfortran


$ spack compiler list

==> Available compilers
-- gcc rocky8-x86_64 --------------------------------------------
[e]  gcc@8.5.0

$ spack config blame repos
---                                               repos:
/opt/spack/etc/spack/defaults/base/repos.yaml:14    builtin:
/opt/spack/etc/spack/defaults/base/repos.yaml:15      git: https://github.com/spack/spack-packages.git
/opt/spack/etc/spack/defaults/base/repos.yaml:16      branch: develop

$ spack repo update
remote: Enumerating objects: 500650, done.
remote: Counting objects: 100% (500650/500650), done.
remote: Compressing objects: 100% (148497/148497), done.
remote: Total 491954 (delta 211058), reused 490287 (delta 209692), pack-reused 0 (from 0)
==> builtin: Updated sucessfully.


$ spack install gcc@13.4.0


$ spack compiler info gcc
[e]  gcc@=8.5.0+binutils+bootstrap~graphite+libsanitizer~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky8 target=x86_64

  prefix: /usr
  compilers:
    cc: /usr/bin/gcc
    cxx: /usr/bin/g++
    fortran: /usr/bin/gfortran

[+]  gcc@=13.4.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' patches:=8b79314 platform=linux os=rocky8 target=x86_64_v4

  prefix: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn
  compilers:
    cc: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gcc
    cxx: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/g++
    fortran: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gfortran


$ spack spec netcdf-fortran target=x86_64 %gcc@13.4.0 
==> Error: failed to concretize `netcdf-fortran target=x86_64 %gcc@13.4.0` for the following reasons:
     1. Only external, or concrete, compilers are allowed for the fortran language
     2. Only external, or concrete, compilers are allowed for the cxx language
     3. Only external, or concrete, compilers are allowed for the c language


[root@3f4d4f3ec5b3 test]# spack spec netcdf-fortran target=x86_64 %gcc@8.5.0
 -   netcdf-fortran@4.6.2~doc+pic+shared build_system=autotools platform=linux os=rocky8 target=x86_64 %c,fortran=gcc@8.5.0

Workaround 1

$ git -C ~/.spack/package_repos/fncqgg4/ revert e6d347de0cecc04e9837ae1e90a42f99354501b7
Auto-merging stacks/tutorial/spack.yaml
[develop 73389f394c] Revert "gcc: default to +binutils (#3106)"
 2 files changed, 14 insertions(+), 2 deletions(-)


$ spack uninstall gcc@13.4.0
    -- linux-rocky8-x86_64_v4 / %c,cxx=gcc@8.5.0 --------------------
    avqaqh3 gcc@13.4.0

==> 1 packages will be uninstalled. Do you want to proceed? [y/N] y
==> Successfully uninstalled gcc@13.4.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' patches:=8b79314 platform=linux os=rocky8 target=x86_64_v4/avqaqh3


$ spack install gcc@13.4.0
...
==> gcc: Successfully installed gcc-13.4.0-7mbv6a7khuafo2xn3ks4u3ztl7bjwfvu
  Stage: 2.12s.  Autoreconf: 0.00s.  Configure: 2.26s.  Build: 15m 10.04s.  Install: 4.81s.  Post-install: 0.98s.  Total: 15m 20.40s


$ spack spec netcdf-fortran target=x86_64 %gcc@13.4.0
...
 -   netcdf-fortran@4.6.2~doc+pic+shared build_system=autotools platform=linux os=rocky8 target=x86_64 %c,fortran=gcc@13.4.0


$ spack compiler list
==> Available compilers
-- gcc rocky8-x86_64 --------------------------------------------
[e]  gcc@8.5.0  [+]  gcc@13.4.0

$ spack compiler info gcc

[e]  gcc@=8.5.0~binutils+bootstrap~graphite+libsanitizer~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky8 target=x86_64

  prefix: /usr
  compilers:
    cc: /usr/bin/gcc
    cxx: /usr/bin/g++
    fortran: /usr/bin/gfortran

[+]  gcc@=13.4.0~binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' patches:=8b79314 platform=linux os=rocky8 target=x86_64_v4

  prefix: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-7mbv6a7khuafo2xn3ks4u3ztl7bjwfvu
  compilers:
    cc: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-7mbv6a7khuafo2xn3ks4u3ztl7bjwfvu/bin/gcc
    cxx: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-7mbv6a7khuafo2xn3ks4u3ztl7bjwfvu/bin/g++
    fortran: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-7mbv6a7khuafo2xn3ks4u3ztl7bjwfvu/bin/gfortran

Workaround 2

$ spack load gcc@13.4.0

$ spack compiler find
==> Added 1 new compiler to /root/.spack/packages.yaml
    gcc@13.4.0
==> Compilers are defined in the following files:
    /root/.spack/packages.yaml

$ spack compiler list
==> Available compilers
-- gcc rocky8-x86_64 --------------------------------------------
[e]  gcc@8.5.0  [e]  gcc@13.4.0  [+]  gcc@13.4.0

$ spack compiler info gcc
[e]  gcc@=8.5.0+binutils+bootstrap~graphite+libsanitizer~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky8 target=x86_64

  prefix: /usr
  compilers:
    cc: /usr/bin/gcc
    cxx: /usr/bin/g++
    fortran: /usr/bin/gfortran

[e]  gcc@=13.4.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky8 target=x86_64

  prefix: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn
  compilers:
    cc: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gcc
    cxx: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/g++
    fortran: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gfortran

[+]  gcc@=13.4.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' patches:=8b79314 platform=linux os=rocky8 target=x86_64_v4

  prefix: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn
  compilers:
    cc: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gcc
    cxx: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/g++
    fortran: /opt/spack/opt/spack/linux-x86_64_v4/gcc-13.4.0-avqaqh3ii6nyo3x4e365zv4hy3o2wqsn/bin/gfortran

$ spack spec netcdf-fortran target=x86_64 %gcc@13.4.0 
 -   netcdf-fortran@4.6.2~doc+pic+shared build_system=autotools platform=linux os=rocky8 target=x86_64 %c,fortran=gcc@13.4.0

[harshula]

Hi @becker33 & @scheibelp , This is what I was describing yesterday. Please let me know if you need any more information.

[rbberger]

Tried to reproduce this on aarch64 (podman on m3 mac) and it didn't show up. Then I noticed a subtle point here:

• gcc@13.4.0 gets built for x86_64_v4
• workaround 2 probably works because the external changes it to only x86_64, which is what was requested for netcdf-fortran.

@harshula does this go away if you just build gcc@13.4.0 with target=x86_64?

[harshula]

Hi @rbberger , Did you use a RHEL8 container?

Thanks, running spack install gcc@13.4.0 target=x86_64 %gcc@8.5.0 instead of
spack install gcc@13.4.0 results in both spack spec netcdf-fortran target=x86_64 %gcc@13.4.0 and spack spec netcdf-fortran target=x86_64_v4 %gcc@13.4.0 concretizing.

Running spack install gcc@13.4.0 results in spack spec netcdf-fortran target=x86_64 %gcc@13.4.0 failing to concretize and spack spec netcdf-fortran target=x86_64_v4 %gcc@13.4.0 succeeding. Is this expected Spack concretizer behaviour?

Please note, I've described a minimal reproducer in this issue. Our actual reproducer involves creating a Docker image.

[rbberger]

@harshula Yes, this was with a RHEL8 container (rocky via podman), just following the steps you outlined. The only difference was that mine was aarch64 due to my laptop, which didn't fail since everything became target=aarch64.

[rbberger]

@harshula Thanks for confirming. So the concretizer seems to not consider a x86_64_v4 compiler for target=x86_64 packages.

What would be interesting is the other way around: if gcc@13.4.0 target=x84_64 works for spack spec netcdf-fortran target=x86_64_v4 %gcc@13.4.0. Basically whether a compiler, compiled for a smaller instruction set, will still be used for a larger instruction set package. (I'm expecting this to work)

[harshula]

Hi @rbberger , Is your question answered in #3505 (comment) ?