Installation issue: trilinos `KeyError: 'cusparse'`

[dabele]

Steps to reproduce the issue

Install trilinos with dependencies already installed, so only trilinos is built.

$ spack spec -I trilinos

spack spec -I trilinos
[+]  trilinos@13.4.1~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_constexpr~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=14 generator=make gotype=long_long arch=linux-ubuntu22.04-cascadelake %c,cxx,fortran=gcc@11.4.0
[e]      ^cmake@3.31.8~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu22.04-cascadelake 
[+]      ^compiler-wrapper@1.0 build_system=generic arch=linux-ubuntu22.04-cascadelake 
[e]      ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-cascadelake 
[+]      ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-cascadelake 
[e]      ^glibc@2.35 build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[e]      ^gmake@4.3~guile build_system=generic patches:=599f134 arch=linux-ubuntu22.04-cascadelake 
[+]      ^hwloc@2.11.1~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]          ^libpciaccess@0.17 build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[e]              ^util-macros@1.19.3 build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[+]          ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]              ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]              ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[e]          ^ncurses@6.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[e]          ^pkgconf@1.6.3 build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[+]      ^openblas@0.3.30~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu22.04-cascadelake %c,cxx,fortran=gcc@11.4.0
[e]          ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-cascadelake 
[+]          ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-cascadelake 
[+]      ^openmpi@5.0.8+atomics~cuda~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none arch=linux-ubuntu22.04-cascadelake %c,cxx,fortran=gcc@11.4.0
[e]          ^autoconf@2.69 build_system=autotools patches:=7793209 arch=linux-ubuntu22.04-cascadelake 
[e]          ^automake@1.16.1 build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[+]          ^libevent@2.1.12+openssl build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[e]              ^openssl@1.1.1f~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-cascadelake 
[e]          ^libtool@2.4.6 build_system=autotools arch=linux-ubuntu22.04-cascadelake 
[+]          ^numactl@2.0.18 build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[e]              ^m4@1.4.18+sigsegv build_system=autotools patches:=3877ab5,fc9b616 arch=linux-ubuntu22.04-cascadelake 
[+]          ^openssh@9.9p1+gssapi build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]              ^krb5@1.21.3+shared build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]                  ^bison@3.8.2~color build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]                  ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                  ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                      ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]                          ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                              ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]                          ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                              ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[e]                      ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-cascadelake 
[+]                      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                          ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]                          ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0
[+]              ^libedit@3.1-20240808 build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]              ^libxcrypt@4.4.38~obsolete_api build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[e]          ^perl@5.30.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-cascadelake 
[+]          ^pmix@5.0.5~munge~python build_system=autotools arch=linux-ubuntu22.04-cascadelake %c=gcc@11.4.0
[+]          ^zlib-ng@2.2.4+compat~new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-cascadelake %c,cxx=gcc@11.4.0

Error message

Error message

spack install trilinos
[+] /localdata1/usr (external cmake-3.31.8-4ypejvib2clrgojj4ra3bex6uo5atcxj)
[+] /localdata1/spack/opt/spack/linux-cascadelake/compiler-wrapper-1.0-kcwqlumnifqw5btnzcexalhpwxqq5oiu
[+] /usr (external gcc-11.4.0-qysvavrd62he77ib7jmbtbmpf57ioxfe)
[+] /usr (external glibc-2.35-fyvp44uolcswcafwhvkxbh3w4qnnrgsh)
[+] /usr (external gmake-4.3-lmzkskk46nzs332cjws5r54dh2zez3zt)
[+] /usr (external ncurses-6.2.0-7gp6wfk224gagms2jhb5im22zha4elac)
[+] /usr (external openssl-1.1.1f-elax25n4lipzo2hwvpvxlirksxeeivbt)
[+] /usr (external bzip2-1.0.8-vjjjvosnne6wg3mfkzvs2fuv45d4bf3w)
[+] /localdata1/spack/opt/spack/linux-cascadelake/gcc-runtime-11.4.0-zyrpmcyc2jahacsklafoctiqgxtp7bya
[+] /localdata1/spack/opt/spack/linux-cascadelake/gcc-runtime-11.4.0-lieukcqodwkkeyiujkk6u5gbi67yxsn4
[+] /localdata1/spack/opt/spack/linux-cascadelake/zlib-ng-2.2.4-d3erpndzrgsaj46me52pynjme777tdkt
[+] /localdata1/spack/opt/spack/linux-cascadelake/numactl-2.0.18-ebawvbcx52s5nksskvbo2nmayltnzuj7
[+] /localdata1/spack/opt/spack/linux-cascadelake/libevent-2.1.12-sivdnmyw42o2vgpupor2cw74r72giwrg
[+] /localdata1/spack/opt/spack/linux-cascadelake/libxcrypt-4.4.38-y5pdd5eixfc3zsv5fsb7luiymqn6qc7r
[+] /localdata1/spack/opt/spack/linux-cascadelake/openblas-0.3.30-2rlyfztnu4fxpxyuxojh4u2jhjzuiyzi
[+] /localdata1/spack/opt/spack/linux-cascadelake/xz-5.6.3-2ojhdqdhixamtqh7eyn35yocnwhrccix
[+] /localdata1/spack/opt/spack/linux-cascadelake/libedit-3.1-20240808-upk3qymfbcz7lrszydmbdstuxeq5mgno
[+] /localdata1/spack/opt/spack/linux-cascadelake/libiconv-1.18-olk4wtafuffiqskdk4xy3ej7lkt4ijyu
[+] /localdata1/spack/opt/spack/linux-cascadelake/libpciaccess-0.17-zd3s6pwheczinzf2dla6cn2nendrnuls
[+] /localdata1/spack/opt/spack/linux-cascadelake/zstd-1.5.7-2a3dn2xakij4gh4cvx2rl7vaymaltj63
[+] /localdata1/spack/opt/spack/linux-cascadelake/pigz-2.8-z3jnlqng4f4c24gq77dirzadm6fxqscl
[+] /localdata1/spack/opt/spack/linux-cascadelake/libxml2-2.13.5-6dg3wkmgnngoks5aykhm7myfijx57qml
[+] /localdata1/spack/opt/spack/linux-cascadelake/tar-1.35-g4xh2l5g5eejtj24g5f5mgfcz2njxhfu
[+] /localdata1/spack/opt/spack/linux-cascadelake/hwloc-2.11.1-n6qmvvvd5ak3nclxtwuyqdwm2zylomat
[+] /localdata1/spack/opt/spack/linux-cascadelake/gettext-0.23.1-x4vp5vzefgfz4hufgjr767zs72zequug
[+] /localdata1/spack/opt/spack/linux-cascadelake/pmix-5.0.5-zym4xafia2alvjzknyj4l3lt2uao66lp
[+] /localdata1/spack/opt/spack/linux-cascadelake/krb5-1.21.3-7bjyihaas3gs53kvuwwfn7fzydoqejof
[+] /localdata1/spack/opt/spack/linux-cascadelake/openssh-9.9p1-qtgj2qwbbv4ovsielo5gh3el6nojky2z
[+] /localdata1/spack/opt/spack/linux-cascadelake/openmpi-5.0.8-o6m3r3w7y3yxu46xxg6jnevgjrerxw6c
==> No binary for trilinos-14.2.0-gnxy7pwm4hc64zd4dxrzjuuxuj3kqbtm found: installing from source
==> Installing trilinos-14.2.0-gnxy7pwm4hc64zd4dxrzjuuxuj3kqbtm [30/30]
==> Using cached archive: /localdata1/spack/var/spack/cache/_source-cache/archive/c9/c96606e5cd7fc9d25b9dc20719cd388658520d7cbbd2b4de77a118440d1e0ccb.tar.gz
==> No patches needed for trilinos
==> trilinos: Executing phase: 'cmake'
==> Error: KeyError: 'cusparse'

/localdata1/spack-packages/repos/spack_repo/builtin/packages/trilinos/package.py:924, in cmake_args:
        921            tpl_variant_map.append(("STRUMPACK", "strumpack", "strumpack"))
        922
        923        for tpl_name, var_name, spec_name in tpl_variant_map:
  >>    924            define_tpl(tpl_name, spec_name, spec.variants[var_name].value)
        925
        926        # Enable these TPLs based on whether they're in our spec; prefer to
        927        # require this way so that packages/features disable availability

See build log for details:
  /tmp/abel_da/spack-stage/spack-stage-trilinos-14.2.0-gnxy7pwm4hc64zd4dxrzjuuxuj3kqbtm/spack-build-out.txt

Information on your system

• Spack: 1.0.1 (spack/spack@17d3428)
• Builtin repo: 3f8d8ed
• Python: 3.10.12
• Platform: linux-ubuntu22.04-cascadelake

Environment spack.yaml:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - dealii@9.6.2
  view: true
  concretizer:
    unify: true

(Note: trilinos is a dependency of dealii)

Additional information

Not sure under which circumstances exactly the cusparse variant is missing. It's not conditional in package.py as far as I can tell, but it's not listed in the spack spec output above. It's also not in the spack.lock file in the environment.

As a workaround, replace in package.py, line 962: define_tpl(tpl_name, spec_name, spec.variants[var_name].value) with define_tpl(tpl_name, spec_name, (var_name in spec.variants) and spec.variants[var_name].value) but a proper fix probably needs to explain why cusparse is missing at all.

cusparse was added not that long ago, maybe I had an install of trilinos around from before it was added that polluted the spec somehow?

Mentioning maintainers of trilinos @elliottslaughter @ccober6 @cgcgcg @fryeguy52 @jwillenbring @keitat @kuberry @psakievich @rppawlo @sebrowne

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
• I have uploaded the build log and environment files
• I have searched the issues of this repo and believe this is not a duplicate

[kuberry]

Trilinos 14.2 was released Jun 21, 2023 and Spack 1.0.1 was released August 11, 2025, but the cusparse variant was only added Nov 16, 2025. Is it possibly to test with a Spack release after the cusparse variant was added?

[dabele]

There doesn't seem to be any issue with a clean install.

I forgot that I had switched to a local development package repo at some point.

Here is how to reproduce the issue:

1. install spack with default packages repo
2. create and concretize an environment, lock in trilinos without cusparse
3. update the package repo to a version with cusparse
4. try to install trilinos
5. error

In reality (I think this happened with me) you probably install and use the environment for a while between steps 2 and 3. And then something happens that you need to update and reinstall.

I'm not sure this is a valid use of spack. When you update the packages, are you expected to recreate all environments? Or at least force-concretize before reinstalling something?

If this is not a valid use of spack, I feel spack should be able to detect it somehow.