diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/LICENSE.md b/easybuild/easyconfigs/l/lumi-CPEtools/LICENSE.md index 29506e2aa..5b0758d46 100644 --- a/easybuild/easyconfigs/l/lumi-CPEtools/LICENSE.md +++ b/easybuild/easyconfigs/l/lumi-CPEtools/LICENSE.md @@ -5,3 +5,9 @@ and licensed under the GNU General Public License version 3.0, a copy of which can be found in the [LICENSE file in the source repository](https://github.com/Lumi-supercomputer/lumi-CPEtools/blob/main/LICENSE). +The `hpcat` tool included in the module is developed by HPE and licensed under an MIT-style +license which can be found in the +[LICENSE file in the source repository](https://github.com/HewlettPackard/hpcat/blob/main/LICENSE). + +After loading the module, both license files are available in their respective subdirectories +in `$EBROOTLUMIMINCPETOOLS/share/licenses`. diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/README.md b/easybuild/easyconfigs/l/lumi-CPEtools/README.md index ecae26b26..ea9a0acee 100644 --- a/easybuild/easyconfigs/l/lumi-CPEtools/README.md +++ b/easybuild/easyconfigs/l/lumi-CPEtools/README.md @@ -4,6 +4,8 @@ lumi-CPEtools is developed by the LUST team. - [lumi-CPEtools on GitHub](https://github.com/Lumi-supercomputer/lumi-CPEtools) +- The [hpcat utility from HPE](https://github.com/HewlettPackard/hpcat) on GitHub + ## EasyBuild @@ -39,3 +41,40 @@ for the CrayEnv software stack. - It looks like the compiler wrappers have changed in 24.03 as unloading the accelerator target module in the cpeAMD version was no longer needed. + + +### Version 1.2 + +- Transformed the EasyConfig from version 1.1 to a Bundle to be able to add `hpcat` + using its own installation procedure. + +- Building `hpcat`: + + - LUMI lacks the `hwloc-devel` package so we simply copied the header files from another system + and download them from LUMI-O. + + - The Makefile was modified to integrate better with EasyBuild and to work around a problem with + finding the `hwloc` library on LUMI. + + Rather than writing a new Makefile or a patch, we actually used a number of `sed` commands to edit + the Makefile: + + - `mpicc` was replaced with `$(CC)` so that the wrappers are used instead. + - `-O3` was replaced with `$(CFLAGS)` to pick up the options from EasyBuild + - '-fopenmp' is managed by the Makefile though and not by EasyBuild. On one hand because the + ultimate goal is to integrate with another packages that sometimes needs and sometimes does not + need the OpenMP flags, on the other hand to use `$(CFLAGS)` also for `hipcc`. + - `-lhwloc` is replaced with `-Wl,/usr/lib64/libhwloc.so.15`. We had to do this through `-Wl` as + the `hipcc` driver thought this was a source file. + - As '-L.' is not needed, it is omitted. + + - As there is no `make install`, we simply use the `MakeCp` EasyBlock, doing the edits to the Makefile in + `prebuiltopts`. + + Not that we copy the `libhip.so` file to the `lib` directory as that is the conventional + place to store shared objects, but it is not found there by `hpcat`, so we also create a + symbolic link to it in the `bin` subidrecitory. + + - Note that the accelerator target module should not be loaded when using the wrappers as the OpenMP offload + options cause a problem in one of the header files used. + diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/USER.md b/easybuild/easyconfigs/l/lumi-CPEtools/USER.md index e5861f297..f71407657 100644 --- a/easybuild/easyconfigs/l/lumi-CPEtools/USER.md +++ b/easybuild/easyconfigs/l/lumi-CPEtools/USER.md @@ -20,11 +20,78 @@ Commands provided: - `gpu_check` (from version 1.1 on): A hybrid MPI/OpenMP program that prints information about thread and GPU binding/mapping on Cray EX Bardpeak nodes as in LUMI-G, based on the ORNL hello_jobstep program. (AMD GPU nodes only) + +- `hpcat` (from version 1.2 on): Another HPC Affinity Tracker program. This program + is [developed by HPE](https://github.com/HewlettPackard/hpcat) and shows for + each MPI rank the core(s) that will be used (and per thread if `OMP_NUM_THREADS` + is set), which GPU(s) are accessible to the task and which network adapter will be + used, indicating the NUMA domain for each so that one can easily check if the resource + mapping is ideal. The various `*_check` programs are designed to test CPU and GPU binding in Slurm and are the LUST recommended way to experiment with those bindings. +## Some interactive examples + +The examples assume the appropriate software stack modules and `lumi-CPEtools` module +are loaded. The examples show one version of modules, but can work with others too. +You'll also need to add the appropriate `-A` flag to the `salloc` commands. + + +### `gpu_check` + +``` +salloc -N2 -pstandard-g -G 16 -t 10:00 +module load LUMI/24.03 partition/G lumi-CPEtools/1.2-cpeGNU-24.03 +srun -n16 -c7 bash -c 'ROCR_VISIBLE_DEVICES=$SLURM_LOCALID gpu_check -l' +srun -n16 -c7 \ + --cpu-bind=mask_cpu:0xfe000000000000,0xfe00000000000000,0xfe0000,0xfe000000,0xfe,0xfe00,0xfe00000000,0xfe0000000000 \ + bash -c 'ROCR_VISIBLE_DEVICES=$SLURM_LOCALID gpu_check -l' +``` + +Note that in the first `srun` command, the mapping of GPU binding is not optimal while +in the second `srunz` command it is. + + +### `hpcat` on a GPU node + +``` +salloc -N2 -pstandard-g -G 16 -t 10:00 +module load LUMI/24.03 partition/G lumi-CPEtools/1.2-cpeGNU-24.03 +srun -n16 -c7 bash -c 'ROCR_VISIBLE_DEVICES=$SLURM_LOCALID OMP_NUM_THREADS=7 hpcat' +srun -n16 -c7 \ + --cpu-bind=mask_cpu:0xfe000000000000,0xfe00000000000000,0xfe0000,0xfe000000,0xfe,0xfe00,0xfe00000000,0xfe0000000000 \ + bash -c 'ROCR_VISIBLE_DEVICES=$SLURM_LOCALID OMP_NUM_THREADS=7 hpcat' +srun -n16 -c7 \ + --cpu-bind=mask_cpu:0xfe000000000000,0xfe00000000000000,0xfe0000,0xfe000000,0xfe,0xfe00,0xfe00000000,0xfe0000000000 \ + bash -c 'ROCR_VISIBLE_DEVICES=$SLURM_LOCALID OMP_NUM_THREADS=7 OMP_PLACES=cores hpcat' +``` + +Note that in the first `srun` command, the mapping of resources is not very good. GPUs +don't map to their closest chiplet, and the network adapters are also linked based +on the CPU NUMA domain. In the second case, the mapping is optimal, but except for the +Cray compilers, the OpenMP threads can still move in the chiplet. In the last case, these +are also fixed with all compilers. + + +### `serial_check`, `omp_check`, `mpi_cehck` and `hybrid_check` + +``` +salloc -N1 -pstandard -t 10:00 +module load LUMI/24.03 partition/C lumi-CPEtools/1.2-cpeGNU-24.03 +srun -n1 -c1 serial_check +srun -n1 -c4 omp_check +srun -n4 -c1 mpi_check +srun -n4 -c4 hybrid_check +``` + +One big difference between these tools and `hpcat` is that this tool shows on which +core a thread is running at the moment that this is measured, while `hpcat` actually +shows the affinity mask, i.e., all cores that can be used by that thread. `gpu_check` +has the same limitation as `omp_check` etc. + + ## Acknowledgements The code for `hybrid_check` and its offsprings `serial_check`, `omp_check` and `mpi_check` is inspired @@ -39,3 +106,7 @@ without reworking on other AMD GPU systems or on NVIDIA GPU systems. The `lumi-CPEtools` code is developed by LUST in the [lumi-CPEtools repository on the LUMI supercomputer GitHub](https://github.com/Lumi-supercomputer/lumi-CPEtools). + +The `hpcat` program (lumi-CPEtools 1.2 and later) is +[developed by HPE](https://github.com/HewlettPackard/hpcat) and provided +unmodified. diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAMD-24.03.eb b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAMD-24.03.eb new file mode 100644 index 000000000..f5fb6a9db --- /dev/null +++ b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAMD-24.03.eb @@ -0,0 +1,145 @@ +easyblock = 'Bundle' + +local_CPEtools_version = '1.2' +local_hpcat_version = '0.4' + +name = 'lumi-CPEtools' +version = local_CPEtools_version + +homepage = 'https://www.lumi-supercomputer.eu' + +whatis = [ + 'Description: Various programs to experiment with starting processes and core affinity and analyse executables.', +] + +description = """ +The LUMI-CPEtools module provides various programs to experiment with starting +applications of various types and with core affinity and to show which Cray PE +libraries are used by an executable. It may be enhanced with additional +features in the future. + +Sources for all but hpcat can be accessed after loading the module in the +directory $EBROOTLUMIMINCPETOOLS/src. +""" + +docurls = [ + "Man pages, start with man lumi-CPEtools", +] + +toolchain = {'name': 'cpeAMD', 'version': '24.03'} +# Note: The Makefile is designed to work with the compiler variables as defined +# when usempi and openmp are both false, as the module contains code with and +# without MPI or OpenMP support, though it would probably still work if these +# are set to true as usually it does no harm to compile with the MPI wrappers +# or OpenMP options enabled even if the sources don't use these. +toolchainopts = {'usempi': False,'openmp': False, 'extra_cxxflags': '-std=c++11'} + +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', SYSTEM), # For make +] + +import os as local_os +local_partition = local_os.getenv('LUMI_STACK_PARTITION') + +start_dir = 'src' + +# +# Options for lumi-CPEtools +# +local_CPEtools_buildopts = 'CC=cc MPICC=cc CFLAGS="-O2" OMPFLAG="-fopenmp" CXX=CC MPICXX=CC CXXFLAGS="-O2 -std=c++11" ' +local_CPEtools_buildopts += 'DEFINES="-D_GNU_SOURCE=" ROCMDEFINES="-D__HIP_PLATFORM_AMD__" ' +local_CPEtools_buildopts += 'LIBS="" ROCMLIBS="-L${ROCM_PATH}/lib -lamdhip64" ' + +if local_partition == 'G': + local_CPEtools_buildopts += 'exe_cpu exe_gpu' +else: + local_CPEtools_buildopts += 'exe_cpu' + +# +# Options for hpcat +# +local_hpcat_prebuildopts = 'sed -i -e \'s|-O3|$(CFLAGS)|\' -e \'s|mpicc|$(CC)|\' -e \'s|gcc|$(CC)|\' Makefile && ' +local_hpcat_prebuildopts += 'sed -i -e \'s|-lhwloc|-Wl,/usr/lib64/libhwloc.so.15|\' -e \'s|-L. ||\' Makefile && ' +if local_partition == 'G': + local_hpcat_prebuildopts += 'module unload craype-accel-amd-gfx90a && ' + +if local_partition == 'G': + local_hpcat_buildopts = 'all amd' +else: + local_hpcat_buildopts = 'all' + +local_hpcat_files_to_copy = [ (['hpcat'], 'bin'), (['LICENSE'], 'share/licenses/hpcat') ] +if local_partition == 'G': + local_hpcat_files_to_copy += [ (['hpcathip.so'], 'lib') ] + + +default_easyblock = 'MakeCp' + +components = [ + ('lumi-CPEtools', local_CPEtools_version, { + 'sources': [{ # https://github.com/Lumi-supercomputer/lumi-CPEtools/archive/refs/tags/1.2.tar.gz + 'download_filename': '%(version)s.tar.gz', + 'filename': 'lumi-CPEtools-%(version)s.tar.gz', + 'source_urls': ['https://github.com/Lumi-supercomputer/%(name)s/archive/refs/tags'] + }], + 'checksums': [{ f'lumi-CPEtools-{version}.tar.gz': '89e1a01d9ecd30da53c70d866aa4a2d60853e16b2cc2da4b23589742f8d84e86' }], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': 'cd src && unset LIBRARY_PATH && ', # Cannot do the directory change via start_dir as otherwise the bin etc directories are not found in the copy step. + 'buildopts': local_CPEtools_buildopts, + 'files_to_copy': [ 'bin', 'man', 'src', 'README.md', (['LICENSE'], 'share/licenses/lumi-CPEtools') ], + }), + ( 'hpcat', local_hpcat_version, { + 'sources': [{ # HPCAT 0.4: https://github.com/HewlettPackard/hpcat/archive/refs/tags/v0.4.tar.gz + 'download_filename': 'v%(version)s.tar.gz', + 'filename': '%(name)s-%(version)s.tar.gz', + 'source_urls': ['https://github.com/HewlettPackard/hpcat/archive/refs/tags'] + },{ # Missing header files for hwloc from https://lumidata.eu/462000008:missing-libraries-headers/hwloc-devel-15SP5.tar.bz2 + 'filename': 'hwloc-devel-15SP5.tar.bz2', + 'source_urls': ['https://lumidata.eu/462000008:missing-libraries-headers'], + 'extract_cmd': 'cd %(name)s-%(version)s && tar xvf %s', + }], + 'checksums': [ + { f'hpcat-{local_hpcat_version}.tar.gz': 'dfad8649a5cc75c07deabbd5682b22fe0fdd650de14b382cccc5244a27b439ab' }, + { 'hwloc-devel-15SP5.tar.bz2': '3ca23fcaca9ba05e44e6816076230f356e658648b7e1747241f4bd3632011582' } + ], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': local_hpcat_prebuildopts, + 'buildopts': local_hpcat_buildopts, + 'files_to_copy': local_hpcat_files_to_copy, + }) +] + +if local_partition == 'G': + # Postinstall for hpcat + postinstallcmds = [ 'cd %(installdir)s/bin && ln -s %(installdir)s/lib/hpcathip.so' ] + +# +# Sanity checks +# + +local_exe = [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd', 'hpcat' ] +if local_partition == 'G': + local_exe += [ 'gpu_check', 'hpcat.so' ] + +local_files = [ 'share/licenses/lumi-CPEtools/LICENSE', 'share/licenses/hpcat/LICENSE' ] +if local_partition == 'G': + local_files += [ 'lib/hpcathip.so' ] + +sanity_check_paths = { + 'files': [ 'bin/%s' % x for x in [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd' ] ] + + local_files, + 'dirs': [ 'man/man1', 'src' ] +} + +sanity_check_commands = [ + 'xldd --help', + 'serial_check', + 'OMP_NUM_THREADS=4 omp_check', + 'mpi_check', + 'OMP_NUM_THREADS=4 hybrid_check', + 'hpcat --version', +] +if local_partition == 'G': + sanity_check_commands += [ 'gpu_check -h' ] + +moduleclass = 'devel' diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAOCC-24.03.eb b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAOCC-24.03.eb new file mode 100644 index 000000000..e2c7327aa --- /dev/null +++ b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeAOCC-24.03.eb @@ -0,0 +1,145 @@ +easyblock = 'Bundle' + +local_CPEtools_version = '1.2' +local_hpcat_version = '0.4' + +name = 'lumi-CPEtools' +version = local_CPEtools_version + +homepage = 'https://www.lumi-supercomputer.eu' + +whatis = [ + 'Description: Various programs to experiment with starting processes and core affinity and analyse executables.', +] + +description = """ +The LUMI-CPEtools module provides various programs to experiment with starting +applications of various types and with core affinity and to show which Cray PE +libraries are used by an executable. It may be enhanced with additional +features in the future. + +Sources for all but hpcat can be accessed after loading the module in the +directory $EBROOTLUMIMINCPETOOLS/src. +""" + +docurls = [ + "Man pages, start with man lumi-CPEtools", +] + +toolchain = {'name': 'cpeAOCC', 'version': '24.03'} +# Note: The Makefile is designed to work with the compiler variables as defined +# when usempi and openmp are both false, as the module contains code with and +# without MPI or OpenMP support, though it would probably still work if these +# are set to true as usually it does no harm to compile with the MPI wrappers +# or OpenMP options enabled even if the sources don't use these. +toolchainopts = {'usempi': False,'openmp': False} + +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', SYSTEM), # For make +] + +import os as local_os +local_partition = local_os.getenv('LUMI_STACK_PARTITION') + +start_dir = 'src' + +# +# Options for lumi-CPEtools +# +local_CPEtools_buildopts = 'CC=cc MPICC=cc CFLAGS="-O2" OMPFLAG="-fopenmp" CXX=CC MPICXX=CC CXXFLAGS="-O2" ' +local_CPEtools_buildopts += 'DEFINES="-D_GNU_SOURCE=" ROCMDEFINES="-D__HIP_PLATFORM_AMD__" ' +local_CPEtools_buildopts += 'LIBS="" ROCMLIBS="-L${ROCM_PATH}/lib -lamdhip64" ' + +if local_partition == 'G': + local_CPEtools_buildopts += 'exe_cpu exe_gpu' +else: + local_CPEtools_buildopts += 'exe_cpu' + +# +# Options for hpcat +# +local_hpcat_prebuildopts = 'sed -i -e \'s|-O3|$(CFLAGS)|\' -e \'s|mpicc|$(CC)|\' -e \'s|gcc|$(CC)|\' Makefile && ' +local_hpcat_prebuildopts += 'sed -i -e \'s|-lhwloc|-Wl,/usr/lib64/libhwloc.so.15|\' -e \'s|-L. ||\' Makefile && ' +if local_partition == 'G': + local_hpcat_prebuildopts += 'module unload craype-accel-amd-gfx90a && ' + +if local_partition == 'G': + local_hpcat_buildopts = 'all amd' +else: + local_hpcat_buildopts = 'all' + +local_hpcat_files_to_copy = [ (['hpcat'], 'bin'), (['LICENSE'], 'share/licenses/hpcat') ] +if local_partition == 'G': + local_hpcat_files_to_copy += [ (['hpcathip.so'], 'lib') ] + + +default_easyblock = 'MakeCp' + +components = [ + ('lumi-CPEtools', local_CPEtools_version, { + 'sources': [{ # https://github.com/Lumi-supercomputer/lumi-CPEtools/archive/refs/tags/1.2.tar.gz + 'download_filename': '%(version)s.tar.gz', + 'filename': 'lumi-CPEtools-%(version)s.tar.gz', + 'source_urls': ['https://github.com/Lumi-supercomputer/%(name)s/archive/refs/tags'] + }], + 'checksums': [{ f'lumi-CPEtools-{version}.tar.gz': '89e1a01d9ecd30da53c70d866aa4a2d60853e16b2cc2da4b23589742f8d84e86' }], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': 'cd src && ', # Cannot do this via start_dir as otherwise the bin etc directories are not found in the copy step. + 'buildopts': local_CPEtools_buildopts, + 'files_to_copy': [ 'bin', 'man', 'src', 'README.md', (['LICENSE'], 'share/licenses/lumi-CPEtools') ], + }), + ( 'hpcat', local_hpcat_version, { + 'sources': [{ # HPCAT 0.4: https://github.com/HewlettPackard/hpcat/archive/refs/tags/v0.4.tar.gz + 'download_filename': 'v%(version)s.tar.gz', + 'filename': '%(name)s-%(version)s.tar.gz', + 'source_urls': ['https://github.com/HewlettPackard/hpcat/archive/refs/tags'] + },{ # Missing header files for hwloc from https://lumidata.eu/462000008:missing-libraries-headers/hwloc-devel-15SP5.tar.bz2 + 'filename': 'hwloc-devel-15SP5.tar.bz2', + 'source_urls': ['https://lumidata.eu/462000008:missing-libraries-headers'], + 'extract_cmd': 'cd %(name)s-%(version)s && tar xvf %s', + }], + 'checksums': [ + { f'hpcat-{local_hpcat_version}.tar.gz': 'dfad8649a5cc75c07deabbd5682b22fe0fdd650de14b382cccc5244a27b439ab' }, + { 'hwloc-devel-15SP5.tar.bz2': '3ca23fcaca9ba05e44e6816076230f356e658648b7e1747241f4bd3632011582' } + ], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': local_hpcat_prebuildopts, + 'buildopts': local_hpcat_buildopts, + 'files_to_copy': local_hpcat_files_to_copy, + }) +] + +if local_partition == 'G': + # Postinstall for hpcat + postinstallcmds = [ 'cd %(installdir)s/bin && ln -s %(installdir)s/lib/hpcathip.so' ] + +# +# Sanity checks +# + +local_exe = [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd', 'hpcat' ] +if local_partition == 'G': + local_exe += [ 'gpu_check', 'hpcat.so' ] + +local_files = [ 'share/licenses/lumi-CPEtools/LICENSE', 'share/licenses/hpcat/LICENSE' ] +if local_partition == 'G': + local_files += [ 'lib/hpcathip.so' ] + +sanity_check_paths = { + 'files': [ 'bin/%s' % x for x in [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd' ] ] + + local_files, + 'dirs': [ 'man/man1', 'src' ] +} + +sanity_check_commands = [ + 'xldd --help', + 'serial_check', + 'OMP_NUM_THREADS=4 omp_check', + 'mpi_check', + 'OMP_NUM_THREADS=4 hybrid_check', + 'hpcat --version', +] +if local_partition == 'G': + sanity_check_commands += [ 'gpu_check -h' ] + +moduleclass = 'devel' diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeCray-24.03.eb b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeCray-24.03.eb new file mode 100644 index 000000000..59ae4c95d --- /dev/null +++ b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeCray-24.03.eb @@ -0,0 +1,145 @@ +easyblock = 'Bundle' + +local_CPEtools_version = '1.2' +local_hpcat_version = '0.4' + +name = 'lumi-CPEtools' +version = local_CPEtools_version + +homepage = 'https://www.lumi-supercomputer.eu' + +whatis = [ + 'Description: Various programs to experiment with starting processes and core affinity and analyse executables.', +] + +description = """ +The LUMI-CPEtools module provides various programs to experiment with starting +applications of various types and with core affinity and to show which Cray PE +libraries are used by an executable. It may be enhanced with additional +features in the future. + +Sources for all but hpcat can be accessed after loading the module in the +directory $EBROOTLUMIMINCPETOOLS/src. +""" + +docurls = [ + "Man pages, start with man lumi-CPEtools", +] + +toolchain = {'name': 'cpeCray', 'version': '24.03'} +# Note: The Makefile is designed to work with the compiler variables as defined +# when usempi and openmp are both false, as the module contains code with and +# without MPI or OpenMP support, though it would probably still work if these +# are set to true as usually it does no harm to compile with the MPI wrappers +# or OpenMP options enabled even if the sources don't use these. +toolchainopts = {'usempi': False,'openmp': False} + +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', SYSTEM), # For make +] + +import os as local_os +local_partition = local_os.getenv('LUMI_STACK_PARTITION') + +start_dir = 'src' + +# +# Options for lumi-CPEtools +# +local_CPEtools_buildopts = 'CC=cc MPICC=cc CFLAGS="-O2" OMPFLAG="-fopenmp" CXX=CC MPICXX=CC CXXFLAGS="-O2" ' +local_CPEtools_buildopts += 'DEFINES="-D_GNU_SOURCE=" ROCMDEFINES="-D__HIP_PLATFORM_AMD__" ' +local_CPEtools_buildopts += 'LIBS="" ROCMLIBS="-L${ROCM_PATH}/lib -lamdhip64" ' + +if local_partition == 'G': + local_CPEtools_buildopts += 'exe_cpu exe_gpu' +else: + local_CPEtools_buildopts += 'exe_cpu' + +# +# Options for hpcat +# +local_hpcat_prebuildopts = 'sed -i -e \'s|-O3|$(CFLAGS)|\' -e \'s|mpicc|$(CC)|\' -e \'s|gcc|$(CC)|\' Makefile && ' +local_hpcat_prebuildopts += 'sed -i -e \'s|-lhwloc|-Wl,/usr/lib64/libhwloc.so.15|\' -e \'s|-L. ||\' Makefile && ' +if local_partition == 'G': + local_hpcat_prebuildopts += 'module unload craype-accel-amd-gfx90a && ' + +if local_partition == 'G': + local_hpcat_buildopts = 'all amd' +else: + local_hpcat_buildopts = 'all' + +local_hpcat_files_to_copy = [ (['hpcat'], 'bin'), (['LICENSE'], 'share/licenses/hpcat') ] +if local_partition == 'G': + local_hpcat_files_to_copy += [ (['hpcathip.so'], 'lib') ] + + +default_easyblock = 'MakeCp' + +components = [ + ('lumi-CPEtools', local_CPEtools_version, { + 'sources': [{ # https://github.com/Lumi-supercomputer/lumi-CPEtools/archive/refs/tags/1.2.tar.gz + 'download_filename': '%(version)s.tar.gz', + 'filename': 'lumi-CPEtools-%(version)s.tar.gz', + 'source_urls': ['https://github.com/Lumi-supercomputer/%(name)s/archive/refs/tags'] + }], + 'checksums': [{ f'lumi-CPEtools-{version}.tar.gz': '89e1a01d9ecd30da53c70d866aa4a2d60853e16b2cc2da4b23589742f8d84e86' }], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': 'cd src && ', # Cannot do this via start_dir as otherwise the bin etc directories are not found in the copy step. + 'buildopts': local_CPEtools_buildopts, + 'files_to_copy': [ 'bin', 'man', 'src', 'README.md', (['LICENSE'], 'share/licenses/lumi-CPEtools') ], + }), + ( 'hpcat', local_hpcat_version, { + 'sources': [{ # HPCAT 0.4: https://github.com/HewlettPackard/hpcat/archive/refs/tags/v0.4.tar.gz + 'download_filename': 'v%(version)s.tar.gz', + 'filename': '%(name)s-%(version)s.tar.gz', + 'source_urls': ['https://github.com/HewlettPackard/hpcat/archive/refs/tags'] + },{ # Missing header files for hwloc from https://lumidata.eu/462000008:missing-libraries-headers/hwloc-devel-15SP5.tar.bz2 + 'filename': 'hwloc-devel-15SP5.tar.bz2', + 'source_urls': ['https://lumidata.eu/462000008:missing-libraries-headers'], + 'extract_cmd': 'cd %(name)s-%(version)s && tar xvf %s', + }], + 'checksums': [ + { f'hpcat-{local_hpcat_version}.tar.gz': 'dfad8649a5cc75c07deabbd5682b22fe0fdd650de14b382cccc5244a27b439ab' }, + { 'hwloc-devel-15SP5.tar.bz2': '3ca23fcaca9ba05e44e6816076230f356e658648b7e1747241f4bd3632011582' } + ], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': local_hpcat_prebuildopts, + 'buildopts': local_hpcat_buildopts, + 'files_to_copy': local_hpcat_files_to_copy, + }) +] + +if local_partition == 'G': + # Postinstall for hpcat + postinstallcmds = [ 'cd %(installdir)s/bin && ln -s %(installdir)s/lib/hpcathip.so' ] + +# +# Sanity checks +# + +local_exe = [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd', 'hpcat' ] +if local_partition == 'G': + local_exe += [ 'gpu_check', 'hpcat.so' ] + +local_files = [ 'share/licenses/lumi-CPEtools/LICENSE', 'share/licenses/hpcat/LICENSE' ] +if local_partition == 'G': + local_files += [ 'lib/hpcathip.so' ] + +sanity_check_paths = { + 'files': [ 'bin/%s' % x for x in [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd' ] ] + + local_files, + 'dirs': [ 'man/man1', 'src' ] +} + +sanity_check_commands = [ + 'xldd --help', + 'serial_check', + 'OMP_NUM_THREADS=4 omp_check', + 'mpi_check', + 'OMP_NUM_THREADS=4 hybrid_check', + 'hpcat --version', +] +if local_partition == 'G': + sanity_check_commands += [ 'gpu_check -h' ] + +moduleclass = 'devel' diff --git a/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeGNU-24.03.eb b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeGNU-24.03.eb new file mode 100644 index 000000000..fffbdbc05 --- /dev/null +++ b/easybuild/easyconfigs/l/lumi-CPEtools/lumi-CPEtools-1.2-cpeGNU-24.03.eb @@ -0,0 +1,145 @@ +easyblock = 'Bundle' + +local_CPEtools_version = '1.2' +local_hpcat_version = '0.4' + +name = 'lumi-CPEtools' +version = local_CPEtools_version + +homepage = 'https://www.lumi-supercomputer.eu' + +whatis = [ + 'Description: Various programs to experiment with starting processes and core affinity and analyse executables.', +] + +description = """ +The LUMI-CPEtools module provides various programs to experiment with starting +applications of various types and with core affinity and to show which Cray PE +libraries are used by an executable. It may be enhanced with additional +features in the future. + +Sources for all but hpcat can be accessed after loading the module in the +directory $EBROOTLUMIMINCPETOOLS/src. +""" + +docurls = [ + "Man pages, start with man lumi-CPEtools", +] + +toolchain = {'name': 'cpeGNU', 'version': '24.03'} +# Note: The Makefile is designed to work with the compiler variables as defined +# when usempi and openmp are both false, as the module contains code with and +# without MPI or OpenMP support, though it would probably still work if these +# are set to true as usually it does no harm to compile with the MPI wrappers +# or OpenMP options enabled even if the sources don't use these. +toolchainopts = {'usempi': False,'openmp': False} + +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', SYSTEM), # For make +] + +import os as local_os +local_partition = local_os.getenv('LUMI_STACK_PARTITION') + +start_dir = 'src' + +# +# Options for lumi-CPEtools +# +local_CPEtools_buildopts = 'CC=cc MPICC=cc CFLAGS="-O2" OMPFLAG="-fopenmp" CXX=CC MPICXX=CC CXXFLAGS="-O2" ' +local_CPEtools_buildopts += 'DEFINES="" ROCMDEFINES="-D__HIP_PLATFORM_AMD__" ' +local_CPEtools_buildopts += 'LIBS="" ROCMLIBS="-L${ROCM_PATH}/lib -lamdhip64" ' + +if local_partition == 'G': + local_CPEtools_buildopts += 'exe_cpu exe_gpu' +else: + local_CPEtools_buildopts += 'exe_cpu' + +# +# Options for hpcat +# +local_hpcat_prebuildopts = 'sed -i -e \'s|-O3|$(CFLAGS)|\' -e \'s|mpicc|$(CC)|\' -e \'s|gcc|$(CC)|\' Makefile && ' +local_hpcat_prebuildopts += 'sed -i -e \'s|-lhwloc|-Wl,/usr/lib64/libhwloc.so.15|\' -e \'s|-L. ||\' Makefile && ' +if local_partition == 'G': + local_hpcat_prebuildopts += 'module unload craype-accel-amd-gfx90a && ' + +if local_partition == 'G': + local_hpcat_buildopts = 'all amd' +else: + local_hpcat_buildopts = 'all' + +local_hpcat_files_to_copy = [ (['hpcat'], 'bin'), (['LICENSE'], 'share/licenses/hpcat') ] +if local_partition == 'G': + local_hpcat_files_to_copy += [ (['hpcathip.so'], 'lib') ] + + +default_easyblock = 'MakeCp' + +components = [ + ('lumi-CPEtools', local_CPEtools_version, { + 'sources': [{ # https://github.com/Lumi-supercomputer/lumi-CPEtools/archive/refs/tags/1.2.tar.gz + 'download_filename': '%(version)s.tar.gz', + 'filename': 'lumi-CPEtools-%(version)s.tar.gz', + 'source_urls': ['https://github.com/Lumi-supercomputer/%(name)s/archive/refs/tags'] + }], + 'checksums': [{ f'lumi-CPEtools-{version}.tar.gz': '89e1a01d9ecd30da53c70d866aa4a2d60853e16b2cc2da4b23589742f8d84e86' }], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': 'cd src && ', # Cannot do this via start_dir as otherwise the bin etc directories are not found in the copy step. + 'buildopts': local_CPEtools_buildopts, + 'files_to_copy': [ 'bin', 'man', 'src', 'README.md', (['LICENSE'], 'share/licenses/lumi-CPEtools') ], + }), + ( 'hpcat', local_hpcat_version, { + 'sources': [{ # HPCAT 0.4: https://github.com/HewlettPackard/hpcat/archive/refs/tags/v0.4.tar.gz + 'download_filename': 'v%(version)s.tar.gz', + 'filename': '%(name)s-%(version)s.tar.gz', + 'source_urls': ['https://github.com/HewlettPackard/hpcat/archive/refs/tags'] + },{ # Missing header files for hwloc from https://lumidata.eu/462000008:missing-libraries-headers/hwloc-devel-15SP5.tar.bz2 + 'filename': 'hwloc-devel-15SP5.tar.bz2', + 'source_urls': ['https://lumidata.eu/462000008:missing-libraries-headers'], + 'extract_cmd': 'cd %(name)s-%(version)s && tar xvf %s', + }], + 'checksums': [ + { f'hpcat-{local_hpcat_version}.tar.gz': 'dfad8649a5cc75c07deabbd5682b22fe0fdd650de14b382cccc5244a27b439ab' }, + { 'hwloc-devel-15SP5.tar.bz2': '3ca23fcaca9ba05e44e6816076230f356e658648b7e1747241f4bd3632011582' } + ], + 'start_dir': '%(name)s-%(version)s', + 'prebuildopts': local_hpcat_prebuildopts, + 'buildopts': local_hpcat_buildopts, + 'files_to_copy': local_hpcat_files_to_copy, + }) +] + +if local_partition == 'G': + # Postinstall for hpcat + postinstallcmds = [ 'cd %(installdir)s/bin && ln -s %(installdir)s/lib/hpcathip.so' ] + +# +# Sanity checks +# + +local_exe = [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd', 'hpcat' ] +if local_partition == 'G': + local_exe += [ 'gpu_check', 'hpcat.so' ] + +local_files = [ 'share/licenses/lumi-CPEtools/LICENSE', 'share/licenses/hpcat/LICENSE' ] +if local_partition == 'G': + local_files += [ 'lib/hpcathip.so' ] + +sanity_check_paths = { + 'files': [ 'bin/%s' % x for x in [ 'serial_check', 'omp_check', 'mpi_check', 'hybrid_check', 'xldd' ] ] + + local_files, + 'dirs': [ 'man/man1', 'src' ] +} + +sanity_check_commands = [ + 'xldd --help', + 'serial_check', + 'OMP_NUM_THREADS=4 omp_check', + 'mpi_check', + 'OMP_NUM_THREADS=4 hybrid_check', + 'hpcat --version', +] +if local_partition == 'G': + sanity_check_commands += [ 'gpu_check -h' ] + +moduleclass = 'devel' diff --git a/easybuild/easystacks/LUMI-24.03/production-G-cpeAMD-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-G-cpeAMD-24.03.yaml index 73e70bf25..8e5bc55f1 100644 --- a/easybuild/easystacks/LUMI-24.03/production-G-cpeAMD-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-G-cpeAMD-24.03.yaml @@ -104,6 +104,7 @@ easyconfigs: # - ESMF-8.6.0-cpeAMD-24.03.eb # Some apps - lumi-CPEtools-1.1-cpeAMD-24.03.eb + - lumi-CPEtools-1.2-cpeAMD-24.03.eb # - VampirServer # - Score-P # diff --git a/easybuild/easystacks/LUMI-24.03/production-G-cpeCray-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-G-cpeCray-24.03.yaml index 1288ef622..6756947b4 100644 --- a/easybuild/easystacks/LUMI-24.03/production-G-cpeCray-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-G-cpeCray-24.03.yaml @@ -106,6 +106,7 @@ easyconfigs: # - ESMF-8.6.0-cpeCray-24.03.eb # Some apps - lumi-CPEtools-1.1-cpeCray-24.03.eb + - lumi-CPEtools-1.2-cpeCray-24.03.eb # - VampirServer # - Score-P # diff --git a/easybuild/easystacks/LUMI-24.03/production-G-cpeGNU-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-G-cpeGNU-24.03.yaml index 5c2f5f54c..70260dec6 100644 --- a/easybuild/easystacks/LUMI-24.03/production-G-cpeGNU-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-G-cpeGNU-24.03.yaml @@ -106,6 +106,7 @@ easyconfigs: - SuiteSparse-5.13.0-cpeGNU-24.03-OpenMP.eb # Some apps - lumi-CPEtools-1.1-cpeGNU-24.03.eb + - lumi-CPEtools-1.2-cpeGNU-24.03.eb # - VampirServer # - Score-P # diff --git a/easybuild/easystacks/LUMI-24.03/production-LC-cpeAOCC-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-LC-cpeAOCC-24.03.yaml index 1716f1db2..dcaa861a0 100644 --- a/easybuild/easystacks/LUMI-24.03/production-LC-cpeAOCC-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-LC-cpeAOCC-24.03.yaml @@ -106,6 +106,7 @@ easyconfigs: # - ESMF-8.6.0-cpeAOCC-24.03.eb # Some apps - lumi-CPEtools-1.1-cpeAOCC-24.03.eb + - lumi-CPEtools-1.2-cpeAOCC-24.03.eb # - VampirServer # - Score-P # diff --git a/easybuild/easystacks/LUMI-24.03/production-LC-cpeCray-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-LC-cpeCray-24.03.yaml index c8d7fc1e3..2459ea8ea 100644 --- a/easybuild/easystacks/LUMI-24.03/production-LC-cpeCray-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-LC-cpeCray-24.03.yaml @@ -106,6 +106,7 @@ easyconfigs: - SuiteSparse-5.13.0-cpeCray-24.03-OpenMP.eb # Some apps - lumi-CPEtools-1.1-cpeCray-24.03.eb + - lumi-CPEtools-1.2-cpeCray-24.03.eb # - VampirServer # - Score-P # diff --git a/easybuild/easystacks/LUMI-24.03/production-LC-cpeGNU-24.03.yaml b/easybuild/easystacks/LUMI-24.03/production-LC-cpeGNU-24.03.yaml index cadfaffd6..9cff7c2c4 100644 --- a/easybuild/easystacks/LUMI-24.03/production-LC-cpeGNU-24.03.yaml +++ b/easybuild/easystacks/LUMI-24.03/production-LC-cpeGNU-24.03.yaml @@ -103,6 +103,7 @@ easyconfigs: - SuiteSparse-5.13.0-cpeGNU-24.03-OpenMP.eb # Some apps - lumi-CPEtools-1.1-cpeGNU-24.03.eb + - lumi-CPEtools-1.2-cpeGNU-24.03.eb # - VampirServer # - Score-P #