Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify vecmem/alpaka interaction #670

Merged
merged 13 commits into from
Nov 11, 2024
103 changes: 103 additions & 0 deletions device/alpaka/include/traccc/alpaka/utils/vecmem_types.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not absolutely necessary to do it in this PR, but this header should eventually be in the src/ directory. Generally we'll have to avoid any public dependence on the Alpaka headers in this library. Just like how we avoid any public CUDA or SYCL dependence in those libraries. (Not counting some coding mishaps, that still appear in some places...)

* traccc library, part of the ACTS project (R&D line)
*
* (c) 2024 CERN for the benefit of the ACTS project
*
* Mozilla Public License Version 2.0
*/

#pragma once

// VecMem include(s).
#ifdef ALPAKA_ACC_GPU_CUDA_ENABLED
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the exact logic should be here... 🤔

Generally, I think I'd try to organise this code like:

// define the generic host_device_types trait

#ifdef TRACCC_HAVE_VECMEM_CUDA
#include <vecmem/memory/cuda/device_memory_resource.hpp>
...
template <>
struct host_device_types<::alpaka::TagGpuCudaRt> {
    using device_memory_resource = ::vecmem::cuda::host_memory_resource;
...
#endif
#ifdef TRACCC_HAVE_VECMEM_SYCL
#include <vecmem/memory/sycl/device_memory_resource.hpp>
...
#endif

I.e. we should just depend on all the headers/libraries in this header that are available to us during the build. And declare the trait specializations for all the available types / environments.

The decision about which environment is being chosen for the Alpaka build, should be a separate one. That choice should be able to select between all the available options, presented by this header.

And yes, the TRACCC_HAVE_VECMEM_CUDA, etc. definitions would have to come from CMake in some clever way.

#include <vecmem/memory/cuda/device_memory_resource.hpp>
#include <vecmem/memory/cuda/host_memory_resource.hpp>
#include <vecmem/memory/cuda/managed_memory_resource.hpp>
#include <vecmem/utils/cuda/copy.hpp>

#elif defined(ALPAKA_ACC_GPU_HIP_ENABLED)
#include <vecmem/memory/hip/device_memory_resource.hpp>
#include <vecmem/memory/hip/host_memory_resource.hpp>
#include <vecmem/memory/hip/managed_memory_resource.hpp>
#include <vecmem/utils/hip/copy.hpp>

#elif defined(ALPAKA_ACC_SYCL_ENABLED)
#include <vecmem/memory/sycl/device_memory_resource.hpp>
#include <vecmem/memory/sycl/host_memory_resource.hpp>
#include <vecmem/utils/sycl/copy.hpp>

#else
#include <vecmem/memory/memory_resource.hpp>
#include <vecmem/utils/copy.hpp>
#endif

#include <alpaka/alpaka.hpp>

// Forward declarations so we can compile the types below
namespace vecmem {
class host_memory_resource;
class copy;
namespace cuda {
class host_memory_resource;
class device_memory_resource;
class managed_memory_resource;
class copy;
} // namespace cuda
namespace hip {
class host_memory_resource;
class device_memory_resource;
class managed_memory_resource;
class copy;
} // namespace hip
namespace sycl {
class host_memory_resource;
class device_memory_resource;
class managed_memory_resource;
class copy;
} // namespace sycl
} // namespace vecmem

namespace traccc::alpaka::vecmem {
// For all CPU accelerators (except SYCL), just use host
template <typename T>
struct host_device_types {
using device_memory_resource = ::vecmem::host_memory_resource;
using host_memory_resource = ::vecmem::host_memory_resource;
using managed_memory_resource = ::vecmem::host_memory_resource;
using device_copy = ::vecmem::copy;
};
template <>
struct host_device_types<::alpaka::TagGpuCudaRt> {
using device_memory_resource = ::vecmem::cuda::host_memory_resource;
using host_memory_resource = ::vecmem::cuda::host_memory_resource;
using managed_memory_resource = ::vecmem::cuda::managed_memory_resource;
using device_copy = ::vecmem::cuda::copy;
};
template <>
struct host_device_types<::alpaka::TagGpuHipRt> {
using device_memory_resource = ::vecmem::hip::device_memory_resource;
using host_memory_resource = ::vecmem::hip::host_memory_resource;
using managed_memory_resource = ::vecmem::hip::managed_memory_resource;
using device_copy = ::vecmem::hip::copy;
};
template <>
struct host_device_types<::alpaka::TagCpuSycl> {
using device_memory_resource = ::vecmem::sycl::device_memory_resource;
using host_memory_resource = ::vecmem::sycl::host_memory_resource;
using managed_memory_resource = ::vecmem::sycl::host_memory_resource;
using device_copy = ::vecmem::sycl::copy;
};
template <>
struct host_device_types<::alpaka::TagFpgaSyclIntel> {
using device_memory_resource = ::vecmem::sycl::device_memory_resource;
using host_memory_resource = ::vecmem::sycl::host_memory_resource;
using managed_memory_resource = ::vecmem::sycl::host_memory_resource;
using device_copy = ::vecmem::sycl::copy;
};
template <>
struct host_device_types<::alpaka::TagGpuSyclIntel> {
using device_memory_resource = ::vecmem::sycl::device_memory_resource;
using host_memory_resource = ::vecmem::sycl::host_memory_resource;
using device_copy = ::vecmem::sycl::copy;
};
} // namespace traccc::alpaka::vecmem
15 changes: 14 additions & 1 deletion examples/run/alpaka/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,21 @@

set(EXTRA_LIBS)

set(TRACCC_ALPAKA_EXAMPLE_SOURCES
seq_example_alpaka.cpp
seeding_example_alpaka.cpp
)

if(alpaka_ACC_GPU_CUDA_ENABLE)
enable_language(CUDA)
include( traccc-compiler-options-cuda )
set_source_files_properties(${TRACCC_ALPAKA_EXAMPLE_SOURCES} PROPERTIES LANGUAGE CUDA)

list (APPEND EXTRA_LIBS vecmem::cuda traccc::cuda)
elseif(alpaka_ACC_GPU_HIP_ENABLE)
enable_language(HIP)
find_package( HIPToolkit REQUIRED )

set_source_files_properties(${TRACCC_ALPAKA_EXAMPLE_SOURCES} PROPERTIES LANGUAGE HIP)
list(APPEND EXTRA_LIBS HIP::hiprt vecmem::hip)
endif()

Expand All @@ -27,3 +33,10 @@ traccc_add_executable( seq_example_alpaka "seq_example_alpaka.cpp"
traccc_add_executable( seeding_example_alpaka "seeding_example_alpaka.cpp"
LINK_LIBRARIES ${LIBRARIES} )

#Can only do this once target is defined, so need another if here
if(alpaka_ACC_GPU_HIP_ENABLE)
set_target_properties( traccc_seq_example_alpaka PROPERTIES
POSITION_INDEPENDENT_CODE TRUE )
set_target_properties( traccc_seeding_example_alpaka PROPERTIES
POSITION_INDEPENDENT_CODE TRUE )
endif()
49 changes: 14 additions & 35 deletions examples/run/alpaka/seeding_example_alpaka.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -34,31 +34,15 @@
#include "traccc/seeding/track_params_estimation.hpp"

// Detray include(s).
#include "alpaka/example/ExampleDefaultAcc.hpp"
#include "detray/core/detector.hpp"
#include "detray/core/detector_metadata.hpp"
#include "detray/detectors/bfield.hpp"
#include "detray/io/frontend/detector_reader.hpp"
#include "detray/navigation/navigator.hpp"
#include "detray/propagator/propagator.hpp"
#include "detray/propagator/rk_stepper.hpp"

// VecMem include(s).
#ifdef ALPAKA_ACC_GPU_CUDA_ENABLED
#include <vecmem/memory/cuda/device_memory_resource.hpp>
#include <vecmem/memory/cuda/host_memory_resource.hpp>
#include <vecmem/memory/cuda/managed_memory_resource.hpp>
#include <vecmem/utils/cuda/copy.hpp>
#endif

#ifdef ALPAKA_ACC_GPU_HIP_ENABLED
#include <vecmem/memory/hip/device_memory_resource.hpp>
#include <vecmem/memory/hip/host_memory_resource.hpp>
#include <vecmem/memory/hip/managed_memory_resource.hpp>
#include <vecmem/utils/hip/copy.hpp>
#endif

#include <vecmem/memory/host_memory_resource.hpp>
#include <vecmem/utils/copy.hpp>
#include "traccc/alpaka/utils/vecmem_types.hpp"

// System include(s).
#include <exception>
Expand All @@ -75,24 +59,19 @@ int seq_run(const traccc::opts::track_seeding& seeding_opts,
const traccc::opts::performance& performance_opts,
const traccc::opts::accelerator& accelerator_opts) {

#if defined(ALPAKA_ACC_GPU_CUDA_ENABLED)
vecmem::cuda::copy copy;
vecmem::cuda::host_memory_resource host_mr;
vecmem::cuda::device_memory_resource device_mr;
vecmem::cuda::managed_memory_resource mng_mr;
traccc::memory_resource mr{device_mr, &host_mr};
#elif defined(ALPAKA_ACC_GPU_HIP_ENABLED)
vecmem::hip::copy copy;
vecmem::hip::host_memory_resource host_mr;
vecmem::hip::device_memory_resource device_mr;
vecmem::hip::managed_memory_resource mng_mr;
using Dim = ::alpaka::DimInt<1>;
using Idx = uint32_t;

using Acc = ::alpaka::ExampleDefaultAcc<Dim, Idx>;
traccc::alpaka::vecmem::host_device_types<
::alpaka::trait::AccToTag<Acc>::type>::device_copy copy;
traccc::alpaka::vecmem::host_device_types<
::alpaka::trait::AccToTag<Acc>::type>::host_memory_resource host_mr;
traccc::alpaka::vecmem::host_device_types<
::alpaka::trait::AccToTag<Acc>::type>::device_memory_resource device_mr;
traccc::alpaka::vecmem::host_device_types<
::alpaka::trait::AccToTag<Acc>::type>::managed_memory_resource mng_mr;
traccc::memory_resource mr{device_mr, &host_mr};
#else
vecmem::copy copy;
vecmem::host_memory_resource host_mr;
vecmem::host_memory_resource mng_mr;
traccc::memory_resource mr{host_mr, &host_mr};
#endif

// Performance writer
traccc::seeding_performance_writer sd_performance_writer(
Expand Down
44 changes: 13 additions & 31 deletions examples/run/alpaka/seq_example_alpaka.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,13 @@
*/

// Project include(s).
#include "alpaka/example/ExampleDefaultAcc.hpp"
#include "traccc/alpaka/clusterization/clusterization_algorithm.hpp"
#include "traccc/alpaka/clusterization/measurement_sorting_algorithm.hpp"
#include "traccc/alpaka/seeding/seeding_algorithm.hpp"
#include "traccc/alpaka/seeding/spacepoint_formation_algorithm.hpp"
#include "traccc/alpaka/seeding/track_params_estimation.hpp"
#include "traccc/alpaka/utils/vecmem_types.hpp"
#include "traccc/clusterization/clusterization_algorithm.hpp"
#include "traccc/efficiency/seeding_performance_writer.hpp"
#include "traccc/io/read_cells.hpp"
Expand All @@ -31,22 +33,6 @@
#include "traccc/seeding/silicon_pixel_spacepoint_formation_algorithm.hpp"
#include "traccc/seeding/track_params_estimation.hpp"

// VecMem include(s).
#ifdef ALPAKA_ACC_GPU_CUDA_ENABLED
#include <vecmem/memory/cuda/device_memory_resource.hpp>
#include <vecmem/memory/cuda/host_memory_resource.hpp>
#include <vecmem/utils/cuda/copy.hpp>
#endif

#ifdef ALPAKA_ACC_GPU_HIP_ENABLED
#include <vecmem/memory/hip/device_memory_resource.hpp>
#include <vecmem/memory/hip/host_memory_resource.hpp>
#include <vecmem/utils/hip/copy.hpp>
#endif

#include <vecmem/memory/host_memory_resource.hpp>
#include <vecmem/utils/copy.hpp>

// System include(s).
#include <exception>
#include <iomanip>
Expand Down Expand Up @@ -74,22 +60,18 @@ int seq_run(const traccc::opts::detector& detector_opts,
const traccc::vector3 field_vec = {0.f, 0.f,
seeding_opts.seedfinder.bFieldInZ};

using Dim = ::alpaka::DimInt<1>;
using Idx = uint32_t;

using Acc = ::alpaka::ExampleDefaultAcc<Dim, Idx>;
// Memory resources used by the application.
vecmem::host_memory_resource host_mr;
#ifdef ALPAKA_ACC_GPU_CUDA_ENABLED
vecmem::cuda::copy copy;
vecmem::cuda::host_memory_resource cuda_host_mr;
vecmem::cuda::device_memory_resource device_mr;
traccc::memory_resource mr{device_mr, &cuda_host_mr};
#elif ALPAKA_ACC_GPU_HIP_ENABLED
vecmem::hip::copy copy;
vecmem::hip::host_memory_resource hip_host_mr;
vecmem::hip::device_memory_resource hip_device_mr;
traccc::memory_resource mr{hip_device_mr, &hip_host_mr};
#else
vecmem::copy copy;
traccc::memory_resource mr{host_mr, &host_mr};
#endif
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::host_memory_resource host_mr;
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::device_copy copy;
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::device_memory_resource device_mr;
traccc::memory_resource mr{device_mr, &host_mr};

// Construct the detector description object.
traccc::silicon_detector_description::host host_det_descr{host_mr};
Expand Down
38 changes: 16 additions & 22 deletions tests/alpaka/test_cca.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,43 +7,37 @@

#include <gtest/gtest.h>

#include <alpaka/alpaka.hpp>
#include <alpaka/example/ExampleDefaultAcc.hpp>
#include <functional>
#include <vecmem/memory/host_memory_resource.hpp>

#include "tests/cca_test.hpp"
#include "traccc/alpaka/clusterization/clusterization_algorithm.hpp"
#include "traccc/alpaka/utils/vecmem_types.hpp"
#include "traccc/geometry/silicon_detector_description.hpp"

#if defined(ALPAKA_ACC_GPU_CUDA_ENABLED)
#include <vecmem/memory/cuda/device_memory_resource.hpp>
#include <vecmem/memory/cuda/host_memory_resource.hpp>
#include <vecmem/utils/cuda/copy.hpp>
#elif defined(ALPAKA_ACC_GPU_HIP_ENABLED)
#include <vecmem/memory/hip/device_memory_resource.hpp>
#include <vecmem/memory/hip/host_memory_resource.hpp>
#include <vecmem/utils/hip/copy.hpp>
#endif

namespace {

// template <TAccTag>
cca_function_t get_f_with(traccc::clustering_config cfg) {
return [cfg](const traccc::edm::silicon_cell_collection::host& cells,
const traccc::silicon_detector_description::host& dd) {
std::map<traccc::geometry_id, vecmem::vector<traccc::measurement>>
result;

vecmem::host_memory_resource host_mr;

#if defined(ALPAKA_ACC_GPU_CUDA_ENABLED)
vecmem::cuda::copy copy;
vecmem::cuda::device_memory_resource device_mr;
#elif defined(ALPAKA_ACC_GPU_HIP_ENABLED)
vecmem::hip::copy copy;
vecmem::hip::device_memory_resource device_mr;
#else
vecmem::copy copy;
vecmem::host_memory_resource device_mr;
#endif
using namespace alpaka;
using Dim = DimInt<1>;
using Idx = uint32_t;

using Acc = ExampleDefaultAcc<Dim, Idx>;
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::host_memory_resource host_mr;
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::device_copy copy;
traccc::alpaka::vecmem::host_device_types<
alpaka::trait::AccToTag<Acc>::type>::device_memory_resource
device_mr;

traccc::alpaka::clusterization_algorithm cc({device_mr}, copy, cfg);

Expand Down
Loading