From c5d573e0d513cc2fca9042cce304da0aea4370bd Mon Sep 17 00:00:00 2001 From: Bernhard Manfred Gruber Date: Mon, 20 Jul 2026 15:49:45 +0200 Subject: [PATCH 1/2] Document CUB environments Fixes: #7712 Fixes: #7276 --- docs/cub/environments.rst | 179 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 docs/cub/environments.rst diff --git a/docs/cub/environments.rst b/docs/cub/environments.rst new file mode 100644 index 00000000000..15eebd115f3 --- /dev/null +++ b/docs/cub/environments.rst @@ -0,0 +1,179 @@ +.. _cub-environments: + +Execution environments +================================================================================ + +.. + TODO(bgruber): We should generalize the std::execution::* parts into a new document in libcu++ and link to it here + +Almost all CUB device-wide algorithms accept an execution environment as their last argument. +Environments are objects responding to queries which return values. +A "query" in this case is usually a (tag) type naming the query, for example `cuda::execution::__get_tuning_t`. +Environments can also be passed to certain customization point objects (CPOs), +which will extract values from the environment. +For example, the CPO `cuda::get_stream` will extract the stream from the given environment. +The purpose of an environment is to provide properties that govern the execution of an algorithm, +like the used CUDA stream, memory resource, required determinism, requested tuning, etc. + +Some types are already environment themselves. +For example, `cudaStream_t` or `cuda::stream_ref` are environments responding to the `cuda::get_stream` CPO. +Similarly, several memory resource types are environments responding to the `cuda::mr::get_memory_resource` CPO. +Objects of such types can be passed directly to a CUB device-scope algorithm, +and the algorithm will query them for the stream or memory resource to use. +No wrapping of the value is necessary. + + +Constructing environments +-------------------------------------------------------------------------------- + +If the type of a value is not queryable directly, +a simple environment can be constructed using `cuda::std::execution::prop` given a value and a query. +For example: + +.. code-block:: c++ + + cudaStream_t stream = ...; + auto env = cuda::std::execution::prop(cuda::get_stream, stream); + +Builds an environment `env` that responds to the `cuda::get_stream` query with the value of `stream`. +If environments with more properties are needed, +they can be constructed from other environments using `cuda::std::execution::env`. +For example: + +.. code-block:: c++ + + cudaStream_t stream = ...; + auto stream_env = cuda::std::execution::prop(cuda::get_stream, stream); + cuda::mr::resource_ref<> mr = ...; + auto mr_env = cuda::std::execution::prop{cuda::mr::get_memory_resource, cuda::mr::resource_ref<>{alloc}} + auto env = cuda::std::execution::env(stream_env, mr_env); + +Here, `env` is constructed from two other environments, `stream_env` and `mr_env`. +`env` now responds to both the `cuda::get_stream` and `cuda::mr::get_memory_resource` queries. +Notice that wrapping environments with `cuda::std::execution::env` does not nest them. + +However, because `cudaStream_t` and `cuda::mr::resource_ref<>` are already environments themselves, +the above can be simplified to: + +.. code-block:: c++ + + cudaStream_t stream = ...; + cuda::mr::resource_ref<> mr = ...; + auto env = cuda::std::execution::env(stream, mr); + +CUB also has a few more convenience functions for constructing environments, +like `cuda::execution::tune(...)` or `cuda::execution::require(...)`, +which build environments containing tuning policies and execution requirements, respectively. +These environments can be freely composed with others again: + +.. code-block:: c++ + + cudaStream_t stream = ...; + cuda::mr::resource_ref<> mr = ...; + auto env = cuda::std::execution::env(stream, mr, cuda::execution::tune(...)); + auto env2 = cuda::std::execution::env(env, cuda::execution::require(...)); + +Here, `env` is an environment containing a stream, a memory resource, and tuning policies. +`env2` additionally contains an execution requirement. +A shorter way to define `env2` would be: + +.. code-block:: c++ + + cudaStream_t stream = ...; + cuda::mr::resource_ref<> mr = ...; + auto env2 = cuda::std::execution::env( + stream, mr, cuda::execution::tune(...), cuda::execution::require(...)); + + +Querying environments +-------------------------------------------------------------------------------- + +There are two ways to get a value out of an environment again. +Using the query (tag), we can query an environment directly using `cuda::std::execution::__query_or`: + +.. code-block:: c++ + + auto env = cuda::std::execution::env(stream, mr); // from above example + cudaStream_t fallback_stream = ...; + cudaStream_t stream = cuda::std::execution::__query_or(env, cuda::get_stream, fallback_stream); + +`__query_or` will look for a value in the environment responding to the `cuda::get_stream` query. +If no such value is found, it will return the fallback value, `fallback_stream`, instead. + +.. + TODO(bgruber) Does cuda::std::execution::__query_or(cudaStream_t{}, cuda::get_stream, fallback_stream); work? + +... + TODO(bgruber): Should we mention `__query_result_or_t`? + +Alternatively, we can use the CPO `cuda::get_stream` to query an environment for a stream: + +.. code-block:: c++ + + auto env = cuda::std::execution::env(stream, mr); // from above example + cudaStream_t stream = cuda::get_stream(env); + +If `env` would not contain any value responding to the `cuda::get_stream` query, the above would not compile. +Using a CPO to query an environment is preferred over using `__query_or` in general. + +CUB currently supports the following CPOs for querying environments: + + - `cuda::get_stream` + - `cuda::mr::get_memory_resource` + - `cuda::execution::__get_requirements_t` (internal) + - `cuda::execution::__get_tuning_t` (internal) + +While combining environments with `cuda::std::execution::env` does not nest them, +environments can contain values which are environments themselves, +and thus respond to queries again. +Execution requirements and tuning policy selectors are examples of values which are environments themselves. +For example: + +.. code-block:: c++ + + auto env = cuda::std::execution::env(cuda::execution::require(cuda::execution::determinism::run_to_run)); + auto requirements = cuda::__get_requirements_t(env); + auto determinism = cuda::__get_requirements_t(requirements); + static_assert(is_same_v): + +.. + TODO(bgruber): Should we mention `__queryable_with`? + TODO(bgruber): Should we mention `__call_or`? + +Structure of the CUB execution environment +-------------------------------------------------------------------------------- + +In CUB, all device wide algorithms with an environment parameter `env` expect that: + + - If the environment contains a stream, it can be queried using `cuda::get_stream(env)`. + - If the environment contains a memory resource, it can be queried using `cuda::mr::get_memory_resource(env)`. + - If the environment contains execution requirements, + they can be queried using `auto requirements = cuda::execution::__get_requirements_t(env)`. + The value `requirements` is another environment, which can be queried for requirements, + e.g. the requested determinism using `cuda::execution::__get_determinism_t`. + - If the environment contains tuning policy selectors, + they can be queried using `auto tunings = cuda::execution::__get_tuning_t(env)`. + The value `tunings` is another environment, which can be queried for tuning policy selectors using + `cuda::execution::__query_or` and the type of a tuning policy + +See also the documentation on `determinism `_ and tuning `_. + + +Properties of environments +-------------------------------------------------------------------------------- + +Since any object (of any type) can potentially be an environment, +we cannot describe the properties of environments in general. +However, in CUB we assume that environments are lightweight objects, +containing reference/pointer like values like stream or memory resource handles. +CUB therefore assumes in many places that an environment is copyable and movable. +We are currently changing this to pass environments between functions by `const&`, +to not rely on these properties. +Since CUB only queries environments, it does not mutate them, store them, +extend their lifetime in any way beyond a CUB API call, +or end their lifetime prematurely. +If an environment manages the lifetime of a contained resource, +and CUB queries and uses this resource for the execution of an algorithm, +it is the responsibility of the user to ensure the lifetime of the resource +exceeds the duration of the asynchronous CUB algorithm execution. +CUB strives for non-owning environments. From 93b4396432778e878eb147a069c250eb61931893 Mon Sep 17 00:00:00 2001 From: Bernhard Manfred Gruber Date: Mon, 20 Jul 2026 15:52:04 +0200 Subject: [PATCH 2/2] Link page from index --- docs/cub/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/cub/index.rst b/docs/cub/index.rst index a131af21747..bc0d7f531c2 100644 --- a/docs/cub/index.rst +++ b/docs/cub/index.rst @@ -12,6 +12,7 @@ CUB warp_wide block_wide device_wide + environments determinism benchmarking tuning