Control threading behavior

[Andlon]

I am experiencing some non-determinism problems. This appears to be due to threading behavior. When running without contacts, it looks as if I am able to get deterministic (enough) output by disabling threading on my end. However, when running simulations with contacts with ipc-toolkit, I'm getting very non-deterministic behavior (seen, for example, through very varying number of solver iterations across runs with identical parameters).

I'd like to disable threading in ipc-toolkit to see if I get deterministic results this way. However, I am not so familiar with TBB and I'm not even sure if TBB is the only thing that's being used in ipc-toolkit, or if there are other sources of parallelism. Is there a way to disable parallelism or otherwise control the number of threads in ipc-toolkit?

Note: The problems I'm looking at are extremely sensitive (presumably due to near-singularity of matrices), I don't believe there's anything wrong with ipc-toolkit.

[zfergus]

With multi-threading enabled, it is expected that the results can be non-deterministic because of rounding differences when adding numbers in different orders. We only utilize TBB in the toolkit (excluding experimental CUDA CCD), so to make the results deterministic you can limit TBB's max number of threads to one. The following code shows how to do this:

#include <tbb/global_control.h>
// ...
tbb::global_control thread_limiter(tbb::global_control::max_allowed_parallelism, 1);

As long as this thread_limiter object stays alive, the number of threads will be limited to 1. This example shows the thread limiter stack allocated, so when the object goes out-of-scope, the limit will be released. If you want this limit for the entire length of your program it is best to define it in main or you can define it statically using a std::shared_ptr<tbb::global_control>. For example,

#include <tbb/info.h>
#include <tbb/global_control.h>

static std::shared_ptr<tbb::global_control> thread_limiter;

void set_num_threads(int nthreads)
{
    if (nthreads <= 0) {
        nthreads = tbb::info::default_concurrency();
    } else if (nthreads > tbb::info::default_concurrency()) {
        logger().warn(
            "Attempting to use more threads than available ({:d} > {:d})!",
            nthreads, tbb::info::default_concurrency());
        nthreads = tbb::info::default_concurrency();
    }
    thread_limiter = std::make_shared<tbb::global_control>(
        tbb::global_control::max_allowed_parallelism, nthreads);
}

[Andlon]

@zfergus: thanks, that's very helpful!

[zfergus]

I added this description here.

[fosskers]

How about controlling this via an environment variable? We have a situation where TBB is consuming more CPUs than we want, but using ipctk.set_num_threads in a dynamic way is not appropriate for us as we want to be able to control this from the global, production environment in a predictable way, and not in any one particular Python file.

[zfergus]

@fosskers, you should now be able to set the number of threads using the TBB_NUM_THREADS environment variable (see #242).

[fosskers]

Amazing, thank you very much!

[fosskers]

Any chance this is worth an official patch release?

[zfergus]

Yes, I'll include it in the v1.6.0 release, which I am putting together now.

[fosskers]

Thank you!