Skip to content

[BUG] Flaky: DeterministicBB deterministic_across_runs/swath1_threads8 — run 3 diverges under work_limit #1793

Description

@ramakrishnap-nv

DeterministicBB/DeterministicBBInstanceTest.deterministic_across_runs/swath1_threads8
fails intermittently in conda-cpp-tests. 1 of 528 tests; the rest pass.

Seen on run 32773153161
(13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on
#1763, which changes only .github/workflows/*.yaml for wheel publishing, so it
is not caused by that PR.

What happened

Params: ("/mip/swath1.mps", 8, 60, 4) — 8 CPU threads, time_limit=60,
work_limit=4. Seed for this occurrence: 480747044 (the test draws from
std::random_device per run and prints it, so each CI run exercises a different
seed).

The test solves the same problem three times with the same seed and compares.
Runs 1 and 2 agreed exactly; run 3 diverged:

Run Nodes Simplex iterations
1 1004 10931
2 1004 10931
3 986 10364

Zero Run 1 vs 2 assertion failures, 128 Run 1 vs 3 failures.

determinism_test.cu:217: Failure
  solution1.get_objective_value()  Which is: 382.30780375000012
  solution3.get_objective_value()  Which is: 384.26733044999986

determinism_test.cu:220: Failure
  solution1.get_solution_bound()   Which is: 341.0518313
  solution3.get_solution_bound()   Which is: 341.16572324999987

Why this looks like a real bug rather than tolerance

All three runs terminated on the work limit, not the clock:

Work limit reached. Stopping the solver...
Explored 986 nodes (10364 simplex iterations) in 3.55s.
Best objective 3.842673e+02, best bound 3.411657e+02, gap 11.22%.

work_limit exists precisely so termination does not depend on wall time. Three
runs with identical settings and seed consuming different amounts of work before
the counter trips means the divergence is upstream of termination — the search
itself took a different path — rather than a formatting or floating-point
tolerance question. The differing node and simplex-iteration counts point the
same way.

The failure is also not marginal: variable 1 differs by 2.0
(1.78e-14 vs 2.0) and variable 11 by 1.0 (1.0 vs 0), i.e. different
integer assignments, not last-bit noise.

Notes

Metadata

Metadata

Assignees

Labels

awaiting responseThis expects a response from maintainer or contributor depending on who requested in last comment.bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions