Skip to content

[SDK Sentinel] Fix replay failure result race - #1502

Closed
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/rust/31847451911-1-1b7e489780b6
Closed

[SDK Sentinel] Fix replay failure result race#1502
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/rust/31847451911-1-1b7e489780b6

Conversation

@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor

Caution

This draft contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.

Summary

Prevent multi-history WorkflowReplayer calls from occasionally omitting a nondeterminism result. A [successful equivalent run](https://github.com/temporalio/sdk-rust/actions/runs/31765593305) preceded the [captured assertion failure](https://github.com/temporalio/sdk-rust/actions/runs/31803647780/job/94777252814).

Root cause

Workflow-task failure handling released the next replay history immediately. That could exhaust the input stream and shut down the worker before the failed history's eviction activation reached the result observer.

Fix

Advance through the existing post-activation hook only after the eviction completes. This preserves the eviction reason before another history is dispatched or shutdown begins.

Validation

The unchanged baseline passed, and the trusted formatting, lint, test-lint, and check command passed twice on the candidate. The focused integration command was attempted but zstd-sys hit a sandbox PermissionDenied before the test ran. The Per merge CI integration matrix is required. Remaining risk is a failure path that does not emit an eviction activation, which could stall rather than advance.

Validation status: validation-incomplete

  • Flake confidence: high
  • Fix confidence: medium
  • Value: medium
  • Patch scope: standard — 3 changed files; 3.4 KiB
  • Local reproduction: unavailable — The focused integration command was attempted once but could not build because zstd-sys encountered a sandbox PermissionDenied.
  • Regression coverage: not-added — The existing workflow_replayer_returns_ordered_results_for_multiple_histories integration test directly covers the regression; it failed in captured CI but could not be executed locally.
  • Unchanged baseline (Cargo formatting, linting, and checks) — passed
  • Independent candidate (Cargo formatting, linting, and checks) — passed

Investigator-run checks

  • trusted unchanged-checkout baseline validation — passed; 1 attempt(s). The supplied baseline conclusion was success.
  • CARGO_BUILD_JOBS=2 timeout 180 cargo integ-test workflow_replayer_returns_ordered_results_for_multiple_histories — not run; 1 attempt(s). The test did not run: zstd-sys failed during compilation with sandbox PermissionDenied.
  • .ci-flake-runtime/input/validate.sh — passed; 2 attempt(s). Both final validation runs passed nightly formatting, cargo lint, cargo test-lint, and cargo check.

Required target CI

  • Per merge CI / Integ tests (ubuntu-latest): run workflow_replayer_returns_ordered_results_for_multiple_histories and workflow_replayer_reports_nondeterminism
  • Per merge CI integration matrix: ubuntu-arm, macos-arm, macos-intel, and windows-latest

Residual risks

  • A replay failure mode that does not produce an eviction activation could now stall instead of advancing; the relevant Core failure paths are expected to evict but were not executable locally.
  • The focused runtime behavior remains dependent on CI validation because local compilation was blocked.

Automation provenance

If this finding should not be fixed, apply the sdk-sentinel:false-positive label and close the PR. Sentinel will suppress the finding until its affected source changes.

SDK Sentinel recurrence history

  • Unique failed CI run attempts: 1
  • First occurrence: 2026-08-14 13:22 UTC
  • Latest occurrence: 2026-08-14 13:22 UTC
  • Recent occurrences:

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Sushisource

Copy link
Copy Markdown
Member

This one causes existing tests to hang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants