Skip to content

[Bug]: Replay mismatch silently consumes a checkpoint from a different operation #692

Description

@zhongkechen

Expected Behavior

During replay, before an SDK operation interprets a checkpoint's status or
payload, it must validate that the checkpointed operation has the same type,
subtype, and name as the operation currently being emitted at that positional
ID.

If the identity does not match, the durable execution should terminate with a
NonDeterministicExecutionError and return a terminal FAILED response. It
must not consume the checkpoint as though it belonged to the current
operation.

Actual Behavior

Replay lookup currently matches only on the positional operation ID.
ExecutionState.get_checkpoint_result() returns any operation stored under
that ID, and each operation executor interprets the returned status without
first validating the stored type, subtype, or name.

For the concrete WAIT to STEP case:

  1. History contains a WAIT at position 1 with status SUCCEEDED.
  2. Non-deterministic replay skips the wait and emits a STEP at position 1.
  3. StepOperationExecutor sees the checkpoint as succeeded.
  4. Because a wait has no step result, the step returns None.
  5. The step body is never executed and no non-determinism error is raised.

The execution can therefore complete successfully even though the step's work
and side effects never happened. This is more severe than returning a generic
failure because the missing work can be reported as successful.

NonDeterministicExecutionError and
TerminationReason.NON_DETERMINISTIC_EXECUTION already exist, but no
production path currently raises that error when replay identity differs.

Steps to Reproduce

The following focused reproduction runs against the current repository source:

from unittest.mock import Mock

from aws_durable_execution_sdk_python.config import StepConfig
from aws_durable_execution_sdk_python.identifier import (
    OperationIdentifier,
    OperationIdNamespace,
)
from aws_durable_execution_sdk_python.lambda_service import (
    Operation,
    OperationStatus,
    OperationSubType,
    OperationType,
)
from aws_durable_execution_sdk_python.operation.step import StepOperationExecutor
from aws_durable_execution_sdk_python.plugin import PluginExecutor
from aws_durable_execution_sdk_python.state import ExecutionState

op_id = OperationIdNamespace().create_id_for_step(1)
state = ExecutionState(
    durable_execution_arn=(
        "arn:aws:lambda:us-east-1:123456789012:"
        "function:test:1/durable-execution/test/id"
    ),
    initial_checkpoint_token="token",
    operations={
        op_id: Operation(
            operation_id=op_id,
            operation_type=OperationType.WAIT,
            status=OperationStatus.SUCCEEDED,
        )
    },
    service_client=Mock(),
    plugin_executor=PluginExecutor([]),
)

step_body_ran = []
executor = StepOperationExecutor(
    func=lambda _ctx: step_body_ran.append(True) or "expired",
    config=StepConfig(),
    state=state,
    operation_identifier=OperationIdentifier(
        operation_id=op_id,
        sub_type=OperationSubType.STEP,
        name="expire-audit",
    ),
    context_logger=Mock(),
)

result = executor.process()

assert result is None
assert step_body_ran == []

Observed output in a direct run:

result= None step_body_ran= False

The same behavior follows from a handler that emits a wait at a position on the
first invocation, then takes a non-deterministic branch and emits a step at
that position after the wait completes.

SDK Version

Reproduced on:

  • latest tagged SDK release 1.7.0
  • current main at 24c75363b2ef86bf5c3af8c1c689fba55a6eb9fa
    (source declares 1.8.0)

Python Version

3.13

Is this a regression?

No known working version.

Last Working Version

N/A

Additional Context

Relevant code paths:

  • state.py: ExecutionState.get_checkpoint_result() retrieves solely by ID.
  • operation/step.py: StepOperationExecutor.check_result_status() treats any
    SUCCEEDED checkpoint as a completed step.
  • exceptions.py: NonDeterministicExecutionError is defined but unused by
    replay validation.

Suggested fix:

  1. Add a shared replay-identity validator that compares checkpointed and
    claimed operation type, subtype, and name.
  2. Invoke it before any operation-specific status or payload handling.
  3. Raise NonDeterministicExecutionError on mismatch so the top-level
    execution wrapper returns terminal FAILED.
  4. Add composed regression tests asserting that type/name/subtype drift never
    returns PENDING or SUCCEEDED and never executes or skips work using
    another operation's checkpoint.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugpkg:sdkPackage: aws-durable-execution-sdk-python

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions