Skip to content

Don't log the caller's own cancellation as a failed broker request - #4645

Open
nitrobass24 wants to merge 1 commit into
actions:mainfrom
nitrobass24:fix/broker-retry-ignores-caller-cancellation
Open

Don't log the caller's own cancellation as a failed broker request#4645
nitrobass24 wants to merge 1 commit into
actions:mainfrom
nitrobass24:fix/broker-retry-ignores-caller-cancellation

Conversation

@nitrobass24

Copy link
Copy Markdown

Fixes #4644.

RetryRequest has no cancellation filter, so a cancellation raised by the caller's own token is caught by the general retry handler, logged as a failed request, and charged a backoff.

BrokerMessageListener polls using a linked token it owns, and already handles the cancellation itself:

_getMessagesTokenSource = CancellationTokenSource.CreateLinkedTokenSource(token);
message = await _brokerServer.GetRunnerMessageAsync(..., _getMessagesTokenSource.Token);
...
catch (OperationCanceledException) when (_getMessagesTokenSource.Token.IsCancellationRequested && !token.IsCancellationRequested)
{
    Trace.Info("Get messages has been cancelled using local token source. Continue to get messages with new status.");
    continue;
}

It cancels that poll on every Busy -> Idle transition, i.e. on every job completion. ShouldRetryException filters only auth and session exceptions, so the cancellation reaches the general catch and produces:

ERR   BrokerServer] Catch exception during request
ERR   BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
ERR   BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled
WARN  BrokerServer] Back off 14.623 seconds before next retry. 4 attempt left.

The announced backoff is never served: Task.Delay is awaited on the same token that was just cancelled, so it throws immediately. On a 12-runner fleet this produced 2,064 of these warnings across 1,475 job completions, and the "Back off" line shares a timestamp with the listener's "cancelled using local token source" line in every occurrence sampled (12 of 12) — confirming no delay elapses.

The cost is diagnostic rather than latency: ERR here does not indicate a fault, and the signature is identical to a genuine broker connectivity failure. #3904 collects reports of this exact pattern where the discussion has gone hunting network causes. Silencing the benign case makes the real ones legible.

The change

A cancellation-specific catch ahead of the general one, rethrowing so the caller's existing handler runs.

The when (cancellationToken.IsCancellationRequested) guard is deliberate: HttpClient surfaces a request timeout as TaskCanceledException while the caller's token is not cancelled. That is a genuine failure and must keep retrying, so it deliberately falls through to the existing handler.

Behaviour is unchanged for every other exception type, and for cancellation the caller sees the same OperationCanceledException it does today — only the spurious ERR/WARN output and the discarded backoff computation go away.

Testing

I do not have a .NET SDK available, so this is not built or tested locally — please treat it accordingly. There is no existing coverage of RetryRequest to extend (ErrorThrottlerL0 covers the separate ErrorThrottler class). Happy to add a test if you can point me at the shape you'd want; RetryRequest is protected, so it would need a test double over RunnerService.

RetryRequest has no cancellation filter, so a cancellation raised by the
caller's own token is caught by the general retry handler, logged at ERR
with a stack trace, and charged a 5-15s backoff.

BrokerMessageListener polls using a linked token it owns and already
handles the cancellation itself, cancelling that poll on every
Busy -> Idle transition -- that is, on every job completion.
ShouldRetryException filters only auth and session exceptions, so the
cancellation reaches the general catch.

The announced backoff is never served: Task.Delay awaits the same token
that was just cancelled, so it throws immediately. The cost is therefore
diagnostic rather than latency -- ERR here does not indicate a fault, and
the output is indistinguishable from a genuine broker connectivity
failure.

Rethrow caller-initiated cancellation ahead of the general handler. The
IsCancellationRequested guard is deliberate: HttpClient reports a request
timeout as TaskCanceledException while the caller's token is uncancelled,
and that is a real failure which must keep retrying.
@nitrobass24
nitrobass24 requested a review from a team as a code owner August 24, 2026 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broker retry logs an ERR stack trace and a phantom backoff on every job completion (listener's own cancellation treated as a request failure)

1 participant