Don't log the caller's own cancellation as a failed broker request - #4645
Open
nitrobass24 wants to merge 1 commit into
Open
Don't log the caller's own cancellation as a failed broker request#4645nitrobass24 wants to merge 1 commit into
nitrobass24 wants to merge 1 commit into
Conversation
RetryRequest has no cancellation filter, so a cancellation raised by the caller's own token is caught by the general retry handler, logged at ERR with a stack trace, and charged a 5-15s backoff. BrokerMessageListener polls using a linked token it owns and already handles the cancellation itself, cancelling that poll on every Busy -> Idle transition -- that is, on every job completion. ShouldRetryException filters only auth and session exceptions, so the cancellation reaches the general catch. The announced backoff is never served: Task.Delay awaits the same token that was just cancelled, so it throws immediately. The cost is therefore diagnostic rather than latency -- ERR here does not indicate a fault, and the output is indistinguishable from a genuine broker connectivity failure. Rethrow caller-initiated cancellation ahead of the general handler. The IsCancellationRequested guard is deliberate: HttpClient reports a request timeout as TaskCanceledException while the caller's token is uncancelled, and that is a real failure which must keep retrying.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4644.
RetryRequesthas no cancellation filter, so a cancellation raised by the caller's own token is caught by the general retry handler, logged as a failed request, and charged a backoff.BrokerMessageListenerpolls using a linked token it owns, and already handles the cancellation itself:It cancels that poll on every
Busy->Idletransition, i.e. on every job completion.ShouldRetryExceptionfilters only auth and session exceptions, so the cancellation reaches the general catch and produces:The announced backoff is never served:
Task.Delayis awaited on the same token that was just cancelled, so it throws immediately. On a 12-runner fleet this produced 2,064 of these warnings across 1,475 job completions, and the "Back off" line shares a timestamp with the listener's "cancelled using local token source" line in every occurrence sampled (12 of 12) — confirming no delay elapses.The cost is diagnostic rather than latency:
ERRhere does not indicate a fault, and the signature is identical to a genuine broker connectivity failure. #3904 collects reports of this exact pattern where the discussion has gone hunting network causes. Silencing the benign case makes the real ones legible.The change
A cancellation-specific catch ahead of the general one, rethrowing so the caller's existing handler runs.
The
when (cancellationToken.IsCancellationRequested)guard is deliberate:HttpClientsurfaces a request timeout asTaskCanceledExceptionwhile the caller's token is not cancelled. That is a genuine failure and must keep retrying, so it deliberately falls through to the existing handler.Behaviour is unchanged for every other exception type, and for cancellation the caller sees the same
OperationCanceledExceptionit does today — only the spurious ERR/WARN output and the discarded backoff computation go away.Testing
I do not have a .NET SDK available, so this is not built or tested locally — please treat it accordingly. There is no existing coverage of
RetryRequestto extend (ErrorThrottlerL0covers the separateErrorThrottlerclass). Happy to add a test if you can point me at the shape you'd want;RetryRequestisprotected, so it would need a test double overRunnerService.