Repository navigation
Persist failed outbound delivery attempts in traces #1257
Description
Background
#1205 aligns federation logs with their operation spans for #1030. Failed outbound deliveries still have no TraceActivityRecord: sendActivityInternal() emits activitypub.activity.sent only after a successful response, and FedifySpanExporter extracts records from sent/received events. A failed delivery therefore has a span and error logs, but no activity record to display beside those logs in #902.
This is an existing limitation, separate from the logging-context fix. This issue tracks the federation instrumentation and exporter changes needed before #902 can show failed outbound attempts with their related logs.
Scope
Persist failed outbound delivery attempts with enough typed data to identify the activity, destination inbox, and delivery outcome. Keep activitypub.activity.sent reserved for successful delivery and preserve the existing trace hierarchy.
Before implementation, settle how failure events or span attributes produce records, how success/failure is represented in TraceActivityRecord, and how retries affect record grouping and TraceSummary.activityCount. Records should retain the delivery attempt's own traceId/spanId so its logs can be matched directly. Multiple attempts for the same activity and inbox must remain distinguishable.
Account for trace-list visibility when every attempt fails and for compatibility with existing successful outbound records. Define how worker-level retry, abandonment, and circuit-breaker logs relate to delivery records, including cases that never enter sendActivity(). Those logs should retain their worker span IDs.
#902 owns log spanId persistence and debugger rendering, including success/failure labels, grouping attempts, and showing logs without a matching activity record. This issue supplies the records and typed outcome data that rendering needs.
Validation
- Exercise a failed delivery inside a real queue worker, export its span through
FedifySpanExporter, and verify that the persisted failure record and emitted delivery error log have the sametraceId/spanId. - Cover HTTP error responses and transport failures, then a retry sequence that fails before succeeding. Verify distinct attempt records, correct outcomes, and consistent trace summaries.
- Verify that an always-failing delivery is discoverable through the exporter's recent-trace listing and that successful delivery records remain compatible.
- Check that worker retry/abandonment logs retain their worker IDs and are not attributed to a delivery attempt that never ran.
Update the public telemetry documentation and add a changelog fragment for the resulting behavior. Run the affected package suites on Deno, Node.js, and Bun. The debugger rendering checks remain in #902.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Activity
u-zzn commentedon Oct 7, 2026
Hi! I’d like to work on this issue as a follow-up to #1205. Could you please assign this issue to me? Thank you 🙂
u-zzn commentedon Oct 19, 2026
I'd like to propose an approach for persisting failed outbound deliveries and confirm a few design points before going further, since none of this has been discussed here yet:
activitypub.activity.send_failedspan event on the delivery's ownactivitypub.send_activityspan, keepingactivitypub.activity.sentreserved for a successful delivery.outcome("success" | "failed"),failureKind("http" | "transport" | "policy"), andstatusCodetoTraceActivityRecord— all outbound-only and optional (absent on inbound records, and on outbound records that predate this change).sendActivityInternal()(oneactivitypub.send_activityspan) — not necessarily one perOutboxMessage.attempt, since a portable actor's gateway fallback can callsendActivity()more than once within a single worker attempt.activityCount: would count every stored record per trace, including failed attempts, not just successful ones — so a delivery that fails twice before succeeding raises it by three. Open to a different definition if that's not the intent.I've verified this against a real queue worker: a fail → fail → succeed retry sequence through
FederationImpl.processQueuedTask()produces three distinct per-attempt records with a consistentactivityCountof 3.I also found a pre-existing issue where sequentially exporting the same span twice increases activityCount twice. I have a separate commit addressing sequential re-exports, but it does not cover concurrent exports of the same span. Would you prefer that change in this PR or a separate PR?
I can open a draft PR if that would make the design easier to review.
Thanks!
dahlia commentedon Oct 19, 2026
@u-zzn Thanks for verifying this against a real worker. Your one-record-per-
activitypub.send_activityspan proposal fits the execution-attempt level. I'd like to expand #1257 to distinguish that from the broadcast invocation and the destination delivery it belongs to. This should give #902 enough information to group retries without conflating separate sends of the same Activity.broadcastIdbefore recipient preparationdeliveryIdallocated before sending or enqueueing fanoutactivitypub.send_activityagainst a concrete URL(traceId, spanId)The Activity ID cannot identify a broadcast because an application can send the same Activity more than once. A host cannot identify a delivery either: shared inbox recipients may share one delivery, while two selected inboxes on one host remain separate. For ordinary portable delivery, fallback gateways serve one destination; one successful gateway can satisfy that obligation. Portable-inbox forwarding is replication instead, so each selected gateway is a separate delivery. Success at one does not satisfy the others.
Keep the existing trace hierarchy. Grouping IDs connect work without replacing its actual trace/span IDs or merging traces when propagation is unavailable. The queue's existing
attemptshould be exported as optionalqueueAttempt: it is a Fedify queue round, not an execution ordinal. Native redelivery can leave it unchanged, and gateway fallback can run several attempts in one round. Immediate sends leave it absent.There are a few implementation boundaries to preserve:
queueMessageIdas separate task context.forwardActivity, and both immediate and queued portable-inbox forwarding. Ordinary sends allocate delivery IDs after inbox deduplication; replication allocates them after selecting/claiming gateway targets. A forwarding deadline does not cancel ongoing sends, so late completion must retain its IDs rather than becoming a synthetic failed attempt.inboxUrl; usedestinationInboxfor the planned destination and optionalportableInboxfor canonical portable metadata. This lets a reader distinguish gateway routes without accidentally splitting one fallback delivery or merging independent replication deliveries.The failure event and private-address policy rejections are in scope. Keep
activitypub.activity.sentsuccess-only and emitactivitypub.activity.send_failedonce for a failed send. Reusingsentfor failures would change the meaning of existing telemetry. Guard the terminal event so an exception after an emitted success cannot append a contradictory failure, and collect common metadata without requiring the full Activity JSON: serialization itself may be the failure.New outbound records should have
outcome: "success" | "failed", optionalstatusCode, and nested failure details with a bounded kind (http,transport,policy,local, orunknown) and phase. Keep these additions optional on the compatibility type so inbound and legacy outbound records remain readable. A missing outcome can be interpreted as legacy success only for outbound records known to come from the old success-only writer.Capture failure provenance before broad error wrapping. Signing, spec lookup, and spec-cache errors can otherwise look like transport failures. In particular, double-knocking can throw from
rememberSpec()after receiving a response. A known final non-OK response stayshttp/responseeven if that bookkeeping also throws; keep the original exception in diagnostics and preserve retry behavior. An observed 2xx followed by that local failure isfailed,local/post-response, with the status retained separately. Never invent status 0 or report an intermediate challenge/redirect status as the final response. Policy rejection inside the send operation is a failed attempt even if no request was made; rejection before entry is not an attempt. Keep the existing metrics and exceptions unchanged. These records describe observed send results, not proof that the remote received nothing or completed its downstream processing.For selected worker dispositions, add a small separate
TraceOutboundWorkerRecordwith the worker's actual IDs andattemptsStartedfor that execution. Cover acknowledged deferral, TTL drops, preparation failures before the first send, and failures of the defer/drop action. This also covers a gateway round that starts an attempt and then defers remaining work. Track actual send entry; missing exported spans may be sampling rather than non-execution. An enqueue error may leave backend acceptance uncertain, so it must not claim either a scheduled hold or definite non-enqueue.Keep worker records in a separate namespace/getter so
getActivitiesByTraceId()still returns inbound and attempt records. A worker-only trace should be discoverable withactivityCount: 0and a separateoutboundWorkerCount. Worker records do not create fake attempts, and the existing workeractivitypub.delivery.failedmust not create a duplicate attempt. Ordinary retry/abandonment logs retain their worker IDs. Exact log matching uses actual trace/span IDs; a shared delivery or message ID provides broader context, not proof that a log belongs to a particular attempt. #902 still owns logspanIdpersistence, labels, and rendering. We do not need a full lifecycle journal or cross-trace index here.activityCountshould retain its record-count meaning and count each retained inbound record or outbound attempt once, including failures. Your fail/fail/succeed example counts as three when those records belong to the same trace and are retained. Making it a broadcast count would change the existing API and obscure retries. Broadcast/delivery grouping uses the new IDs. Sampling, export loss, and expiry mean these are observed retained counts, not lifetime totals or enough evidence to declare a broadcast complete.Please put the re-export counting bug in a separate prerequisite PR, targeting the oldest affected maintenance branch after checking it. It predates this feature, and the record write and summary increment are separate operations: a CAS claim does not make the pair atomic. The sequential regression is a useful start, but the fix needs concurrent same-span and distinct-span exports across exporter instances, partial-write recovery, and expiry coverage on CAS and non-CAS stores.
Primary records should determine authoritative counts and discovery, including when a summary write failed. Read-time reduction is the correctness baseline, but a full scan transfers retained values, including inbound JSON, on each recent-list poll. Measure that cost before choosing the implementation. An indexing optimization needs an explicit repair and retention contract; a finite extra TTL on a prewritten marker cannot bound the delay before the record write. The #1257 draft can proceed in parallel, with this prerequisite made clear.
A draft with these boundaries would be welcome. Please include real-worker log/record correlation, fanout replay, native retries, fallback and replication, legacy messages, worker-only traces and partial gateway deferral, and the HTTP/transport/policy/local failure cases, including post-response cache failure. Update the public telemetry documentation and changelog, and run the affected suites on Deno, Node.js, and Bun through mise.
Link #1257 and the predicted PR number in the changelog entry
Cover trace summary recovery and reexports