Skip to content

Keep the upstream sampling decision for a propagated trace - #264

Merged
mrproliu merged 2 commits into
mainfrom
fix-propagated-trace-sampling
Nov 7, 2026
Merged

mrproliu merged 2 commits into
mainfrom
fix-propagated-trace-sampling

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes the sampling bug reported in apache/skywalking#14201.

createSpan0 checked len(ds.Refs) > 0 and ran the local sampler before it applied the core options, so the upstream sw8 ref from withRef was not attached yet. Every entry span that starts a tracing context was sampled locally, even with a valid upstream context. With agent.sampler below 1, a downstream Go service dropped traces that the upstream service had already sampled. It then sent no sw8 header onward, so its own downstream services could start unrelated traces. The order changed in #27, so every release since v0.1.0 is affected.

  • Apply the core options before the sampling decision. A valid sw8 context forces sampling, as in the Java agent (ContextManager#createEntrySpan). The sampler now receives the operation name (it received an empty one), and sw_go_created_tracing_context_counter{created_by="propagated"} counts again.
  • Accept a sw8 header only when every required field is set, like the Java agent's ContextCarrier#isValid. A header such as 1---0---- used to be accepted and created a segment with an empty trace ID. With the reorder alone, it would also have bypassed sampling. It is now ignored, and the request starts a new trace through the local sampler. Headers written by the Go agent always carry every field, because the plugin API rejects empty operation names and peers.
  • Document the sampling policy in the agent.sampler row of docs/en/agent/tracing-metrics-logging.md, and fix its environment key: the agent reads SW_AGENT_SAMPLE, not SW_AGENT_SAMPLER.

The Sample flag in sw8 is still not checked, which matches the other SkyWalking agents. apache/skywalking#14204 documents that convention in the protocol doc, including the #sampling anchor that the agent.sampler row links to.

Tests

  • New: TestPropagatedContextIsNotResampled, TestSamplerReceivesOperationName, and TestIncompletePropagatedContextIsIgnored. Each fails on main and passes with this change.
  • In plugins/core, go test ./..., the race tests, go vet, and golangci-lint v2.15.0 with the repository config pass.

createSpan0 checked len(ds.Refs) and ran the local sampler before it
applied the core options, so the upstream sw8 ref was not attached yet
and every entry span that starts a tracing context was sampled locally.
With agent.sampler below 1, a downstream service dropped traces that the
upstream service had already sampled. Reported in
apache/skywalking#14201.

- Apply the core options before the sampling decision. A valid sw8
  context forces sampling, as in the Java agent. The sampler gets the
  real operation name, and the propagated so11y counter counts again.
- Accept a sw8 header only when every required field is set, like the
  Java agent's ContextCarrier#isValid, so a header such as 1---0----
  no longer forces sampling or creates a segment with an empty trace ID.
- Add regression tests.
- Document the sampling policy for agent.sampler, and fix its
  environment key, which is SW_AGENT_SAMPLE, not SW_AGENT_SAMPLER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The /propagated and /ignored.html entry spans carry a valid sw8 ref
from /so11y, so sw_go_created_tracing_context_counter now counts them
as created_by=propagated (2) instead of created_by=sampler, which
keeps the /health and /so11y entry spans and the two exit spans (4).
The scenario expected 0 propagated contexts because of the bug this
change fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mrproliu
mrproliu merged commit 1ec150a into main Nov 7, 2026
49 checks passed
@mrproliu
mrproliu deleted the fix-propagated-trace-sampling branch November 7, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants