Add head-based sampling for LLM Observability traces [MLOB-7815] - #12277
Add head-based sampling for LLM Observability traces [MLOB-7815]#12277ncybul wants to merge 2 commits into
Conversation
ncybul
added
tag: ai generated
ncybul
changed the title
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9f5c5c40a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| boolean stamped = rawSamplingDecision instanceof String && rawSampleRate instanceof String; | ||
| String samplingDecision = stamped ? (String) rawSamplingDecision : SAMPLING_DECISION_SAMPLED; | ||
| String sampleRate = stamped ? (String) rawSampleRate : SAMPLE_RATE_ALL; |
There was a problem hiding this comment.
When an OpenAI Java request creates an LLMObs span without a surrounding manual DDLLMObsSpan, OpenAiDecorator.doAfterStart is the other production span-creation path and never stamps either sampling tag. This fallback consequently emits sampling_decision=1 and sample_rate=1, so even DD_LLMOBS_SAMPLE_RATE=0 retains every standalone auto-instrumented OpenAI trace. Apply the configured decision in that path or in a shared LLMObs span initializer.
Useful? React with 👍 / 👎.
| // Remove unconditionally: the generic _ml_obs_tag. sweep below would otherwise surface these | ||
| // internal fields in the user-visible tags[] array. | ||
| span.removeTag(SAMPLING_DECISION_TAG_INTERNAL_FULL); | ||
| span.removeTag(SAMPLE_RATE_TAG_INTERNAL_FULL); |
There was a problem hiding this comment.
When the MsgPack buffer fills after these removals, MsgPackWriter.format resets the buffer and invokes map(..., true) on the same spans. The retry then reads missing sampling tags and falls back to the retain-all 1/1 pair, so a trace originally marked dropped can be retained merely because serialization crossed a payload boundary or overflowed while writing later metadata. Preserve the original values for retries or defer mutation until serialization succeeds.
Useful? React with 👍 / 👎.
| private final boolean llmObsAgentlessEnabled; | ||
| private final String llmObsAgentlessUrl; | ||
| private final String llmObsMlApp; | ||
| private final double llmObsSampleRate; |
There was a problem hiding this comment.
The new llmObsSampleRate field is initialized and exposed but never appended to Config.toString(), so startup configuration diagnostics cannot show whether DD_LLMOBS_SAMPLE_RATE was loaded or clamped. The repository's required new-configuration procedure explicitly includes adding the value to Config.toString(); include it with the other diagnostic fields.
AGENTS.md reference: AGENTS.md:L37-L44
Useful? React with 👍 / 👎.
1 participant
What Does This Do
Adds head-based retention sampling for LLM Observability traces, controlled by a new
DD_LLMOBS_SAMPLE_RATE(dd.llmobs.sample.rate) config, defaulting to1.0.At the root of an LLMObs trace, the tracer makes a deterministic keep/drop decision keyed on the APM trace ID, then stamps
sampling_decisionandsample_rateinto the_ddblock of every span in that trace. Descendant spans inherit the decision unchanged, so a trace is retained or dropped as a whole.The SDK never drops a span. Every LLMObs span is still sent to the intake at 100%; the intake performs the drop. This is deliberate: token and cost metrics are computed over full volume regardless of the configured sample rate.
Since Java does not currently propagate LLMObs contexts, communicating the sampling decision across process boundaries is follow-up work. For now, services configured at the same rate independently reach the same decision for the same trace, because the algorithm is deterministic in the APM trace ID and sample rate.
Motivation
Python and Node already support LLMObs head-based sampling; Java did not. Customers running high-volume LLM workloads have no way to control retention volume from the Java tracer.
Claude session:
f94e7cd8-1c7d-41b1-835f-553f557f2d04Resume:
claude --resume f94e7cd8-1c7d-41b1-835f-553f557f2d04