Add OTLP runtime metrics system test by link04 · Pull Request #6715 · DataDog/system-tests

link04 · 2026-04-09T16:58:44Z

Summary

Related One Pager: Sending Runtime Metrics via OTLP from dd-trace-*

Add OTLP_RUNTIME_METRICS end-to-end scenario and test for validating that dd-trace libraries export runtime metrics via OTLP using OTel semantic convention names (dotnet.*, jvm.*, go.*, v8js.*) instead of DD-proprietary names (runtime.dotnet.*, runtime.go.*, etc.)
Register the new scenario in CI workflow and orchestrator
Fix parametric test_otel_metrics tests leaking runtime metrics by explicitly setting DD_RUNTIME_METRICS_ENABLED=false in test env vars

How it works

When DD_RUNTIME_METRICS_ENABLED=true + DD_METRICS_OTEL_ENABLED=true, tracers should send runtime metrics via OTLP.
The scenario routes OTLP metrics through the proxy to the agent using OTEL_EXPORTER_OTLP_METRICS_HEADERS: "dd-protocol=otlp,dd-otlp-path=agent" (same pattern as otel_integrations), then validates metric names from interfaces.agent.get_metrics() (same pattern as RUNTIME_METRICS_ENABLED).

Test coverage

One test class (Test_OtlpRuntimeMetrics) asserts:

All expected OTel-named metrics are present for the language under test
No DD-proprietary metric names appear

Expected metrics are defined per language (19 dotnet, 8 go, 13 nodejs, 18 java). All languages are missing_feature until their respective tracer PRs land.

Test plan

OTLP_RUNTIME_METRICS scenario build from dd-trace-dotnet#8457 passes in System Test CI #6797 Dev run
Existing RUNTIME_METRICS_ENABLED scenario unaffected
Parametric test_otel_metrics failures fixed with DD_RUNTIME_METRICS_ENABLED=false
CI green on system-tests

🤖 Generated with Claude Code

github-actions · 2026-04-09T16:59:18Z

CODEOWNERS have been resolved as:

tests/test_otlp_runtime_metrics.py                                      @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/parametric/test_otel_metrics.py                                   @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core

datadog-prod-us1-5 · 2026-04-20T14:32:56Z

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f74c21d | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 483e4b93ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Add OTLP_RUNTIME_METRICS scenario that validates dd-trace libraries export runtime metrics via OTLP with OTel semantic convention names instead of DD-proprietary names. - New scenario routes OTLP metrics through proxy to agent using standard dd-otlp-path=agent header pattern - Test reads from interfaces.agent.get_metrics(), matching how RUNTIME_METRICS_ENABLED tests work - Expected metrics defined per language (dotnet, go, nodejs, java), all set to missing_feature until tracer PRs land - Registered in CI workflow and orchestrator - Fix parametric test_otel_metrics leaking runtime metrics by setting DD_RUNTIME_METRICS_ENABLED=false in DEFAULT_ENVVARS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

## Summary of changes Add support for exporting .NET runtime metrics via OTLP, using OpenTelemetry semantic convention names. When both `DD_RUNTIME_METRICS_ENABLED=true` and `DD_METRICS_OTEL_ENABLED=true`, runtime metrics (`System.Runtime`) are collected through the existing OTLP metrics pipeline instead of DogStatsD. **Related One Pager:** [Sending Runtime Metrics via OTLP from dd-trace-* ](https://docs.google.com/document/d/12DhKgK_cu5PMD54J-Ij2xFixUmVgRi-MvMEjb9mbNZw/edit?pli=1&tab=t.0#heading=h.dtlburq1um7q) ## Reason for change Enables runtime visibility for users on OTLP-native pipelines or without a local Datadog Agent. Aligns .NET runtime metric names with OTel semantic conventions (`dotnet.gc.collections`, `dotnet.process.cpu.time`, etc.). ## Implementation details - **`TracerManagerFactory`**: Disables the StatsD `RuntimeMetricsWriter` when OTLP runtime metrics are active to prevent duplicate reporting. - **`MetricReaderHandler`**: Force-enables the `System.Runtime` meter when `OtlpRuntimeMetricsEnabled` is true, even if not listed in `DD_METRICS_OTEL_METER_NAMES`. - **`RuntimeMetricsPolyfill`** (new): Provides the .NET 9 native `System.Runtime` instrument set on .NET 6–8 hosts. Names, units, descriptions, and tag keys mirror [dotnet/runtimev9.0.0](https://github.com/dotnet/runtime/blob/v9.0.0/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/RuntimeMetrics.cs). - **`MeterObservableUpDownCounterReflection`** (new): Resolves `Meter.CreateObservableUpDownCounter<T>` at runtime so .NET 7/8 hosts (with `System.Diagnostics.DiagnosticSource ≥ 7.0` via roll-forward) emit `ObservableUpDownCounter` for the 7 point-in-time metrics that match native .NET 9. .NET 6 hosts without DiagnosticSource ≥ 7 fall back to `ObservableGauge`. - **`Datadog.Trace.Trimming.xml`**: Preserves the `System.Diagnostics.Metrics` types the polyfill creates (`Counter`1`, `ObservableCounter`1`, `ObservableGauge`1`, `ObservableUpDownCounter`1`) so they survive aggressive trimming/AOT in customer apps. - **`MetricsRuntime`**: Starts the polyfill on .NET < 9 with thread-safe initialization. - **`MetricPoint`**: Detects counter resets on `ObservableCounter` (treats negative deltas as resets and reports the current value as the delta). ## Test coverage - **`OpenTelemetrySdkTests.SubmitsOtlpRuntimeMetrics`** (new): end-to-end integration test verifying the OTLP runtime metrics payload via a Verify snapshot. A single snapshot covers all TFMs because the test sample transitively references `OpenTelemetry` 1.13.x → `System.Diagnostics.DiagnosticSource` 9.0, so the polyfill produces identical wire output on .NET 6/7/8 (and .NET 9+ uses native runtime metrics with the same wire format). Also asserts that StatsD is not emitted when OTLP runtime metrics are active. - **`TracerSettingsTests.OpenTelemetryMetricsEnabled`**: parameterized unit test confirming the `DD_TRACE_OPEN_TELEMETRY_METRICS_ENABLED` (or whatever the actual config key name resolves to) boolean parses correctly with default `false`. - Added `DD_RUNTIME_METRICS_ENABLED=false` to existing `SubmitsOtlpMetrics` test to prevent runtime metrics from interfering with custom meter assertions. ### Verification of `.NET 6` in-box behavior (out-of-band) `MeterObservableUpDownCounterReflection.TryRegister` was verified directly against each in-box `System.Diagnostics.DiagnosticSource` (no OTel package installed), since the integration test masks the .NET 6 in-box scenario by transitively pulling in `DiagnosticSource` 9.0: | TFM | DiagnosticSource | `CreateObservableUpDownCounter` overloads | `TryRegister` returns | |---|---|---|---| | net6.0 (6.0.36) | 6.0.0 | 0 | `false` → omit branch | | net7.0 (7.0.20) | 7.0.0 | 3 | `true` → UpDownCounter | | net8.0 (8.0.8) | 8.0.0 | 6 | `true` → UpDownCounter | | net9.0 (9.0.9) | 9.0.0 | 6 | `true` → UpDownCounter | The 4-param `Length == 4` pin in `FindOverload` correctly disambiguates on .NET 8/9 where 5-param overloads (with `tags`) also exist. ## Other details - Built against system tests [PR](DataDog/system-tests#6715) (`system-tests` repo) , added need `DD_RUNTIME_METRICS_ENABLED=false` added to their `DEFAULT_ENVVARS` to avoid unexpected runtime metrics in OTel metric assertions and created new test based on language expectations.

link04 changed the title ~~Add OTLP runtime metrics system test~~ [dotnet@maximo/otlp-runtime-metrics] Add OTLP runtime metrics system test Apr 15, 2026

link04 added dotnet Pull requests that update .NET code ai-generated The pull request includes a significant amount of AI-generated code labels Apr 15, 2026

link04 changed the title ~~[dotnet@maximo/otlp-runtime-metrics] Add OTLP runtime metrics system test~~ Add OTLP runtime metrics system test Apr 21, 2026

link04 force-pushed the maximo/otlp-runtime-metrics-test branch from c2be460 to 483e4b9 Compare April 21, 2026 20:48

link04 removed the dotnet Pull requests that update .NET code label Apr 21, 2026

link04 marked this pull request as ready for review April 22, 2026 15:18

link04 requested review from a team as code owners April 22, 2026 15:18

link04 requested review from claponcet, manuel-alvarez-alvarez, rachelyangdog and vlad-scherbich and removed request for a team April 22, 2026 15:18

link04 requested a review from Anilm3 April 22, 2026 15:18

chatgpt-codex-connector Bot reviewed Apr 22, 2026

View reviewed changes

Comment thread utils/_context/_scenarios/__init__.py

link04 force-pushed the maximo/otlp-runtime-metrics-test branch from 483e4b9 to f74c21d Compare April 22, 2026 15:28

link04 mentioned this pull request Apr 22, 2026

[Runtime Metrics] Support for OTLP Runtime Metrics DataDog/dd-trace-dotnet#8457

Merged

cbeauchesne reviewed Apr 22, 2026

View reviewed changes

Comment thread tests/parametric/test_otel_metrics.py

cbeauchesne approved these changes Apr 23, 2026

View reviewed changes

Anilm3 removed their request for review April 27, 2026 09:59

link04 merged commit 68b5c06 into main Apr 27, 2026
2820 of 2822 checks passed

link04 deleted the maximo/otlp-runtime-metrics-test branch April 27, 2026 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OTLP runtime metrics system test#6715

Add OTLP runtime metrics system test#6715
link04 merged 1 commit intomainfrom
maximo/otlp-runtime-metrics-test

link04 commented Apr 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-5 Bot commented Apr 20, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

link04 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Test coverage

Test plan

Uh oh!

github-actions Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-5 Bot commented Apr 20, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

link04 commented Apr 9, 2026 •

edited

Loading

github-actions Bot commented Apr 9, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Apr 20, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading