Skip to content

Add feature flag span enrichment system tests#6828

Draft
sameerank wants to merge 1 commit intomainfrom
sameerank/FFL-2201/feature-flag-apm-enrichment-tests
Draft

Add feature flag span enrichment system tests#6828
sameerank wants to merge 1 commit intomainfrom
sameerank/FFL-2201/feature-flag-apm-enrichment-tests

Conversation

@sameerank
Copy link
Copy Markdown
Contributor

@sameerank sameerank commented Apr 29, 2026

Motivation

Add system tests for feature flag event enrichment (Feature ID 551) that validates APM span chunk tags:

  • ffe_flags_enc: base64(delta varint encoded serial IDs) for all evaluated flags
  • ffe_subjects_enc: subject hash → serial IDs mapping (when doLog=true)
  • ffe_defaults: flag name → "coded-default: " mapping (when flag not found in UFC)

Example span tags:

{
  "ffe_defaults": {
    "flag-not-found": "coded-default: my-default-value"
  },
  "ffe_flags_enc": "ZAgUAg==",
  "ffe_subjects_enc": {
    "4208f8d8017ce252df51f51e8c5b558a": "ZA==",
    "8e526f09f0531278f8bd60f224adf60a": "bAg="
  }
}

In this example: "ZAgUAg==" decodes to serial IDs [100, 108, 128, 130]

JIRA: https://datadoghq.atlassian.net/browse/FFL-2201

Changes

New test files (consolidated in parametric):

  • tests/parametric/test_ffe/test_span_enrichment.py - All span enrichment tests
  • tests/parametric/test_ffe/span-enrichment-flags.json - UFC fixture with serialId in splits
  • tests/parametric/test_ffe/utils.py - Delta varint encoding/decoding utilities + SHA256 hashing

Test coverage:

Test Class Test Purpose
Test_Span_Enrichment_Serial_IDs test_multiple_flags_serial_ids_combined Multiple flags combine serial IDs correctly
Test_Span_Enrichment_Child_Span_Propagation test_child_span_flag_evaluation_propagates_to_root Child span propagates to root
Test_Span_Enrichment_Max_Serial_IDs test_max_128_serial_ids_enforced 128 serial ID limit (generates 150 flags)
Test_Span_Enrichment_Default_Fallback test_flag_not_found_adds_ffe_defaults_tag Flag not in UFC adds coded-default prefix
Test_Span_Enrichment_Default_Fallback test_ffe_defaults_value_truncated_at_64_chars coded-default value truncated to 64 chars
Test_Span_Enrichment_Default_Fallback test_max_5_flag_keys_in_ffe_defaults Max 5 flag keys in ffe_defaults
Test_Span_Enrichment_Max_Subjects test_max_25_subjects_enforced 25 subject limit
Test_Span_Enrichment_Subjects test_do_log_true_adds_subjects_encoded doLog=true adds subjects
Test_Span_Enrichment_Subjects test_do_log_false_no_subjects_encoded doLog=false skips subjects
Test_Span_Enrichment_Subjects test_subjects_encoded_uses_sha256_hashed_targeting_key Targeting keys are SHA256 hashed
Test_Span_Enrichment_Delta_Varint 5 encoding tests Delta varint algorithm correctness

Manifest updates:

  • Added missing_feature entries to language manifests (tests written before tracer implementation)

Why parametric only (no E2E)?

  • Span enrichment is internal tracer behavior, not weblog behavior
  • Weblog FFE integration is already tested by test_exposures.py and test_dynamic_evaluation.py
  • Parametric tests provide precise control for edge cases (limits, child span propagation)

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 29, 2026

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/span-enrichment-flags.json                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_span_enrichment.py                       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/utils.py                                      @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core

@sameerank sameerank force-pushed the sameerank/FFL-2201/feature-flag-apm-enrichment-tests branch 6 times, most recently from dfe6168 to 537d098 Compare April 29, 2026 16:09
Copy link
Copy Markdown
Contributor Author

@sameerank sameerank Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File name enforced by CI rule

if file in ("utils.py", "conftest.py", "__init__.py"):

Comment on lines +6 to +7
3. Max serial ID limit (128)
4. Max subjects limit (25)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just picked these numbers randomly, provisionally so I could write tests. What limits should we actually use?

@sameerank sameerank force-pushed the sameerank/FFL-2201/feature-flag-apm-enrichment-tests branch from 537d098 to 7370a3b Compare April 29, 2026 17:39
@datadog-datadog-prod-us1-2
Copy link
Copy Markdown

datadog-datadog-prod-us1-2 Bot commented Apr 29, 2026

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2fa1e11 | Docs | Datadog PR Page | Give us feedback!

@sameerank sameerank force-pushed the sameerank/FFL-2201/feature-flag-apm-enrichment-tests branch 3 times, most recently from e63833a to 12da3a9 Compare May 5, 2026 00:06
Add parametric tests for feature flag APM span enrichment:

- Test_Span_Enrichment_Serial_IDs: verify multiple flag evaluations combine
  serial IDs correctly in feature_flags_encoded using delta varint encoding
- Test_Span_Enrichment_Child_Span_Propagation: verify flag evaluations in
  child spans propagate to root span
- Test_Span_Enrichment_Max_Serial_IDs: verify 128 serial ID limit enforcement
- Test_Span_Enrichment_Max_Subjects: verify 25 subject limit enforcement
- Test_Span_Enrichment_Default_Fallback: verify feature_flags tag with
  coded-default prefix when flag not found in UFC, including 64 char truncation
- Test_Span_Enrichment_Subjects: verify feature_flag_subjects_encoded behavior
  based on doLog flag, including SHA256 hashing of targeting keys
- Test_Span_Enrichment_Delta_Varint: unit tests for encoding utilities

Add utilities for delta varint encoding/decoding and SHA256 targeting key
hashing. Mark tests as missing_feature in all library manifests.
@sameerank sameerank force-pushed the sameerank/FFL-2201/feature-flag-apm-enrichment-tests branch from 12da3a9 to 2fa1e11 Compare May 5, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant