Skip to content

ai guard telemetry metrics#6843

Open
IlyasShabi wants to merge 3 commits intomainfrom
ishabi/ai-guard-telemetry
Open

ai guard telemetry metrics#6843
IlyasShabi wants to merge 3 commits intomainfrom
ishabi/ai-guard-telemetry

Conversation

@IlyasShabi
Copy link
Copy Markdown
Contributor

@IlyasShabi IlyasShabi commented May 4, 2026

Motivation

Improve AI Guard telemetry metrics coverage

Changes

  • Validate request tags
  • Validate truncation tags

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026 •
edited
Loading

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ai_guard/test_ai_guard_sdk.py                                     @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

IlyasShabi
IlyasShabi commented May 4, 2026
View reviewed changes
Comment thread utils/_context/_scenarios/__init__.py
)

ai_guard_telemetry = AIGuardScenario(
"AI_GUARD_TELEMETRY",
Copy link
Copy Markdown
Contributor Author

@IlyasShabi IlyasShabi May 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding new scenario for low truncation thresholds

@IlyasShabi IlyasShabi marked this pull request as ready for review May 4, 2026 12:56
@IlyasShabi IlyasShabi requested review from a team as code owners May 4, 2026 12:56
@IlyasShabi IlyasShabi requested review from claponcet, florentinl, manuel-alvarez-alvarez, quinna-h and tabgok and removed request for a team May 4, 2026 12:56
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 4, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1678adb31c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread manifests/nodejs.yml Outdated
@IlyasShabi IlyasShabi marked this pull request as draft May 4, 2026 14:51
@IlyasShabi IlyasShabi force-pushed the ishabi/ai-guard-telemetry branch from 1678adb to aff6f02 Compare May 4, 2026 14:53
@IlyasShabi IlyasShabi marked this pull request as ready for review May 5, 2026 08:42
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 5, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aff6f021bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread manifests/nodejs.yml
Comment on lines +96 to +97
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryRequests: missing_feature
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryTruncated: missing_feature
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot May 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enable at least one telemetry manifest entry

Both new telemetry classes are declared missing_feature here, and the same two entries were added to every other language manifest in this commit (cpp_httpd, cpp_nginx, dotnet, golang, java, php, python, and ruby), so AI_GUARD_TELEMETRY has no library where these tests can actually run. In CI jobs that include this new scenario, the telemetry assertions are therefore skipped/xfail everywhere instead of providing the intended AI Guard telemetry coverage.

Useful? React with 👍 / 👎.

@IlyasShabi IlyasShabi requested review from avara1986 and smola May 5, 2026 09:29
avara1986
avara1986 reviewed May 5, 2026
View reviewed changes
Comment thread tests/ai_guard/test_ai_guard_sdk.py
avara1986
avara1986 reviewed May 5, 2026
View reviewed changes
Comment thread tests/ai_guard/test_ai_guard_sdk.py
)


@features.ai_guard
Copy link
Copy Markdown
Member

@avara1986 avara1986 May 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it okay to use the same feature? Should we create a new entry in the Feature Parity Dashboard? cc: @smola

Copy link
Copy Markdown
Contributor Author

@IlyasShabi IlyasShabi May 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we don't use new feature for telemetry on appsec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avara1986 avara1986 avara1986 left review comments

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@florentinl florentinl Awaiting requested review from florentinl florentinl is a code owner automatically assigned from DataDog/apm-python

@quinna-h quinna-h Awaiting requested review from quinna-h quinna-h is a code owner automatically assigned from DataDog/apm-python

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/asm-java

@claponcet claponcet Awaiting requested review from claponcet claponcet is a code owner automatically assigned from DataDog/asm-java

@tabgok tabgok Awaiting requested review from tabgok tabgok is a code owner automatically assigned from DataDog/dd-trace-cpp

@smola smola Awaiting requested review from smola

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@IlyasShabi @avara1986