Skip to content

Smola/wait conditions#6795

Draft
smola wants to merge 10 commits intomainfrom
smola/wait-conditions
Draft

Smola/wait conditions#6795
smola wants to merge 10 commits intomainfrom
smola/wait-conditions

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Apr 22, 2026

Motivation

Changes

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 22, 2026

CODEOWNERS have been resolved as:

PLAN.md                                                                 @DataDog/system-tests-core
TODO.md                                                                 @DataDog/system-tests-core
tests/test_the_test/test_wait_conditions.py                             @DataDog/system-tests-core
utils/scripts/run_isolated_matrix.py                                    @DataDog/system-tests-core
utils/wait_conditions.py                                                @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-27/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-gunicorn-alpine/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-gunicorn/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-preinstalled/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-unsupported-package-force/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django-uvicorn/Dockerfile  @DataDog/system-tests-core
lib-injection/build/docker/python/dd-lib-python-init-test-django/Dockerfile  @DataDog/system-tests-core
tests/appsec/waf/test_telemetry.py                                      @DataDog/asm-libraries @DataDog/system-tests-core
tests/test_profiling.py                                                 @DataDog/system-tests-core
tests/test_telemetry.py                                                 @DataDog/libdatadog-telemetry @DataDog/apm-sdk-capabilities @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/build/docker/cpp_httpd/httpd.Dockerfile                           @DataDog/system-tests-core
utils/build/docker/java/spring-boot-openliberty.Dockerfile              @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/spring-boot/app.sh                              @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java_otel/spring-boot-native.Dockerfile              @DataDog/opentelemetry @DataDog/system-tests-core
utils/build/docker/java_otel/spring-boot/app.sh                         @DataDog/opentelemetry @DataDog/system-tests-core
utils/build/docker/nodejs/anthropic-js.Dockerfile                       @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4-typescript.Dockerfile                @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4.Dockerfile                           @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express5.Dockerfile                           @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/fastify.Dockerfile                            @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/google_genai-js.Dockerfile                    @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/nextjs.Dockerfile                             @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/openai-js.Dockerfile                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/app.sh                             @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/uds-express4.Dockerfile                       @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs_otel/express4-otel.Dockerfile                 @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.0-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.0.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.1-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.1.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.2-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.2.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.3-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.3.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.4-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.4.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.0-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.0.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.1-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.1.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.2-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.2.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/app.sh                                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.0.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.1.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.2.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.3.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.4.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.0.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.1.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.2.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.5.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/proxy.Dockerfile                                     @DataDog/system-tests-core
utils/build/docker/python/django-poc.Dockerfile                         @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/django-py3.13.Dockerfile                      @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/fastapi.Dockerfile                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask-poc.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/python3.12.Dockerfile                         @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/tornado.Dockerfile                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uds-flask.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python_otel/flask-poc-otel.Dockerfile                @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python_otel/flask-poc-otel/app.sh                    @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/graphql23/app.sh                                @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rack/app.sh                                     @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails42/app.sh                                  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails52/app.sh                                  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails61/app.sh                                  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app.sh                                  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails80/app.sh                                  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/sinatra14/app.sh                                @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/sinatra22/app.sh                                @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/sinatra32/app.sh                                @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/sinatra41/app.sh                                @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/uds-rails.Dockerfile                            @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/uds-sinatra.Dockerfile                          @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/runner.Dockerfile                                    @DataDog/system-tests-core
utils/build/ssi/nodejs/js-app.Dockerfile                                @DataDog/system-tests-core
utils/build/ssi/python/py-app.Dockerfile                                @DataDog/system-tests-core
utils/build/ssi/ruby/rails6-app-deployment-mode.Dockerfile              @DataDog/system-tests-core
utils/build/ssi/ruby/rails6-app-vendored-mode.Dockerfile                @DataDog/system-tests-core
utils/build/ssi/ruby/rails6-app.Dockerfile                              @DataDog/system-tests-core
utils/build/ssi/ruby/ruby-app-deployment-mode.Dockerfile                @DataDog/system-tests-core
utils/build/ssi/ruby/ruby-app-vendored-mode.Dockerfile                  @DataDog/system-tests-core
utils/build/ssi/ruby/ruby-app.Dockerfile                                @DataDog/system-tests-core
utils/build/virtual_machine/microvm/Dockerfile.amazonlinux2023          @DataDog/system-tests-core
utils/build/virtual_machine/microvm/Dockerfile.ubuntu22                 @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-alpine/Dockerfile.template  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multialpine/Dockerfile.python_3_10-alpine  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multialpine/Dockerfile.python_3_11-alpine  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multialpine/Dockerfile.python_3_12-alpine  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multialpine/Dockerfile.python_3_8-alpine  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multialpine/Dockerfile.python_3_9-alpine  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multicontainer/Dockerfile.python_3_10  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multicontainer/Dockerfile.python_3_11  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multicontainer/Dockerfile.python_3_12  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multicontainer/Dockerfile.python_3_8  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python-multicontainer/Dockerfile.python_3_9  @DataDog/system-tests-core
utils/build/virtual_machine/weblogs/python/test-app-python37-container/Dockerfile.template  @DataDog/system-tests-core
utils/interfaces/_core.py                                               @DataDog/system-tests-core

@datadog-prod-us1-3

This comment has been minimized.

@smola smola force-pushed the smola/wait-conditions branch 2 times, most recently from 407b075 to d71fb78 Compare April 30, 2026 16:31
smola and others added 10 commits May 4, 2026 21:24
…) after watermark

After the watermark trace is observed, check whether any telemetry was
arriving before it. If so, wait until every (runtime_id, user_agent) pair
that sent telemetry before the watermark has sent at least one telemetry
message after it.

Two axes require tracking both dimensions:
- runtime_id: weblogs with multiple worker processes (gunicorn, OpenLiberty)
  flush independently per process; a single watermark trace only proves the
  worker that handled it has flushed its traces, not that every worker has
  flushed its pending telemetry.
- user_agent: some tracers run two concurrent telemetry emitters under the
  same runtime_id (tracer emitter + libdatadog emitter, identified by their
  User-Agent). Each has its own flush schedule and must be awaited separately.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@smola smola force-pushed the smola/wait-conditions branch from d71fb78 to a7321bb Compare May 4, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant