Skip to content

Stabilize OpenFeature provider event tests - #12278

Open
leoromanovsky wants to merge 2 commits into
masterfrom
leo.romanovsky/fix-provider-test-readiness-flake
Open

Stabilize OpenFeature provider event tests#12278
leoromanovsky wants to merge 2 commits into
masterfrom
leo.romanovsky/fix-provider-test-readiness-flake

Conversation

@leoromanovsky

Copy link
Copy Markdown
Contributor

Motivation

ProviderTest is marked flaky after repeated failures on master ARM64 CI jobs under Semeru 17 and JDK 27. The affected tests, testSetProvider() and testNullConfigurationAfterReadyTransitionsToErrorAndRecovers(), polled provider state with a one-second deadline. The configuration gateway already retains and replays the latest configuration, so the failures did not indicate a lost update; they occurred when shared-runner scheduling delayed OpenFeature's asynchronous worker or event handler past that deadline. The tests were therefore measuring executor latency rather than the provider contract.

Changes

The asynchronous tests now complete futures from the actual OpenFeature ready, error, and configuration-changed events and assert provider state after the expected event arrives. testSetProviderAndWait() waits for its submitted initialization operation, while the recovery-state test preloads configuration and initializes synchronously before exercising error and recovery transitions. The ten-second future deadline remains only as hang protection, not as a polling window. Since this was the module's last Awaitility usage, the unused test dependency is removed.

Decisions

Production code is unchanged because the live failures point to test orchestration, not a runtime provider defect. Synchronizing on semantic provider events preserves the behavior under test and gives a precise failure when an expected transition never occurs; simply increasing the Awaitility timeout would retain the scheduler-dependent race and make the suite slower to diagnose.

Verification

  • Ran ProviderTest successfully in 10 fresh Gradle test executions.
  • Ran ProviderTest with JDK 17.
  • Ran ProviderTest with JDK 25, the newest locally available toolchain and closest proxy for the observed JDK 27 failure.
  • Ran all 392 feature-flagging-api tests.
  • Ran :products:feature-flagging:feature-flagging-api:spotlessCheck.

Wait for provider events instead of polling asynchronous SDK state with a one-second deadline. Initialize recovery coverage synchronously and remove the unused Awaitility test dependency.

Environment: Datadog workspace
@leoromanovsky leoromanovsky added type: bug fix Bug fix tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Aug 24, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.80 s 14.76 s [-0.7%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 13.68 s 13.67 s [-0.8%; +0.9%] (no difference)
startup:petclinic:appsec:Agent 17.51 s 16.83 s [-0.5%; +8.6%] (no difference)
startup:petclinic:iast:Agent 17.41 s 17.55 s [-1.7%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 17.40 s 17.29 s [-0.2%; +1.5%] (no difference)
startup:petclinic:sca:Agent 17.46 s 17.16 s [+1.0%; +2.5%] (maybe worse)
startup:petclinic:tracing:Agent 16.57 s 16.72 s [-2.1%; +0.3%] (no difference)

Commit: 41cdfdfb · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@leoromanovsky
leoromanovsky marked this pull request as ready for review August 24, 2026 22:26
@leoromanovsky
leoromanovsky requested review from a team as code owners August 24, 2026 22:26
@leoromanovsky
leoromanovsky requested review from amarziali, danyal002 and greghuels and removed request for a team August 24, 2026 22:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3eebb73ff0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@leoromanovsky
leoromanovsky requested review from AlexeyKuznetsov-DD and removed request for amarziali August 24, 2026 22:28

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The tests now wait for provider events instead of repeated state checks. The changed logic keeps the same state checks and has no reportable defect.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 3eebb73 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Wrap the provider event futures with verifiable consumers so deterministic waiting continues to reject duplicate transition events.

Environment: Datadog workspace

@amarziali amarziali left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants