feat: emit TaskBlock events for JVM blocking states#665
Conversation
CI Test ResultsRun: #30016752921 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Failed Testsglibc-aarch64/debug / 25-graalJob: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 25-graalJob: View logs No detailed failure information available. Check the job logs. Summary: Total: 32 | Passed: 30 | Failed: 2 Updated: 2026-07-23 15:29:29 UTC |
1beeda2 to
38efbf8
Compare
Benchmark Results (commit 38efbf8)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125051342 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10365 ms (21 iters) | ✅ 10195 ms (21 iters) | ≈ -1.6% (±11.4%) | — / — |
| akka-uct | 25 | ✅ 8859 ms (24 iters) | ✅ 8925 ms (24 iters) | ≈ +0.7% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5945 ms (33 iters) | ✅ 5953 ms (33 iters) | ≈ +0.1% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5495 ms (36 iters) | ✅ 5431 ms (36 iters) | ≈ -1.2% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2719 ms (68 iters) | ✅ 2830 ms (66 iters) | 🔴 +4.1% | — / — |
| fj-kmeans | 25 | ✅ 2737 ms (68 iters) | ✅ 2815 ms (66 iters) | 🔴 +2.8% | — / — |
| future-genetic | 21 | ✅ 2114 ms (88 iters) | ✅ 2140 ms (87 iters) | ≈ +1.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2080 ms (90 iters) | ✅ 2126 ms (88 iters) | ≈ +2.2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1299 ms (131 iters) | ✅ 1232 ms (139 iters) | ≈ -5.2% (±31.9%) | — / — |
| naive-bayes | 25 | ✅ 997 ms (172 iters) | ✅ 1004 ms (170 iters) | ≈ +0.7% (±31.8%) | — / — |
| reactors | 21 | ✅ 16202 ms (15 iters) | ✅ 15908 ms (15 iters) | ≈ -1.8% (±7.6%) | — / — |
| reactors | 25 | ✅ 18524 ms (15 iters) | ✅ 18484 ms (15 iters) | ≈ -0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2045 / 1958 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 2353 / 2059 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8437 / 8382 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8800 / 8439 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1248 / 1279 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1262 / 1271 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2963 / 2980 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2955 / 2936 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 7 | 3466 / 3539 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 1 | 3478 / 3531 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1700 / 1609 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1873 | ✅ / ✅ | ✅ / ✅ |
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/126552005 |
38efbf8 to
cdb8274
Compare
Benchmark Results (commit cdb8274)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125162043 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| finagle-chirper | 21 | ✅ 5956 ms (33 iters) | ✅ 6031 ms (33 iters) | ≈ +1.3% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5457 ms (36 iters) | ✅ 5444 ms (36 iters) | ≈ -0.2% (±23.9%) | |
| fj-kmeans | 21 | ✅ 2683 ms (70 iters) | ✅ 2821 ms (66 iters) | 🔴 +5.1% | — / — |
| fj-kmeans | 25 | ✅ 2786 ms (67 iters) | ✅ 2839 ms (66 iters) | ≈ +1.9% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2063 ms (90 iters) | ✅ 2095 ms (89 iters) | ≈ +1.6% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2072 ms (90 iters) | ✅ 2125 ms (87 iters) | ≈ +2.6% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1257 ms (135 iters) | ✅ 1250 ms (136 iters) | ≈ -0.6% (±32.8%) | — / — |
| reactors | 21 | ✅ 16137 ms (15 iters) | ✅ 15974 ms (15 iters) | ≈ -1% (±7.5%) | — / — |
| reactors | 25 | ✅ 18395 ms (15 iters) | ✅ 18505 ms (15 iters) | ≈ +0.6% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2260 / 2182 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8741 / 8673 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 8238 / 8373 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1241 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2971 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2968 / 2956 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3468 / 3474 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 9 / 3 | 3464 / 3465 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1531 / 1536 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1857 / 1856 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit e13ec27)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125211559 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
e13ec27 to
c111812
Compare
Benchmark Results (commit c111812)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125236325 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
c111812 to
df472c0
Compare
Benchmark Results (commit df472c0)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125568035 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | ✅ 8806 ms (24 iters) | ✅ 8924 ms (24 iters) | ≈ +1.3% (±11.1%) | — / — |
| finagle-chirper | 21 | ✅ 6033 ms (33 iters) | ✅ 5987 ms (33 iters) | ≈ -0.8% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5500 ms (36 iters) | ✅ 5495 ms (36 iters) | ≈ -0.1% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2692 ms (70 iters) | ✅ 2827 ms (66 iters) | 🔴 +5% | — / — |
| future-genetic | 21 | ✅ 2075 ms (89 iters) | ✅ 2042 ms (91 iters) | ≈ -1.6% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2036 ms (91 iters) | ✅ 2023 ms (91 iters) | ≈ -0.6% (±2.8%) | — / — |
| naive-bayes | 21 | ✅ 1228 ms (138 iters) | ✅ 1283 ms (133 iters) | ≈ +4.5% (±33.5%) | — / — |
| reactors | 21 | ✅ 16202 ms (15 iters) | ✅ 16354 ms (15 iters) | ≈ +0.9% (±6.6%) | — / — |
| reactors | 25 | ✅ 18212 ms (15 iters) | ✅ 18060 ms (15 iters) | ≈ -0.8% (±4.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 2176 / 2305 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8602 / 8811 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 8746 / 8149 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1255 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 1280 / 1287 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 2966 / 2985 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 2906 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3486 / 3512 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3483 / 3521 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1654 / 1756 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1822 / 1798 | ✅ / ✅ | ✅ / ✅ |
df472c0 to
46fdadd
Compare
Benchmark Results (commit 46fdadd)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125595112 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
46fdadd to
658df77
Compare
Benchmark Results (commit 658df77)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125689745 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10260 ms (21 iters) | ✅ 10296 ms (21 iters) | ≈ +0.4% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8840 ms (24 iters) | ✅ 8942 ms (24 iters) | ≈ +1.2% (±10.6%) | — / — |
| finagle-chirper | 21 | ✅ 6010 ms (33 iters) | ✅ 5968 ms (33 iters) | ≈ -0.7% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5480 ms (36 iters) | ✅ 5467 ms (36 iters) | ≈ -0.2% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2679 ms (69 iters) | ✅ 2773 ms (67 iters) | 🔴 +3.5% | — / — |
| fj-kmeans | 25 | ✅ 2769 ms (68 iters) | ✅ 2839 ms (66 iters) | ≈ +2.5% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2051 ms (90 iters) | ✅ 2133 ms (87 iters) | 🔴 +4% | — / — |
| future-genetic | 25 | ✅ 2047 ms (91 iters) | ✅ 2037 ms (91 iters) | ≈ -0.5% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1254 ms (136 iters) | ✅ 1314 ms (131 iters) | ≈ +4.8% (±33.9%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 983 ms (173 iters) | ≈ -3.2% (±31%) | — / — |
| reactors | 21 | ✅ 16594 ms (15 iters) | ✅ 16990 ms (15 iters) | ≈ +2.4% (±9.1%) | — / — |
| reactors | 25 | ✅ 18557 ms (15 iters) | ✅ 18660 ms (15 iters) | ≈ +0.6% (±4.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1856 / 2016 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2373 / 2253 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 7 | 8704 / 8448 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8503 / 8277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1260 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1282 / 1284 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2984 / 2927 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2889 / 2841 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 3476 / 3519 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 3442 / 3468 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1680 / 1865 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1868 / 1860 | ✅ / ✅ | ✅ / ✅ |
658df77 to
746edc2
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds first-class TaskBlock emission for common JVM blocking states (park, Object.wait, monitor contention), and reworks wall-clock precheck/suppression to use lifecycle-owned blocked intervals (instead of the prior once-per-run suppression model) while preserving context-window sampling semantics.
Changes:
- Introduces a new
datadog.TaskBlockJFR event and a public paired Java API (beginTaskBlock/endTaskBlock) plus internal park/monitor producers. - Implements native lifecycle ownership + unfiltered-thread registry tracking and a bounded candidate backfill strategy to prevent O(N) wall-tick regressions.
- Adds broad Java + C++ test coverage (including restart/pre-existing-thread scenarios, JVMTI-delegated variants, and benchmark hooks).
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/WallclockMitigationsCombinedTest.java | Updates combined mitigation test expectations to ensure context-scoped threads remain sampled. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java | New integration tests for owned-block suppression behavior when filter= samples all threads. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java | New tests ensuring unfiltered tracking/registry state is correctly reset across recording restarts. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/TaskBlockAssertions.java | New shared assertions for the datadog.TaskBlock event contract. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java | Refactors precheck tests to the new lifecycle-owned suppression model and unowned blocked weighting counters. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckEfficiencyTest.java | Updates efficiency tests to match lifecycle ownership semantics and explicit unfiltered sampling configuration. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/ParkTaskBlockTest.java | New tests verifying TaskBlock emission from Java-owned park hooks and suppression counters. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/MonitorTaskBlockTest.java | New tests verifying TaskBlock emission from JVMTI monitor callbacks and restart recovery. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java | New subclass to exercise unfiltered precheck via delegated JVMTI stack collection. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedPrecheckTest.java | Updates command strings to use explicit filter= under new behavior. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedParkTaskBlockTest.java | New JVMTI-delegated variant of park TaskBlock tests. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedMonitorTaskBlockTest.java | New JVMTI-delegated variant of monitor TaskBlock tests. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockPreExistingThreadTest.java | New test covering TaskBlock TLS initialization for threads created before profiler startup. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockDisabledTest.java | New test verifying TaskBlock API stays inactive outside any sampling scope. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockApiTest.java | New end-to-end tests for the paired TaskBlock API semantics and rejection rules. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java | New test ensuring J9 wall sampling doesn’t activate unfiltered precheck tracking. |
| ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerTest.java | Adds child-JVM agentpath initialization tests and pre-existing monitor callback coverage. |
| ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerApiSurfaceTest.java | Updates API-surface assertions: TaskBlock API public, internal hooks stay non-public, monitor delegation APIs are public. |
| ddprof-test/src/test/java/com/datadoghq/profiler/ExternalLauncher.java | Extends launcher scenarios to cover agent compatibility, delegation conflicts, and pre-existing monitor callbacks. |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/OtelContextStorageModeTest.java | Ensures trace context is cleared during cleanup alongside thread context reset. |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/AllNativeContextTest.java | Ensures trace context is cleared during cleanup alongside thread context reset. |
| ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java | Adds beforeProfilerStart() hook and improves teardown to clear trace context reliably. |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java | New helper exposing package-scoped owned-block hooks to JMH benchmarks. |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java | New benchmark measuring wall-clock timer overhead under growing owned-block populations. |
| ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp | Adds capability + filter parsing tests relevant to unfiltered tracking (filter omitted vs filter=). |
| ddprof-lib/src/test/cpp/wallClockCounters_ut.cpp | Updates counter tests to the renamed owned-block suppression metric. |
| ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp | New unit tests for randomized bounded candidate selection/backfill. |
| ddprof-lib/src/test/cpp/taskBlockRecorder_ut.cpp | New unit tests for TaskBlock eligibility, rotation gating, and error accounting. |
| ddprof-lib/src/test/cpp/park_state_ut.cpp | Expands state-machine tests for park and monitor snapshot/exit behavior and owned-block transitions. |
| ddprof-lib/src/test/cpp/jvmSupport_ut.cpp | New unit tests for platform-thread classification (virtual thread rejection) across JNI versions. |
| ddprof-lib/src/test/cpp/frame_ut.cpp | Adds unit test for safe JVMTI->ASGCT frame copying with overlapping buffers. |
| ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java | Adds monitor-delegation-aware getInstance, TaskBlock public API, and thread-parameterized native hooks. |
| ddprof-lib/src/main/cpp/wallClockCounters.h | Renames and redefines wall-clock suppression counter to owned-block semantics. |
| ddprof-lib/src/main/cpp/wallClockCandidateSelector.h | New header implementing bounded randomized candidate visitation/backfill. |
| ddprof-lib/src/main/cpp/wallClock.h | Integrates candidate selector into wall timer loop and adds engine capability declaration. |
| ddprof-lib/src/main/cpp/wallClock.cpp | Reworks precheck logic to rely on owned-block registry suppression and bounded slot lookup backfill. |
| ddprof-lib/src/main/cpp/vmEntry.h | Adds monitor delegation ownership model and initialization result enum. |
| ddprof-lib/src/main/cpp/vmEntry.cpp | Implements JVMTI monitor callbacks producing TaskBlocks, plus one-time bridge ownership negotiation. |
| ddprof-lib/src/main/cpp/threadLocalData.h | Adds per-thread state for park/monitor/taskblock lifecycles and snapshotting. |
| ddprof-lib/src/main/cpp/threadFilter.h | Adds unfiltered registry tracking, TID index, recording epochs, and owned-block suppression candidate checks. |
| ddprof-lib/src/main/cpp/threadFilter.cpp | Implements registry activation, per-recording epoch publication, lock-protected indexing, and slot lookup APIs. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.h | New TaskBlock recorder API + eligibility helpers + rotation-safe activity RAII. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.cpp | Implements TaskBlock duration threshold and recording/cleanup at exit paths. |
| ddprof-lib/src/main/cpp/profiler.h | Adds TaskBlock recording API, rotation gating fields, and bootstrap registration hook. |
| ddprof-lib/src/main/cpp/profiler.cpp | Implements TaskBlock recording through JVMTI stack capture, rotation gating, registry bootstrap for existing threads, and start/stop integration. |
| ddprof-lib/src/main/cpp/jvmThread.h | Adds supportsNativeThreadIdLookup() used for registry bootstrap logic. |
| ddprof-lib/src/main/cpp/jvmThread.cpp | Implements native-thread-id lookup capability check. |
| ddprof-lib/src/main/cpp/jvmSupport.h | Adds platform-thread classification API. |
| ddprof-lib/src/main/cpp/jvmSupport.cpp | Implements platform-thread classification (virtual-thread rejection) using JNI version + IsVirtualThread. |
| ddprof-lib/src/main/cpp/jfrMetadata.h | Adds new JFR type id for datadog.TaskBlock. |
| ddprof-lib/src/main/cpp/jfrMetadata.cpp | Defines datadog.TaskBlock metadata schema. |
| ddprof-lib/src/main/cpp/javaApi.cpp | Wires new JNI entrypoints for TaskBlock API and updates park/block hooks to be thread-aware and emit TaskBlocks. |
| ddprof-lib/src/main/cpp/frames.h | Adds copyJvmtiFrames() helper to safely convert overlapping JVMTI frame buffers. |
| ddprof-lib/src/main/cpp/flightRecorder.h | Adds TaskBlock recording API to the JFR writer interface. |
| ddprof-lib/src/main/cpp/flightRecorder.cpp | Implements TaskBlock event serialization and a rotation-safe record path. |
| ddprof-lib/src/main/cpp/event.h | Adds TaskBlockEvent struct and updates wall-clock epoch suppressed counter naming. |
| ddprof-lib/src/main/cpp/engine.h | Adds wall-engine capability hook supportsUnfilteredWallPrecheck(). |
| ddprof-lib/src/main/cpp/counters.h | Adds new counters for registry/backfill and TaskBlock lifecycle/accounting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark Results (commit 746edc2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125879034 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10315 ms (21 iters) | ✅ 10281 ms (21 iters) | ≈ -0.3% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8869 ms (24 iters) | ✅ 8962 ms (24 iters) | ≈ +1% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6014 ms (33 iters) | ✅ 5992 ms (33 iters) | ≈ -0.4% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5424 ms (36 iters) | ≈ -0.5% (±24%) | |
| fj-kmeans | 21 | ✅ 2634 ms (72 iters) | ✅ 2749 ms (68 iters) | 🔴 +4.4% | — / — |
| fj-kmeans | 25 | ✅ 2757 ms (68 iters) | ✅ 2825 ms (66 iters) | ≈ +2.5% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2097 ms (88 iters) | ✅ 2137 ms (87 iters) | ≈ +1.9% (±2.8%) | — / — |
| future-genetic | 25 | ✅ 2067 ms (89 iters) | ✅ 2031 ms (91 iters) | ≈ -1.7% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1253 ms (136 iters) | ✅ 1253 ms (137 iters) | ≈ 0% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1008 ms (169 iters) | ✅ 1002 ms (170 iters) | ≈ -0.6% (±31.9%) | — / — |
| reactors | 21 | ✅ 17619 ms (15 iters) | ✅ 15712 ms (16 iters) | 🟢 -10.8% | — / — |
| reactors | 25 | ✅ 18550 ms (15 iters) | ✅ 18609 ms (15 iters) | ≈ +0.3% (±4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1987 / 1971 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2310 / 2435 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8921 / 8512 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8447 / 8030 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1274 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1280 / 1270 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 2912 / 3053 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2767 / 2834 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 3517 / 3509 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 6 | 3483 / 3497 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1802 / 1666 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1883 / 1920 | ✅ / ✅ | ✅ / ✅ |
746edc2 to
1db6b94
Compare
1db6b94 to
e9b2308
Compare
This comment has been minimized.
This comment has been minimized.
e9b2308 to
cdbed88
Compare
Benchmark Results (commit cdbed88)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126290898 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10260 ms (21 iters) | ✅ 10222 ms (21 iters) | ≈ -0.4% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8935 ms (24 iters) | ✅ 8930 ms (24 iters) | ≈ -0.1% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 6038 ms (33 iters) | ✅ 5962 ms (33 iters) | ≈ -1.3% (±24.4%) | |
| finagle-chirper | 25 | ✅ 5468 ms (36 iters) | ✅ 5427 ms (36 iters) | ≈ -0.7% (±24%) | |
| fj-kmeans | 21 | ✅ 2764 ms (68 iters) | ✅ 2816 ms (66 iters) | ≈ +1.9% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2802 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2130 ms (87 iters) | ✅ 2084 ms (89 iters) | ≈ -2.2% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2023 ms (91 iters) | ✅ 2105 ms (89 iters) | 🔴 +4.1% | — / — |
| naive-bayes | 21 | ✅ 1254 ms (137 iters) | ✅ 1294 ms (132 iters) | ≈ +3.2% (±32.7%) | — / — |
| naive-bayes | 25 | ✅ 965 ms (176 iters) | ✅ 1011 ms (169 iters) | ≈ +4.8% (±32.6%) | — / — |
| reactors | 21 | ✅ 17025 ms (15 iters) | ✅ 16431 ms (15 iters) | ≈ -3.5% (±7.3%) | — / — |
| reactors | 25 | ✅ 18745 ms (15 iters) | ✅ 18914 ms (15 iters) | ≈ +0.9% (±4.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1951 / 2053 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2293 / 2284 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 8904 / 8718 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8249 / 8246 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1302 / 1261 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1251 / 1245 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 2947 / 2965 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 2754 / 3026 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 8 | 3535 / 3486 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 3472 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1677 / 1597 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1961 / 1906 | ✅ / ✅ | ✅ / ✅ |
cdbed88 to
3208f71
Compare
Benchmark Results (commit 3208f71)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126387908 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3208f71ed4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| Error Profiler::start(Arguments &args, bool reset) { | ||
| MutexLocker ml(_state_lock); | ||
| _task_block_enabled.store(false, std::memory_order_release); |
There was a problem hiding this comment.
Move TaskBlock reset after start state validation
When start is called while the profiler is already RUNNING, checkState() returns Profiler already started; because this store happens before that check, the failed start attempt disables TaskBlock emission for the active recording. In contexts that retry or issue start defensively while profiling is already active, subsequent park/monitor/API exits see taskBlockEnabled()==false and only clean up without recording until a full restart, so move this reset until after checkState() succeeds.
Useful? React with 👍 / 👎.
| public static synchronized JavaProfiler getInstance(String libLocation, String scratchDir, | ||
| boolean delegateMonitorWaitEvents) throws IOException { |
There was a problem hiding this comment.
Honor delegation requests when reusing singleton
For callers that first initialize with the default false and later explicitly request getInstance(..., true), this overload returns the existing singleton before invoking init0(delegateMonitorWaitEvents), so the documented ownership conflict is silently ignored and Object.wait remains JVMTI-owned despite the explicit request. Check the requested ownership against monitorEventsDelegated0() or persist the effective ownership before returning the cached instance.
Useful? React with 👍 / 👎.
3208f71 to
c789f17
Compare
Benchmark Results (commit c789f17)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126491885 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
c789f17 to
d45870a
Compare
Benchmark Results (commit d45870a)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126522777 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
d45870a to
f87e382
Compare
Benchmark Results (commit f87e382)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126552063 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10390 ms (21 iters) | ✅ 10267 ms (21 iters) | ≈ -1.2% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8842 ms (24 iters) | ✅ 8904 ms (24 iters) | ≈ +0.7% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 5926 ms (33 iters) | ✅ 5984 ms (33 iters) | ≈ +1% (±25%) | |
| fj-kmeans | 21 | ✅ 2638 ms (72 iters) | ✅ 2783 ms (67 iters) | 🔴 +5.5% | — / — |
| fj-kmeans | 25 | ✅ 2805 ms (66 iters) | ✅ 2811 ms (66 iters) | ≈ +0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2131 ms (87 iters) | ✅ 2054 ms (90 iters) | 🟢 -3.6% | — / — |
| future-genetic | 25 | ✅ 2047 ms (91 iters) | ✅ 2061 ms (90 iters) | ≈ +0.7% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1272 ms (135 iters) | ✅ 1275 ms (134 iters) | ≈ +0.2% (±32.3%) | — / — |
| reactors | 21 | ✅ 16384 ms (15 iters) | ✅ 15994 ms (15 iters) | ≈ -2.4% (±7.8%) | — / — |
| reactors | 25 | ✅ 18446 ms (15 iters) | ✅ 18728 ms (15 iters) | ≈ +1.5% (±5.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2030 / 2110 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2281 / 2374 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 6 | 8596 / 8613 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1264 / 1248 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1273 / 1259 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2981 / 2924 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2862 / 2843 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3506 / 3500 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1621 / 1786 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1842 / 1896 | ✅ / ✅ | ✅ / ✅ |
What does this PR do?:
Adds JVM TaskBlock producers for
LockSupport.parkand monitor contention/wait states, wiring JVM state transitions into the synchronous TaskBlock recording API.Motivation:
Common JVM blocking operations can be represented as TaskBlock events instead of relying on wall-clock signals against blocked threads.
Additional Notes:
Virtual threads are not explicitly emitted by this PR. This change focuses on platform-thread JVM blocking states exposed through the existing VM entry paths.
How to test the change?:
Automated coverage includes native park-state tests and Java integration tests for park, monitor wait, monitor contention, JVMTI-backed variants, and combined wall-clock mitigation behavior.
./.claude/commands/build-and-summarize testDebugFor Datadog employees:
dd:platform-security-reviewskill, or file a request via the PSEC review form).bewairealso runs automatically on every PR.