Skip to content

uploader: reduce validation work and compact batch metadata - #305

Merged
tonyredondo merged 2 commits into
mainfrom
perf/uploader-throughput
Nov 13, 2026
Merged

tonyredondo merged 2 commits into
mainfrom
perf/uploader-throughput

Conversation

@tonyredondo

@tonyredondo tonyredondo commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Large uploads repeat the same Git, CI, OS and Bazel tags in every event. This PR stores common string tags once in the shared metadata of an opt-in batch and avoids an unnecessary keyword scan during schema validation.

Stacked on #304. Batching stays disabled by default.

What changes

  • Only tags beginning with bazel., git., ci., os. or runtime. move, and only when every supported event has the same string value. A key mentioned in another metadata scope stays in the events so its precedence cannot change.
  • Per-event runtime-id, test fields, hierarchy IDs and services keep their existing locations. Varying target tags stay per event. The global runtime fallback is preserved.
  • The final request is validated after compaction and remains bounded by the existing byte limit. Original payload files, source accounting, retries, rejection fallback and retention keep their current behavior.
  • The schema validator uses one set-disjointness check before its existing unsupported-keyword loop. Its validation rules, warnings and errors are unchanged.

Coverage, telemetry, ordinary per-file uploads, gzip settings and HTTP transport are unchanged. Compaction adds no setting: the existing batching switch controls it. Its timing appears in performance.worker_phases.metadata_compaction.

Measurements

Controlled local A-B-B-A runs used CPython 3.14.10 on macOS arm64, eight workers and the same 4,096-file, 94.3 MB synthetic corpus derived from four Mini E2E payloads. Each source tree ran dry-run, HTTP upload, HTTP upload, then dry-run. The receiver was a separate loopback process. Source manifests and corpus hashes were checked; all 24,576 effective events and original source bytes matched.

These are two separate comparisons, not cumulative estimates:

Change Measurement Before After
Schema guard, ordinary uploads Validation CPU per upload pass 2.959 s 2.621 s
Schema guard, ordinary uploads Total upload wall time 6.037 s 5.623 s
Compaction, with the guard in both trees Decoded JSON bytes 92,863,484 41,388,284
Compaction, with the guard in both trees Gzip bytes 1,910,263 1,402,891
Compaction, with the guard in both trees Process CPU per upload pass 7.283 s 6.113 s
Compaction, with the guard in both trees Total upload wall time 6.979 s 5.930 s

The compaction comparison sent 64 requests in both versions. The lower byte count comes from compaction, not fewer requests. Timings are means of four HTTP passes per version. This small-file scale fixture does not represent a production size distribution and excludes TLS, remote staging and intake latency. A gzip-level experiment did not show enough benefit to justify a change. No connection-pooling change is proposed without supporting measurements.

Validation

  • 662 Python tests passed (one platform-specific skip).
  • Schema differential: 1,092 cases preserved diagnostics and statistics; deliberate broken-guard controls failed as expected.
  • Metadata differential: 2,000 seeded mixed-scope/type/value documents preserved complete effective events and idempotence.
  • Actual Mini shard/retry/empty/cached consumer E2E passed in Bzlmod and WORKSPACE, including complete effective-event and original-carrier comparisons. The release bundle generated from this exact commit also passed checksum-pinned HTTP download, doctor and enriched mock-upload checks for four targets in each frontend.
  • Local Bazel execution of the Python suite failed in the macOS sandbox: PowerShell aborts in its assembly loader. The same 662 tests passed directly, including PowerShell cases. GitHub Actions at 07ff0296 passed all 34 checks, including the native suite on Linux/macOS/Windows and rules_go 0.59.0–0.65.0 in both resolution modes. CI run.
  • Independent auditor found one E2E proof gap: an uncompressed fallback could masquerade as gzip savings. The check now requires gzip on every request; six failing-before controls and all 64 E2E helper tests pass.

Manual staging validation

A separate check used the Python uploader workers and HTTP transport from 07ff02963a79485da96de842d44540fd461a7b17 with synthetic Mini-derived payloads. Both ordinary uploads (four requests) and batching with compaction (one request) indexed all 24 source events: 12 tests, four suites, four modules and four sessions. Gzip bodies totalled 4,301 and 1,873 bytes respectively. Readback preserved the four process runtime IDs, varying targets and promoted Git/CI/OS/runtime tags. Complete effective outbound events matched after normalizing the intentionally distinct experiment identities; original files were unchanged.

Real rejection checks also passed:

Case Intake responses Result
Invalid API key 403 No retry or per-file fan-out; both source files retained byte-for-byte.
Shared batch rejected 400, 202, 202 Per-file fallback recovered both sources; all 12 events were indexed. Files were deleted only after their successful uploads.
One source also rejected 400, 400, 202 The rejected source stayed intact; the other uploaded successfully and all six of its events were indexed.

The 400s came from deliberately invalid gzip injected only at the test transport boundary; the intake responses were real. These checks exercised the uploader workers directly, with the normal retry policy and keep_payloads=False for rejection tests. They do not cover every semantic rejection, HTTP 413/422/429/5xx or network loss.

Batching remains disabled by default. Representative throughput measurement is deferred until consumer adoption and does not block this opt-in delivery. No production throughput improvement is claimed.

@tonyredondo
tonyredondo marked this pull request as ready for review November 13, 2026 12:07
@tonyredondo
tonyredondo requested a review from a team as a code owner November 13, 2026 12:07

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Compaction remains confined to opt-in batches, with per-event runtime IDs and metadata precedence preserved. Real intake acceptance and rejection behavior remain unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 07ff029

Base automatically changed from fix/reporting-service-ownership to main November 13, 2026 20:32
@tonyredondo
tonyredondo merged commit e1a6e1b into main Nov 13, 2026
35 checks passed
@tonyredondo
tonyredondo deleted the perf/uploader-throughput branch November 13, 2026 20:32

@brunnojob brunnojob left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I independently exercised compact_shared_metadata from this head for each supported event kind (test, test_suite_end, test_module_end, test_session_end, span), using two events with an identical git.commit.sha, distinct runtime-id values and distinct bazel.target values.

For all five kinds, reconstructing effective tags as global defaults plus each event's overrides gives exactly the same metadata before and after compaction. Only the common commit tag moves; process IDs and varying target tags remain per-event. A second compaction returns zero in every case.

This was isolated execution of the exact function with its declared prefix/type constants, not an intake integration test or the full Bazel suite. It adds bounded evidence for metadata preservation and idempotence; it does not establish real intake acceptance or the performance benefit. No issue found in these checked cases. Prepared with AI assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants