Repository navigation
uploader: reduce validation work and compact batch metadata - #305
Conversation
There was a problem hiding this comment.
More details
Compaction remains confined to opt-in batches, with per-event runtime IDs and metadata precedence preserved. Real intake acceptance and rejection behavior remain unverified.
🤖 Bits Code Review · Commit 07ff029
brunnojob
left a comment
There was a problem hiding this comment.
I independently exercised compact_shared_metadata from this head for each supported event kind (test, test_suite_end, test_module_end, test_session_end, span), using two events with an identical git.commit.sha, distinct runtime-id values and distinct bazel.target values.
For all five kinds, reconstructing effective tags as global defaults plus each event's overrides gives exactly the same metadata before and after compaction. Only the common commit tag moves; process IDs and varying target tags remain per-event. A second compaction returns zero in every case.
This was isolated execution of the exact function with its declared prefix/type constants, not an intake integration test or the full Bazel suite. It adds bounded evidence for metadata preservation and idempotence; it does not establish real intake acceptance or the performance benefit. No issue found in these checked cases. Prepared with AI assistance.
Large uploads repeat the same Git, CI, OS and Bazel tags in every event. This PR stores common string tags once in the shared metadata of an opt-in batch and avoids an unnecessary keyword scan during schema validation.
Stacked on #304. Batching stays disabled by default.
What changes
bazel.,git.,ci.,os.orruntime.move, and only when every supported event has the same string value. A key mentioned in another metadata scope stays in the events so its precedence cannot change.runtime-id, test fields, hierarchy IDs and services keep their existing locations. Varying target tags stay per event. The global runtime fallback is preserved.Coverage, telemetry, ordinary per-file uploads, gzip settings and HTTP transport are unchanged. Compaction adds no setting: the existing batching switch controls it. Its timing appears in
performance.worker_phases.metadata_compaction.Measurements
Controlled local A-B-B-A runs used CPython 3.14.10 on macOS arm64, eight workers and the same 4,096-file, 94.3 MB synthetic corpus derived from four Mini E2E payloads. Each source tree ran dry-run, HTTP upload, HTTP upload, then dry-run. The receiver was a separate loopback process. Source manifests and corpus hashes were checked; all 24,576 effective events and original source bytes matched.
These are two separate comparisons, not cumulative estimates:
The compaction comparison sent 64 requests in both versions. The lower byte count comes from compaction, not fewer requests. Timings are means of four HTTP passes per version. This small-file scale fixture does not represent a production size distribution and excludes TLS, remote staging and intake latency. A gzip-level experiment did not show enough benefit to justify a change. No connection-pooling change is proposed without supporting measurements.
Validation
07ff0296passed all 34 checks, including the native suite on Linux/macOS/Windows and rules_go 0.59.0–0.65.0 in both resolution modes. CI run.Manual staging validation
A separate check used the Python uploader workers and HTTP transport from
07ff02963a79485da96de842d44540fd461a7b17with synthetic Mini-derived payloads. Both ordinary uploads (four requests) and batching with compaction (one request) indexed all 24 source events: 12 tests, four suites, four modules and four sessions. Gzip bodies totalled 4,301 and 1,873 bytes respectively. Readback preserved the four process runtime IDs, varying targets and promoted Git/CI/OS/runtime tags. Complete effective outbound events matched after normalizing the intentionally distinct experiment identities; original files were unchanged.Real rejection checks also passed:
The 400s came from deliberately invalid gzip injected only at the test transport boundary; the intake responses were real. These checks exercised the uploader workers directly, with the normal retry policy and
keep_payloads=Falsefor rejection tests. They do not cover every semantic rejection, HTTP 413/422/429/5xx or network loss.Batching remains disabled by default. Representative throughput measurement is deferred until consumer adoption and does not block this opt-in delivery. No production throughput improvement is claimed.