Skip to content

feat(kernel): send per-statement query tags on the kernel backend - #470

Merged
rahuls-db merged 2 commits into
mainfrom
feat/kernel-query-tags
Sep 4, 2026
Merged

feat(kernel): send per-statement query tags on the kernel backend#470
rahuls-db merged 2 commits into
mainfrom
feat/kernel-query-tags

Conversation

@rahuls-db

@rahuls-db rahuls-db commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What

Brings the kernel (SEA) backend to parity with the Thrift backend for
per-statement query tags. The Thrift backend already reads tags from the
request context and sends them via confOverlay["query_tags"]; the kernel backend
ignored them because the kernel C ABI had no per-statement tags setter. It now does
(kernel_statement_set_query_tags, merged as
databricks-sql-kernel 21504eae), so this wires it up.

No new user-facing API — the same driverctx.NewContextWithQueryTags(ctx, …)
that already works on Thrift now also works on the kernel backend.

How

  • internal/backend/kernel/operation.go: after binding parameters and before
    execute, read driverctx.QueryTagsFromContext(ctx), serialize with the shared
    querytags.Serialize, and pass the wire string to kernel_statement_set_query_tags.
    Empty/absent → skipped.
  • Interior-NUL guard (internal/backend/kernel/querytags_nul.go):
    checkQueryTags/errQueryTagsNUL rejects a serialized tags string with an
    interior NUL before the length-less C setter would silently truncate it — the
    query-tags counterpart to checkQueryText (set_sql) and checkParamValue
    (bind_parameter), keeping the kernel path at parity with the whole-string Thrift
    path. Pure Go, unit-tested under CGO_ENABLED=0 (querytags_nul_test.go).
  • KERNEL_REV + committed header: bumped to 21504eae and the C-ABI header
    synced to that rev. The databricks_kernel CI leg source-builds the kernel .a
    from KERNEL_REV (make kernel-lib), so this is what lets it link the new setter.
    The header sync also carries the other C-ABI decls that landed since the prior pin
    (max_connections, telemetry circuit breaker, a result-stream canceller type, the
    kernel_session_test timeout param); the Go backend calls none of them, so no
    binding code changes were needed.

Note on the published bindings

go.mod still pins the databricks-sql-kernel-bindings modules at v1.0.0 — a
matching bindings release (built from a kernel rev ≥ 21504eae) is a fast-follow
for go get -tags databricks_kernel consumers. CI and this repo's tagged tests build
the kernel from KERNEL_REV and are unaffected; the pure-Go default build never
touches the kernel.

Validation (local)

Built the kernel static/shared lib from 21504eae (the new KERNEL_REV) and, with
the synced header, ran the databricks_kernel-tagged path against it:

  • go build / go test (-tags databricks_kernel, CGO on) — pass.
  • checkQueryTags guard test under CGO_ENABLED=0 — pass.
  • Default pure-Go go build ./... + gofmt — clean.
  • Live warehouse E2E: connected on the kernel backend with statement-level tags
    in context and ran a tagged query end to end — statement id
    01f1a7e7-5834-15c7-bd90-bb1c7e0f144d on warehouse 153fb3a25b8051a7.

This pull request and its description were written by Isaac.

@rahuls-db
rahuls-db requested a review from vuanhphung September 3, 2026 22:36
Comment thread internal/backend/kernel/include/databricks_kernel.h
@rahuls-db
rahuls-db marked this pull request as ready for review September 4, 2026 16:52

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-scoped wiring that mirrors the Thrift query-tags path closely (same source, serializer, empty-string skip) and handles kernel error/cleanup correctly. One medium concern: unlike SQL text and bound params, the serialized query-tags string reaches a length-less kernel C ABI setter without the interior-NUL guard this repo explicitly maintains for parity — a NUL would silently truncate tags on the kernel path while Thrift sends them whole.

Comment thread internal/backend/kernel/operation.go
@rahuls-db
rahuls-db force-pushed the feat/kernel-query-tags branch from 3a63b2b to 6cea7c5 Compare September 4, 2026 18:55

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

rahuls-db and others added 2 commits September 4, 2026 19:11
The Thrift backend already forwards per-statement query tags (from
driverctx.QueryTagsFromContext) via confOverlay["query_tags"]; the kernel
(SEA) backend did not, because the kernel C ABI had no per-statement tags
setter. It now does (kernel_statement_set_query_tags), so wire it up:

- internal/backend/kernel/operation.go: after binding params, read the
  context query tags, serialize with the shared querytags.Serialize, and
  pass the wire string to kernel_statement_set_query_tags before execute.
  Reuses the same user-facing API and serializer as the Thrift path.
- internal/backend/kernel/querytags_nul.go: checkQueryTags/errQueryTagsNUL
  guards the serialized string against an interior NUL before the length-less
  C setter silently truncates it — the query-tags counterpart to
  checkQueryText (set_sql) and checkParamValue (bind_parameter), keeping the
  kernel path at parity with Thrift (which sends tags whole). Pure Go, tested
  under CGO_ENABLED=0 (querytags_nul_test.go).
- internal/backend/kernel/include/databricks_kernel.h: add the declaration.

NOTE (do not merge until unblocked): the databricks_kernel-tagged CI leg can
only link once the databricks-sql-kernel-bindings modules are rebuilt at a
kernel rev that includes kernel_statement_set_query_tags (merged as
21504eae074c7f5b11888a4ebcc7468780b2f844), and KERNEL_REV + the go.mod
bindings versions are bumped to that rev in a follow-up commit. Validated
locally by building/linking/vetting the kernel-tagged package + running the
guard test against a kernel lib built from that rev.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>
Advance the pinned kernel to 21504eae074c7f5b11888a4ebcc7468780b2f844
(databricks-sql-kernel main, which merged kernel_statement_set_query_tags
via #314) and sync the committed C-ABI header to that rev. The
databricks_kernel CI leg source-builds the kernel .a from KERNEL_REV
(make kernel-lib), so this is what lets it link the new setter and run the
tagged tests.

The header sync also pulls in the other C-ABI declarations that landed
since the previous pin (max_connections, telemetry circuit breaker, a
result-stream canceller type, and the kernel_session_test timeout param);
the Go backend calls none of them, so no binding code changes are needed.

Note: the published databricks-sql-kernel-bindings modules (go.mod, still
v1.0.0) are NOT bumped here — a matching bindings release is a fast-follow
for `go get -tags databricks_kernel` consumers. CI and this repo's tagged
tests build from KERNEL_REV and are unaffected.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>
@rahuls-db
rahuls-db force-pushed the feat/kernel-query-tags branch from 446f9dd to 949edc8 Compare September 4, 2026 19:12

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — a faithful port of the Thrift per-statement query-tags behavior to the kernel backend, reusing the shared serializer and mirroring the existing set_sql/bind_parameter NUL guards; C-string lifetimes and error/eviction handling are correct. One low-severity heads-up: the tagged build now needs a symbol only present at KERNEL_REV≥21504eae while go.mod still pins bindings at v1.0.0 (acknowledged fast-follow in the PR).

Other findings

  • 🔵 Low — The tagged build now references kernel_statement_set_query_tags, a symbol added at KERNEL_REV=21504eae, but go.mod still pins the published databricks-sql-kernel-bindings modules at v1.0.0 (built from a pre-21504eae rev). Any consumer running go get -tags databricks_kernel against the prebuilt v1.0.0 archives will hit an undefined-symbol link failure — the setter is not in those .a files. CI and this repo's tagged tests source-build from KERNEL_REV (make kernel-lib) and are unaffected, and the default pure-Go build never touches the kernel, so the blast radius is limited to opt-in prebuilt-bindings consumers.

The PR description explicitly acknowledges this as an intended fast-follow, so this is a heads-up rather than a blocker: the matching bindings release must land before this new capability is usable via prebuilt archives, otherwise the tagged build breaks for those users.

@rahuls-db
rahuls-db enabled auto-merge September 4, 2026 19:37
@rahuls-db
rahuls-db added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 1b6e8c5 Sep 4, 2026
12 of 13 checks passed
@rahuls-db
rahuls-db deleted the feat/kernel-query-tags branch September 4, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants