feat(kernel): send per-statement query tags on the kernel backend - #470
Conversation
There was a problem hiding this comment.
Verdict: 1 Medium
Solid, well-scoped wiring that mirrors the Thrift query-tags path closely (same source, serializer, empty-string skip) and handles kernel error/cleanup correctly. One medium concern: unlike SQL text and bound params, the serialized query-tags string reaches a length-less kernel C ABI setter without the interior-NUL guard this repo explicitly maintains for parity — a NUL would silently truncate tags on the kernel path while Thrift sends them whole.
3a63b2b to
6cea7c5
Compare
The Thrift backend already forwards per-statement query tags (from driverctx.QueryTagsFromContext) via confOverlay["query_tags"]; the kernel (SEA) backend did not, because the kernel C ABI had no per-statement tags setter. It now does (kernel_statement_set_query_tags), so wire it up: - internal/backend/kernel/operation.go: after binding params, read the context query tags, serialize with the shared querytags.Serialize, and pass the wire string to kernel_statement_set_query_tags before execute. Reuses the same user-facing API and serializer as the Thrift path. - internal/backend/kernel/querytags_nul.go: checkQueryTags/errQueryTagsNUL guards the serialized string against an interior NUL before the length-less C setter silently truncates it — the query-tags counterpart to checkQueryText (set_sql) and checkParamValue (bind_parameter), keeping the kernel path at parity with Thrift (which sends tags whole). Pure Go, tested under CGO_ENABLED=0 (querytags_nul_test.go). - internal/backend/kernel/include/databricks_kernel.h: add the declaration. NOTE (do not merge until unblocked): the databricks_kernel-tagged CI leg can only link once the databricks-sql-kernel-bindings modules are rebuilt at a kernel rev that includes kernel_statement_set_query_tags (merged as 21504eae074c7f5b11888a4ebcc7468780b2f844), and KERNEL_REV + the go.mod bindings versions are bumped to that rev in a follow-up commit. Validated locally by building/linking/vetting the kernel-tagged package + running the guard test against a kernel lib built from that rev. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>
Advance the pinned kernel to 21504eae074c7f5b11888a4ebcc7468780b2f844 (databricks-sql-kernel main, which merged kernel_statement_set_query_tags via #314) and sync the committed C-ABI header to that rev. The databricks_kernel CI leg source-builds the kernel .a from KERNEL_REV (make kernel-lib), so this is what lets it link the new setter and run the tagged tests. The header sync also pulls in the other C-ABI declarations that landed since the previous pin (max_connections, telemetry circuit breaker, a result-stream canceller type, and the kernel_session_test timeout param); the Go backend calls none of them, so no binding code changes are needed. Note: the published databricks-sql-kernel-bindings modules (go.mod, still v1.0.0) are NOT bumped here — a matching bindings release is a fast-follow for `go get -tags databricks_kernel` consumers. CI and this repo's tagged tests build from KERNEL_REV and are unaffected. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>
446f9dd to
949edc8
Compare
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — a faithful port of the Thrift per-statement query-tags behavior to the kernel backend, reusing the shared serializer and mirroring the existing set_sql/bind_parameter NUL guards; C-string lifetimes and error/eviction handling are correct. One low-severity heads-up: the tagged build now needs a symbol only present at KERNEL_REV≥21504eae while go.mod still pins bindings at v1.0.0 (acknowledged fast-follow in the PR).
Other findings
- 🔵 Low — The tagged build now references
kernel_statement_set_query_tags, a symbol added atKERNEL_REV=21504eae, butgo.modstill pins the publisheddatabricks-sql-kernel-bindingsmodules atv1.0.0(built from a pre-21504eaerev). Any consumer runninggo get -tags databricks_kernelagainst the prebuilt v1.0.0 archives will hit an undefined-symbol link failure — the setter is not in those.afiles. CI and this repo's tagged tests source-build fromKERNEL_REV(make kernel-lib) and are unaffected, and the default pure-Go build never touches the kernel, so the blast radius is limited to opt-in prebuilt-bindings consumers.
The PR description explicitly acknowledges this as an intended fast-follow, so this is a heads-up rather than a blocker: the matching bindings release must land before this new capability is usable via prebuilt archives, otherwise the tagged build breaks for those users.
What
Brings the kernel (SEA) backend to parity with the Thrift backend for
per-statement query tags. The Thrift backend already reads tags from the
request context and sends them via
confOverlay["query_tags"]; the kernel backendignored them because the kernel C ABI had no per-statement tags setter. It now does
(
kernel_statement_set_query_tags, merged asdatabricks-sql-kernel
21504eae), so this wires it up.No new user-facing API — the same
driverctx.NewContextWithQueryTags(ctx, …)that already works on Thrift now also works on the kernel backend.
How
internal/backend/kernel/operation.go: after binding parameters and beforeexecute, read
driverctx.QueryTagsFromContext(ctx), serialize with the sharedquerytags.Serialize, and pass the wire string tokernel_statement_set_query_tags.Empty/absent → skipped.
internal/backend/kernel/querytags_nul.go):checkQueryTags/errQueryTagsNULrejects a serialized tags string with aninterior NUL before the length-less C setter would silently truncate it — the
query-tags counterpart to
checkQueryText(set_sql) andcheckParamValue(
bind_parameter), keeping the kernel path at parity with the whole-string Thriftpath. Pure Go, unit-tested under
CGO_ENABLED=0(querytags_nul_test.go).KERNEL_REV+ committed header: bumped to21504eaeand the C-ABI headersynced to that rev. The
databricks_kernelCI leg source-builds the kernel.afrom
KERNEL_REV(make kernel-lib), so this is what lets it link the new setter.The header sync also carries the other C-ABI decls that landed since the prior pin
(max_connections, telemetry circuit breaker, a result-stream canceller type, the
kernel_session_testtimeout param); the Go backend calls none of them, so nobinding code changes were needed.
Note on the published bindings
go.modstill pins thedatabricks-sql-kernel-bindingsmodules atv1.0.0— amatching bindings release (built from a kernel rev ≥
21504eae) is a fast-followfor
go get -tags databricks_kernelconsumers. CI and this repo's tagged tests buildthe kernel from
KERNEL_REVand are unaffected; the pure-Go default build nevertouches the kernel.
Validation (local)
Built the kernel static/shared lib from
21504eae(the newKERNEL_REV) and, withthe synced header, ran the
databricks_kernel-tagged path against it:go build/go test(-tags databricks_kernel, CGO on) — pass.checkQueryTagsguard test underCGO_ENABLED=0— pass.go build ./...+gofmt— clean.in context and ran a tagged query end to end — statement id
01f1a7e7-5834-15c7-bd90-bb1c7e0f144don warehouse153fb3a25b8051a7.This pull request and its description were written by Isaac.