Repository navigation
Add experimental skill/<name> user-agent pairs from DATABRICKS_SKILLS_INVOKED - #7730
Conversation
…KILLS_INVOKED Agent skills that drive a CLI call can list themselves in the DATABRICKS_SKILLS_INVOKED environment variable (or a hidden --skills-invoked flag, which wins). The CLI normalizes the list and sends it as X-Databricks-Skills-Invoked on Databricks API requests. Invalid entries are dropped silently, with debug logging only. The value is resolved once in the root PersistentPreRunE and stored in the command context. Every SDK config site applies a header hook that chains any existing hook, so liteswap keeps working. The files client gets the header through a wrapping transport. Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=1 unreleased=0 coverage=0 failing=0
TestAccept/cmd/skills-invoked: branch ✅ | main (e4e790c) ❌ | latest (v1.26.0) ❌main (e4e790c): latest (v1.26.0): |
✅ Integration test reportCommit: b1c946b
Top 48 slowest tests (at least 2 minutes):
|
Every SDK config site applies liteswap before skillsinvoked, and skillsinvoked.Apply chains the existing header hook, so liteswap does not need to change. Shorten doc comments to the contract and the reasons. Co-authored-by: Isaac <no-reply@databricks.com>
The HTTP access log truncates the header at 1024 bytes. Stop adding names before the joined value would exceed that, so a partial name is never logged. Co-authored-by: Isaac <no-reply@databricks.com>
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 954546c. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking]; CI failures caused by this PR: task test (windows, direct), task test (windows, directdms), test-result.
This PR sends the agent skills that drove a CLI call as X-Databricks-Skills-Invoked, read from DATABRICKS_SKILLS_INVOKED or a hidden flag, and attaches it to the SDK clients, bundles, databricks api, and the files/v2 client. The parsing, the size caps, the chaining onto liteswap's hook, and the request-cloning transport all look correct. The new acceptance test fails on both Windows CI jobs because Git Bash rewrites the /api/2.0/... argument, so this needs one MSYS_NO_PATHCONV=1 fix before merging. The rest are small coverage and test-hygiene suggestions, plus a question about whether the hidden flag is needed.
Questions
- Is the hidden
--skills-invokedflag needed? The comment incmd/root/skills_invoked.gosays skills setDATABRICKS_SKILLS_INVOKEDbecause older CLIs reject unknown flags, so nothing in this PR uses the flag. It still adds a root persistent flag to every command, along with precedence logic and tests. Invalid values are dropped without a message, which AGENTS.md discourages for flags ("Never silently ignore a flag…"). If there is no planned user, would you drop the flag and keep only the env var? If there is one, could you name it in the description? (non-blocking)
Accepted answers
- Skills header depends on liteswap being applied first
- Invalid skill names are dropped silently
| trace print_requests.py --get //clusters/list | ||
|
|
||
| title "Env var sends the header on databricks api" | ||
| DATABRICKS_SKILLS_INVOKED=databricks-core trace $CLI api get /api/2.0/clusters/list > /dev/null |
There was a problem hiding this comment.
[blocking] On Windows, Git Bash rewrites the leading-slash argument /api/2.0/clusters/list into /Program Files/Git/api/2.0/clusters/list. The testserver then 404s, api get exits 1, and the rest of the script never runs. Both Windows task test jobs fail on TestAccept/cmd/skills-invoked with No stub found for pattern: GET /Program Files/Git/api/2.0/clusters/list. Prefix only this command with MSYS_NO_PATHCONV=1, as acceptance/cmd/api/*/script does: DATABRICKS_SKILLS_INVOKED=databricks-core MSYS_NO_PATHCONV=1 trace $CLI api get .... Don't set it test-wide in test.toml. The script runs print_requests.py, and .agents/rules/testing.md explains that the helper needs path conversion to work on Windows.
There was a problem hiding this comment.
Fixed in 5de7d1b: the api get line now has MSYS_NO_PATHCONV=1 on that command only. The rest of the script still runs print_requests.py with path conversion on.
| // Apply sets the header on every request made with cfg, keeping any header | ||
| // hook already on cfg (such as liteswap's). The value comes from the request | ||
| // context, so it is only sent when the command context reaches the request. | ||
| func Apply(cfg *config.Config) { |
There was a problem hiding this comment.
[nice to have] databricks auth profiles builds its own config.Config in profileMetadata.Load (cmd/auth/profiles.go:48) and never calls Apply. Its CurrentUser.Me and Workspaces.List validation calls run on the command context but go out without the header. The SSH release-upload client has the same gap: newHTTP11WorkspaceClient (experimental/ssh/internal/client/releases.go:61) rebuilds the config from ConfigAttributes, which drops cfg.Headers. Skills often run auth profiles to check auth, and neither path is in the PR's "Known gaps". Either call skillsinvoked.Apply(cfg) in profileMetadata.Load and copy src.Headers in newHTTP11WorkspaceClient, or list both under "Known gaps".
There was a problem hiding this comment.
Fixed in 5de7d1b: profileMetadata.Load now calls skillsinvoked.Apply(cfg), and newHTTP11WorkspaceClient copies src.Headers, since it's a func and the attribute loop skips it.
| client.WithoutConfigFile(), | ||
| client.WithoutEnv(), | ||
| // files/v2 doesn't use cfg.Headers. http.DefaultTransport is its default. | ||
| client.WithHTTPClient(&http.Client{Transport: skillsinvoked.Transport(http.DefaultTransport)}), |
There was a problem hiding this comment.
[nice to have] The transport added here sets only the skills header, so other cfg.Headers hooks still don't reach the files/v2 client. For example, with DATABRICKS_LITESWAP_TARGET set, fs ls dbfs:/Volumes/... sends no x-databricks-traffic-id, while the other requests in the same command do. A transport in libs/filer that clones the request and calls cfg.Headers (when it is set) would send both headers. skillsinvoked.Transport would then not be needed.
There was a problem hiding this comment.
Done in 5de7d1b: libs/filer now has a headersTransport that clones the request and runs cfg.Headers when it's set, so liteswap's x-databricks-traffic-id reaches files/v2 too. skillsinvoked.Transport is gone. Unit tests cover the no-hook, hook and hook-error cases.
| // the environment variable. | ||
| func withSkillsInvoked(ctx context.Context, cmd *cobra.Command) context.Context { | ||
| raw := env.Get(ctx, envSkillsInvoked) | ||
| if v := cmd.Flag(flagSkillsInvoked).Value.String(); v != "" { |
There was a problem hiding this comment.
[nit] The check uses Value.String() != "" instead of whether the flag was set. With DATABRICKS_SKILLS_INVOKED=from-env, --skills-invoked= still sends from-env, while --skills-invoked='!!' sends nothing (the unit test asserts that case). So the flag wins over the env var only when its value is non-empty. Use cmd.Flag(flagSkillsInvoked).Changed and add a test case for an explicit empty value. This goes away if the flag is dropped.
There was a problem hiding this comment.
This went away with the flag: in 5de7d1b the CLI reads only DATABRICKS_SKILLS_INVOKED.
| if tt.env != "" { | ||
| ctx = env.Set(ctx, envSkillsInvoked, tt.env) | ||
| } |
There was a problem hiding this comment.
[nit] When tt.env is empty, the test doesn't override the variable, so env.Get falls back to os.LookupEnv. Running go test ./cmd/root from a shell where DATABRICKS_SKILLS_INVOKED is exported makes the neither case fail. The acceptance test has the same exposure: acceptance/acceptance_test.go unsets the host's agent env vars (including CLAUDE_CODE_CHILD_SESSION) but not this one. With it exported, "No header when unset" and the header-less fs mkdir PUT pick up the host value. Always call env.Set(ctx, envSkillsInvoked, tt.env): an empty value in the context overrides the OS environment. Also add DATABRICKS_SKILLS_INVOKED to the unset list in acceptance_test.go.
There was a problem hiding this comment.
Fixed in 5de7d1b: the unit test always calls env.Set for both env vars, and acceptance_test.go now unsets DATABRICKS_SKILLS_INVOKED next to CLAUDE_CODE_CHILD_SESSION.
| [[Server]] | ||
| Pattern = "GET /api/2.0/clusters/list" | ||
| Response.Body = '{}' |
There was a problem hiding this comment.
[nit] This stub exists only because the script calls /api/2.0/clusters/list, and libs/testserver already models GET /api/2.1/clusters/list. .agents/rules/testing.md reserves [[Server]] for routes the testserver doesn't model. Call api get /api/2.1/clusters/list in the script, with the MSYS_NO_PATHCONV=1 fix, and drop the stub.
There was a problem hiding this comment.
Fixed in 5de7d1b: the script calls api get /api/2.1/clusters/list and the [[Server]] stub is gone.
Only DATABRICKS_SKILLS_INVOKED is read now, so the feature can be removed later without changing the CLI's interface. No header is sent when DATABRICKS_CLI_DISABLE_TELEMETRY is set. Review fixes: - Acceptance test: MSYS_NO_PATHCONV=1 for the leading-slash api get so it passes on Windows, call the modeled /api/2.1/clusters/list and drop the stub, and unset a host DATABRICKS_SKILLS_INVOKED in acceptance_test.go. - Unit test always sets the env vars in the context so a host value can't leak in. - `auth profiles` validation calls and the SSH HTTP/1.1 release client now send the header. - The files/v2 client applies the whole cfg.Headers hook in its transport, so liteswap's traffic header reaches it too; skillsinvoked.Transport is gone. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
@simonfaltum thanks for the review. Everything is addressed as of 6268ba7:
I also merged main into the branch. Could you take another look? |
databricks/cli#7730 now reads only DATABRICKS_SKILLS_INVOKED and sends nothing when DATABRICKS_CLI_DISABLE_TELEMETRY is set. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Chris Stephens <chris.stephens@databricks.com>
DATABRICKS_SKILLS_INVOKED now adds one skill/<name> pair per skill to the user agent, next to the other user-agent dimensions in cmd/root. The SDK already sends context user-agent pairs on every request, so the per-client header hooks (auth, bundles, api, auth profiles, SSH releases, files/v2) are reverted. The DECO usage pipeline parses repeated keys into unified_user_agent.other_info_multi, as it does for aitools/. Names are limited to [a-z0-9_.-], a subset of what the SDK accepts in a user agent value. This is experimental tracking and is commented as such. Co-authored-by: Isaac <no-reply@databricks.com>
|
@simonfaltum heads-up, the design changed after my last comment: skills now go into the user agent as
The PR is now 217 lines across 9 files. Could you take another look? |
Co-authored-by: Isaac <no-reply@databricks.com>
simonfaltum
left a comment
There was a problem hiding this comment.
Reviewed 704ed07f7f872f4a3d704e29ef8d5a9b5ede01a7 with the pr-review workflow; all five reviewers completed, and each reported finding was checked against the code.
Requesting changes: one blocking issue, one nice-to-have test fix, and four minor nits, detailed inline.
The user-agent approach, repeated skill/ pairs, bare-name contract, and telemetry opt-out look consistent with the intended design. The remaining blocker is the aggregate length: supported input can push existing auth and agent attribution beyond the usage-log user-agent limit. There is also a host-environment dependency in the acceptance fixture.
CI is green on this head, including all six Linux/macOS/Windows direct/directdms test jobs and Integration Tests; some other jobs were skipped. The earlier Windows path-conversion issue is fixed. The removal of the hidden flag and the documented files/v2 scope are accepted.
The length finding was verified through source tracing and a static calculation against the 512-character usage-log tag cap. No live ingestion reproduction or full local Go test run was performed.
This review was generated with GitHub MCP.
| const skillUserAgentKey = "skill" | ||
|
|
||
| const ( | ||
| maxSkills = 20 |
There was a problem hiding this comment.
[blocking] Twenty accepted 64-character names produce 1,419 characters of skill/ pairs alone. The usage-log path feeding deco_usage_logs_normalized caps userAgent at 512 characters, and SDK v0.200.0 appends auth/, agent/, and other dimensions after these skills. This input therefore loses later skills and existing attribution metadata before the analytics parser sees it.
Please add an aggregate serialized-size budget with room for SDK metadata, or increase the receiving limit before enabling this, with boundary coverage. The 512-character cap is specific to usage logging; this was verified from source and a static calculation, without a live ingestion test.
There was a problem hiding this comment.
Fixed in d6888e2: the 20-skill cap is now a 128-byte budget for all skill/<name> pairs together (maxSkillsUserAgentLength). Names are kept in order while they fit, so the auth/, agent/ and other SDK pairs still have room under the 512-character usage-log cap. About five typical skill names fit. Boundary tests cover fits, overflow, and exactly 128 bytes, and every case asserts the result stays within the budget.
There was a problem hiding this comment.
Follow-up in b1c946b: I raised the budget to 225 bytes. 128 fit only two of the longest current skill names; 225 fits any five of today's 38 skills (the longest pair is 45 bytes, so five of those is 225). Without skills, the CLI's user agent is about 200 characters, so the total stays under the 512-character usage-log cap with room to spare.
There was a problem hiding this comment.
@chrisst could you pick up a follow-up fix for this user-agent length guard? The merged code still uses the 225-byte budget. I checked the merge: the relevant CLI code is unchanged from b1c946b, and the SDK upgrade from 0.200.0 to 0.201.0 doesn't change user-agent construction.
A local reproduction using the actual CLI skills helpers and SDK request visitors produces a 514-byte user agent for bundle deploy with an existing direct-engine lineage, Claude Code as caller, PAT auth, and Databricks plugins for Claude Code 0.2.9, Codex 0.2.8, and Copilot 0.2.9. These six existing skill names are all accepted and occupy exactly 225 bytes including the skill/ prefixes:
databricks-serverless-migration-api,databricks-lakeflow-connect-saas,databricks-serverless-migration,databricks-synthetic-data-gen,databricks-mlflow-evaluation,databricks-execution-compute
The other fields total 289 bytes, so the 512-character logging prefix cuts agent/claude-code to agent/claude-co. The previous 128-byte budget produced a 408-byte header for the same input and preserved the agent. This verifies header construction and the truncation boundary; it isn't a live ingestion test.
Could you restore the conservative budget for now, or account for the complete request user agent when choosing the skill allowance, and add a regression test covering bundle engine/lineage plus SDK auth/agent fields? The current boundary tests check only the skill portion.
This comment was generated with GitHub MCP.
| @@ -0,0 +1,3 @@ | |||
| RecordRequests = true | |||
There was a problem hiding this comment.
[nice to have] With DATABRICKS_CLI_DISABLE_TELEMETRY=1 exported in the invoking shell, this fixture inherits the opt-out through auth.ProcessEnv. The CLI correctly omits the skills, so the first positive contains.py assertion fails. Set Env.DATABRICKS_CLI_DISABLE_TELEMETRY = "" in this fixture; the final scenario can still set it to 1 inline.
There was a problem hiding this comment.
Fixed in d6888e2: the fixture sets Env.DATABRICKS_CLI_DISABLE_TELEMETRY = "", and the last scenario still sets it to 1 inline. I checked it passes with the opt-out exported in my shell.
|
|
||
| title "Env var adds the skills to the user agent on a workspace GET" | ||
| DATABRICKS_SKILLS_INVOKED=databricks-core,databricks-jobs trace $CLI current-user me > /dev/null | ||
| trace print_requests.py --get //scim/v2/Me | contains.py " skill/databricks-core skill/databricks-jobs" > /dev/null |
There was a problem hiding this comment.
[nit] Keep the request dump in a LOG.<name> file before asserting it. The current redirection discards the actual user agent, and print_requests.py deletes the recording, so a missing-skill failure has little diagnostic output. .agents/rules/testing.md:209 asks for LOG.* instead of /dev/null.
There was a problem hiding this comment.
Fixed in d6888e2: each request dump goes to a LOG.<name> file, and contains.py reads from it, so a failed assertion keeps the actual user agent.
| @@ -0,0 +1,72 @@ | |||
| package root | |||
There was a problem hiding this comment.
[nit] .agents/rules/testing.md:31 asks new tests to use the _test package suffix. Could these use root_test, with test-only aliases in the existing export_test.go if direct helper access is needed? Existing sibling user-agent tests also use package root, so this is only a nit against the written rule.
There was a problem hiding this comment.
Done in d6888e2: the tests are in package root_test, with aliases in export_test.go.
| "github.com/databricks/databricks-sdk-go/useragent" | ||
| ) | ||
|
|
||
| // Experimental skill-usage tracking: agent skills prefix CLI calls with this |
There was a problem hiding this comment.
[nit] Add a public source link for the skills producer or its environment-variable contract to this comment. .agents/rules/comments.md:62 requires a reference URL when integrating external tools or detecting their environment variables, and it would make the bare-name format easy to trace.
There was a problem hiding this comment.
Done in d6888e2: the comment links the databricks-core skill that tells agents to set the env var.
| {"trims and lowercases", " Databricks-Core , JOBS ", []string{"databricks-core", "jobs"}}, | ||
| {"drops empty entries", ",,a,,", []string{"a"}}, | ||
| {"allows underscore and dot", "a_b.c-d", []string{"a_b.c-d"}}, | ||
| {"drops characters the user agent rejects", "ok,bad name,a:b,a/b,a+b", []string{"ok"}}, |
There was a problem hiding this comment.
[nit] Rename this case to refer to disallowed skill-name characters. The input includes a+b, which the pinned SDK accepts in a user-agent value; the CLI intentionally uses a stricter pattern.
There was a problem hiding this comment.
Renamed in d6888e2 to "drops disallowed skill-name characters".
The usage logs that feed skill analytics cap the user agent at 512 characters, and the SDK appends auth/, agent/ and other pairs after the skills. Replace the 20-skill cap with a 128-byte budget for all " skill/<name>" pairs so those pairs survive, with boundary tests. Also: - Link the databricks-core skill that sets DATABRICKS_SKILLS_INVOKED. - Move the unit tests to package root_test via export_test.go, and rename the disallowed-characters case. - Acceptance: pin DATABRICKS_CLI_DISABLE_TELEMETRY empty so a host opt-out can't hide the skills, and keep request dumps in LOG.* files. Co-authored-by: Isaac <no-reply@databricks.com>
|
@simonfaltum all six points from your latest review are addressed in d6888e2. The blocker is fixed with a 128-byte budget for the skill pairs, so the other user-agent pairs keep room under the 512-character usage-log cap. The fixture, LOG files, package, link and test name are done too; details are in each thread. Could you take another look? |
Co-authored-by: Isaac <no-reply@databricks.com>
128 bytes fit only two of the longest current skill names. 225 bytes fits five of the longest (45-byte " skill/<name>" pairs) and still leaves room under the 512-character usage-log cap for the rest of the CLI's user agent, which is about 200 characters. Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Experimental skill-usage tracking. When
DATABRICKS_SKILLS_INVOKED=databricks-core,databricks-jobsis set, the CLI adds oneskill/<name>pair per skill to its user agent:... skill/databricks-core skill/databricks-jobs. Databricks agent skills will prefix their CLI calls with this env var (databricks/databricks-agent-skills#311).DATABRICKS_CLI_DISABLE_TELEMETRYis set.^[a-z0-9][a-z0-9_.-]*$(a subset of what the SDK allows in a user agent value) or longer than 64 bytes are dropped. Names are kept in order while allskill/<name>pairs fit in 225 bytes (five of the longest current skill names), so the SDK'sauth/,agent/and other pairs stay within the 512-character usage-log cap. Bad input never fails or warns a command; dropped entries are only logged at debug level.cmd/root/user_agent_skills.gonext to the other user-agent dimensions and is applied in the rootPersistentPreRunE. Every request that already carriescmd/carries the skills too.The DECO usage pipeline (
deco/bundles/http_access_logs→deco_usage_logs_normalized) parses repeated keys intounified_user_agent.other_info_multi, as it already does for the repeatedaitools/pairs. So the skills land inother_info_multi['skill']with no pipeline change.Known gap: the files/v2 client used by
fson/Volumesbuilds its own user agent without any of the CLI's context pairs (nocli/orcmd/either), so those requests carry no skills.Why
To measure which Databricks agent skills drive CLI usage, per request, from the existing HTTP access logs, with no new header or log column.
Tests
cmd/root_test): parsing table (normalization, disallowed characters including:, name length, and 225-byte budget boundaries), and the user-agent pairs with and without the telemetry opt-out. Both env vars are always set in the context so a host value can't leak in.acceptance/cmd/skills-invoked): assertsskill/pairs in the recorded user agent forcurrent-user me,api get, andbundle validate. It also covers normalization, no pairs when unset, and no pairs with telemetry disabled. A hostDATABRICKS_SKILLS_INVOKEDis unset inacceptance_test.go.This pull request and its description were written by Isaac.