Skip to content

feat(cli): record command outcomes and first-touch surface in telemetry - #372

Merged
designcode merged 6 commits into
mainfrom
katie/onboarding-funnel
Nov 3, 2026
Merged

designcode merged 6 commits into
mainfrom
katie/onboarding-funnel

Conversation

@katieschilling

@katieschilling katieschilling commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

cli_command used to be sent when a command started, so a failed command was counted the same as a successful one. It is now sent when the command ends, carrying the outcome. The event name and every existing property stay the same. $identify (and cli_command's person update) also sets the first-touch surface once with $set_once.

Neither PostHog nor Sentry does network I/O in the CLI process any more. Both queue ready-made requests, and when the command ends the queue goes to a detached background copy of the CLI that posts it after the command has exited. This also removes a pre-existing ~300ms Sentry delay on every naturally exiting command.

Properties

New on every cli_command:

  • success: boolean, exit_code === 0
  • exit_code: integer process exit code (130 for Ctrl-C)
  • duration_ms: from dispatch to exit (excludes CLI startup)

Only when success is false:

  • error_category: the Sentry categories from utils/errors.ts (auth, permission, not_found, rate_limit, network, general) plus usage (rejected by local validation before any work) and interrupted (Ctrl-C or a cancelled prompt)
  • error_code: only when the error carries a machine code: AWS Code/name (e.g. NoSuchKey), Node errno, commander code (commander.unknownOption), or a CLI code (unknown_command, missing_required_argument, removed_command, removed_argument, prompt_cancelled, command_load_failed). Never derived from the message.

Person properties, $set_once: first_interface: 'cli', first_cli_install_method, first_cli_version, on $identify and on cli_command. The existing $set is unchanged.

Delivery: a detached sender (utils/telemetry-sender.ts)

  1. During the run, analytics and Sentry append fully prepared POSTs (URL, headers, body) to an in-memory queue. $identify is queued too, so login no longer waits on it.
  2. When the command ends, the queue is written to os.tmpdir()/tigris-telemetry-<pid>-<random>.json (mode 0600, wx), and the CLI spawns itself with spawn(exe, args, { detached: true, stdio: 'ignore', windowsHide: true }).unref(), then exits without waiting. Under Node, exe is process.execPath with the entry script (argv[1]); the compiled binary runs itself.
  3. The child (tigris __telemetry-send <file>) reads and deletes the file (in a finally), POSTs each request with a 5s timeout, prints nothing, swallows every error, and exits. It only accepts a tigris-telemetry-* file directly in the temp dir. The entry points branch on it before the commander program exists, so it never appears in help and never initializes telemetry. In cli.ts the rest of the CLI is now imported lazily, so the child loads only the sender module (~40ms of CPU instead of ~200ms).
  4. If the child cannot start (spawn throws, no pid, async error, unwritable temp dir), the file is removed and the queue is posted in-process instead with a 1s cap. That survives a command that returns normally, but not process.exit(), so restricted sandboxes can still lose events.

Anything queued after the command's event (an error report with no tracked command, a late $identify) is sent by a process.on('exit') listener through the same path. With telemetry off, nothing is queued, so no file is written and no child is spawned.

The payload file holds the same scrubbed bodies that were sent before: arguments go through scrubArgv, Sentry events through beforeSend. No access key or secret appeared in any payload in the runs below.

Sentry

The ~300ms came from the ProcessSession integration: on beforeExit it ended the release-health session and sent it, so the SDK's transport held every naturally exiting command open for a TLS round trip. Changes in utils/telemetry.ts:

  • Custom transport (makeQueuedTransport) built on Sentry.createTransport: each serialized envelope is queued for the child, posted to the DSN's envelope endpoint with an X-Sentry-Auth header. beforeSend scrubbing is unchanged.
  • ProcessSession (the delay above), Context and ContextLines join main's DISABLED_INTEGRATIONS. Context/ContextLines are asynchronous, so an error captured just before process.exit() never reached the transport; without them capture is synchronous, and the OS release is now in the cli context.
  • enableRuntimeChannelInjection: false (Sentry 11's successor to registerEsmLoaderHooks): the module hooks only serve instrumenting third-party packages. Measured on 11.9.0, they add ~56ms to init.
  • sendClientReports: false, so a clean run sends no envelope at all.

A side effect: handled general/network failures from exitWithError now actually reach Sentry. Before, they were captured but the process exited before the event was processed.

Rebased onto Sentry v11

main moved @sentry/node to v11 (#378, #390) and rewrote utils/telemetry.ts: the SDK is imported lazily only when telemetry is on, with a loading/on/failed state machine, a bounded pending queue replayed once the SDK is up, dataCollection: DATA_COLLECTION, and default integrations filtered through DISABLED_INTEGRATIONS. The rebase keeps all of that and re-applies this PR on top. configure() passes the queued transport (makeQueuedTransport(sdk.createTransport), so the SDK stays out of the static imports), sendClientReports: false and enableRuntimeChannelInjection: false. The three extra integrations go into DISABLED_INTEGRATIONS, and the name-pinning test now guards them too. I dropped the old explicit allow-list: the filter costs ~10ms more in init (12ms vs 2ms on 11.9.0), but that runs in the background while the command works. flushTelemetry waits for loading (capped at its timeout) and then flushes into the queue, with no network. So a crash reported while the SDK was still loading is replayed into the transport before exitWithError → process.exit → finishCommand spawns the sender. Tests now cover that replay.

Remaining gap, inherited from main's lazy load: a handled general/network error that exits through exitWithError before the SDK has loaded (~120ms import) is queued in pending and lost, because that path cannot await. Commands that reach the network take longer than that, so in practice this only affects very early failures.

How it hooks every exit

Path Hook
Command returns finishCommand(process.exitCode) in loadAndExecuteCommand
exitWithError / failWithError (~370 sites) recordCommandError(error) in the helper, then the process.exit wrapper sends
Thrown error / unhandled rejection crash handler captures, awaits the (now local) Sentry flush, then exitWithError → same as above
Direct process.exit(0/1) in commands and the dispatcher process.exit wrapper (installed only while a command is tracked)
Local validation: missing required option, unknown subcommand, removed command/argument tracked at the top of the action handler, before validation
commander usage errors (unknown option, missing argument) per-command exitOverride callback records, then commander's own process.exit is wrapped
Ctrl-C SIGINT listener spawns the sender with exit_code: 130, then re-raises SIGINT so the shell still sees a signal death. It stands aside when a command has its own SIGINT handler (buckets migrate), whose process.exit(130) is wrapped

telemetry commands stay untracked. The browser build's no-op shim gains the two new exports.

Not covered

  • SIGKILL, a native crash, or a second Ctrl-C before the first is handled.
  • Environments where the child cannot be spawned: falls back to in-process sending, which a hard process.exit() cuts off.
  • Exits in the same tick as dispatch (local validation) are sent before the stored login has been read. They carry the machine's anonymous id and auth_method: 'unknown'; PostHog still puts them on the right person once $identify has linked that id.
  • The storage SDK replaces S3 errors with plain Error(message), so SDK failures usually have an error_category but no error_code.
  • The Bun binary was typechecked (tsconfig.binary.json) but not built or run here (no Bun on this machine). It relies on Bun honoring detached in child_process.spawn.

Cost

tigris buckets list, built CLI, macOS, real ph.tigrisdata.com and Sentry, 7 interleaved runs each, median wall time:

Build Telemetry Median Runs (ms)
this PR, rebased (Sentry 11, lazy import) on 604ms 521, 547, 563, 604, 618, 638, 692
this PR before rebase (Sentry 10, static import) on 620ms 595, 605, 610, 620, 634, 670, 672
this PR, rebased off (TIGRIS_NO_TELEMETRY=1) 473ms 445, 451, 462, 473, 505, 539, 660
this PR before rebase off 712ms 589, 598, 602, 712, 755, 761, 788

With telemetry on, the rebase changes nothing (604 vs 620ms, within jitter). On and off are no longer equal because main made off cheaper: an opted-out run never loads the SDK, and that import costs ~120ms of CPU. With telemetry on, that import overlaps the command's network wait. A separate 5-run set had buckets list at 667 vs 581ms and whoami at 501 vs 422ms (on vs off). frobnicate, which exits before the SDK finishes loading, took 112 vs 113ms. Before the rebase, d9552c6 (worker + Atomics.wait) took 1035ms with telemetry on. The parent still spends ~2ms writing the file and spawning.

Verification

  • pnpm --filter @tigrisdata/cli test (vitest 5, after the rebase): 52 files, 1332 passed (225 integration tests skipped as before). analytics.test.ts mocks child_process.spawn: it checks the spawn command and options, the payload file's name, location and 0600 mode, the binary runtime launching itself, the spawn-failure fallback (event still sent, no file left behind), and that nothing is spawned or written with telemetry off. All the outcome tests from the first commit are kept. The new telemetry-sender.test.ts runs the __telemetry-send handler in-process against a local HTTP server (posts, then deletes the file; malformed payloads; refuses files it did not write), checks the Sentry transport moves auth into X-Sentry-Auth, and checks a real Sentry.init queues a scrubbed envelope synchronously on capture and nothing on a clean run.
  • pnpm --filter @tigrisdata/cli build passes. biome check packages/cli/src packages/cli/test is clean apart from specs.yaml. Root pnpm check fails on main itself after the biome 2.6.3 bump (chore(deps): update dependency @biomejs/biome to v2.6.3 #389): it now formats Markdown and YAML, and 18 files fail, including specs.yaml. On this branch the changeset's frontmatter is a 19th file. I left those files alone because they are outside this PR.
  • End to end: the built CLI with a fresh HOME, PostHog and the Sentry DSN pointed at a local sink:
command success exit_code duration_ms error_category error_code Sentry envelope
login credentials true 0 349 no
buckets list true 0 351 no
objects put (small file) true 0 429 no
buckets create (already exists) false 1 393 general yes
objects get does-not-exist false 1 646 general yes
buckets list --bogus-flag false 1 0 usage commander.unknownOption no
frobnicate (unknown command) false 1 0 usage unknown_command no
objects put 300MB, Ctrl-C after 3s false 130 2784 interrupted no
objects put, uncaught throw mid-upload false 1 1324 general yes

Envelopes arrived at /api/42/envelope/ with X-Sentry-Auth: Sentry sentry_version=7, sentry_key=…, sentry_client=tigris-cli/3.17.0. No tigris-telemetry-* files were left in the temp dir afterwards.

After the rebase (Sentry 11), the same sink run gave: login credentials success (480ms), buckets list success (576ms, no envelope), buckets create general exit 1 (727ms, one envelope), frobnicate usage/unknown_command (no envelope), and uncaught throw mid-upload general exit 1 (1456ms, one envelope). Each envelope reached the sink in the same child POST batch as its cli_command. No payload files were left behind.

  • Real PostHog: buckets list (success) and buckets create (failure) at 2026-10-14T18:35:12Z, distinct_id 7f6041da-d92e-461f-81b0-71ffcb0e4f1d (anonymous, credentials login). Both POSTs from the child got 200 from ph.tigrisdata.com/i/v0/e/. After the rebase: buckets list at 2026-10-30T20:08:50Z and failing buckets create at 2026-10-30T20:08:51Z, distinct_id 349bfcaa-17c5-44bb-97cd-5350c8141128.

Depends on #392

pnpm check fails on main with biome 2.6.3 (it now formats Markdown/YAML), and this branch adds one more hit on its changeset file. #392 turns those formatters off and should merge first; after that pnpm check and the pre-commit hook pass here without HUSKY=0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Hqe2uWQDRZXDwo7DZEu5En


Note

Medium Risk
Wraps process.exit and SIGINT, spawns detached CLI children, and changes when telemetry fires—designed not to block exits, but exit-hook and spawn behavior is easy to get wrong in edge environments.

Overview
cli_command is emitted when a command finishes, not at start, with success, exit_code, duration_ms, and on failure error_category / error_code (machine codes only, never messages). $set_once records first CLI touch (first_interface, install method, version) on identify and command events.

PostHog and Sentry no longer do HTTP in the user-facing process. Events queue to a temp file and a detached tigris __telemetry-send child posts after exit; cli.ts lazy-loads the main CLI so the sender stays light. process.exit is wrapped (while tracking), finishCommand / recordCommandError hook returns, exitWithError, commander usage errors, validation failures, and Ctrl-C (with commandHandlesSigint for migrate). Sentry uses a queued transport, drops ProcessSession (fixes ~300ms beforeExit delay) and async context integrations, and exitWithError reports attach to analytics before exit.

Docs and telemetry status copy reflect outcomes and background delivery; browser shim stubs the new exports.

Reviewed by Cursor Bugbot for commit a5cf701. Bugbot is set up for automated code reviews on this repo. Configure here.

@katieschilling
katieschilling marked this pull request as draft October 14, 2026 17:39
@greptile-apps

greptile-apps Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest changes appear safe to merge and fix the previous JSON-mode finding.

Summary

The PR records command outcomes and first-touch person fields, then hands queued analytics and error reports to a detached sender.

  • Command events now show whether each run succeeded or failed.
  • The CLI sends queued PostHog and Sentry requests after commands exit.

Reviews (5) · Last reviewed commit: "fix(cli): set json mode before tracking ..." · Reviewed by Greptile

Comment thread packages/cli/src/utils/analytics.ts Resolved
@katieschilling
katieschilling marked this pull request as ready for review October 30, 2026 19:53
Comment thread packages/cli/src/cli-core.ts Resolved
katieschilling and others added 5 commits November 2, 2026 15:28
cliCommand is now captured after the command finishes with success,
exit_code, duration_ms and, on failure, error_category / error_code, so
funnels can count outcomes rather than invocations. $identify sets
first_interface / first_cli_install_method / first_cli_version once.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hqe2uWQDRZXDwo7DZEu5En
…g at exit

Replace the worker-thread sender with a detached child process that
posts the cli_command event and any Sentry envelopes after the parent
exits, so commands no longer pay network latency on completion. Sentry
uses a transport that hands envelopes to the same child.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hqe2uWQDRZXDwo7DZEu5En
When a command with its own SIGINT handler (buckets migrate) is
cancelled and then exits non-zero, the outcome now reads
error_category=interrupted rather than general.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hqe2uWQDRZXDwo7DZEu5En
Node's request timeout is socket inactivity, so a stalled connect or a
trickling response could keep the sender child, or the in-process
fallback, alive past the promised cap. post() now destroys the request
at an elapsed-time deadline as well.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hqe2uWQDRZXDwo7DZEu5En
- Spawn the detached sender with its cwd in the temp dir. It outlives the
  command, and on Windows a directory that is any process's cwd cannot be
  deleted, so a script running the CLI in a temp folder could not remove it.
- Replace the SIGINT listener-count heuristic with an explicit
  commandHandlesSigint() that `buckets migrate` calls. Any other listener
  (the prompt spinner) no longer stops Ctrl-C from ending the run.
- Re-raise SIGINT on the next turn of the loop. Delivered synchronously, a
  second listener removing itself in the same event made Node drop the
  pending signal, and the process kept running.
- On Windows exit 130 instead of process.kill(pid, 'SIGINT'), which is
  TerminateProcess with exit code 1 there.

Also applies the Biome 2.6 formatting now on main to the changeset and README.

Assisted-by: Claude Fable 5.1 via Claude Code
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@designcode
designcode force-pushed the katie/onboarding-funnel branch from 7ee5ba8 to 6db0693 Compare November 2, 2026 14:45

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6db0693. Configure here.

…path

- JSON mode was entered inside loadAndExecuteCommand, after trackCommand had
  run and after local validation could exit. The first-run notice could print
  human text under --json, and events for invocations rejected by validation
  reported json_mode: false. Apply it from the parsed options at the top of
  both action paths.
- Resolve argv[1] before handing it to the sender child, which now runs from
  the temp dir. Node and Bun already report an absolute path; this removes
  the dependency on that.

Assisted-by: Claude Fable 5.1 via Claude Code
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@designcode
designcode merged commit f7600ff into main Nov 3, 2026
3 checks passed
@designcode
designcode deleted the katie/onboarding-funnel branch November 3, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants