Skip to content

feat(sdk): measure elapsed time for each control check - #291

Open
josjeon wants to merge 1 commit into
mainfrom
codex/control-check-elapsed
Open

josjeon wants to merge 1 commit into
mainfrom
codex/control-check-elapsed

Conversation

@josjeon

@josjeon josjeon commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements a directly measured SDK control-check elapsed metric for one pre/post invocation. This is the caller's actual wait for the check, including local evaluation and the conditional remote call, rather than an average/max/sum of individual control durations.

Context: aggregate control time request and per-step average discussion.

Behavior and boundary

  • Adds nullable EvaluationResult.control_check_elapsed_ms and control_check_id. New SDK helper invocations measure elapsed with a monotonic clock; legacy/manual results remain null unless measured.
  • Measures one pre/post check's evaluation-helper preparation, SDK-local evaluation, conditional server call, and existing per-control event handling. Includes network/token-refresh/retry delays that occur inside that call.
  • Excludes SDK initialization, high-level step construction/client setup and parent context resolution before the helper, decorator payload preparation before the helper, the protected tool/LLM function, and whole-check event export. The uncached decorator server fallback measures its HTTP evaluation attempt and preserves raw response extensions.
  • Local deny, handled evaluator errors and no-applicable-control completion still have measured timing. Check exceptions/cancellation propagate, with separate telemetry status when supported. The existing decorator fallback after a failed local attempt records separate attempt IDs.
  • Preserves per-control execution_duration_ms and the server evaluation response schema.

Observability

  • Introduces typed ControlCheckEvent and optional ControlCheckEventSink capability. Existing control-only sinks remain compatible; the HTTP control-execution ingestion endpoint receives no whole-check payload.
  • Reuses the selected OTEL sink/provider to emit one agent_control.control_check span with captured invocation start and monotonic-derived end, correlated to its parent step. Metadata contains only safe check ID/stage/elapsed/status; no input/output or exception text.
  • Existing Orbit ingestion treats this as a workflow span, not an individual control span. Dedicated UI/session-summary presentation is a follow-up. This PR does not add a server-side engine/pipeline elapsed metric.
  • Overlapping checks cannot be summed to infer session wall time or counterfactual agent overhead.

Validation

  • 15 new deterministic contract tests cover independent batch timing vs five child durations, mixed local/server, pre/post, local deny, empty/measured-zero compatibility, original error/cancellation preservation, telemetry failure, old sinks, actual native OTEL provider export, server-only helper and uncached decorator fallback with preserved raw extensions.
  • 179 targeted SDK tests passed (15 new + 164 existing).
  • make models-test: 76 passed.
  • make telemetry-test: 20 passed.
  • make prepush: all repository lint/type checks and Galileo extra checks passed.
  • git diff --check passed. No dependency or lockfile changes.

Full DB-backed server/SDK integration, deployment and remote load testing were not run for this SDK/telemetry draft.

@codecov

codecov Bot commented Oct 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.56198% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
sdks/python/src/agent_control/observability.py 76.47% 4 Missing ⚠️
models/src/agent_control_models/observability.py 83.33% 3 Missing ⚠️
sdks/python/src/agent_control/otel_sink.py 88.88% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@josjeon
josjeon marked this pull request as ready for review October 9, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant