feat(sdk): add create_decision_evaluator and DecisionEvaluator (SAO-17973) by john-weiler · Pull Request #273 · splunk/splunk-ao-python

john-weiler · 2026-10-16T13:53:51Z

Why

Decision-model evaluators (scorer_type: "decision", for non-LLM judges such as Jev) are live in the API and UI. This adds the Python SDK side: SAO-17973. It also ports SAO-18652 (galileo-python #639), without which the API leaves decision scorers out of list/get responses and the SDK cannot decode them.

What

create_decision_evaluator(name, question, node_level, model_name, output_type=None, answer_options, input_type, description, tags, ground_truth) as an Evaluators method and module function. Inputs are validated up front. If output_type is omitted it is inferred from answer_options (None to boolean_probability, strings to categorical, levels to discrete); a mismatch raises ValueError.
Rollback: the new scorer is deleted only when the version POST is definitively rejected (400/403/404/422). On a timeout, network error, 5xx or other failure the scorer is kept and DecisionVersionOutcomeUnknownError (names the scorer id, chains the cause, exported from splunk_ao) is raised.
DecisionEvaluator(Evaluator): create(), get/list populate question, answer options (including discrete levels the decoder returns as raw dicts), model, node level and output type.
SAO-18652: X-Galileo-Scorer-Capabilities: decision header on /scorers*, */scorer-settings, */metric_settings (codegen template and get_scorer_capabilities_header); decision added to ScorerTypes and ModelType.

Spec and generator notes

Spec: the full regeneration yields an unrelated ~45k-line diff, so only the decision path, CreateDecisionScorerVersionRequest, DecisionLevel, DecisionScorerDB, OutputTypeEnum += boolean_probability and decision_scorer on BaseScorerVersionResponse/BaseScorerVersionDB were spliced into openapi.yaml from the prod client spec, then scripts/auto-generate-api-client.sh was run. resources/ is generated only. (Main's generated control_applies_to.py already drifts from the spec; that regen noise is left out.)
Generator bug (follow-up): the generated CreateDecisionScorerVersionRequest.to_dict has identical isinstance(list) branches for list[str] | list[DecisionLevel], so levels would never serialize. Worked around by sending levels through additional_properties (_build_decision_version_request). Fixing the union_property template is a follow-up because it touches many generated models.

Where to start

src/splunk_ao/evaluators.py (helpers, create_decision_evaluator), then src/splunk_ao/evaluator.py (DecisionEvaluator), then tests/test_decision_scorers.py. Commit 1 is the SAO-18652 port, commit 2 the spec splice and regeneration (generated).

Tests

poetry run pytest (2386 passed, 4 skipped); poetry run invoke type-check clean.

Related PRs: galileo-python #639 (SAO-18652 original), galileo-js#727 (parity).

🤖 Generated with Claude Code

…-18652) Send X-Galileo-Scorer-Capabilities on /scorers*, */scorer-settings and */metric_settings endpoints so the API includes decision scorers in responses, and add decision to the ScorerTypes and ModelType enums. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…nerated client (SAO-17973) Splice the decision path, CreateDecisionScorerVersionRequest, DecisionLevel, DecisionScorerDB, boolean_probability and decision_scorer into openapi.yaml from the prod client spec (full regeneration yields an unrelated ~45k-line diff) and regenerate. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…7973) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

john-weiler · 2026-10-16T13:57:38Z

baz review

john-weiler · 2026-10-16T13:59:46Z


 _logger = logging.getLogger(__name__)

+DEFAULT_DECISION_MODEL_NAME = "jev-1.13.0"


should this be defined in code like this?

Changed in f450fb0 to match how LLM-as-a-judge handles its default model:

DecisionEvaluator now falls back to Configuration.default_decision_model (env SPLUNK_AO_DEFAULT_DECISION_MODEL, default jev-1.13.0), the same way LlmEvaluator uses default_scorer_model / SPLUNK_AO_DEFAULT_SCORER_MODEL. Users can move to a new Jev version without waiting for an SDK release.

create_decision_evaluator keeps a literal model_name="jev-1.13.0" default, as create_custom_llm_evaluator does with "gpt-4.1-mini". The module constant is gone.

Follow-up (not in this PR): Jev names pin a version that TypeSafe will deprecate and retire, so the longer-term fix is for the API to default to the registry's recommended model (recommended_decision_model) when model_name is omitted. Both SDKs could then stop shipping a default.

…creating the scorer (SAO-17973) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…973) DecisionEvaluator falls back to Configuration.default_decision_model (env SPLUNK_AO_DEFAULT_DECISION_MODEL, default jev-1.13.0), the same way LlmEvaluator uses default_scorer_model. create_decision_evaluator keeps a literal default, like create_custom_llm_evaluator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

etserend

🤖 This review was generated by the Astra agent (claude-opus-5-5). It may contain mistakes.

Verdict: approve — The create, rollback and decode paths are correct and well tested. Only minor issues remain: env-var default parity, status-code classification and a CHANGELOG gap.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

codegen_templates/endpoint_module.py.jinja:17-18: The PR description names a generator bug: the union to_dict branches for list[str] | list[DecisionLevel] are identical. DecisionScorerDB.to_dict (src/splunk_ao/resources/models/decision_scorer_db.py:63-70) has the same bug, so a round-trip of a discrete scorer never serializes levels through DecisionLevel.to_dict. Fix the union_property template and drop the additional_properties workaround in _build_decision_version_request.

etserend · 2026-10-16T20:22:47Z

+        name: str,
+        question: str,
+        node_level: StepType = StepType.llm,
+        model_name: str = "jev-1.13.0",


🟡 minor (design): The thread reply says SPLUNK_AO_DEFAULT_DECISION_MODEL lets users "move to a new Jev version without waiting for an SDK release". That holds only for DecisionEvaluator (src/splunk_ao/evaluator.py:946). Evaluators.create_decision_evaluator and the module function (line 493) hard-code "jev-1.13.0". As a result, a user who sets the env var and calls create_decision_evaluator(...) still gets jev-1.13.0. The ConfigKey description, "Default model for decision evaluators" (src/splunk_ao/configuration.py:113), suggests that both paths use it. The LLM functions copy this gap, but this API is new, so it is cheap to fix now. Suggested fix: default model_name to None in both signatures and resolve it with model_name or Configuration.default_decision_model. The literal then exists in one place only.

Suggested change

model_name: str = "jev-1.13.0",

model_name: str | None = None,

🤖 Generated by the Astra agent

etserend · 2026-10-16T20:22:48Z

+        )
+
+
+REJECTION_STATUS_CODES = frozenset({400, 403, 404, 422})


🟡 minor (question): The generated endpoint raises AuthenticationError for 401 and ConflictError for 409 (create_decision_scorer_version_scorers_scorer_id_version_decision_post.py:60-67). Both mean the server refused the request. Neither is in this set, so both fall through to DecisionVersionOutcomeUnknownError. The result is an orphan scorer with no version and an error that says "may have a version", which misleads the user. Please confirm that this choice is intentional. If it is not, add 401 and 409. The delete may also fail on 401, but _delete_scorer_quietly already logs and swallows that failure.

🤖 Generated by the Astra agent

etserend · 2026-10-16T20:22:49Z

+  from `answer_options`. If the version request is clearly rejected (400, 403, 404, 422) the new scorer is
+  deleted; on a timeout, network error or 5xx it is kept and the raised error names its id.
+- Scorer requests now send `X-Galileo-Scorer-Capabilities: decision`, and `ScorerTypes` and `ModelType` include
+  `decision`, so decision scorers appear in list and get responses and decode.


🟡 minor (documentation): This change adds a public configuration key: Configuration.default_decision_model / SPLUNK_AO_DEFAULT_DECISION_MODEL (src/splunk_ao/configuration.py:110-115). The Added section does not mention it. AGENTS.md asks for CHANGELOG updates when public behavior changes. Please add one bullet that names the env var and its jev-1.13.0 default.

🤖 Generated by the Astra agent

john-weiler and others added 5 commits October 16, 2026 09:36

feat(sdk): add create_decision_evaluator and DecisionEvaluator (SAO-1…

9b1c43e

…7973) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(sdk): harden decision evaluator validation and read path (SAO-17973)

6e451ed

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore: tidy comments

8388e52

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

john-weiler marked this pull request as ready for review October 16, 2026 13:58

john-weiler commented Oct 16, 2026

View reviewed changes

fix(sdk): validate decision question and answer option limits before …

5945ba8

…creating the scorer (SAO-17973) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

john-weiler marked this pull request as draft October 16, 2026 14:07

john-weiler marked this pull request as ready for review October 16, 2026 14:19

quinn-galileo approved these changes Oct 16, 2026

View reviewed changes

etserend reviewed Oct 16, 2026

View reviewed changes

etserend approved these changes Oct 16, 2026

View reviewed changes

john-weiler merged commit 59cd6b4 into main Oct 16, 2026
13 checks passed

john-weiler deleted the jaweiler/sao-17973-create-decision-evaluator branch October 16, 2026 21:16

github-actions Bot locked and limited conversation to collaborators Oct 16, 2026

Navigation Menu

Repository navigation

feat(sdk): add create_decision_evaluator and DecisionEvaluator (SAO-17973) - #273

feat(sdk): add create_decision_evaluator and DecisionEvaluator (SAO-17973)#273
john-weiler merged 7 commits into
mainfrom
jaweiler/sao-17973-create-decision-evaluator

john-weiler commented Oct 16, 2026

Uh oh!

john-weiler commented Oct 16, 2026

Uh oh!

john-weiler Oct 16, 2026

Uh oh!

john-weiler Oct 16, 2026

Uh oh!

etserend left a comment

Uh oh!

etserend Oct 16, 2026

Uh oh!

etserend Oct 16, 2026

Uh oh!

etserend Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		_logger = logging.getLogger(__name__)

		DEFAULT_DECISION_MODEL_NAME = "jev-1.13.0"

	model_name: str = "jev-1.13.0",
	model_name: str \| None = None,

Conversation

john-weiler commented Oct 16, 2026

Why

What

Spec and generator notes

Where to start

Tests

Uh oh!

john-weiler commented Oct 16, 2026

Uh oh!

john-weiler Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

john-weiler Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

etserend left a comment

Choose a reason for hiding this comment

Follow-ups

Uh oh!

etserend Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

etserend Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

etserend Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants