feat(datagen): close the fold/overview gaps and raise typed store errors - #223
Merged
Merged
Conversation
`DatagenTrajectory` already tracks `started` and `completed` position sets alongside `ordered`, but `folded_item_to_dto` projected only `ordered`, so neither set was reachable from the API, the client, or Python. Spec resume needs both: `completed` gates STEP_COMPLETED re-emission, `started` gates STEP_STARTED, and `started \ completed` is exactly the set of driver frames that were open when a writer died. Add `DatagenStreamPositionDto` and carry both sets on `FoldedDatagenItemDto`, projected in a deterministic order (the core sets are unordered). Both fields are `#[serde(default, skip_serializing_if = "Vec::is_empty")]`, so this is a backward-compatible wire addition: an older server simply yields empty sets rather than failing to decode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…temId Two follow-ups on the datagen fold/binding surface: - fold now rejects a field whose value kind drifts between writes, for both FIELD_SET (last-writer-wins) and FIELD_APPEND (within one list). Previously only the SET-vs-APPEND mix was caught, so a field could silently hold mixed-kind values. - DatagenItemId is registered as a pyclass and re-exported from the Python package, so callers compose and parse sub-item ids through the store's own path format instead of hand-splicing strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps in the datagen delta-log read surface:
* `DatagenRunOverview` aggregates a whole store (one experiment) into
root-granularity status counts, per-step completion counts, and a
failure roll-up grouped by `run_id`. Each run bucket carries its own
error-type counts plus a capped, deterministic sample of failing root
item ids to drill into. Exposed as `DatagenStore::overview`,
`GET /datagen/{name}/overview`, and `DatagenStore.overview()` in Python.
* `DatagenBlobProjection` makes the blob projection explicit. `fold_item`
stays lazy (bytes absent, resolved through `get_blob`);
`fold_item_with`/`fold_item_with_blobs`/`fold_item(load_blobs=True)`
materialize them inline, reading the payload column only when asked.
Also fixes the lint CI failure by wrapping the PyO3 `DatagenItemId` in a
thin Python class, matching the existing `DatagenStreamWriter` pattern.
The wire additions are backward compatible: new DTO fields default, so no
lockstep server upgrade is required.
…eError `to_py_err` flattened every `ContextError` to `PyRuntimeError`, discarding the distinction the enum already carries. Callers that retry appends could not tell a transport fault from a request the store rejected outright, so a deterministic rejection burned the whole retry budget before failing. Add a `ContextStoreError` hierarchy (rooted at `RuntimeError`, so `except RuntimeError` keeps working) with one leaf per `ContextError` variant, and route the `DatagenStore` methods through it. `InternalError` and `CompactionInProgressError` are the retryable pair; the rest are verdicts.
YangjunZ
changed the title
…es them CI type-checks without a built `_internal`, so every name imported from it needs the per-name `pyright: ignore` the rest of this module already uses. Re-exporting the error classes directly leaked the unresolved symbols into `__init__`.
beinan
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four changes the datagen delta-log needed before a Python client could drive it end to end.
started/completedsets — the fold now projects both, so a caller canderive the driver frames that were opened and never closed. That difference is what
attributes a crash to a step.
intis rejected if a later eventwrites it as
str, instead of silently reshaping the folded state.DatagenItemIdis alsoexported so callers compose fan-out child ids through the store's own path format rather
than by string splicing.
overview()aggregates one store intoroot-granularity status counts, completed-step counts, and a failure roll-up grouped by
run_id, each bucket carrying a capped deterministic sample of failing root item ids.fold_itemgains a lazy/eager blob switch; lazy stays the default, so folding an item doesnot read blob payloads unless asked.
to_py_errflattened everyContextErrortoRuntimeError,discarding a distinction the enum already carried. Callers retrying an append could not
tell a transport fault from a request the store rejected outright, so a deterministic
rejection burned the whole retry budget. There is now a
ContextStoreErrorhierarchy withone leaf per variant; it is rooted at
RuntimeError, soexcept RuntimeErrorkeepscatching everything it used to.
Note for remote deployments:
overview()is a new endpoint (GET /datagen/{name}/overview),so remote mode needs a server upgrade. The existing fold calls stay wire-compatible, and
embedded mode bypasses client/server entirely.