diff --git a/docs/adr/0002-decision-model-and-rl-environment.md b/docs/adr/0002-decision-model-and-rl-environment.md index ce09fda6..5191e265 100644 --- a/docs/adr/0002-decision-model-and-rl-environment.md +++ b/docs/adr/0002-decision-model-and-rl-environment.md @@ -124,13 +124,14 @@ The list leaves out models above 10 billion parameters and models built on a bas A trained model should replace the base model only when a repository administrator approves it. The evaluation command writes a report with two numbers to compare against the base model on the test set: the mean reward, which should be higher, and the calibration error, which should not be worse. The command does not record a promotion decision or serve a model. -The customer-operated, single-node reference Replica decision runtime records that decision outside Hub. It accepts a taskset, tracks a job bounded by update, wall-time, and required credit limits, writes authoritative connector usage with the immutable candidate and durable job, and serves only the candidate an operator explicitly promotes. Promotion and rollback require the generation the operator read; the connector compares and advances that generation atomically, preserves exact idempotent retries, and appends every successful decision. A deterministic fixture backend runs this lifecycle without a GPU or external model download. Fixture candidates are for development and tests only and cannot be promoted in production. The real local backend trains d1-omni on NVIDIA CUDA or Apple silicon MPS and serves the verified pinned base model plus the promoted adapter; CPU mode is inference-only and does not claim model-quality validation. Provider-neutral infrastructure contracts separate taskset and artifact access, durable jobs and leases, candidates and promotion generations, training, inference, credentials, and usage reporting. Local and hosted implementations must preserve tenant and repository scope, hard-budget failure behavior, and restart recovery, and pass the same conformance suite. +The customer-operated, single-node reference Replica decision runtime records that decision outside Hub. It accepts a taskset, tracks a job bounded by update, wall-time, and required credit limits, writes authoritative connector usage with the immutable candidate and durable job, and serves only the candidate an operator explicitly promotes. Promotion and rollback require the generation the operator read; the connector compares and advances that generation atomically, preserves exact idempotent retries, and appends every successful decision. A deterministic fixture backend runs this lifecycle without a GPU or external model download. Fixture candidates are for development and tests only and cannot be promoted in production. The real local backend trains d1-omni on NVIDIA CUDA or Apple silicon MPS and serves the verified pinned base model plus the promoted adapter; CPU mode is inference-only and does not claim model-quality validation. Provider-neutral infrastructure contracts separate taskset and artifact access, durable jobs and leases, candidates and promotion generations, content-free copy inventory and deletion receipts, training, inference, credentials, and usage reporting. Local and hosted implementations must preserve tenant and repository scope, hard-budget failure behavior, consent-withdrawal revocation, restart recovery, and the same conformance suite. ## Consequences - Teams can train on their own review outcomes without sending them anywhere they did not choose. - Teams can practice and test the complete lifecycle with a deterministic backend before spending GPU budget. Its report proves the orchestration and scoring path, not real-model quality. -- A taskset copies screenshots and learning text. It lives under `.replica/` in a checkout, or outside any checkout. When someone withdraws training consent, `replica-learning` lists the export the taskset came from, and the owner must delete the taskset too. +- A taskset copies screenshots and learning text. It lives under `.replica/` in a checkout, or outside any checkout. The runtime records its source export, permitting grants and purposes, review and evidence digests, connector reference, retention metadata, and derived candidates and reports without copying raw evidence into receipts. Withdrawal revokes new and current runtime use before connector deletion begins. +- Runtime deletion cannot prove deletion from infrastructure outside its configured connectors. Operators must separately clear customer-derived model and trainer caches, merged weights, external registries, backups, and provider logs. Unmodified shared base models are not customer-derived copies. - The Builds file holds screenshots of every published Build of each reviewed prototype. The operator should delete it after writing the taskset. - A "follows" label rests on the reviewer's check of the fix, which may have covered one width and theme. The other screenshots of that page count as fixed unless a later thread reports the same theme there. - A "breaks" label can still be wrong when the reviewer found the problem after scrolling, opening a menu, or resizing the window to another width, because the screenshots show each page as it first loads at two widths. Older Builds that kept light screenshots only give "follows" labels alone. diff --git a/docs/decision-model.md b/docs/decision-model.md index e7c1202a..d4693774 100644 --- a/docs/decision-model.md +++ b/docs/decision-model.md @@ -91,4 +91,6 @@ See [Train d1-omni-600M](../packages/learning/environments/brand-check/README.md ## 4. Run the customer-specific model in the reference runtime -The [single-node reference runtime](../packages/learning/environments/brand-check/README.md#run-the-single-node-reference-runtime) runs the complete job, candidate, promotion, rollback, and inference lifecycle on customer-controlled infrastructure. Its default deterministic fixture backend is for development and tests only and remains ineligible for production promotion. Install the optional model dependencies and configure `--d1-model` to train d1-omni on NVIDIA CUDA or Apple silicon MPS, keep only the adapter and evaluation report in runtime storage, promote it explicitly, and serve the pinned base model plus that adapter. CPU mode supports an opt-in forward-pass smoke check and low-volume inference, not real training or a model-quality claim. +The [single-node reference runtime](../packages/learning/environments/brand-check/README.md#run-the-single-node-reference-runtime) runs the complete job, candidate, promotion, rollback, inference, disclosure, and consent-withdrawal deletion lifecycle on customer-controlled infrastructure. Its default deterministic fixture backend is for development and tests only and remains ineligible for production promotion. Install the optional model dependencies and configure `--d1-model` to train d1-omni on NVIDIA CUDA or Apple silicon MPS, keep only the adapter and evaluation report in runtime storage, promote it explicitly, and serve the pinned base model plus that adapter. CPU mode supports an opt-in forward-pass smoke check and low-volume inference, not real training or a model-quality claim. + +The runtime records content-free provenance for tasksets, candidates, and reports. An authenticated deletion request can select copies by source export, permitting grant, evidence digest, or taskset digest. It revokes use first, cancels matching jobs, removes connector-backed copies, and keeps a durable receipt until every item is deleted or retained for a documented legal reason. Operators remain responsible for copies outside those connectors, including trainer and serving caches, merged weights, external model registries, backups, and provider logs. diff --git a/packages/learning/README.md b/packages/learning/README.md index 408e973d..ace26a4b 100644 --- a/packages/learning/README.md +++ b/packages/learning/README.md @@ -220,7 +220,7 @@ The output directory must be new and, inside a checkout, under `.replica/`. It h | File | Contents | | --- | --- | -| `taskset.json` | The export it came from, task counts for each split and label, and how much evidence it skipped and why | +| `taskset.json` | The export and export digest it came from, permitting grant IDs and purposes, review and evidence digests, task counts for each split and label, retention metadata, and how much evidence it skipped and why | | `tasks.jsonl` | One task per line: its split, images, the questions, labels for the yes-or-no questions the evidence answers, and where the labels came from | | `images/` | A copy of each screenshot, and of the image `--brand-sheet` names, which every task then shows first as a reference | @@ -243,7 +243,7 @@ The command skips evidence it cannot turn into a label, and reports each reason: | `renderings-incomplete` | The page lacks a screenshot at one width or theme, so it gets no "breaks" label | | `learning-over-limit` | More than 64 learnings were accepted, and recall ranks this one below the first 64 | -The [single-node reference Replica decision runtime](environments/brand-check/README.md#run-the-single-node-reference-runtime) is customer-operated. Its authenticated API registers private content-addressed tasksets, tracks cancellable jobs with hard credit and time limits, stores immutable candidates with authoritative connector usage, requires generation-checked promotion or rollback, and serves the promoted result through an authenticated scoring endpoint. Durable job limits, usage, promotion history, and idempotency records survive restart. The fixture proves the lifecycle without a GPU or model download, but its candidates are development/test-only and cannot be promoted in production. Provider-neutral [infrastructure connectors](environments/brand-check/README.md#replace-the-infrastructure) separate tasksets, artifacts, durable jobs and leases, candidates and promotions, training, inference, credentials, and usage reporting so local and hosted implementations can share one conformance suite. +The [single-node reference Replica decision runtime](environments/brand-check/README.md#run-the-single-node-reference-runtime) is customer-operated. Its authenticated API registers private content-addressed tasksets, tracks cancellable jobs with hard credit and time limits, stores immutable candidates with authoritative connector usage, requires generation-checked promotion or rollback, and serves the promoted result through an authenticated scoring endpoint. It also discloses content-free runtime copy metadata and accepts idempotent consent-withdrawal deletion requests. A deletion revokes readiness, inference, promotion, uploads, and new training first; cancels matching jobs; removes or retention-tombstones tasksets, candidates, and reports through their connectors; and remains pending until every copy is gone or has a documented legal reason. Durable job limits, usage, promotion history, privacy receipts, and idempotency records survive restart. The fixture proves the lifecycle without a GPU or model download, but its candidates are development/test-only and cannot be promoted in production. Provider-neutral [infrastructure connectors](environments/brand-check/README.md#replace-the-infrastructure) separate tasksets, artifacts, durable jobs and leases, candidates and promotions, privacy inventory and deletion receipts, training, inference, credentials, and content-free usage reporting so local and hosted implementations can share one conformance suite. ## Local storage diff --git a/packages/learning/environments/brand-check/.dockerignore b/packages/learning/environments/brand-check/.dockerignore index ac6ae6ec..1879de1f 100644 --- a/packages/learning/environments/brand-check/.dockerignore +++ b/packages/learning/environments/brand-check/.dockerignore @@ -4,3 +4,4 @@ !README.md !*.py !server/*.py +!examples/runtime_deletion_demo.py diff --git a/packages/learning/environments/brand-check/README.md b/packages/learning/environments/brand-check/README.md index 2b628dae..6204752e 100644 --- a/packages/learning/environments/brand-check/README.md +++ b/packages/learning/environments/brand-check/README.md @@ -297,6 +297,9 @@ The HTTP contract stays independent of the local store and trainer implementatio | `GET /v1/jobs/` | Returns `queued`, `running`, `succeeded`, `failed`, or `cancelled` | | `POST /v1/jobs//cancel` | Idempotently cancels a queued or running job; completed jobs return `409 job_terminal` | | `GET /v1/candidates/` | Returns the immutable manifest and evaluation report | +| `GET /v1/runtime-copies` | Discloses bounded, content-free copy metadata for exact export, grant, evidence-digest, or taskset-digest selectors | +| `POST /v1/deletions` | Revokes matching use, cancels work, deletes or retains copies, and returns a durable idempotent receipt | +| `GET /v1/deletions/` | Returns the durable content-free deletion receipt | | `POST /v1/promotions` | Moves the pointer when required `expectedGeneration` still matches; exact idempotent retries return the applied generation | | `POST /v1/rollbacks` | Restores the previous candidate as a new generation when required `expectedGeneration` still matches | | `GET /v1/promotion` | Returns the promoted candidate and generation | @@ -334,6 +337,15 @@ docker compose up --build Compose binds the service only to localhost and mounts the private runtime state read-write. Upload a taskset, submit a hard-budget job that names its digest, cancel a queued or running job, and repeat promotion or rollback requests with the same generation and idempotency key to exercise restart-safe retries. Keep `REPLICA_RUNTIME_STATE_DIR` on durable storage: it holds job budgets, connector-reported usage, generation history, and retry records. The local factory binds the directory permanently to the configured tenant and repository, so reopening it under another scope fails instead of exposing the first scope's tasksets, jobs, or candidates. +Run the complete consent-withdrawal proof with the optional Compose demo service: + +```sh +docker compose --profile demo up --build \ + --abort-on-container-exit --exit-code-from lifecycle-demo +``` + +The demo uploads a synthetic taskset, trains and promotes the fixture candidate, discloses its runtime copies, withdraws the training grant, verifies that inference is refused, and exits only after the deletion receipt is `completed`. + ### Replace the infrastructure The runtime contract follows the lifecycle rather than the services of one cloud. [`connectors.py`](connectors.py) defines the interfaces, and [`local_connectors.py`](local_connectors.py) composes the complete single-node implementation: @@ -344,12 +356,13 @@ The runtime contract follows the lifecycle rather than the services of one cloud | Artifact | `LocalArtifactConnector` keeps verified immutable bytes under private SHA-256 paths | Keep immutable candidate inputs and outputs in private content-addressed storage | | Job | `LocalJobConnector` keeps atomic JSON records, compare-and-swap versions, durable cancellation, idempotency records, and requires the active expiring worker lease for each state update | Keep durable job state and fence each transactional update through one worker lease | | Candidate | `LocalCandidateConnector` keeps read-only candidate files, verifies their digests, and appends generation-checked promotion or rollback decisions with durable idempotency records | Keep immutable candidate manifests and artifacts, with a transactional generation-checked promotion pointer and exact retry result | +| Privacy | `LocalPrivacyConnector` keeps scoped content-free copy inventory, revocations, retention tombstones, and versioned deletion receipts | Enumerate exact matching copies, durably revoke use before deletion, delete or tombstone through the owning connector, and expose retryable pending items without raw evidence | | Training | `LocalTrainingConnector` runs the fixture or real d1 backend inside the bounded local worker, enforces supported hard credit budgets before work, and returns persistable adapter bytes, evaluation evidence, and authoritative usage | Reject unsupported hard budgets before scheduling, stop before exceeding measurable limits, and return the same candidate evidence, authoritative usage, and billed cost | | Inference | `LocalInferenceConnector` serves the explicitly promoted fixture adapter or verified pinned d1 base plus promoted adapter | Send evaluation requests only to the explicitly promoted model deployment | | Credential | `LocalCredentialConnector` mints short-lived operation-bound credentials from a private installation key | Acquire workload or brokered credentials restricted to the operation, audience, tenant, and repository | | Observability | `LocalObservabilityConnector` appends scoped usage, optional cost, and metadata to private JSON Lines | Send the same scoped usage and cost records to the operator's telemetry and billing sinks | -`create_local_connector_bundle` is the composition boundary. `LocalRuntime` uses that factory for tasksets, artifacts, jobs, candidates, training, inference, credentials, and observability; it no longer keeps a second filesystem implementation beside the connectors. Existing `jobs/`, `candidates/`, `promotions/`, and `promotion.json` data remains readable. The factory adds private connector metadata beside it and migrates an existing candidate descriptor when the candidate is first read. +`create_local_connector_bundle` is the composition boundary. `LocalRuntime` uses that factory for tasksets, artifacts, jobs, candidates, privacy receipts, training, inference, credentials, and observability; it no longer keeps a second filesystem implementation beside the connectors. Existing `jobs/`, `candidates/`, `promotions/`, and `promotion.json` data remains readable. The factory adds private connector metadata beside it and migrates an existing candidate descriptor when the candidate is first read. Every call receives an `OperationContext`. It names the tenant, repository, owner, operation, idempotency key, monotonic deadline, and cooperative cancellation signal. A retry with the same idempotency key and input returns the first result. Reusing the key for different input is a conflict. Connectors must check cancellation and deadlines before causing side effects. @@ -357,7 +370,7 @@ Artifacts and candidates are immutable. Their identities include content digests Connector failures use stable codes plus one category: invalid input, not found, conflict, unauthorized, integrity, cancelled, deadline exceeded, unavailable, or internal. Only a connector that knows an operation is safe to repeat marks its error retryable. Usage records always name the operation and customer scope; providers may add quantities and cost, but must not add provider-specific fields to the shared lifecycle records. -[`connector_conformance.py`](connector_conformance.py) returns one reusable `unittest` suite for a fresh connector bundle. Local and hosted implementations run that suite unchanged. Its contract-only fixture proves tenant and repository isolation across every stored record, immutable content, idempotent retries, lease expiry and fencing, promotion generations, cancellation and deadlines, persistable training output, credential boundaries, error classification, and usage/cost attribution. +[`connector_conformance.py`](connector_conformance.py) returns one reusable `unittest` suite for a fresh connector bundle. Local and hosted implementations run that suite unchanged. Its contract-only fixture proves tenant and repository isolation across every stored record, immutable content, bounded copy enumeration, deletion and retention tombstones, versioned receipts, idempotent retries, lease expiry and fencing, promotion generations, cancellation and deadlines, persistable training output, credential boundaries, error classification, and usage/cost attribution. The single-node runtime preserves its HTTP API and lifecycle behavior while every infrastructure operation goes through the local bundle. Installed runtimes execute training and each inference request in a spawned process, so a backend and its configuration must remain serializable. Each inference request pays process-startup cost, and model state loaded inside one inference child is not retained for the next request. A future hosted bundle replaces the implementations, not the lifecycle records or routes, and must pass the same suite without provider-specific fields in the shared contracts. @@ -365,7 +378,10 @@ The single-node runtime preserves its HTTP API and lifecycle behavior while ever - The operator chooses the host. The reference runtime never calls Replica or a cloud service. A later hosted implementation can keep the same job, candidate, promotion, and inference contracts. - The operator or hosting layer assigns the tenant and repository scope. A storage connector must return not found, rather than reveal another scope's records. A credential connector must reject another tenant or repository as unauthorized without minting a credential. -- The taskset contains private screenshots, learning text, and labels. Mount it read-only. A training-consent withdrawal still requires the operator to delete the taskset and every candidate derived from that export. +- The taskset contains private screenshots, learning text, and labels. The runtime keeps content-free provenance for uploaded and mounted tasksets and every derived candidate and report. A matching deletion request revokes readiness, promotion, inference, and new training before it removes connector-backed copies. +- A receipt reaches `completed` only after every known copy is deleted or has a documented legal or retention reason. A hosted connector must preserve retryable pending items across restart and must not translate an unavailable delete into success. +- Model-serving caches, trainer caches, downloaded base models, external model registries, snapshots, backups, and provider logs are outside the reference runtime's connectors. Operators must inventory and clear customer-derived adapters, prompts, embeddings, compiled artifacts, and cache entries there. A shared unmodified base model may be retained only when it contains no customer training data; any merged or fine-tuned weights are a derived copy and must be disclosed and deleted or explicitly retained. +- Connector implementations must return stable, scope-safe references; implement exact enumeration, deletion, and retention tombstones; make retries idempotent; and keep receipts free of screenshots, learning text, labels, prompts, adapter bytes, and raw evidence. - The runtime state contains adapters, evaluation reports, job errors, the promotion pointer, `credentials/local.key`, and the append-only `observability/usage.jsonl` usage and cost log. Back it up or delete it under the customer's retention rules. Candidate files and the local credential key are created read-only and never updated in place. The runtime does not rotate or prune the usage log; the operator must do that while the runtime is stopped. - The runtime token authorizes training, promotion, candidate reads, and inference. Keep it in a protected file or secret store and pass it through the environment, not the command line or image. - The service uses plain HTTP. Bind it to localhost as shown, or put customer-operated TLS and authentication in front of it before another host can connect. diff --git a/packages/learning/environments/brand-check/compose.yaml b/packages/learning/environments/brand-check/compose.yaml index b3380721..ce370a74 100644 --- a/packages/learning/environments/brand-check/compose.yaml +++ b/packages/learning/environments/brand-check/compose.yaml @@ -26,3 +26,18 @@ services: user: ${REPLICA_RUNTIME_USER:-10001:10001} volumes: - ${REPLICA_RUNTIME_STATE_DIR:?set REPLICA_RUNTIME_STATE_DIR}:/data/runtime + lifecycle-demo: + profiles: [demo] + build: + context: . + dockerfile: server/runtime.Dockerfile + depends_on: + decision-runtime: + condition: service_healthy + environment: + REPLICA_RUNTIME_TOKEN: ${REPLICA_RUNTIME_TOKEN:?set REPLICA_RUNTIME_TOKEN} + command: + - python + - /app/runtime_deletion_demo.py + - --url + - http://decision-runtime:8010 diff --git a/packages/learning/environments/brand-check/connector_conformance.py b/packages/learning/environments/brand-check/connector_conformance.py index 34be1f16..cb13c838 100644 --- a/packages/learning/environments/brand-check/connector_conformance.py +++ b/packages/learning/environments/brand-check/connector_conformance.py @@ -62,6 +62,18 @@ def case(label: str, body: Callable[[unittest.TestCase], None]) -> unittest.Func case("isolates tenant and repository scope", _scope_isolation(create_fixture)), case("stores immutable content by digest", _immutable_artifacts(create_fixture)), case("stores immutable scoped candidates", _immutable_candidates(create_fixture)), + case( + "enumerates and tombstones content-free runtime copies", + _privacy_inventory(create_fixture), + ), + case( + "deletes tasksets and candidates through their owning connectors", + _copy_deletion(create_fixture), + ), + case( + "persists idempotent versioned deletion receipts", + _deletion_receipts(create_fixture), + ), case( "makes retries idempotent and conflicting reuse explicit", _idempotency(create_fixture), @@ -114,7 +126,7 @@ def run(case): context, ) case.assertEqual(registered.digest, fixture.taskset_digest) - case.assertEqual(registered.reference, f"taskset:{registered.digest.value}") + case.assertTrue(registered.reference.endswith(registered.digest.value)) case.assertEqual( _validated_taskset_digest(registered.taskset), fixture.taskset_digest, @@ -366,6 +378,255 @@ def run(case): return run +def _privacy_inventory(create_fixture): + def run(case): + fixture = create_fixture() + context = _context(fixture.scope, "privacy-copy") + record = { + "schemaVersion": "replica.runtime-copy/v1", + "id": "copy_contract", + "kind": "taskset", + "scope": { + "tenantId": fixture.scope.tenant_id, + "repository": fixture.scope.repository, + }, + "sourceTrainingExportId": "export_contract", + "sourceTrainingExportDigest": "sha256:" + "1" * 64, + "permittingGrantIds": ["grant_contract"], + "permittedPurposes": ["training"], + "reviewDigests": ["sha256:" + "2" * 64], + "evidenceDigests": ["sha256:" + "3" * 64], + "evidenceDigest": "sha256:" + "4" * 64, + "tasksetDigest": str(fixture.taskset_digest), + "runtimeCopyId": fixture.taskset_reference, + "storage": { + "connector": type(fixture.connectors.tasksets).__name__, + "reference": fixture.taskset_reference, + }, + "createdAt": "2026-11-18T00:00:00Z", + "useState": "active", + "promotionState": "not_applicable", + "retention": {"mode": "delete_on_withdrawal"}, + "parentRuntimeCopyIds": [], + } + case.assertEqual(fixture.connectors.privacy.put_copy(record, context), record) + second = { + **record, + "id": "copy_contract_second", + "runtimeCopyId": "taskset:second", + "storage": {**record["storage"], "reference": "taskset:second"}, + } + case.assertEqual(fixture.connectors.privacy.put_copy(second, context), second) + case.assertEqual( + fixture.connectors.privacy.get_copy("copy_contract", context), + record, + ) + page, cursor = fixture.connectors.privacy.list_copies( + {"grantId": "grant_contract"}, + cursor=None, + limit=1, + context=context, + ) + case.assertEqual(tuple(page), (record,)) + case.assertEqual(cursor, record["id"]) + second_page, end = fixture.connectors.privacy.list_copies( + { + "exportId": "export_contract", + "grantId": "grant_contract", + "evidenceDigest": "sha256:" + "4" * 64, + }, + cursor=cursor, + limit=1, + context=context, + ) + case.assertEqual(tuple(second_page), (second,)) + case.assertIsNone(end) + no_match, no_cursor = fixture.connectors.privacy.list_copies( + {"exportId": "export_other"}, + cursor=None, + limit=10, + context=context, + ) + case.assertEqual(tuple(no_match), ()) + case.assertIsNone(no_cursor) + for invalid_limit in (0, 101): + with case.assertRaises(ConnectorError) as invalid: + fixture.connectors.privacy.list_copies( + {"grantId": "grant_contract"}, + cursor=None, + limit=invalid_limit, + context=context, + ) + case.assertEqual(invalid.exception.category, "invalid_input") + revoked = fixture.connectors.privacy.tombstone_copy( + "copy_contract", + state="revoked", + retention=None, + context=context, + ) + case.assertEqual(revoked["useState"], "revoked") + case.assertNotIn("payload", json.dumps(revoked)) + for label, scope in ( + ("repository", fixture.other_repository_scope), + ("tenant", fixture.other_tenant_scope), + ): + other = _context(scope, f"privacy-copy-other-{label}") + with case.assertRaises(ConnectorError) as hidden: + fixture.connectors.privacy.get_copy("copy_contract", other) + case.assertIn(hidden.exception.category, ("not_found", "unauthorized")) + try: + other_page, _cursor = fixture.connectors.privacy.list_copies( + {"grantId": "grant_contract"}, + cursor=None, + limit=10, + context=other, + ) + except ConnectorError as error: + case.assertIn(error.category, ("not_found", "unauthorized")) + else: + case.assertFalse(other_page) + with case.assertRaises(ConnectorError) as refused: + fixture.connectors.privacy.tombstone_copy( + "copy_contract", + state="deleted", + retention=None, + context=other, + ) + case.assertIn(refused.exception.category, ("not_found", "unauthorized")) + + return run + + +def _copy_deletion(create_fixture): + def run(case): + fixture = create_fixture() + context = _context(fixture.scope, "copy-delete") + deletable_bundle = replace( + fixture.taskset_bundle, + manifest={**fixture.taskset_bundle.manifest, "exportId": "export_delete"}, + digest=None, + ) + registered = fixture.connectors.tasksets.register( + deletable_bundle, + context, + ) + candidate = fixture.connectors.candidates.put( + _candidate("delete"), context + ) + kept_candidate = fixture.connectors.candidates.put( + _candidate("kept"), context + ) + case.assertTrue( + fixture.connectors.tasksets.delete(registered.reference, context) + ) + case.assertTrue(fixture.connectors.candidates.delete(candidate.id, context)) + case.assertFalse( + fixture.connectors.tasksets.delete(registered.reference, context) + ) + case.assertFalse(fixture.connectors.candidates.delete(candidate.id, context)) + with case.assertRaises(ConnectorError) as missing_taskset: + fixture.connectors.tasksets.resolve(registered.reference, context) + case.assertEqual(missing_taskset.exception.category, "not_found") + with case.assertRaises(ConnectorError) as missing_candidate: + fixture.connectors.candidates.get(candidate.id, context) + case.assertEqual(missing_candidate.exception.category, "not_found") + other = _context(fixture.other_tenant_scope, "copy-delete-other") + with case.assertRaises(ConnectorError) as hidden_taskset: + fixture.connectors.tasksets.delete(fixture.taskset_reference, other) + case.assertIn(hidden_taskset.exception.category, ("not_found", "unauthorized")) + with case.assertRaises(ConnectorError) as hidden_candidate: + fixture.connectors.candidates.delete(kept_candidate.id, other) + case.assertIn(hidden_candidate.exception.category, ("not_found", "unauthorized")) + case.assertEqual( + fixture.connectors.tasksets.resolve( + fixture.taskset_reference, + _context(fixture.scope, "copy-delete-owner-read"), + ).reference, + fixture.taskset_reference, + ) + case.assertEqual( + fixture.connectors.candidates.get( + kept_candidate.id, + _context(fixture.scope, "copy-delete-owner-candidate-read"), + ), + kept_candidate, + ) + + return run + + +def _deletion_receipts(create_fixture): + def run(case): + fixture = create_fixture() + context = _context( + fixture.scope, + "privacy-delete", + idempotency_key="privacy-delete", + ) + request = { + "selectors": {"exportId": "export_contract"}, + "requestedBy": "operator", + "reason": "Training consent was withdrawn.", + "retention": None, + } + receipt = fixture.connectors.privacy.create_deletion(request, context) + case.assertEqual( + fixture.connectors.privacy.create_deletion(request, context), + receipt, + ) + with case.assertRaises(ConnectorError) as reused: + fixture.connectors.privacy.create_deletion( + {**request, "selectors": {"grantId": "grant_other"}}, + context, + ) + case.assertEqual(reused.exception.category, "conflict") + updated = dict(receipt, state="pending", updatedAt="2026-11-18T00:01:00Z") + saved = fixture.connectors.privacy.update_deletion( + receipt["id"], + updated, + expected_version=receipt["version"], + context=context, + ) + case.assertEqual(saved["version"], receipt["version"] + 1) + with case.assertRaises(ConnectorError) as stale: + fixture.connectors.privacy.update_deletion( + receipt["id"], + saved, + expected_version=receipt["version"], + context=context, + ) + case.assertEqual(stale.exception.category, "conflict") + case.assertEqual( + fixture.connectors.privacy.get_deletion(receipt["id"], context), + saved, + ) + case.assertEqual( + tuple(fixture.connectors.privacy.list_deletions(("pending",), context)), + (saved,), + ) + for label, scope in ( + ("repository", fixture.other_repository_scope), + ("tenant", fixture.other_tenant_scope), + ): + other = _context(scope, f"privacy-deletion-other-{label}") + with case.assertRaises(ConnectorError) as hidden: + fixture.connectors.privacy.get_deletion(receipt["id"], other) + case.assertIn(hidden.exception.category, ("not_found", "unauthorized")) + case.assertFalse( + fixture.connectors.privacy.list_deletions(("pending",), other) + ) + with case.assertRaises(ConnectorError) as refused: + fixture.connectors.privacy.update_deletion( + receipt["id"], + saved, + expected_version=saved["version"], + context=other, + ) + case.assertIn(refused.exception.category, ("not_found", "unauthorized")) + + return run + + def _leases(create_fixture): def run(case): fixture = create_fixture() diff --git a/packages/learning/environments/brand-check/connectors.py b/packages/learning/environments/brand-check/connectors.py index 959e50d1..14f9a2b2 100644 --- a/packages/learning/environments/brand-check/connectors.py +++ b/packages/learning/environments/brand-check/connectors.py @@ -162,6 +162,7 @@ class TasksetArtifact: reference: str digest: ContentDigest taskset: Taskset + created: bool = field(default=False, compare=False) @dataclass(frozen=True) @@ -336,6 +337,10 @@ def register( def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: ... + def delete(self, reference: str, context: OperationContext) -> bool: ... + + def list(self, context: OperationContext) -> Sequence[TasksetArtifact]: ... + class ArtifactConnector(Protocol): def put(self, artifact: Artifact, context: OperationContext) -> Artifact: ... @@ -403,6 +408,61 @@ def promote( def current(self, context: OperationContext) -> Promotion | None: ... + def delete(self, candidate_id: str, context: OperationContext) -> bool: ... + + def list(self, context: OperationContext) -> Sequence[Candidate]: ... + + +class PrivacyConnector(Protocol): + """Content-free runtime copy inventory, revocations, and deletion receipts.""" + + def put_copy( + self, record: Mapping[str, Any], context: OperationContext + ) -> Mapping[str, Any]: ... + + def get_copy( + self, copy_id: str, context: OperationContext + ) -> Mapping[str, Any]: ... + + def list_copies( + self, + selectors: Mapping[str, str], + *, + cursor: str | None, + limit: int, + context: OperationContext, + ) -> tuple[Sequence[Mapping[str, Any]], str | None]: ... + + def tombstone_copy( + self, + copy_id: str, + *, + state: str, + retention: Mapping[str, Any] | None, + context: OperationContext, + ) -> Mapping[str, Any]: ... + + def create_deletion( + self, request: Mapping[str, Any], context: OperationContext + ) -> Mapping[str, Any]: ... + + def get_deletion( + self, deletion_id: str, context: OperationContext + ) -> Mapping[str, Any]: ... + + def list_deletions( + self, states: Sequence[str], context: OperationContext + ) -> Sequence[Mapping[str, Any]]: ... + + def update_deletion( + self, + deletion_id: str, + record: Mapping[str, Any], + *, + expected_version: int, + context: OperationContext, + ) -> Mapping[str, Any]: ... + class TrainingConnector(Protocol): name: str @@ -423,6 +483,12 @@ def train( class InferenceConnector(Protocol): name: str + def delete(self, reference: str, context: OperationContext) -> bool: ... + + def list_results( + self, context: OperationContext + ) -> Sequence[Mapping[str, Any]]: ... + def predict( self, adapter: Path, @@ -451,6 +517,7 @@ class RuntimeConnectors: artifacts: ArtifactConnector jobs: JobConnector candidates: CandidateConnector + privacy: PrivacyConnector training: Mapping[str, TrainingConnector] inference: Mapping[str, InferenceConnector] credentials: CredentialConnector diff --git a/packages/learning/environments/brand-check/examples/runtime_deletion_demo.py b/packages/learning/environments/brand-check/examples/runtime_deletion_demo.py new file mode 100644 index 00000000..8ef4aeee --- /dev/null +++ b/packages/learning/environments/brand-check/examples/runtime_deletion_demo.py @@ -0,0 +1,270 @@ +"""Exercise upload, training, promotion, disclosure, withdrawal, and deletion over HTTP.""" + +from __future__ import annotations + +import argparse +import base64 +import hashlib +import json +import os +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + +from replica_brand_check.local_connectors import taskset_digest +from replica_brand_check.taskset import load_taskset + +PNG = base64.b64decode( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=" +) +EXPORT_ID = "export_compose_deletion_demo" +GRANT_ID = "grant_compose_training" +RULE_ID = "mem_111111111111111111111111" + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--url", default="http://127.0.0.1:8010") + args = parser.parse_args() + token = os.environ["REPLICA_RUNTIME_TOKEN"] + with tempfile.TemporaryDirectory() as temporary: + taskset = write_taskset(Path(temporary)) + digest, upload = upload_request(taskset) + call( + args.url, + token, + "PUT", + f"/v1/tasksets/{digest}", + upload, + key=f"upload-{digest}", + ) + job = call( + args.url, + token, + "POST", + "/v1/jobs", + { + "backend": "fixture", + "tasksetDigest": f"sha256:{digest}", + "maxUpdates": 8, + "maxSeconds": 30, + "maxCredits": 8, + "budgetMode": "hard", + }, + ) + job = wait_for_job(args.url, token, job["id"]) + promotion = call( + args.url, + token, + "POST", + "/v1/promotions", + { + "candidateId": job["candidateId"], + "promotedBy": "compose-demo", + "expectedGeneration": 0, + }, + key="promote-compose-demo", + ) + disclosure = call( + args.url, + token, + "GET", + "/v1/runtime-copies?" + + urllib.parse.urlencode({"exportId": EXPORT_ID, "limit": 100}), + ) + before = call(args.url, token, "POST", "/score", evaluation_request()) + receipt = call( + args.url, + token, + "POST", + "/v1/deletions", + { + "selectors": {"grantId": GRANT_ID}, + "requestedBy": "compose-demo", + "reason": "The training grant was withdrawn.", + }, + key="delete-compose-demo", + ) + receipt = wait_for_deletion(args.url, token, receipt["id"]) + if receipt["state"] != "completed": + raise RuntimeError(f"Deletion did not complete: {receipt['state']}") + refused = call_error(args.url, token, "POST", "/score", evaluation_request()) + if refused != (409, "deletion_revoked"): + raise RuntimeError(f"Inference was not refused after deletion: {refused}") + print( + json.dumps( + { + "uploadedTasksetDigest": f"sha256:{digest}", + "jobState": job["state"], + "promotedGeneration": promotion["generation"], + "disclosedCopies": len(disclosure["items"]), + "servedCandidateId": before["candidateId"], + "deletionReceiptId": receipt["id"], + "deletionState": receipt["state"], + "inferenceAfterWithdrawal": "refused", + }, + sort_keys=True, + ) + ) + + +def write_taskset(directory: Path) -> Path: + (directory / "images").mkdir() + (directory / "images" / "screen.png").write_bytes(PNG) + tasks = [task(1, "train", True), task(2, "train", False), task(3, "test", True)] + manifest = { + "schemaVersion": "replica.decision-taskset/v1", + "exportId": EXPORT_ID, + "createdAt": "2026-11-19T00:00:00Z", + "provenance": { + "exportDigest": "sha256:" + "1" * 64, + "grantIds": [GRANT_ID], + "purposes": ["training"], + "reviewDigests": ["sha256:" + "2" * 64], + "evidenceDigests": ["sha256:" + "3" * 64], + }, + "retention": {"mode": "delete_on_withdrawal"}, + "counts": {"tasks": len(tasks)}, + } + (directory / "taskset.json").write_text(json.dumps(manifest), encoding="utf-8") + (directory / "tasks.jsonl").write_text( + "".join(json.dumps(item) + "\n" for item in tasks), + encoding="utf-8", + ) + return directory + + +def task(number: int, split: str, answer: bool) -> dict: + image_digest = "sha256:" + hashlib.sha256(PNG).hexdigest() + return { + "schemaVersion": "replica.decision-task/v1", + "id": f"task_{number:024x}", + "split": split, + "state": { + "rules": [{"id": RULE_ID, "text": "Put a visible label above every form field."}], + "imagesNote": "Image 1 is the candidate: route / at desktop width in light theme.", + }, + "images": [ + { + "role": "candidate", + "path": "images/screen.png", + "mediaType": "image/png", + "digest": image_digest, + "size": "desktop", + "theme": "light", + } + ], + "questions": { + RULE_ID: { + "type": "noul", + "instructions": "Does the candidate follow this team learning?", + "criteria": {"true": "It follows", "false": "It breaks"}, + } + }, + "labels": {RULE_ID: {"answer": answer}}, + "provenance": { + "exportId": EXPORT_ID, + "prototypeId": number, + "buildNumber": 1, + "route": "/", + }, + } + + +def upload_request(directory: Path) -> tuple[str, dict]: + loaded = load_taskset(directory) + manifest = json.loads((directory / "taskset.json").read_text(encoding="utf-8")) + tasks = [ + json.loads(line) + for line in (directory / "tasks.jsonl").read_text(encoding="utf-8").splitlines() + ] + return taskset_digest(loaded).value, { + "schemaVersion": "replica.decision-taskset-upload/v1", + "manifest": manifest, + "tasks": tasks, + "images": [ + { + "path": "images/screen.png", + "mediaType": "image/png", + "digest": "sha256:" + hashlib.sha256(PNG).hexdigest(), + "dataBase64": base64.b64encode(PNG).decode(), + } + ], + } + + +def evaluation_request() -> dict: + return { + "state": {"imagesNote": "Image 1 is the candidate."}, + "images": ["data:image/png;base64," + base64.b64encode(PNG).decode()], + "questions": { + RULE_ID: { + "type": "noul", + "instructions": "Does the candidate follow this team learning?", + } + }, + } + + +def wait_for_job(url: str, token: str, job_id: str) -> dict: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + job = call(url, token, "GET", f"/v1/jobs/{job_id}") + if job["state"] in ("succeeded", "failed", "cancelled"): + if job["state"] != "succeeded": + raise RuntimeError(f"Training job ended as {job['state']}.") + return job + time.sleep(0.05) + raise RuntimeError("Training job did not finish.") + + +def wait_for_deletion(url: str, token: str, deletion_id: str) -> dict: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + receipt = call(url, token, "GET", f"/v1/deletions/{deletion_id}") + if receipt["state"] == "completed": + return receipt + time.sleep(0.05) + raise RuntimeError("Deletion did not finish.") + + +def call( + url: str, + token: str, + method: str, + path: str, + body: dict | None = None, + *, + key: str | None = None, +) -> dict: + headers = {"Authorization": f"Bearer {token}"} + data = None + if body is not None: + headers["Content-Type"] = "application/json" + data = json.dumps(body).encode() + if key is not None: + headers["Idempotency-Key"] = key + with urllib.request.urlopen( + urllib.request.Request(url + path, data=data, headers=headers, method=method), + timeout=30, + ) as response: + return json.load(response) + + +def call_error( + url: str, token: str, method: str, path: str, body: dict +) -> tuple[int, str]: + try: + call(url, token, method, path, body) + except urllib.error.HTTPError as error: + with error: + response = json.load(error) + return error.code, response["error"]["code"] + raise RuntimeError("The request unexpectedly succeeded.") + + +if __name__ == "__main__": + main() diff --git a/packages/learning/environments/brand-check/examples/runtime_demo.py b/packages/learning/environments/brand-check/examples/runtime_demo.py index c57ff691..36b73c17 100644 --- a/packages/learning/environments/brand-check/examples/runtime_demo.py +++ b/packages/learning/environments/brand-check/examples/runtime_demo.py @@ -98,6 +98,22 @@ def write_taskset(directory: Path) -> None: "schemaVersion": "replica.decision-taskset/v1", "exportId": "export_fixture", "createdAt": "2026-11-10T00:00:00Z", + "provenance": { + "exportDigest": "sha256:" + "1" * 64, + "grantIds": ["grant_fixture_training"], + "purposes": ["training"], + "reviewDigests": ["sha256:" + "2" * 64], + "evidenceDigests": ["sha256:" + "3" * 64], + "evidenceDigest": "sha256:" + + hashlib.sha256( + json.dumps( + ["sha256:" + "3" * 64], + sort_keys=True, + separators=(",", ":"), + ).encode() + ).hexdigest(), + }, + "retention": {"mode": "delete_on_withdrawal"}, "counts": {"tasks": len(tasks)}, } (directory / "taskset.json").write_text(json.dumps(manifest), encoding="utf-8") diff --git a/packages/learning/environments/brand-check/local_connectors.py b/packages/learning/environments/brand-check/local_connectors.py index 3ef3e6e0..1f066d46 100644 --- a/packages/learning/environments/brand-check/local_connectors.py +++ b/packages/learning/environments/brand-check/local_connectors.py @@ -234,6 +234,16 @@ def __init__( os.chmod(self.root, 0o700) os.chmod(self.idempotency, 0o700) self._lock = threading.RLock() + for staging in self.root.glob(".register-*"): + try: + shutil.rmtree(staging) + except OSError as error: + raise ConnectorError( + "taskset_staging_unavailable", + "Staged taskset data could not be removed.", + category="unavailable", + retryable=True, + ) from error self.taskset = None self.digest = None self.reference = None @@ -241,7 +251,7 @@ def __init__( if taskset_directory is not None: self.taskset = load_taskset(taskset_directory) self.digest = taskset_digest(self.taskset) - self.reference = f"taskset:{self.digest.value}" + self.reference = f"mounted-taskset:{self.digest.value}" self._mounted[self.reference] = self.taskset def register( @@ -309,7 +319,7 @@ def register( else "The taskset contents do not match the claimed digest.", category="conflict" if immutable else "invalid_input", ) - reference = f"taskset:{actual.value}" + reference = f"uploaded-taskset:{actual.value}" destination = self.root / actual.value if destination.exists(): existing = self.resolve(reference, context) @@ -345,13 +355,22 @@ def register( category="invalid_input", ) from None finally: - shutil.rmtree(staging, ignore_errors=True) - return TasksetArtifact(reference, actual, load_taskset(destination)) + if staging.exists(): + try: + shutil.rmtree(staging) + except OSError as error: + raise ConnectorError( + "taskset_staging_unavailable", + "Staged taskset data could not be removed.", + category="unavailable", + retryable=True, + ) from error + return TasksetArtifact(reference, actual, load_taskset(destination), created=True) def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: self._check(context, hide=True) - if reference.startswith("taskset:"): - value = reference.removeprefix("taskset:") + if reference.startswith(("taskset:", "uploaded-taskset:")): + value = reference.split(":", 1)[1] directory = self.root / value try: digest = ContentDigest("sha256", value) @@ -373,8 +392,19 @@ def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: category="integrity", ) return TasksetArtifact(reference, digest, taskset) - mounted = self._mounted.get(reference) - if mounted is not None: + if reference.startswith("taskset:"): + mounted_reference = f"mounted-taskset:{value}" + mounted = self._mounted.get(mounted_reference) + if mounted is not None: + return self.resolve(mounted_reference, context) + if reference.startswith("mounted-taskset:"): + mounted = self._mounted.get(reference) + if mounted is None: + raise ConnectorError( + "taskset_not_found", + "The taskset was not found.", + category="not_found", + ) try: mounted = load_taskset(mounted.directory) except TasksetError: @@ -383,21 +413,14 @@ def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: "The taskset no longer matches its immutable identity.", category="integrity", ) from None - digest = taskset_digest(mounted) - if reference != f"taskset:{digest.value}": - raise ConnectorError( - "taskset_damaged", - "The taskset no longer matches its immutable identity.", - category="integrity", - ) - return TasksetArtifact(reference, digest, mounted) - if not reference.startswith("taskset:"): + return TasksetArtifact(reference, taskset_digest(mounted), mounted) + if not reference.startswith(("taskset:", "uploaded-taskset:")): raise ConnectorError( "taskset_not_found", "The taskset was not found.", category="not_found", ) - value = reference.removeprefix("taskset:") + value = reference.split(":", 1)[1] directory = self.root / value try: digest = ContentDigest("sha256", value) @@ -429,6 +452,68 @@ def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: ) return TasksetArtifact(reference, digest, taskset) + def delete(self, reference: str, context: OperationContext) -> bool: + """Delete one uploaded taskset; mounted tasksets remain operator-owned.""" + self._check(context, hide=True) + if reference.startswith("mounted-taskset:"): + raise ConnectorError( + "taskset_retention_required", + "A mounted taskset remains under operator-controlled retention.", + category="conflict", + ) + if reference.startswith("uploaded-taskset:"): + value = reference.removeprefix("uploaded-taskset:") + elif reference.startswith("taskset:"): + if reference in self._mounted: + raise ConnectorError( + "taskset_retention_required", + "A mounted taskset remains under operator-controlled retention.", + category="conflict", + ) + value = reference.removeprefix("taskset:") + else: + raise ConnectorError( + "taskset_not_found", + "The taskset was not found.", + category="not_found", + ) + try: + digest = ContentDigest("sha256", value) + except ValueError: + raise ConnectorError( + "taskset_not_found", + "The taskset was not found.", + category="not_found", + ) from None + directory = self.root / digest.value + with self._lock: + if not directory.exists(): + return False + try: + shutil.rmtree(directory) + _sync_directory(self.root) + except OSError as error: + raise ConnectorError( + "taskset_delete_unavailable", + "The taskset could not be deleted.", + category="unavailable", + retryable=True, + ) from error + return True + + def list(self, context: OperationContext) -> Sequence[TasksetArtifact]: + self._check(context) + records = [] + for directory in sorted(self.root.iterdir()): + if not directory.is_dir() or directory.name.startswith("."): + continue + records.append( + self.resolve(f"uploaded-taskset:{directory.name}", context) + ) + for reference in sorted(self._mounted): + records.append(self.resolve(reference, context)) + return tuple(records) + def _materialize_taskset_bundle( directory: Path, bundle: TasksetBundle, context: OperationContext @@ -990,6 +1075,16 @@ def __init__(self, root: Path, scope: RuntimeScope): os.chmod(directory, 0o700) os.chmod(self.promotion_owners.parent, 0o700) self._lock = threading.RLock() + for staging in self.candidates.glob(".candidate_*.*"): + try: + shutil.rmtree(staging) + except OSError as error: + raise ConnectorError( + "candidate_staging_unavailable", + "Staged candidate data could not be removed.", + category="unavailable", + retryable=True, + ) from error self._recover_promotion() def put(self, candidate: Candidate, context: OperationContext) -> Candidate: @@ -1059,7 +1154,15 @@ def put(self, candidate: Candidate, context: OperationContext) -> Candidate: _sync_directory(self.candidates) finally: if staging.exists(): - shutil.rmtree(staging) + try: + shutil.rmtree(staging) + except OSError as error: + raise ConnectorError( + "candidate_staging_unavailable", + "Staged candidate data could not be removed.", + category="unavailable", + retryable=True, + ) from error return candidate def get(self, candidate_id: str, context: OperationContext) -> Candidate: @@ -1343,6 +1446,34 @@ def current(self, context: OperationContext) -> Promotion | None: ) return promotion + def delete(self, candidate_id: str, context: OperationContext) -> bool: + """Delete one immutable candidate and its colocated evaluation report.""" + self._check(context, hide=True) + candidate_id = _checked_identifier(candidate_id, "candidate") + directory = self.candidates / candidate_id + with self._lock: + if not directory.exists(): + return False + try: + shutil.rmtree(directory) + _sync_directory(self.candidates) + except OSError as error: + raise ConnectorError( + "candidate_delete_unavailable", + "The candidate could not be deleted.", + category="unavailable", + retryable=True, + ) from error + return True + + def list(self, context: OperationContext) -> Sequence[Candidate]: + self._check(context) + return tuple( + self.get(directory.name, context) + for directory in sorted(self.candidates.iterdir()) + if directory.is_dir() and not directory.name.startswith(".") + ) + def artifact_path( self, candidate_id: str, @@ -1409,6 +1540,34 @@ def pending_promotions_json( pending.append(promotion) return tuple(sorted(pending, key=lambda item: item["generation"])) + def discard_pending_promotion( + self, + candidate_id: str, + context: OperationContext, + ) -> int: + self._check(context) + removed = 0 + with self._lock: + for path in self.promotion_idempotency.glob("*.json"): + saved = _read_object( + path, + code="promotion_idempotency_not_found", + message="The promotion retry was not found.", + ) + promotion = saved.get("promotion") + generation = saved.get("generation") + if ( + isinstance(promotion, dict) + and promotion.get("candidateId") == candidate_id + and isinstance(generation, int) + and not (self.promotions / f"{generation:08d}.json").exists() + ): + path.unlink() + removed += 1 + if removed: + _sync_directory(self.promotion_idempotency) + return removed + def promotion_retry_json( self, context: OperationContext, @@ -1977,10 +2136,87 @@ def predict( saved_result = dict(_json_data(result)) _write_immutable( path, - _json_bytes({"requestDigest": request_digest, "result": saved_result}), + _json_bytes( + { + "requestDigest": request_digest, + "candidateId": adapter.parent.name, + "createdAt": _timestamp(), + "result": saved_result, + } + ), ) return saved_result + def reference(self, idempotency_key: str) -> str: + return f"inference:{self.name}:{hashlib.sha256(idempotency_key.encode()).hexdigest()}" + + def delete(self, reference: str, context: OperationContext) -> bool: + check_context(context) + prefix = f"inference:{self.name}:" + if not reference.startswith(prefix): + raise ConnectorError( + "inference_result_not_found", + "The inference result was not found.", + category="not_found", + ) + digest = reference.removeprefix(prefix) + if not re.fullmatch(r"[0-9a-f]{64}", digest): + raise ConnectorError( + "inference_result_not_found", + "The inference result was not found.", + category="not_found", + ) + path = self.idempotency / digest + try: + path.unlink() + _sync_directory(self.idempotency) + except FileNotFoundError: + return False + except OSError as error: + raise ConnectorError( + "inference_result_delete_unavailable", + "The inference retry result could not be deleted.", + category="unavailable", + retryable=True, + ) from error + return True + + def list_results( + self, context: OperationContext + ) -> Sequence[Mapping[str, Any]]: + check_context(context) + records = [] + for path in sorted(self.idempotency.iterdir()): + if not path.is_file(): + continue + saved = _read_object( + path, + code="inference_idempotency_not_found", + message="The inference retry record was not found.", + ) + candidate_id = saved.get("candidateId") + created_at = saved.get("createdAt") + if not isinstance(candidate_id, str) or not isinstance(created_at, str): + try: + path.unlink() + _sync_directory(self.idempotency) + except OSError as error: + raise ConnectorError( + "inference_provenance_cleanup_unavailable", + "A legacy inference retry result could not be removed.", + category="unavailable", + retryable=True, + ) from error + continue + records.append( + { + "reference": f"inference:{self.name}:{path.name}", + "candidateId": candidate_id, + "createdAt": created_at, + } + ) + return tuple(records) + class LocalCredentialConnector(_ScopedConnector): """Mint operation-bound local credentials from a private installation key.""" @@ -2032,6 +2268,345 @@ def acquire( ) +class LocalPrivacyConnector(_ScopedConnector): + """Keep content-free copy inventory, revocations, and deletion receipts.""" + + def __init__(self, root: Path, scope: RuntimeScope): + super().__init__(scope) + self.root = root / "privacy" + self.copies = self.root / "copies" + self.deletions = self.root / "deletions" + self.idempotency = self.root / ".idempotency" + for directory in (self.root, self.copies, self.deletions, self.idempotency): + directory.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(directory, 0o700) + self._lock = threading.RLock() + + def put_copy( + self, record: Mapping[str, Any], context: OperationContext + ) -> Mapping[str, Any]: + self._check(context) + saved = self._validated_copy(record) + path = self._copy_path(saved["id"]) + with self._lock: + if path.exists(): + existing = self._validated_copy( + _read_object( + path, + code="runtime_copy_not_found", + message="The runtime copy was not found.", + ) + ) + if existing != saved: + raise ConnectorError( + "runtime_copy_immutable", + "The runtime copy ID already identifies different metadata.", + category="conflict", + ) + return existing + _write_immutable(path, _json_bytes(saved)) + return saved + + def get_copy( + self, copy_id: str, context: OperationContext + ) -> Mapping[str, Any]: + self._check(context, hide=True) + return self._validated_copy( + _read_object( + self._copy_path(copy_id), + code="runtime_copy_not_found", + message="The runtime copy was not found.", + ) + ) + + def list_copies( + self, + selectors: Mapping[str, str], + *, + cursor: str | None, + limit: int, + context: OperationContext, + ) -> tuple[Sequence[Mapping[str, Any]], str | None]: + self._check(context) + if isinstance(limit, bool) or not isinstance(limit, int) or not 1 <= limit <= 100: + raise ConnectorError( + "pagination_invalid", + "Copy pagination limit must be from 1 to 100.", + category="invalid_input", + ) + if cursor is not None: + _checked_identifier(cursor, "runtime copy cursor") + allowed = { + "exportId", + "grantId", + "evidenceDigest", + "tasksetDigest", + } + if not selectors or set(selectors) - allowed or any( + not isinstance(value, str) or not value for value in selectors.values() + ): + raise ConnectorError( + "disclosure_selector_invalid", + "Disclosure needs one or more supported exact selectors.", + category="invalid_input", + ) + selected = [] + for path in sorted(self.copies.glob("*.json")): + record = self._validated_copy( + _read_object( + path, + code="runtime_copy_not_found", + message="The runtime copy was not found.", + ) + ) + if cursor is not None and record["id"] <= cursor: + continue + if _copy_matches(record, selectors): + selected.append(record) + if len(selected) > limit: + break + next_cursor = selected[limit - 1]["id"] if len(selected) > limit else None + return tuple(selected[:limit]), next_cursor + + def tombstone_copy( + self, + copy_id: str, + *, + state: str, + retention: Mapping[str, Any] | None, + context: OperationContext, + ) -> Mapping[str, Any]: + self._check(context) + if state not in ("registering", "active", "revoked", "deleted", "retained"): + raise ConnectorError( + "runtime_copy_state_invalid", + "A runtime copy state must be registering, active, revoked, deleted, or retained.", + category="invalid_input", + ) + path = self._copy_path(copy_id) + with self._lock: + current = self._validated_copy( + _read_object( + path, + code="runtime_copy_not_found", + message="The runtime copy was not found.", + ) + ) + if current["useState"] == state and ( + retention is None or current.get("retention") == _json_data(retention) + ): + return current + saved = dict(current) + saved["useState"] = state + saved["state"] = state + saved["updatedAt"] = _timestamp() + if retention is not None: + saved["retention"] = _json_data(retention) + _write_json_atomic(path, saved) + return self._validated_copy(saved) + + def create_deletion( + self, request: Mapping[str, Any], context: OperationContext + ) -> Mapping[str, Any]: + self._check(context) + request_data = _json_data(request) + request_digest = str(_digest(_json_bytes(request_data))) + retry_path = self.idempotency / ( + hashlib.sha256(context.idempotency_key.encode()).hexdigest() + ".json" + ) + with self._lock: + if retry_path.exists(): + retry = _read_object( + retry_path, + code="deletion_retry_not_found", + message="The deletion retry record was not found.", + ) + if retry.get("requestDigest") != request_digest: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + return self.get_deletion(retry.get("deletionId", ""), context) + for path in sorted(self.deletions.glob("*.json")): + existing = self.get_deletion(path.stem, context) + if ( + existing.get("selectors") == request_data.get("selectors") + and existing.get("requestedBy") == request_data.get("requestedBy") + and existing.get("reason") == request_data.get("reason") + and existing.get("retention") == request_data.get("retention") + ): + _write_immutable( + retry_path, + _json_bytes( + { + "deletionId": existing["id"], + "requestDigest": request_digest, + } + ), + ) + return existing + deletion_id = "deletion_" + hashlib.sha256( + ( + _scope_key(context.scope) + + "\0" + + context.idempotency_key + + "\0" + + request_digest + ).encode() + ).hexdigest()[:32] + receipt = { + "schemaVersion": "replica.runtime-deletion-receipt/v1", + "id": deletion_id, + "scope": { + "tenantId": context.scope.tenant_id, + "repository": context.scope.repository, + }, + "selectors": request_data["selectors"], + "requestedBy": request_data["requestedBy"], + "reason": request_data["reason"], + "state": "revoking", + "createdAt": _timestamp(), + "updatedAt": _timestamp(), + "items": [], + "pendingItems": [], + "failures": [], + "version": 1, + } + if request_data.get("retention") is not None: + receipt["retention"] = request_data["retention"] + _write_immutable(self._deletion_path(deletion_id), _json_bytes(receipt)) + _write_immutable( + retry_path, + _json_bytes( + { + "deletionId": deletion_id, + "requestDigest": request_digest, + } + ), + ) + return receipt + + def get_deletion( + self, deletion_id: str, context: OperationContext + ) -> Mapping[str, Any]: + self._check(context, hide=True) + saved = _read_object( + self._deletion_path(deletion_id), + code="deletion_not_found", + message="The deletion receipt was not found.", + ) + if ( + saved.get("schemaVersion") != "replica.runtime-deletion-receipt/v1" + or saved.get("id") != deletion_id + or not isinstance(saved.get("version"), int) + ): + raise ConnectorError( + "deletion_receipt_damaged", + "The deletion receipt is invalid.", + category="integrity", + ) + return saved + + def list_deletions( + self, states: Sequence[str], context: OperationContext + ) -> Sequence[Mapping[str, Any]]: + check_context(context) + if context.scope != self.scope: + return () + selected = set(states) + return tuple( + receipt + for path in sorted(self.deletions.glob("*.json")) + if (receipt := self.get_deletion(path.stem, context)).get("state") + in selected + ) + + def update_deletion( + self, + deletion_id: str, + record: Mapping[str, Any], + *, + expected_version: int, + context: OperationContext, + ) -> Mapping[str, Any]: + self._check(context) + with self._lock: + current = dict(self.get_deletion(deletion_id, context)) + if current["version"] != expected_version: + raise ConnectorError( + "deletion_version_conflict", + "The deletion receipt changed before this update.", + category="conflict", + ) + saved = dict(_json_data(record), version=expected_version + 1) + if ( + saved.get("id") != deletion_id + or saved.get("schemaVersion") + != "replica.runtime-deletion-receipt/v1" + ): + raise ConnectorError( + "deletion_receipt_invalid", + "The deletion update must preserve the receipt identity.", + category="invalid_input", + ) + _write_json_atomic(self._deletion_path(deletion_id), saved) + return saved + + def _validated_copy(self, value: Mapping[str, Any]) -> dict[str, Any]: + saved = dict(_json_data(value)) + if ( + saved.get("schemaVersion") != "replica.runtime-copy/v1" + or not isinstance(saved.get("id"), str) + or saved.get("kind") not in ("taskset", "candidate", "report", "inference") + or saved.get("useState") + not in ("registering", "active", "revoked", "deleted", "retained") + or not isinstance(saved.get("sourceTrainingExportId"), str) + or not isinstance(saved.get("tasksetDigest"), str) + or not isinstance(saved.get("storage"), dict) + or saved.get("scope") + != { + "tenantId": self.scope.tenant_id, + "repository": self.scope.repository, + } + or not isinstance(saved.get("permittingGrantIds"), list) + or not isinstance(saved.get("permittedPurposes"), list) + or not isinstance(saved.get("reviewDigests"), list) + or not isinstance(saved.get("evidenceDigests"), list) + or not isinstance(saved.get("evidenceDigest"), str) + or not isinstance(saved.get("retention"), dict) + or not isinstance(saved.get("createdAt"), str) + ): + raise ConnectorError( + "runtime_copy_invalid", + "The runtime copy metadata is invalid.", + category="invalid_input", + ) + _checked_identifier(saved["id"], "runtime copy") + return saved + + def _copy_path(self, copy_id: str) -> Path: + return self.copies / f"{_checked_identifier(copy_id, 'runtime copy')}.json" + + def _deletion_path(self, deletion_id: str) -> Path: + return self.deletions / f"{_checked_identifier(deletion_id, 'deletion')}.json" + + +def _copy_matches(record: Mapping[str, Any], selectors: Mapping[str, str]) -> bool: + values = { + "exportId": record.get("sourceTrainingExportId"), + "grantId": record.get("permittingGrantIds", ()), + "evidenceDigest": record.get("evidenceDigest"), + "tasksetDigest": record.get("tasksetDigest"), + } + return all( + selector in values[name] + if isinstance(values[name], (list, tuple)) + else values[name] == selector + for name, selector in selectors.items() + ) + + class LocalObservabilityConnector(_ScopedConnector): """Append scoped usage and cost records to durable private JSON Lines.""" @@ -2129,6 +2704,7 @@ class LocalConnectorBundle: taskset_digest: ContentDigest | None tasksets: LocalTasksetConnector candidates: LocalCandidateConnector + privacy: LocalPrivacyConnector observability: LocalObservabilityConnector _owner: Any @@ -2198,6 +2774,7 @@ def create_local_connector_bundle( artifacts = LocalArtifactConnector(root, scope) jobs = LocalJobConnector(root, scope) candidates = LocalCandidateConnector(root, scope) + privacy = LocalPrivacyConnector(root, scope) training = { name: LocalTrainingConnector(backend) for name, backend in backends.items() @@ -2213,6 +2790,7 @@ def create_local_connector_bundle( artifacts=artifacts, jobs=jobs, candidates=candidates, + privacy=privacy, training=training, inference=inference, credentials=credentials or local_credentials, @@ -2226,6 +2804,7 @@ def create_local_connector_bundle( tasksets.digest, tasksets, candidates, + privacy, local_observability, owner, ) diff --git a/packages/learning/environments/brand-check/runtime.py b/packages/learning/environments/brand-check/runtime.py index 83417e47..149dabb6 100644 --- a/packages/learning/environments/brand-check/runtime.py +++ b/packages/learning/environments/brand-check/runtime.py @@ -62,6 +62,9 @@ CANDIDATE_REPORT_SCHEMA = "replica.decision-candidate-evaluation/v1" INFERENCE_SCHEMA = "replica.decision-evaluation-response/v1" TASKSET_UPLOAD_SCHEMA = "replica.decision-taskset-upload/v1" +DISCLOSURE_SCHEMA = "replica.runtime-copy-disclosure/v1" +DELETION_RECEIPT_SCHEMA = "replica.runtime-deletion-receipt/v1" +RUNTIME_COPY_SCHEMA = "replica.runtime-copy/v1" MIN_UPDATES = 1 MAX_UPDATES = 100_000 MIN_SECONDS = 1 @@ -226,6 +229,12 @@ def __init__( } self._lock = threading.RLock() self._job_cancellations: dict[str, Cancellation] = {} + self._inference_cancellations: dict[ + str, tuple[Cancellation, Mapping[str, Any]] + ] = {} + self._inference_idempotency_locks: dict[ + str, tuple[threading.Lock, int] + ] = {} self._tasksets: OrderedDict[str, TasksetArtifact] = OrderedDict() self._verified_taskset_directories: dict[str, Path] = {} try: @@ -242,14 +251,31 @@ def __init__( ) self.taskset = resolved.taskset self.taskset_digest = str(resolved.digest) + self._record_taskset_copy( + resolved, + self._operation_context( + f"mounted_taskset_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ), + mounted=True, + ) self._executor = ThreadPoolExecutor( max_workers=1, thread_name_prefix="replica-training" ) work = self.state_directory / "work" work.mkdir(parents=True, exist_ok=True, mode=0o700) os.chmod(work, 0o700) + for stale in work.iterdir(): + if stale.is_dir(): + shutil.rmtree(stale) + else: + stale.unlink() + self._backfill_local_privacy_inventory() + self._reconcile_completed_deletions() self._recover_pending_promotions() self._recover_interrupted_jobs() + self._recover_deletions() except Exception: if hasattr(self, "_executor"): self._executor.shutdown(wait=False, cancel_futures=True) @@ -296,45 +322,73 @@ def submit( "unmetered_local training is available only in development and test.", 409, ) - selected_taskset = _taskset_digest_value( - taskset_digest if taskset_digest is not None else self.taskset_digest - ) - self._resolve_taskset(selected_taskset) - submission_key = idempotency_key or f"submit_{uuid.uuid4().hex}" - job_id = f"job_{hashlib.sha256(submission_key.encode()).hexdigest()[:32]}" - job = { - "schemaVersion": JOB_SCHEMA, - "id": job_id, - "state": "queued", - "backend": backend, - "tasksetDigest": selected_taskset, - "limits": { - "maxUpdates": max_updates, - "maxSeconds": float(max_seconds), - "maxCredits": float(max_credits), - "budgetMode": budget_mode, - }, - "createdAt": _now(), - "updatedAt": _now(), - } - context = self._operation_context( - job_id, - max_seconds=max_seconds, - owner=RuntimeOwner("service", "decision-runtime"), - max_credits=max_credits, - budget_mode=budget_mode, - idempotency_key=submission_key, - ) with self._lock: - existing = False + selected_taskset = _taskset_digest_value( + taskset_digest if taskset_digest is not None else self.taskset_digest + ) + try: + privacy_context = self._operation_context( + f"taskset_privacy_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + for copy_id in _taskset_runtime_copy_ids(selected_taskset): + try: + copy = self.connectors.privacy.get_copy( + copy_id, + privacy_context, + ) + except ConnectorError as error: + if error.category == "not_found": + continue + raise + self._assert_not_revoked( + _copy_provenance(copy), operation="training" + ) + except ConnectorError as error: + if error.category != "not_found": + raise _runtime_connector_failure(error) from None + taskset = self._resolve_taskset(selected_taskset) + provenance = _taskset_provenance(taskset) + self._assert_not_revoked(provenance, operation="training") + if "training" not in provenance["purposes"]: + raise RuntimeFailure( + "training_not_permitted", + "The taskset provenance does not permit training.", + 409, + ) + submission_key = idempotency_key or f"submit_{uuid.uuid4().hex}" + job_id = f"job_{hashlib.sha256(submission_key.encode()).hexdigest()[:32]}" + job = { + "schemaVersion": JOB_SCHEMA, + "id": job_id, + "state": "queued", + "backend": backend, + "tasksetDigest": selected_taskset, + "limits": { + "maxUpdates": max_updates, + "maxSeconds": float(max_seconds), + "maxCredits": float(max_credits), + "budgetMode": budget_mode, + }, + "createdAt": _now(), + "updatedAt": _now(), + } + context = self._operation_context( + job_id, + max_seconds=max_seconds, + owner=RuntimeOwner("service", "decision-runtime"), + max_credits=max_credits, + budget_mode=budget_mode, + idempotency_key=submission_key, + ) try: try: stored = dict(self.connectors.jobs.get(job_id, context)) - existing = True except ConnectorError as error: if error.category != "not_found": raise - if existing: + else: if ( stored.get("backend") != job["backend"] or stored.get("tasksetDigest") != job["tasksetDigest"] @@ -369,21 +423,37 @@ def register_taskset( owner=RuntimeOwner("service", "decision-runtime"), idempotency_key=idempotency_key, ) + registered = None try: - registered = self.connectors.tasksets.register(bundle, context) - self._cache_taskset(registered) - self._record_usage( - context, - connector=type(self.connectors.tasksets).__name__, - kind="taskset_bytes", - quantity=sum(len(image.data) for image in bundle.images), - unit="bytes", - metadata={ - "taskCount": len(bundle.tasks), - "imageCount": len(bundle.images), - }, - ) + with self._lock: + self._assert_not_revoked( + _manifest_provenance(bundle.manifest, str(claimed)), + operation="taskset upload", + ) + registered = self.connectors.tasksets.register(bundle, context) + self._record_taskset_copy(registered, context, mounted=False) + self._cache_taskset(registered) + try: + self._record_usage( + context, + connector=type(self.connectors.tasksets).__name__, + kind="taskset_bytes", + quantity=sum(len(image.data) for image in bundle.images), + unit="bytes", + metadata={ + "taskCount": len(bundle.tasks), + "imageCount": len(bundle.images), + }, + ) + except ConnectorError: + pass except ConnectorError as error: + if registered is not None and registered.created: + try: + self.connectors.tasksets.delete(registered.reference, context) + self._evict_taskset(str(registered.digest)) + except ConnectorError: + pass raise _runtime_connector_failure(error) from None return { "schemaVersion": TASKSET_UPLOAD_SCHEMA, @@ -392,6 +462,486 @@ def register_taskset( "tasks": len(registered.taskset.tasks), } + def disclose( + self, + selectors: Mapping[str, str], + *, + limit: int = 50, + cursor: str | None = None, + ) -> dict[str, Any]: + """Return bounded, content-free metadata for matching runtime copies.""" + selected = _disclosure_selectors(selectors) + context = self._operation_context( + f"disclosure_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + copies, next_cursor = self.connectors.privacy.list_copies( + selected, + cursor=cursor, + limit=limit, + context=context, + ) + self._record_usage( + context, + connector=type(self.connectors.privacy).__name__, + kind="privacy_disclosures", + quantity=1, + unit="requests", + metadata={"copies": len(copies), "paginated": next_cursor is not None}, + ) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + return { + "schemaVersion": DISCLOSURE_SCHEMA, + "scope": { + "tenantId": self.scope.tenant_id, + "repository": self.scope.repository, + }, + "items": [self._disclosed_copy(copy) for copy in copies], + "nextCursor": next_cursor, + } + + def request_deletion( + self, + selectors: Mapping[str, str], + *, + requested_by: str, + reason: str, + idempotency_key: str, + retention: Mapping[str, Any] | None = None, + ) -> dict[str, Any]: + """Revoke matching runtime use, remove copies, and return a durable receipt.""" + selected = _disclosure_selectors(selectors) + if not isinstance(requested_by, str) or not requested_by.strip(): + raise RuntimeFailure("invalid_input", "requestedBy must name the operator.") + if not isinstance(reason, str) or not reason.strip(): + raise RuntimeFailure("invalid_input", "reason must explain the withdrawal.") + retained = _retention_request(retention) + context = self._operation_context( + f"deletion_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("human", requested_by.strip()), + idempotency_key=idempotency_key, + ) + request = { + "selectors": selected, + "requestedBy": requested_by.strip(), + "reason": reason.strip(), + "retention": retained, + } + try: + with self._lock: + receipt = dict(self.connectors.privacy.create_deletion(request, context)) + if receipt["state"] != "completed": + receipt = self._process_deletion(receipt, context) + try: + self._record_usage( + context, + connector=type(self.connectors.privacy).__name__, + kind="privacy_deletions", + quantity=1, + unit="requests", + metadata={ + "state": receipt["state"], + "items": len(receipt.get("items", ())), + "pending": len(receipt.get("pendingItems", ())), + }, + ) + except ConnectorError: + pass + return receipt + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + + def deletion(self, deletion_id: str) -> dict[str, Any]: + """Return one durable content-free deletion receipt.""" + context = self._operation_context( + f"deletion_read_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + receipt = dict(self.connectors.privacy.get_deletion(deletion_id, context)) + if receipt["state"] != "completed": + try: + return self._process_deletion(receipt, context) + except ConnectorError as error: + if error.category != "conflict": + raise + return dict( + self.connectors.privacy.get_deletion(deletion_id, context) + ) + return receipt + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + + def _process_deletion( + self, receipt: dict[str, Any], context: OperationContext + ) -> dict[str, Any]: + copies = self._matching_copies(receipt["selectors"], context) + taskset_digests = {copy["tasksetDigest"] for copy in copies} + pending: list[dict[str, Any]] = [] + failures: list[dict[str, Any]] = [] + items: list[dict[str, Any]] = [] + + for copy in copies: + try: + self.connectors.privacy.tombstone_copy( + copy["id"], + state="revoked", + retention=None, + context=context, + ) + except ConnectorError as error: + pending.append(_pending_item(copy, error)) + + for job in self.connectors.jobs.list(("queued", "running"), context): + if job.get("tasksetDigest") not in taskset_digests: + continue + try: + cancelled = self.connectors.jobs.cancel( + job["id"], + expected_version=job["version"], + context=self._operation_context( + f"delete_cancel_{job['id']}", + max_seconds=30, + owner=context.owner, + idempotency_key=f"{receipt['id']}:{job['id']}", + ), + ) + signal = self._job_cancellations.get(job["id"]) + if signal is not None: + signal.cancel() + items.append( + { + "kind": "job", + "runtimeCopyId": job["id"], + "state": cancelled["state"], + } + ) + except ConnectorError as error: + if error.category == "conflict": + current = self._job_record(job["id"]) + if current.get("state") in ("cancelled", "failed", "succeeded"): + continue + pending.append( + { + "kind": "job", + "runtimeCopyId": job["id"], + "code": error.code, + "retryable": error.retryable or error.category in ("conflict", "unavailable"), + } + ) + + for operation_id, (signal, provenance) in tuple(self._inference_cancellations.items()): + if _selectors_match_provenance(receipt["selectors"], provenance): + signal.cancel() + pending.append( + { + "kind": "inference", + "runtimeCopyId": operation_id, + "code": "operation_cancelling", + "retryable": True, + } + ) + + for job_id, signal in tuple(self._job_cancellations.items()): + job = self._job_record(job_id) + if job.get("tasksetDigest") in taskset_digests: + signal.cancel() + pending.append( + { + "kind": "job", + "runtimeCopyId": job_id, + "code": "operation_cancelling", + "retryable": True, + } + ) + + retention = receipt.get("retention") + operations_pending = any( + item["code"] == "operation_cancelling" for item in pending + ) + deleted_candidate_references: set[str] = set() + for copy in sorted( + copies, + key=lambda item: ( + {"taskset": 0, "candidate": 1, "report": 2, "inference": 3}.get( + item["kind"], 4 + ), + item["id"], + ), + ): + if operations_pending: + continue + if any(item["runtimeCopyId"] == copy["id"] for item in pending): + continue + try: + effective_retention = retention + if ( + effective_retention is None + and copy["kind"] == "taskset" + and copy["storage"]["reference"].startswith("mounted-taskset:") + ): + effective_retention = { + **copy["retention"], + "legalReason": ( + "The mounted source remains under operator-controlled retention." + ), + "reason": ( + "The mounted source remains under operator-controlled retention." + ), + "authority": "runtime configuration", + } + elif ( + effective_retention is None + and copy.get("retention", {}).get("mode") + == "superseded_registration" + ): + effective_retention = copy["retention"] + if effective_retention is not None: + updated = self.connectors.privacy.tombstone_copy( + copy["id"], + state="retained", + retention=effective_retention, + context=context, + ) + state = updated["useState"] + else: + storage = copy["storage"] + if copy["kind"] == "taskset": + self.connectors.tasksets.delete(storage["reference"], context) + self._evict_taskset(copy["tasksetDigest"]) + elif copy["kind"] in ("candidate", "report"): + if storage["reference"] not in deleted_candidate_references: + self.connectors.candidates.delete(storage["reference"], context) + deleted_candidate_references.add(storage["reference"]) + elif copy["kind"] == "inference": + connector = self.inference_connectors.get(storage["connector"]) + if connector is None: + raise ConnectorError( + "inference_connector_unavailable", + "The inference connector is unavailable.", + category="unavailable", + retryable=True, + ) + connector.delete(storage["reference"], context) + else: + raise ConnectorError( + "runtime_copy_kind_unsupported", + "The runtime copy kind cannot be deleted.", + category="invalid_input", + ) + self.connectors.privacy.tombstone_copy( + copy["id"], + state="deleted", + retention={"mode": "deleted", "deletedAt": _now()}, + context=context, + ) + state = "deleted" + items.append( + { + "kind": copy["kind"], + "runtimeCopyId": copy["id"], + "storage": { + "connector": copy["storage"]["connector"], + "reference": copy["storage"]["reference"], + }, + "state": state, + **( + {"retention": updated["retention"]} + if state == "retained" + else {} + ), + } + ) + except ConnectorError as error: + target = _pending_item(copy, error) + if error.category in ("invalid_input", "integrity", "unauthorized"): + failures.append(target) + else: + pending.append(target) + + updated = dict(receipt) + updated["items"] = sorted(items, key=lambda item: (item["kind"], item["runtimeCopyId"])) + updated["pendingItems"] = sorted( + pending, key=lambda item: (item["kind"], item["runtimeCopyId"]) + ) + updated["failures"] = sorted( + failures, key=lambda item: (item["kind"], item["runtimeCopyId"]) + ) + updated["updatedAt"] = _now() + updated.pop("pending", None) + updated["state"] = ( + "completed" + if not updated["pendingItems"] and not updated["failures"] + else "pending" + ) + updated.setdefault("revokedAt", updated["updatedAt"]) + if updated["state"] == "completed": + updated["completedAt"] = updated["updatedAt"] + return dict( + self.connectors.privacy.update_deletion( + receipt["id"], + updated, + expected_version=receipt["version"], + context=context, + ) + ) + + def _matching_copies( + self, selectors: Mapping[str, str], context: OperationContext + ) -> list[dict[str, Any]]: + cursor = None + copies: list[dict[str, Any]] = [] + while True: + page, cursor = self.connectors.privacy.list_copies( + selectors, + cursor=cursor, + limit=100, + context=context, + ) + copies.extend(dict(copy) for copy in page) + if cursor is None: + return copies + + def _record_taskset_copy( + self, + taskset: TasksetArtifact, + context: OperationContext, + *, + mounted: bool, + ) -> None: + provenance = _taskset_provenance(taskset) + withdrawn = False + if mounted: + receipts = self.connectors.privacy.list_deletions( + ("revoking", "pending", "completed"), + context, + ) + withdrawn = any( + _selectors_match_provenance(receipt["selectors"], provenance) + for receipt in receipts + ) + identity = ( + f"mounted:{taskset.reference}" + if mounted + else f"uploaded:{taskset.reference}" + ) + record = _runtime_copy( + "taskset", + identity, + taskset.reference, + provenance, + scope=self.scope, + connector=type(self.connectors.tasksets).__name__, + created_at=taskset.taskset.metadata.get("createdAt"), + retention={ + **dict(provenance.get("retention", {})), + "mode": ( + "operator_managed" + if mounted + else "delete_on_withdrawal" + ), + **( + { + "reason": ( + "The mounted source remains under operator-controlled retention " + "after training consent withdrawal." + ), + "legalReason": ( + "The mounted source remains under operator-controlled retention " + "after training consent withdrawal." + ), + "authority": "runtime configuration", + } + if withdrawn + else {} + ), + }, + use_state="retained" if withdrawn else "active", + ) + try: + existing = self.connectors.privacy.get_copy(record["id"], context) + except ConnectorError as error: + if error.category != "not_found": + raise + self.connectors.privacy.put_copy(record, context) + else: + if mounted and ( + existing.get("retention", {}).get("mode") != "operator_managed" + or withdrawn + ): + self.connectors.privacy.tombstone_copy( + record["id"], + state="retained" if withdrawn else existing["useState"], + retention=record["retention"], + context=context, + ) + if not mounted: + legacy_copy_id = _runtime_copy_id( + "taskset", + f"uploaded:taskset:{taskset.digest.value}", + ) + if legacy_copy_id != record["id"]: + try: + self.connectors.privacy.get_copy(legacy_copy_id, context) + except ConnectorError as error: + if error.category != "not_found": + raise + else: + self.connectors.privacy.tombstone_copy( + legacy_copy_id, + state="retained", + retention={ + "mode": "superseded_registration", + "reason": ( + "A verified taskset registration supersedes this legacy " + "registration record." + ), + "authority": "runtime migration", + }, + context=context, + ) + + def _disclosed_copy(self, copy: Mapping[str, Any]) -> dict[str, Any]: + safe = dict(copy) + safe["reviewDigest"] = (safe.get("reviewDigests") or [None])[0] + safe["evidenceDigest"] = safe.get("evidenceDigest") + if safe["kind"] == "candidate": + promotion = self.promotion(required=False) + safe["promotionState"] = ( + "promoted" + if promotion is not None + and promotion.get("candidateId") == safe["storage"]["reference"] + and safe["useState"] == "active" + else safe.get("promotionState", "eligible") + ) + return safe + + def _assert_not_revoked( + self, provenance: Mapping[str, Any], *, operation: str + ) -> None: + context = self._operation_context( + f"revocation_check_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + receipts = self.connectors.privacy.list_deletions( + ("revoking", "pending", "completed"), context + ) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + if any(_selectors_match_provenance(item["selectors"], provenance) for item in receipts): + raise RuntimeFailure( + "deletion_revoked", + f"Training consent is withdrawn; {operation} is not allowed.", + 409, + ) + def cancel(self, job_id: str, *, idempotency_key: str) -> dict[str, Any]: """Cooperatively cancel one queued or running job.""" identifier = _checked_id(job_id, "job_") @@ -676,24 +1226,131 @@ def evaluate( ) -> dict[str, Any]: """Answer one prototype page with the promoted candidate.""" _check_request(request) - promotion = self.promotion() - candidate_id = promotion["candidateId"] - manifest = self._promotable_candidate(candidate_id) - connector = self.inference_connectors.get(manifest["backend"]) - if connector is None: - raise RuntimeFailure( - "backend_unavailable", - f"Promoted candidate backend {manifest['backend']} is not configured.", - 503, + retry_lock = None + retry_reference = None + retry_copy = None + inference_succeeded = False + with self._lock: + promotion = self.promotion() + candidate_id = promotion["candidateId"] + manifest = self._promotable_candidate(candidate_id) + connector = self.inference_connectors.get(manifest["backend"]) + if connector is None: + raise RuntimeFailure( + "backend_unavailable", + f"Promoted candidate backend {manifest['backend']} is not configured.", + 503, + ) + privacy_context = self._operation_context( + f"inference_copy_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), ) - context = self._operation_context( - f"inference_{uuid.uuid4().hex}", - max_seconds=timeout_seconds, - owner=RuntimeOwner("service", "decision-runtime"), - idempotency_key=idempotency_key, - ) + try: + copy = self.connectors.privacy.get_copy( + _runtime_copy_id("candidate", candidate_id), + privacy_context, + ) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + provenance = _copy_provenance(copy) + if idempotency_key is not None: + lock, users = self._inference_idempotency_locks.get( + idempotency_key, + (threading.Lock(), 0), + ) + self._inference_idempotency_locks[idempotency_key] = ( + lock, + users + 1, + ) + retry_lock = lock + retry_reference = getattr(connector, "reference", lambda _key: None)( + idempotency_key + ) + if retry_lock is not None: + retry_lock.acquire() + try: + self._assert_not_revoked(provenance, operation="inference") + except Exception: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) + raise + try: + with self._lock: + self._assert_not_revoked(provenance, operation="inference") + if isinstance(retry_reference, str): + retry_copy = _runtime_copy( + "inference", + retry_reference, + retry_reference, + provenance, + scope=self.scope, + connector=manifest["backend"], + created_at=_now(), + retention={"mode": "delete_on_withdrawal"}, + parent_copy_ids=( + _runtime_copy_id("candidate", candidate_id), + ), + use_state="registering", + ) + try: + self.connectors.privacy.get_copy( + retry_copy["id"], privacy_context + ) + except ConnectorError as error: + if error.category != "not_found": + raise + self.connectors.privacy.put_copy(retry_copy, privacy_context) + cancellation = Cancellation() + context = self._operation_context( + f"inference_{uuid.uuid4().hex}", + max_seconds=timeout_seconds, + owner=RuntimeOwner("service", "decision-runtime"), + cancellation=cancellation, + ) + self._inference_cancellations[context.operation_id] = ( + cancellation, + provenance, + ) + if idempotency_key is not None: + context = OperationContext( + operation_id=context.operation_id, + idempotency_key=idempotency_key, + scope=context.scope, + owner=context.owner, + deadline=context.deadline, + cancellation=context.cancellation, + ) + except Exception: + if retry_lock is not None: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) + raise work = self.state_directory / "work" / context.operation_id - work.mkdir(mode=0o700) + try: + work.mkdir(mode=0o700) + except OSError: + with self._lock: + self._inference_cancellations.pop(context.operation_id, None) + self._cleanup_inference_retry_reservation( + connector, + retry_reference, + retry_copy, + ) + if retry_lock is not None: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) + raise RuntimeFailure( + "work_directory_unavailable", + "The inference work directory could not be created.", + 500, + ) from None try: adapter = self.local_connectors.candidates.artifact_path( candidate_id, @@ -701,6 +1358,19 @@ def evaluate( context, ) except ConnectorError as error: + shutil.rmtree(work, ignore_errors=True) + with self._lock: + self._inference_cancellations.pop(context.operation_id, None) + self._cleanup_inference_retry_reservation( + connector, + retry_reference, + retry_copy, + ) + if retry_lock is not None: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) raise _runtime_connector_failure(error) from None started = time.monotonic() operation_error: Exception | None = None @@ -708,6 +1378,18 @@ def evaluate( acquired = capacity is None or capacity.acquire(blocking=False) if not acquired: shutil.rmtree(work, ignore_errors=True) + with self._lock: + self._inference_cancellations.pop(context.operation_id, None) + self._cleanup_inference_retry_reservation( + connector, + retry_reference, + retry_copy, + ) + if retry_lock is not None: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) raise RuntimeFailure( "capacity_temporarily_unavailable", f"Promoted candidate backend {manifest['backend']} is already serving " @@ -732,7 +1414,15 @@ def evaluate( ) ) except ConnectorError as error: - operation_error = _runtime_connector_failure(error) + if error.category == "cancelled": + try: + self._assert_not_revoked(provenance, operation="inference") + except RuntimeFailure as revoked: + operation_error = revoked + else: + operation_error = _runtime_connector_failure(error) + else: + operation_error = _runtime_connector_failure(error) except Exception as error: operation_error = error finally: @@ -750,36 +1440,112 @@ def evaluate( except ConnectorError as error: if operation_error is None: operation_error = _runtime_connector_failure(error) - if operation_error is not None: - raise operation_error - evaluations = [] - for key, question in request["questions"].items(): - if question.get("type") != "noul": - continue - probabilities = answers.get(key, {}).get("probabilities") - if not _valid_yes_no(probabilities): - raise RuntimeFailure( - "inference_failed", - f"The promoted candidate did not answer question {json.dumps(key)}.", - 500, + try: + if operation_error is not None: + raise operation_error + evaluations = [] + for key, question in request["questions"].items(): + if question.get("type") != "noul": + continue + probabilities = answers.get(key, {}).get("probabilities") + if not _valid_yes_no(probabilities): + raise RuntimeFailure( + "inference_failed", + f"The promoted candidate did not answer question {json.dumps(key)}.", + 500, + ) + total = probabilities.get("yes", 0) + probabilities.get("no", 0) + yes = probabilities.get("yes", 0) / total + answers[key] = {"probabilities": {"yes": yes, "no": 1.0 - yes}} + evaluations.append( + { + "learningId": key, + "followsProbability": yes, + "breaksProbability": 1.0 - yes, + } ) - total = probabilities.get("yes", 0) + probabilities.get("no", 0) - yes = probabilities.get("yes", 0) / total - answers[key] = {"probabilities": {"yes": yes, "no": 1.0 - yes}} - evaluations.append( - { - "learningId": key, - "followsProbability": yes, - "breaksProbability": 1.0 - yes, - } + if idempotency_key is not None: + if retry_copy is not None: + with self._lock: + self._assert_not_revoked(provenance, operation="inference") + self.connectors.privacy.tombstone_copy( + retry_copy["id"], + state="active", + retention=retry_copy["retention"], + context=context, + ) + else: + with self._lock: + self._assert_not_revoked(provenance, operation="inference") + inference_succeeded = True + return { + "schemaVersion": INFERENCE_SCHEMA, + "candidateId": candidate_id, + "promotionGeneration": promotion["generation"], + "answers": answers, + "evaluations": evaluations, + } + finally: + with self._lock: + self._inference_cancellations.pop(context.operation_id, None) + if ( + isinstance(retry_reference, str) + and retry_copy is not None + and not inference_succeeded + ): + self._cleanup_inference_retry_reservation( + connector, + retry_reference, + retry_copy, + ) + if retry_lock is not None: + self._release_inference_idempotency_lock( + idempotency_key, + retry_lock, + ) + + def _release_inference_idempotency_lock( + self, + idempotency_key: str | None, + lock: threading.Lock, + ) -> None: + lock.release() + assert idempotency_key is not None + with self._lock: + current, users = self._inference_idempotency_locks[idempotency_key] + assert current is lock and users > 0 + if users == 1: + del self._inference_idempotency_locks[idempotency_key] + else: + self._inference_idempotency_locks[idempotency_key] = ( + lock, + users - 1, + ) + + def _cleanup_inference_retry_reservation( + self, + connector: InferenceConnector, + reference: str | None, + copy: Mapping[str, Any] | None, + ) -> None: + if not isinstance(reference, str) or copy is None: + return + context = self._operation_context( + f"inference_retry_cleanup_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + connector.delete(reference, context) + self.connectors.privacy.tombstone_copy( + copy["id"], + state="deleted", + retention={"mode": "deleted", "deletedAt": _now()}, + context=context, ) - return { - "schemaVersion": INFERENCE_SCHEMA, - "candidateId": candidate_id, - "promotionGeneration": promotion["generation"], - "answers": answers, - "evaluations": evaluations, - } + except ConnectorError: + # The registering inventory remains visible and retryable for deletion. + pass def _run_job(self, job_id: str) -> None: job: dict[str, Any] = {} @@ -999,6 +1765,9 @@ def _persist_candidate( context: OperationContext, ) -> str: check_context(context) + taskset = self._resolve_taskset(taskset_digest) + provenance = _taskset_provenance(taskset) + self._assert_not_revoked(provenance, operation="candidate persistence") if ( Path(result.adapter_name).name != result.adapter_name or result.adapter_name in ("manifest.json", "evaluation.json") @@ -1052,23 +1821,78 @@ def _persist_candidate( {"name": "evaluation.json"}, ), ) - self.connectors.candidates.put( - Candidate( + candidate_copy = _runtime_copy( + "candidate", + candidate_id, + candidate_id, + provenance, + scope=self.scope, + connector=type(self.connectors.candidates).__name__, + created_at=manifest["createdAt"], + retention=dict( + provenance.get("retention", {"mode": "delete_on_withdrawal"}) + ), + promotion_state="eligible", + use_state="registering", + ) + self.connectors.privacy.put_copy(candidate_copy, context) + self.connectors.privacy.put_copy( + _runtime_copy( + "report", + f"{candidate_id}:evaluation.json", candidate_id, - ContentDigest( - "sha256", - hashlib.sha256( - _json_bytes(manifest) - + b"\0" - + result.adapter - + _json_bytes(report) - ).hexdigest(), + provenance, + scope=self.scope, + connector=type(self.connectors.candidates).__name__, + created_at=manifest["createdAt"], + retention=dict( + provenance.get("retention", {"mode": "delete_on_withdrawal"}) ), - manifest, - artifacts, + parent_copy_ids=(candidate_copy["id"],), + use_state="registering", ), context, ) + try: + self.connectors.candidates.put( + Candidate( + candidate_id, + ContentDigest( + "sha256", + hashlib.sha256( + _json_bytes(manifest) + + b"\0" + + result.adapter + + _json_bytes(report) + ).hexdigest(), + ), + manifest, + artifacts, + ), + context, + ) + for copy_id in ( + candidate_copy["id"], + _runtime_copy_id("report", f"{candidate_id}:evaluation.json"), + ): + self.connectors.privacy.tombstone_copy( + copy_id, + state="active", + retention=provenance["retention"], + context=context, + ) + except ConnectorError: + for copy_id in ( + candidate_copy["id"], + _runtime_copy_id("report", f"{candidate_id}:evaluation.json"), + ): + self.connectors.privacy.tombstone_copy( + copy_id, + state="deleted", + retention={"mode": "deleted", "deletedAt": _now()}, + context=context, + ) + raise except ConnectorError as error: raise _runtime_connector_failure(error) from None return candidate_id @@ -1108,6 +1932,160 @@ def _recover_interrupted_jobs(self) -> None: except ConnectorError as error: raise _runtime_connector_failure(error) from None + def _backfill_local_privacy_inventory(self) -> None: + context = self._operation_context( + f"privacy_backfill_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + for taskset in self.connectors.tasksets.list(context): + self._record_taskset_copy( + taskset, + context, + mounted=taskset.reference.startswith("mounted-taskset:"), + ) + stored_candidates = { + candidate.id: candidate + for candidate in self.connectors.candidates.list(context) + } + for candidate in stored_candidates.values(): + candidate_copy_id = _runtime_copy_id("candidate", candidate.id) + report_copy_id = _runtime_copy_id( + "report", f"{candidate.id}:evaluation.json" + ) + try: + self.connectors.privacy.get_copy(candidate_copy_id, context) + self.connectors.privacy.get_copy(report_copy_id, context) + except ConnectorError as error: + if error.category != "not_found": + raise + else: + continue + manifest = candidate.manifest + taskset_digest_value = manifest.get("tasksetDigest") + if not isinstance(taskset_digest_value, str): + raise RuntimeFailure( + "candidate_provenance_missing", + f"Candidate {candidate.id} does not name its taskset.", + 500, + ) + taskset = self._resolve_taskset(taskset_digest_value) + provenance = _taskset_provenance(taskset) + candidate_copy = _runtime_copy( + "candidate", + candidate.id, + candidate.id, + provenance, + scope=self.scope, + connector=type(self.connectors.candidates).__name__, + created_at=manifest.get("createdAt"), + retention=provenance["retention"], + promotion_state="eligible", + ) + try: + self.connectors.privacy.get_copy(candidate_copy["id"], context) + except ConnectorError as error: + if error.category != "not_found": + raise + self.connectors.privacy.put_copy(candidate_copy, context) + report_copy = _runtime_copy( + "report", + f"{candidate.id}:evaluation.json", + candidate.id, + provenance, + scope=self.scope, + connector=type(self.connectors.candidates).__name__, + created_at=manifest.get("createdAt"), + retention=provenance["retention"], + parent_copy_ids=(candidate_copy["id"],), + ) + try: + self.connectors.privacy.get_copy(report_copy["id"], context) + except ConnectorError as error: + if error.category != "not_found": + raise + self.connectors.privacy.put_copy(report_copy, context) + for taskset in self.connectors.tasksets.list(context): + cursor = None + while True: + copies, cursor = self.connectors.privacy.list_copies( + {"tasksetDigest": str(taskset.digest)}, + cursor=cursor, + limit=100, + context=context, + ) + for copy in copies: + if ( + copy["kind"] in ("candidate", "report") + and copy["useState"] == "registering" + and copy["storage"]["reference"] not in stored_candidates + ): + self.connectors.privacy.tombstone_copy( + copy["id"], + state="deleted", + retention={"mode": "deleted", "deletedAt": _now()}, + context=context, + ) + if cursor is None: + break + for backend, connector in self.inference_connectors.items(): + for result in connector.list_results(context): + candidate_id = result["candidateId"] + candidate_copy = self.connectors.privacy.get_copy( + _runtime_copy_id("candidate", candidate_id), + context, + ) + provenance = _copy_provenance(candidate_copy) + retry_copy = _runtime_copy( + "inference", + result["reference"], + result["reference"], + provenance, + scope=self.scope, + connector=backend, + created_at=result["createdAt"], + retention={"mode": "delete_on_withdrawal"}, + parent_copy_ids=(candidate_copy["id"],), + ) + try: + self.connectors.privacy.get_copy(retry_copy["id"], context) + except ConnectorError as error: + if error.category != "not_found": + raise + self.connectors.privacy.put_copy(retry_copy, context) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + + def _recover_deletions(self) -> None: + context = self._operation_context( + f"deletion_recovery_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + for receipt in self.connectors.privacy.list_deletions( + ("revoking", "pending"), context + ): + self._process_deletion(dict(receipt), context) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + + def _reconcile_completed_deletions(self) -> None: + context = self._operation_context( + f"completed_deletion_recovery_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + for receipt in self.connectors.privacy.list_deletions( + ("completed",), context + ): + if self._matching_copies(receipt["selectors"], context): + self._process_deletion(dict(receipt), context) + except ConnectorError as error: + raise _runtime_connector_failure(error) from None + def _recover_pending_promotions( self, context: OperationContext | None = None ) -> None: @@ -1119,12 +2097,41 @@ def _recover_pending_promotions( try: pending = self.local_connectors.candidates.pending_promotions_json(selected) for promotion in pending: - self._promotable_candidate(promotion["candidateId"]) + try: + self._promotable_candidate(promotion["candidateId"]) + except RuntimeFailure as error: + if error.code != "deletion_revoked": + raise + self.local_connectors.candidates.discard_pending_promotion( + promotion["candidateId"], + selected, + ) self.local_connectors.candidates.recover_promotion(selected) except ConnectorError as error: raise _runtime_connector_failure(error) from None def _promotable_candidate(self, candidate_id: str) -> dict[str, Any]: + context = self._operation_context( + f"candidate_privacy_{uuid.uuid4().hex}", + max_seconds=30, + owner=RuntimeOwner("service", "decision-runtime"), + ) + try: + copy = self.connectors.privacy.get_copy( + _runtime_copy_id("candidate", candidate_id), + context, + ) + self._assert_not_revoked( + _copy_provenance(copy), operation="promotion or inference" + ) + except ConnectorError as error: + if error.category == "not_found": + raise RuntimeFailure( + "candidate_provenance_missing", + f"Candidate {candidate_id} has no runtime provenance.", + 500, + ) from None + raise _runtime_connector_failure(error) from None candidate = self.candidate(candidate_id)["manifest"] try: self._verify_taskset(candidate["tasksetDigest"]) @@ -1224,6 +2231,12 @@ def _cache_taskset(self, artifact: TasksetArtifact) -> None: while len(self._tasksets) > TASKSET_CACHE_LIMIT: self._tasksets.popitem(last=False) + def _evict_taskset(self, digest: str) -> None: + normalized = _taskset_digest_value(digest) + with self._lock: + self._tasksets.pop(normalized, None) + self._verified_taskset_directories.pop(normalized, None) + def _acquire_credential( self, audience: str, context: OperationContext ) -> Credential | None: @@ -1391,6 +2404,330 @@ def _taskset_digest_value(value: str | None) -> str: ) from None +def _manifest_provenance( + manifest: Mapping[str, Any], taskset_digest: str +) -> dict[str, Any]: + source = manifest.get("provenance", {}) + if not isinstance(source, Mapping): + source = {} + + def strings(name: str) -> list[str]: + value = source.get(name, ()) + if not isinstance(value, (list, tuple)) or not value or not all( + isinstance(item, str) and item for item in value + ): + raise RuntimeFailure( + "taskset_provenance_invalid", + f"The taskset manifest must name non-empty {name}.", + ) + return sorted(set(value)) + + permitting = manifest.get("permittingGrants", ()) + permitting_grants = ( + [item for item in permitting if isinstance(item, Mapping)] + if isinstance(permitting, list) + else [] + ) + grant_ids = ( + strings("grantIds") + if "grantIds" in source + else sorted( + { + item["id"] + for item in permitting_grants + if isinstance(item.get("id"), str) and item["id"] + } + ) + ) + purposes = ( + strings("purposes") + if "purposes" in source + else sorted( + { + purpose + for item in permitting_grants + for purpose in item.get("purposes", ()) + if isinstance(purpose, str) and purpose + } + ) + ) + review_digests = ( + strings("reviewDigests") + if "reviewDigests" in source + else ( + [manifest["reviewDigest"]] + if isinstance(manifest.get("reviewDigest"), str) + else [] + ) + ) + evidence_digests = ( + strings("evidenceDigests") + if "evidenceDigests" in source + else ( + [manifest["evidenceDigest"]] + if isinstance(manifest.get("evidenceDigest"), str) + else [] + ) + ) + if not grant_ids or not purposes or not review_digests or not evidence_digests: + raise RuntimeFailure( + "taskset_provenance_invalid", + "The taskset manifest must name permitting grants, purposes, review digests, " + "and evidence digests.", + ) + if any(not _is_digest(value) for value in review_digests + evidence_digests): + raise RuntimeFailure( + "taskset_provenance_invalid", + "Taskset review and evidence digests must be SHA-256 digests.", + ) + aggregate_evidence_digest = "sha256:" + hashlib.sha256( + _canonical_json_bytes(evidence_digests) + ).hexdigest() + declared_evidence_digest = source.get("evidenceDigest") + if declared_evidence_digest is not None and declared_evidence_digest != aggregate_evidence_digest: + raise RuntimeFailure( + "taskset_provenance_invalid", + "The taskset aggregate evidence digest does not match its evidence digests.", + ) + + export_id = manifest.get("exportId") + if not isinstance(export_id, str) or not export_id: + raise RuntimeFailure( + "taskset_provenance_invalid", + "The taskset manifest must name its source training export.", + ) + export_digest = source.get("exportDigest") + if not _is_digest(export_digest): + raise RuntimeFailure( + "taskset_provenance_invalid", + "The taskset manifest must name its source export digest.", + ) + return { + "sourceExportId": export_id, + "sourceExportDigest": export_digest, + "grantIds": grant_ids, + "purposes": purposes, + "reviewDigests": review_digests, + "evidenceDigests": evidence_digests, + "evidenceDigest": aggregate_evidence_digest, + "tasksetDigest": taskset_digest, + "retention": _normalize_retention( + manifest.get("retention"), + default_mode="delete_on_withdrawal", + ), + } + + +def _taskset_provenance(taskset: TasksetArtifact) -> dict[str, Any]: + return _manifest_provenance(taskset.taskset.metadata, str(taskset.digest)) + + +def _runtime_copy_id(kind: str, reference: str) -> str: + return "copy_" + hashlib.sha256(f"{kind}\0{reference}".encode()).hexdigest()[:32] + + +def _taskset_runtime_copy_ids(taskset_digest: str) -> tuple[str, str]: + digest = taskset_digest.removeprefix("sha256:") + return ( + _runtime_copy_id("taskset", f"uploaded:uploaded-taskset:{digest}"), + _runtime_copy_id("taskset", f"mounted:mounted-taskset:{digest}"), + ) + + +def _runtime_copy( + kind: str, + identity: str, + reference: str, + provenance: Mapping[str, Any], + *, + scope: RuntimeScope, + connector: str, + created_at: Any, + retention: Mapping[str, Any], + promotion_state: str = "not_applicable", + parent_copy_ids: tuple[str, ...] = (), + use_state: str = "active", +) -> dict[str, Any]: + return { + "schemaVersion": RUNTIME_COPY_SCHEMA, + "id": _runtime_copy_id(kind, identity), + "kind": kind, + "scope": { + "tenantId": scope.tenant_id, + "repository": scope.repository, + }, + "sourceTrainingExportId": provenance["sourceExportId"], + "sourceTrainingExportDigest": provenance.get("sourceExportDigest"), + "permittingGrantIds": list(provenance.get("grantIds", ())), + "permittedPurposes": list(provenance.get("purposes", ())), + "reviewDigests": list(provenance.get("reviewDigests", ())), + "evidenceDigests": list(provenance.get("evidenceDigests", ())), + "evidenceDigest": provenance.get("evidenceDigest"), + "tasksetDigest": provenance["tasksetDigest"], + "runtimeCopyId": identity, + "storage": {"connector": connector, "reference": reference}, + "createdAt": created_at if isinstance(created_at, str) else _now(), + "useState": use_state, + "promotionState": promotion_state, + "retention": _normalize_retention(retention), + "parentRuntimeCopyIds": list(parent_copy_ids), + } + + +def _disclosure_selectors(selectors: Mapping[str, str]) -> dict[str, str]: + if not isinstance(selectors, Mapping): + raise RuntimeFailure("invalid_input", "Disclosure selectors must be an object.") + allowed = {"exportId", "grantId", "evidenceDigest", "tasksetDigest"} + selected = { + key: value + for key, value in selectors.items() + if key in allowed and isinstance(value, str) and value + } + if not selected or len(selected) != len(selectors): + raise RuntimeFailure( + "invalid_input", + "Name one or more exact exportId, grantId, evidenceDigest, or tasksetDigest selectors.", + ) + return selected + + +def _retention_request(retention: Mapping[str, Any] | None) -> dict[str, Any] | None: + if retention is None: + return None + if ( + not isinstance(retention, Mapping) + or retention.get("mode") != "retain" + or not isinstance(retention.get("legalReason"), str) + or not retention["legalReason"].strip() + or set(retention) - {"mode", "legalReason", "expiresAt"} + ): + raise RuntimeFailure( + "invalid_input", + "retention must document mode retain and a non-empty legalReason.", + ) + result = { + "mode": "retain", + "legalReason": retention["legalReason"].strip(), + } + if "expiresAt" in retention: + if not isinstance(retention["expiresAt"], str) or not retention["expiresAt"]: + raise RuntimeFailure("invalid_input", "expiresAt must be a timestamp string.") + result["expiresAt"] = retention["expiresAt"] + return _normalize_retention(result) + + +def _selectors_match_provenance( + selectors: Mapping[str, str], provenance: Mapping[str, Any] +) -> bool: + values = { + "exportId": provenance.get("sourceExportId"), + "grantId": provenance.get("grantIds", ()), + "evidenceDigest": provenance.get("evidenceDigest"), + "tasksetDigest": provenance.get("tasksetDigest"), + } + return all( + selector in values[name] + if isinstance(values[name], (list, tuple)) + else values[name] == selector + for name, selector in selectors.items() + ) + + +def _copy_provenance(copy: Mapping[str, Any]) -> dict[str, Any]: + return { + "sourceExportId": copy.get("sourceTrainingExportId"), + "sourceExportDigest": copy.get("sourceTrainingExportDigest"), + "grantIds": copy.get("permittingGrantIds", ()), + "purposes": copy.get("permittedPurposes", ()), + "reviewDigests": copy.get("reviewDigests", ()), + "evidenceDigests": copy.get("evidenceDigests", ()), + "evidenceDigest": copy.get("evidenceDigest"), + "tasksetDigest": copy.get("tasksetDigest"), + "retention": copy.get("retention", {}), + } + + +def _pending_item(copy: Mapping[str, Any], error: ConnectorError) -> dict[str, Any]: + return { + "kind": copy["kind"], + "runtimeCopyId": copy["id"], + "code": error.code, + "retryable": error.retryable + or error.category in ("conflict", "deadline_exceeded", "unavailable"), + } + + +def _is_digest(value: Any) -> bool: + if not isinstance(value, str): + return False + try: + ContentDigest("sha256", value.removeprefix("sha256:")) + except ValueError: + return False + return value.startswith("sha256:") + + +def _canonical_json_bytes(value: Any) -> bytes: + return json.dumps( + value, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + ).encode() + + +def _normalize_retention( + value: Mapping[str, Any] | None, + *, + default_mode: str | None = None, +) -> dict[str, Any]: + source = value if isinstance(value, Mapping) else {} + mode = source.get("mode", default_mode) + if not isinstance(mode, str) or not mode: + raise RuntimeFailure("invalid_input", "Retention needs a non-empty mode.") + normalized = {"mode": mode} + aliases = {"retainUntil": "expiresAt", "policy": None} + for key in ("reason", "legalReason", "authority", "expiresAt", "deletedAt"): + candidate = source.get(key) + if candidate is not None: + if not isinstance(candidate, str) or not candidate: + raise RuntimeFailure("invalid_input", f"Retention {key} must be a string.") + normalized[key] = candidate + if "retainUntil" in source and "expiresAt" not in normalized: + candidate = source["retainUntil"] + if not isinstance(candidate, str) or not candidate: + raise RuntimeFailure("invalid_input", "Retention retainUntil must be a string.") + normalized["expiresAt"] = candidate + for key in ("expiresAt", "deletedAt"): + candidate = normalized.get(key) + if candidate is not None: + try: + parsed = datetime.fromisoformat(candidate.replace("Z", "+00:00")) + except ValueError: + raise RuntimeFailure( + "invalid_input", + f"Retention {key} must be an ISO datetime.", + ) from None + if parsed.tzinfo is None: + raise RuntimeFailure( + "invalid_input", + f"Retention {key} must include a timezone.", + ) + unknown = set(source) - { + "mode", + "policy", + "reason", + "legalReason", + "authority", + "expiresAt", + "retainUntil", + "deletedAt", + } + if unknown: + raise RuntimeFailure("invalid_input", "Retention has unsupported fields.") + return normalized + + def _taskset_bundle( digest: ContentDigest, request: Mapping[str, Any] ) -> TasksetBundle: @@ -1951,7 +3288,12 @@ def _run_inference( ) process.start() try: - process.join(max(0.0, context.deadline - time.monotonic())) + while process.is_alive(): + context.cancellation.raise_if_requested() + remaining = context.deadline - time.monotonic() + if remaining <= 0: + break + process.join(min(0.05, remaining)) if process.is_alive(): context.cancellation.cancel() _stop_process(process) diff --git a/packages/learning/environments/brand-check/runtime_server.py b/packages/learning/environments/brand-check/runtime_server.py index 3f3ed131..b3efb29a 100644 --- a/packages/learning/environments/brand-check/runtime_server.py +++ b/packages/learning/environments/brand-check/runtime_server.py @@ -14,7 +14,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from typing import Any -from urllib.parse import urlparse +from urllib.parse import parse_qs, urlparse from .connectors import RuntimeScope from .d1_connector import D1OmniBackend, taskset_learning_texts @@ -30,6 +30,7 @@ _SAFE_TOKEN = re.compile(r"^[\x21-\x7e]+$") _TASKSET_ROUTE = re.compile(r"^/v1/tasksets/([0-9a-f]{64})$") _CANCEL_ROUTE = re.compile(r"^/v1/jobs/(job_[A-Za-z0-9]+)/cancel$") +_DELETION_ROUTE = re.compile(r"^/v1/deletions/(deletion_[A-Za-z0-9]+)$") _IDEMPOTENCY_KEY = re.compile(r"^[\x21-\x7e]{1,200}$") @@ -132,6 +133,49 @@ def do_GET(self) -> None: promotion = self.server.runtime.promotion(required=False) self._send(200, promotion or {"candidateId": None}) return + if path == "/v1/runtime-copies": + query = parse_qs(urlparse(self.path).query, keep_blank_values=True) + selectors = { + public: query[name][0] + for public, name in ( + ("exportId", "exportId"), + ("grantId", "grantId"), + ("evidenceDigest", "evidenceDigest"), + ("tasksetDigest", "tasksetDigest"), + ) + if name in query and len(query[name]) == 1 + } + try: + limit = int(query.get("limit", ["50"])[0]) + except ValueError: + raise RuntimeFailure("invalid_input", "limit must be an integer.") from None + cursor = query.get("cursor", [None])[0] + known = { + "exportId", + "grantId", + "evidenceDigest", + "tasksetDigest", + "limit", + "cursor", + } + if set(query) - known or any(len(values) != 1 for values in query.values()): + raise RuntimeFailure( + "invalid_input", + "Runtime copy disclosure accepts one value per supported query parameter.", + ) + self._send( + 200, + self.server.runtime.disclose( + selectors, + limit=limit, + cursor=cursor, + ), + ) + return + deletion = _DELETION_ROUTE.fullmatch(path) + if deletion: + self._send(200, self.server.runtime.deletion(deletion.group(1))) + return if path.startswith("/v1/jobs/"): self._send(200, self.server.runtime.job(path.removeprefix("/v1/jobs/"))) return @@ -182,6 +226,20 @@ def do_POST(self) -> None: ) self._send(202, job) return + if path == "/v1/deletions": + self._send( + 200, + self.server.runtime.request_deletion( + body.get("selectors"), + requested_by=body.get("requestedBy"), + reason=body.get("reason"), + retention=body.get("retention"), + idempotency_key=self._idempotency_key( + _request_fingerprint("runtime-deletion", body) + ), + ), + ) + return if path == "/v1/promotions": self._send( 200, diff --git a/packages/learning/environments/brand-check/server/runtime.Dockerfile b/packages/learning/environments/brand-check/server/runtime.Dockerfile index 760c2915..a8d14c3d 100644 --- a/packages/learning/environments/brand-check/server/runtime.Dockerfile +++ b/packages/learning/environments/brand-check/server/runtime.Dockerfile @@ -8,6 +8,7 @@ ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONPATH=/app COPY *.py /app/replica_brand_check/ +COPY examples/runtime_deletion_demo.py /app/runtime_deletion_demo.py RUN useradd --uid 10001 --create-home --shell /usr/sbin/nolog replica USER replica diff --git a/packages/learning/environments/brand-check/tests/support.py b/packages/learning/environments/brand-check/tests/support.py index 88764b9d..3b8aad0d 100644 --- a/packages/learning/environments/brand-check/tests/support.py +++ b/packages/learning/environments/brand-check/tests/support.py @@ -69,10 +69,26 @@ def write_taskset(case: unittest.TestCase, tasks: list) -> Path: directory = Path(holder.name) (directory / "images").mkdir() (directory / "images" / "screen.png").write_bytes(PNG) + evidence_digests = [digest(b"evidence")] manifest = { "schemaVersion": "replica.decision-taskset/v1", "exportId": "export_test", "createdAt": "2026-11-07T00:00:00Z", + "provenance": { + "exportDigest": digest(b"export"), + "grantIds": ["grant_training"], + "purposes": ["training"], + "reviewDigests": [digest(b"review")], + "evidenceDigests": evidence_digests, + "evidenceDigest": digest( + json.dumps( + evidence_digests, + sort_keys=True, + separators=(",", ":"), + ).encode() + ), + }, + "retention": {"mode": "delete_on_withdrawal"}, "counts": {"tasks": len(tasks)}, } (directory / "taskset.json").write_text(json.dumps(manifest), encoding="utf-8") diff --git a/packages/learning/environments/brand-check/tests/test_connector_conformance.py b/packages/learning/environments/brand-check/tests/test_connector_conformance.py index 924f51f8..6a9576d0 100644 --- a/packages/learning/environments/brand-check/tests/test_connector_conformance.py +++ b/packages/learning/environments/brand-check/tests/test_connector_conformance.py @@ -209,9 +209,9 @@ def register(self, bundle, context): category="conflict", ) if previous is not None: - return previous[1] + return replace(previous[1], created=False) self._holders.append(holder) - artifact = TasksetArtifact(reference, digest, taskset) + artifact = TasksetArtifact(reference, digest, taskset, created=True) self._items[(_scope_key(context.scope), reference)] = taskset self._registrations[(_scope_key(context.scope), reference)] = ( identity, @@ -236,6 +236,29 @@ def resolve(self, reference, context): taskset, ) + def delete(self, reference, context): + _check_context(context) + if context.scope != self._scope: + raise ConnectorError( + "scope_unauthorized", + "The operation scope does not match this connector bundle.", + category="unauthorized", + ) + key = (_scope_key(context.scope), reference) + existed = key in self._items + self._items.pop(key, None) + self._registrations.pop(key, None) + return existed + + def list(self, context): + _check_context(context) + scope = _scope_key(context.scope) + return tuple( + self.resolve(reference, context) + for record_scope, reference in sorted(self._items) + if record_scope == scope + ) + class _MemoryArtifacts: def __init__(self): @@ -449,6 +472,7 @@ def __init__(self): self._items = {} self._promotions = {} self._promotion_idempotency = {} + self._owner_scope = None def put(self, candidate, context): _check_context(context) @@ -460,6 +484,8 @@ def put(self, candidate, context): category="integrity", ) key = (_scope_key(context.scope), candidate.id) + if self._owner_scope is None: + self._owner_scope = key[0] existing = self._items.get(key) if existing is not None and existing != candidate: raise ConnectorError( @@ -532,6 +558,169 @@ def current(self, context): _check_context(context) return self._promotions.get(_scope_key(context.scope)) + def delete(self, candidate_id, context): + _check_context(context) + scope = _scope_key(context.scope) + if self._owner_scope is not None and scope != self._owner_scope: + raise ConnectorError( + "candidate_not_found", + "The candidate was not found.", + category="not_found", + ) + key = (scope, candidate_id) + existed = key in self._items + self._items.pop(key, None) + return existed + + def list(self, context): + _check_context(context) + scope = _scope_key(context.scope) + return tuple( + candidate + for (record_scope, _candidate_id), candidate in sorted(self._items.items()) + if record_scope == scope + ) + + +class _MemoryPrivacy: + def __init__(self): + self._copies = {} + self._deletions = {} + self._retries = {} + + def put_copy(self, record, context): + _check_context(context) + key = (_scope_key(context.scope), record["id"]) + existing = self._copies.get(key) + if existing is not None and existing != record: + raise ConnectorError( + "runtime_copy_immutable", + "The runtime copy ID already identifies different metadata.", + category="conflict", + ) + self._copies[key] = dict(record) + return dict(record) + + def get_copy(self, copy_id, context): + _check_context(context) + try: + return dict(self._copies[(_scope_key(context.scope), copy_id)]) + except KeyError: + raise ConnectorError( + "runtime_copy_not_found", + "The runtime copy was not found.", + category="not_found", + ) from None + + def list_copies(self, selectors, *, cursor, limit, context): + _check_context(context) + if not selectors or not 1 <= limit <= 100: + raise ConnectorError( + "pagination_invalid", + "Copy pagination is invalid.", + category="invalid_input", + ) + values = [] + for (scope, _copy_id), record in sorted(self._copies.items()): + if scope != _scope_key(context.scope) or (cursor and record["id"] <= cursor): + continue + matches = { + "exportId": record.get("sourceTrainingExportId"), + "grantId": record.get("permittingGrantIds", ()), + "evidenceDigest": record.get("evidenceDigest"), + "tasksetDigest": record.get("tasksetDigest"), + } + if all( + value in matches[name] + if isinstance(matches[name], (list, tuple)) + else matches[name] == value + for name, value in selectors.items() + ): + values.append(dict(record)) + next_cursor = values[limit - 1]["id"] if len(values) > limit else None + return tuple(values[:limit]), next_cursor + + def tombstone_copy(self, copy_id, *, state, retention, context): + saved = self.get_copy(copy_id, context) + saved["useState"] = state + saved["state"] = state + if retention is not None: + saved["retention"] = dict(retention) + self._copies[(_scope_key(context.scope), copy_id)] = saved + return dict(saved) + + def create_deletion(self, request, context): + _check_context(context) + retry = (_scope_key(context.scope), context.idempotency_key) + prior = self._retries.get(retry) + if prior is not None: + request_identity, deletion_key = prior + identity = json.dumps(request, sort_keys=True, separators=(",", ":")) + if request_identity != identity: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + return dict(self._deletions[deletion_key]) + deletion_id = f"deletion_{len(self._deletions) + 1}" + receipt = { + "schemaVersion": "replica.runtime-deletion-receipt/v1", + "id": deletion_id, + "scope": { + "tenantId": context.scope.tenant_id, + "repository": context.scope.repository, + }, + **dict(request), + "state": "revoking", + "createdAt": "2026-11-18T00:00:00Z", + "updatedAt": "2026-11-18T00:00:00Z", + "items": [], + "pendingItems": [], + "failures": [], + "version": 1, + } + key = (_scope_key(context.scope), deletion_id) + self._deletions[key] = receipt + self._retries[retry] = ( + json.dumps(request, sort_keys=True, separators=(",", ":")), + key, + ) + return dict(receipt) + + def get_deletion(self, deletion_id, context): + _check_context(context) + try: + return dict(self._deletions[(_scope_key(context.scope), deletion_id)]) + except KeyError: + raise ConnectorError( + "deletion_not_found", + "The deletion receipt was not found.", + category="not_found", + ) from None + + def list_deletions(self, states, context): + _check_context(context) + return tuple( + dict(receipt) + for (scope, _deletion_id), receipt in self._deletions.items() + if scope == _scope_key(context.scope) and receipt["state"] in states + ) + + def update_deletion( + self, deletion_id, record, *, expected_version, context + ): + current = self.get_deletion(deletion_id, context) + if current["version"] != expected_version: + raise ConnectorError( + "deletion_version_conflict", + "The deletion receipt changed.", + category="conflict", + ) + saved = dict(record, version=expected_version + 1) + self._deletions[(_scope_key(context.scope), deletion_id)] = saved + return dict(saved) + class _MemoryExecution: name = "contract" @@ -589,6 +778,15 @@ def predict(self, adapter, request, *, context, credential): self.inference_contexts.append(context) return {"operationId": context.operation_id} + def delete(self, reference, context): + del reference + _check_context(context) + return True + + def list_results(self, context): + _check_context(context) + return () + def _check_credential( credential: Credential | None, @@ -680,6 +878,7 @@ def __init__(self): artifacts=_MemoryArtifacts(), jobs=_MemoryJobs(), candidates=_MemoryCandidates(), + privacy=_MemoryPrivacy(), training={"contract": self.execution}, inference={"contract": self.execution}, credentials=_MemoryCredentials(self.scope), diff --git a/packages/learning/environments/brand-check/tests/test_connector_deletion_conformance.py b/packages/learning/environments/brand-check/tests/test_connector_deletion_conformance.py new file mode 100644 index 00000000..a92d2294 --- /dev/null +++ b/packages/learning/environments/brand-check/tests/test_connector_deletion_conformance.py @@ -0,0 +1,347 @@ +"""Deletion conformance for provider-neutral runtime storage connectors.""" + +from __future__ import annotations + +import hashlib +import json +import shutil +import time +import unittest + +from replica_brand_check.connectors import ( + Artifact, + Cancellation, + Candidate, + ConnectorError, + ContentDigest, + OperationContext, + RuntimeOwner, + RuntimeScope, +) +from replica_brand_check.local_connectors import create_local_connector_bundle +from replica_brand_check.runtime import FixtureBackend +from tests.support import make_task, taskset_bundle, write_taskset + + +def _digest(data: bytes) -> ContentDigest: + return ContentDigest("sha256", hashlib.sha256(data).hexdigest()) + + +class ConnectorDeletionConformanceTest(unittest.TestCase): + def setUp(self): + self.taskset_dir = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + self.state_dir = self.taskset_dir.parent / f"{self.taskset_dir.name}-state" + self.addCleanup(shutil.rmtree, self.state_dir, True) + self.scope = RuntimeScope("tenant-one", "github/example") + self.bundle = create_local_connector_bundle( + self.state_dir, + self.taskset_dir, + {"fixture": FixtureBackend()}, + scope=self.scope, + ) + self.addCleanup(self.bundle.close) + + def test_enumerates_scoped_copies_with_exact_filters_and_stable_cursors(self): + privacy = self.bundle.connectors.privacy + first = privacy.put_copy( + self._copy("copy_taskset", "taskset", "sha256:" + "a" * 64), + self._context("put-taskset"), + ) + second = privacy.put_copy( + self._copy("copy_candidate", "candidate", "sha256:" + "a" * 64), + self._context("put-candidate"), + ) + + page_one, cursor = privacy.list_copies( + { + "exportId": "export_contract", + "grantId": "grant_contract", + "evidenceDigest": "sha256:" + "e" * 64, + "tasksetDigest": "sha256:" + "a" * 64, + }, + cursor=None, + limit=1, + context=self._context("list-first"), + ) + page_two, end = privacy.list_copies( + { + "exportId": "export_contract", + "grantId": "grant_contract", + "evidenceDigest": "sha256:" + "e" * 64, + "tasksetDigest": "sha256:" + "a" * 64, + }, + cursor=cursor, + limit=1, + context=self._context("list-second"), + ) + + self.assertEqual( + {item["id"] for item in [*page_one, *page_two]}, + {first["id"], second["id"]}, + ) + self.assertIsNotNone(cursor) + self.assertIsNone(end) + no_match, no_cursor = privacy.list_copies( + {"exportId": "export_contract-other"}, + cursor=None, + limit=10, + context=self._context("list-no-match"), + ) + self.assertEqual(no_match, ()) + self.assertIsNone(no_cursor) + with self.assertRaises(ConnectorError) as unbounded: + privacy.list_copies( + {}, + cursor=None, + limit=1_000_000, + context=self._context("list-unbounded"), + ) + self.assertEqual(unbounded.exception.category, "invalid_input") + with self.assertRaises(ConnectorError) as hidden: + privacy.list_copies( + {}, + cursor=None, + limit=10, + context=self._context( + "list-other-scope", + scope=RuntimeScope("tenant-two", "github/example"), + ), + ) + self.assertIn(hidden.exception.category, ("not_found", "unauthorized")) + + def test_delete_and_tombstone_are_idempotent_without_losing_receipt_metadata(self): + uploaded_directory = write_taskset( + self, + [ + make_task(11, answer=True), + make_task(12, split="test", answer=True), + ], + ) + context = self._context("register") + registered = self.bundle.connectors.tasksets.register( + taskset_bundle(uploaded_directory), + context, + ) + artifact = Artifact( + _digest(b"adapter"), + "application/octet-stream", + b"adapter", + {"name": "adapter.bin"}, + ) + candidate = Candidate( + "candidate_contract", + _digest(b"candidate"), + {"tasksetDigest": str(registered.digest)}, + (artifact,), + ) + self.bundle.connectors.candidates.put(candidate, self._context("put-candidate")) + taskset_copy = self.bundle.connectors.privacy.put_copy( + self._copy( + "copy_taskset", + "taskset", + str(registered.digest), + storage_reference=registered.reference, + ), + self._context("put-taskset-copy"), + ) + candidate_copy = self.bundle.connectors.privacy.put_copy( + self._copy( + "copy_candidate", + "candidate", + str(registered.digest), + storage_reference=candidate.id, + ), + self._context("put-candidate-copy"), + ) + + self.assertTrue( + self.bundle.connectors.tasksets.delete( + registered.reference, + self._context("delete-taskset"), + ) + ) + self.assertFalse( + self.bundle.connectors.tasksets.delete( + registered.reference, + self._context("delete-taskset-retry"), + ) + ) + self.assertTrue( + self.bundle.connectors.candidates.delete( + candidate.id, + self._context("delete-candidate"), + ) + ) + self.assertFalse( + self.bundle.connectors.candidates.delete( + candidate.id, + self._context("delete-candidate-retry"), + ) + ) + for copy in (taskset_copy, candidate_copy): + tombstone = self.bundle.connectors.privacy.tombstone_copy( + copy["id"], + state="deleted", + retention=None, + context=self._context(f"tombstone-{copy['id']}"), + ) + repeated = self.bundle.connectors.privacy.tombstone_copy( + copy["id"], + state="deleted", + retention=None, + context=self._context(f"tombstone-{copy['id']}-retry"), + ) + self.assertEqual(repeated, tombstone) + self.assertEqual(tombstone["useState"], "deleted") + self.assertEqual(tombstone["sourceTrainingExportId"], "export_contract") + self.assertEqual(tombstone["permittingGrantIds"], ["grant_contract"]) + self.assertEqual(tombstone["evidenceDigests"], ["sha256:" + "b" * 64]) + self.assertNotIn("content", json.dumps(tombstone).lower()) + + with self.assertRaises(ConnectorError) as taskset_missing: + self.bundle.connectors.tasksets.resolve( + registered.reference, + self._context("resolve-deleted-taskset"), + ) + self.assertEqual(taskset_missing.exception.category, "not_found") + with self.assertRaises(ConnectorError) as candidate_missing: + self.bundle.connectors.candidates.get( + candidate.id, + self._context("read-deleted-candidate"), + ) + self.assertEqual(candidate_missing.exception.category, "not_found") + + def test_deletion_receipt_compare_and_swap_survives_restart(self): + request_record = { + "schemaVersion": "replica.runtime-deletion/v1", + "id": "deletion_contract", + "scope": {"tenantId": "tenant-one", "repository": "github/example"}, + "selectors": {"exportId": "export_contract"}, + "requestedBy": "privacy-operator", + "reason": "consent withdrawn", + "retention": None, + } + created = self.bundle.connectors.privacy.create_deletion( + request_record, + self._context("create-deletion", key="stable-deletion"), + ) + retried = self.bundle.connectors.privacy.create_deletion( + request_record, + self._context("create-deletion-retry", key="stable-deletion"), + ) + self.assertEqual(retried, created) + other_request = { + **request_record, + "selectors": {"grantId": "grant_other"}, + "reason": "another withdrawal", + } + self.bundle.connectors.privacy.create_deletion( + other_request, + self._context("create-other-deletion", key="other-deletion"), + ) + with self.assertRaises(ConnectorError) as reused: + self.bundle.connectors.privacy.create_deletion( + other_request, + self._context("reuse-stable-deletion", key="stable-deletion"), + ) + self.assertEqual(reused.exception.code, "idempotency_conflict") + completed_record = dict( + created, + state="completed", + revokedAt="2026-11-19T00:00:01Z", + completedAt="2026-11-19T00:00:02Z", + pendingItems=[], + ) + completed = self.bundle.connectors.privacy.update_deletion( + created["id"], + completed_record, + expected_version=created["version"], + context=self._context("complete-deletion"), + ) + with self.assertRaises(ConnectorError) as stale: + self.bundle.connectors.privacy.update_deletion( + created["id"], + dict(completed_record, reason="changed"), + expected_version=created["version"], + context=self._context("stale-deletion-update"), + ) + self.assertEqual(stale.exception.category, "conflict") + self.bundle.close() + + reopened = create_local_connector_bundle( + self.state_dir, + self.taskset_dir, + {"fixture": FixtureBackend()}, + scope=self.scope, + ) + self.addCleanup(reopened.close) + self.assertEqual( + reopened.connectors.privacy.get_deletion( + created["id"], + self._context("read-deletion-after-restart"), + ), + completed, + ) + self.assertIn( + completed, + reopened.connectors.privacy.list_deletions( + ("completed",), + self._context("list-deletions-after-restart"), + ), + ) + + def _copy( + self, + copy_id, + kind, + taskset_digest, + *, + storage_reference=None, + ): + return { + "schemaVersion": "replica.runtime-copy/v1", + "id": copy_id, + "kind": kind, + "scope": {"tenantId": "tenant-one", "repository": "github/example"}, + "sourceTrainingExportId": "export_contract", + "sourceTrainingExportDigest": "sha256:" + "d" * 64, + "permittingGrantIds": ["grant_contract"], + "permittedPurposes": ["training"], + "reviewDigests": ["sha256:" + "c" * 64], + "evidenceDigests": ["sha256:" + "b" * 64], + "evidenceDigest": "sha256:" + "e" * 64, + "tasksetDigest": taskset_digest, + "runtimeCopyId": copy_id, + "storage": { + "connector": "local-filesystem", + "reference": storage_reference or copy_id, + }, + "createdAt": "2026-11-19T00:00:00Z", + "promotionState": "not_promoted", + "useState": "active", + "retention": { + "mode": "delete_on_withdrawal", + "expiresAt": "2026-12-19T00:00:00Z", + }, + "parentRuntimeCopyIds": [], + } + + def _context(self, operation_id, *, key=None, scope=None): + return OperationContext( + operation_id=operation_id, + idempotency_key=key or operation_id, + scope=scope or self.scope, + owner=RuntimeOwner("worker", "deletion-conformance"), + deadline=time.monotonic() + 5, + cancellation=Cancellation(), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/learning/environments/brand-check/tests/test_runtime.py b/packages/learning/environments/brand-check/tests/test_runtime.py index 56e4a7ac..d498022c 100644 --- a/packages/learning/environments/brand-check/tests/test_runtime.py +++ b/packages/learning/environments/brand-check/tests/test_runtime.py @@ -2012,7 +2012,7 @@ def test_http_upload_registers_a_private_taskset_for_jobs_and_restart(self): self.assertIn(retry_status, (200, 201)) self.assertEqual(retried, first) self.assertEqual(first["digest"], f"sha256:{digest}") - self.assertEqual(first["reference"], f"taskset:{digest}") + self.assertEqual(first["reference"], f"uploaded-taskset:{digest}") self.assertIn( "taskset_bytes", [usage.kind for usage in runtime.local_connectors.observability.records()], diff --git a/packages/learning/environments/brand-check/tests/test_runtime_deletion.py b/packages/learning/environments/brand-check/tests/test_runtime_deletion.py new file mode 100644 index 00000000..2ff50a48 --- /dev/null +++ b/packages/learning/environments/brand-check/tests/test_runtime_deletion.py @@ -0,0 +1,1450 @@ +"""Privacy disclosure and deletion behavior for the customer-operated runtime.""" + +from __future__ import annotations + +import base64 +import json +import shutil +import threading +import time +import unittest +from pathlib import Path +from unittest import mock +from urllib import error, parse, request + +from replica_brand_check.connectors import ConnectorError, RuntimeOwner, RuntimeScope +from replica_brand_check.runtime import FixtureBackend, LocalRuntime, RuntimeFailure +from replica_brand_check.runtime_server import RuntimeHTTPServer +from replica_brand_check.taskset import TasksetError +from tests.support import PNG, RULE, make_task, taskset_upload, write_taskset + +TOKEN = "runtime-deletion-test-token-0123456789" +EXPORT_ID = "export_privacy_contract" +GRANT_ID = "grant_training_contract" +REVIEW_DIGEST = "sha256:" + "1" * 64 +RAW_EVIDENCE_DIGEST = "sha256:" + "2" * 64 +EVIDENCE_DIGEST = "sha256:" + __import__("hashlib").sha256( + json.dumps( + [RAW_EVIDENCE_DIGEST], + sort_keys=True, + separators=(",", ":"), + ).encode() +).hexdigest() + + +class BlockingBackend(FixtureBackend): + """Keep a training job active until deletion has a chance to revoke it.""" + + def __init__(self, started: Path, release: Path): + self.started = started + self.release = release + + def train(self, *args, **kwargs): + self.started.write_text("started", encoding="utf-8") + deadline = time.monotonic() + 5 + while not self.release.exists() and time.monotonic() < deadline: + time.sleep(0.01) + if not self.release.exists(): + raise AssertionError("blocking backend was not released") + return super().train(*args, **kwargs) + + +class BlockingInferenceBackend(FixtureBackend): + """Keep inference active until deletion cancels its worker.""" + + def __init__(self, started: Path, release: Path): + self.started = started + self.release = release + + def predict(self, *args, **kwargs): + self.started.write_text("started", encoding="utf-8") + deadline = time.monotonic() + 5 + while not self.release.exists() and time.monotonic() < deadline: + time.sleep(0.01) + return super().predict(*args, **kwargs) + + +class CountingBlockingInferenceBackend(BlockingInferenceBackend): + """Record backend executions while concurrent retries share one result.""" + + max_concurrent_inference = 1 + + def __init__(self, started: Path, release: Path, calls: Path): + super().__init__(started, release) + self.calls = calls + + def predict(self, *args, **kwargs): + with self.calls.open("a", encoding="utf-8") as handle: + handle.write("predict\n") + return super().predict(*args, **kwargs) + + +class InvalidInferenceResultBackend(FixtureBackend): + """Persist a result that runtime response validation must reject.""" + + def predict(self, *args, **kwargs): + del args, kwargs + return {RULE: {"probabilities": {"yes": 2, "no": -1}}} + + +class RuntimeDeletionTest(unittest.TestCase): + def setUp(self): + self.taskset_dir = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, answer=False), + make_task(3, split="test", answer=True), + ], + ) + manifest_path = self.taskset_dir / "taskset.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest.update( + { + "exportId": EXPORT_ID, + "provenance": { + "exportDigest": "sha256:" + "4" * 64, + "grantIds": [GRANT_ID], + "purposes": ["training", "evaluation"], + "reviewDigests": [REVIEW_DIGEST], + "evidenceDigests": [RAW_EVIDENCE_DIGEST], + "evidenceDigest": EVIDENCE_DIGEST, + }, + "retention": { + "policy": "delete_on_withdrawal", + "expiresAt": "2026-12-19T00:00:00Z", + }, + } + ) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + self.state_dir = self.taskset_dir.parent / f"{self.taskset_dir.name}-state" + self.addCleanup(shutil.rmtree, self.state_dir, True) + + def test_disclosure_api_requires_auth_and_exact_scoped_bounded_filters(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-for-disclosure", + ) + _server, _thread, base_url = self._serve(runtime) + query = parse.urlencode( + { + "exportId": EXPORT_ID, + "grantId": GRANT_ID, + "evidenceDigest": EVIDENCE_DIGEST, + "tasksetDigest": finished["tasksetDigest"], + "limit": 1, + } + ) + url = f"{base_url}/v1/runtime-copies?{query}" + + with self.assertRaises(error.HTTPError) as unauthorized: + request.urlopen(url) + self.assertEqual(unauthorized.exception.code, 401) + unauthorized.exception.close() + + first = self._http_json( + request.Request(url, headers={"Authorization": f"Bearer {TOKEN}"}) + ) + second = self._http_json( + request.Request( + f"{url}&cursor={parse.quote(first['nextCursor'])}", + headers={"Authorization": f"Bearer {TOKEN}"}, + ) + ) + + self.assertEqual(len(first["items"]), 1) + self.assertTrue(second["items"]) + self.assertNotEqual(first["items"][0]["id"], second["items"][0]["id"]) + for item in first["items"] + second["items"]: + self.assertEqual( + item["scope"], + {"tenantId": "tenant-one", "repository": "github/example"}, + ) + self.assertEqual(item["sourceTrainingExportId"], EXPORT_ID) + self.assertIn(GRANT_ID, item["permittingGrantIds"]) + self.assertIn("training", item["permittedPurposes"]) + self.assertEqual(item["evidenceDigest"], EVIDENCE_DIGEST) + self.assertEqual(item["reviewDigest"], REVIEW_DIGEST) + self.assertEqual(item["tasksetDigest"], finished["tasksetDigest"]) + self.assertTrue(item["runtimeCopyId"]) + self.assertTrue(item["storage"]["connector"]) + self.assertTrue(item["storage"]["reference"]) + self.assertTrue(item["createdAt"]) + self.assertIn("useState", item) + self.assertIn("promotionState", item) + self.assertIn("retention", item) + self._assert_content_free(item) + + no_match = self._http_json( + request.Request( + f"{base_url}/v1/runtime-copies?" + + parse.urlencode({"exportId": EXPORT_ID + "-other"}), + headers={"Authorization": f"Bearer {TOKEN}"}, + ) + ) + self.assertEqual(no_match["items"], []) + status, body = self._http_error( + request.Request( + f"{base_url}/v1/runtime-copies?limit=1000000", + headers={"Authorization": f"Bearer {TOKEN}"}, + ) + ) + self.assertEqual(status, 400) + self.assertEqual(body["error"]["code"], "invalid_input") + + def test_full_lifecycle_deletion_revokes_use_and_writes_private_receipt(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-deletion", + ) + self.assertEqual( + runtime.evaluate(self._evaluation())["candidateId"], + finished["candidateId"], + ) + + _server, _thread, base_url = self._serve(runtime) + deletion_url = f"{base_url}/v1/deletions" + deletion_body = json.dumps( + { + "selectors": {"exportId": EXPORT_ID}, + "requestedBy": "privacy-operator", + "reason": "training consent withdrawn", + } + ).encode() + status, unauthorized = self._http_error( + request.Request( + deletion_url, + data=deletion_body, + headers={"Content-Type": "application/json"}, + method="POST", + ) + ) + self.assertEqual(status, 401) + self.assertEqual(unauthorized["error"]["code"], "unauthorized") + requested = self._http_json( + request.Request( + deletion_url, + data=deletion_body, + headers={ + "Authorization": f"Bearer {TOKEN}", + "Content-Type": "application/json", + "Idempotency-Key": "delete-export", + }, + method="POST", + ) + ) + receipt = self._wait_for_deletion(runtime, requested["id"]) + + self.assertEqual(receipt["state"], "completed") + self.assertTrue(receipt["revokedAt"]) + self.assertTrue(receipt["completedAt"]) + self.assertEqual(receipt["selectors"], {"exportId": EXPORT_ID}) + self.assertTrue(receipt["items"]) + self.assertTrue( + all(item["state"] in {"deleted", "retained"} for item in receipt["items"]) + ) + for item in receipt["items"]: + if item["state"] == "retained": + self.assertTrue(item["retention"]["reason"]) + self.assertTrue(item["retention"]["authority"]) + self.assertTrue(item["retention"]["expiresAt"]) + self._assert_content_free(receipt) + telemetry = [ + { + "kind": record.kind, + "quantity": record.quantity, + "unit": record.unit, + "metadata": dict(record.metadata), + } + for record in runtime.local_connectors.observability.records() + ] + self._assert_content_free(telemetry) + + with self.assertRaises(RuntimeFailure) as not_ready: + runtime.ready() + self.assertEqual(not_ready.exception.code, "deletion_revoked") + with self.assertRaises(RuntimeFailure) as inference: + runtime.evaluate(self._evaluation()) + self.assertEqual(inference.exception.code, "deletion_revoked") + with self.assertRaises(RuntimeFailure) as promotion: + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=1, + idempotency_key="promote-after-deletion", + ) + self.assertEqual(promotion.exception.code, "deletion_revoked") + + def test_explicit_retention_keeps_bytes_but_revokes_every_use(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-retention", + ) + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="legal hold", + retention={ + "mode": "retain", + "legalReason": "A documented litigation hold applies.", + "expiresAt": "2027-01-01T00:00:00Z", + }, + idempotency_key="retain-export", + ) + + self.assertEqual(receipt["state"], "completed") + self.assertTrue(receipt["items"]) + self.assertTrue(all(item["state"] == "retained" for item in receipt["items"])) + self.assertTrue((self.state_dir / "candidates" / finished["candidateId"]).is_dir()) + self.assertTrue(any((self.state_dir / "tasksets").iterdir())) + for operation in ( + lambda: runtime.submit( + backend="fixture", + max_updates=1, + max_seconds=5, + max_credits=1, + ), + lambda: runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=1, + idempotency_key="promote-retained", + ), + lambda: runtime.evaluate(self._evaluation()), + ): + with self.assertRaises(RuntimeFailure) as blocked: + operation() + self.assertEqual(blocked.exception.code, "deletion_revoked") + + def test_rejects_malformed_retention_requests(self): + runtime = self._runtime() + for retention in ( + {"mode": "retain"}, + {"mode": "retain", "legalReason": "hold", "retainUntil": "2027-01-01"}, + {"mode": "retain", "legalReason": "hold", "expiresAt": "not-a-date"}, + {"mode": "delete_on_withdrawal", "legalReason": "hold"}, + ): + with self.subTest(retention=retention): + with self.assertRaises(RuntimeFailure) as invalid: + runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="invalid retention", + retention=retention, + idempotency_key="invalid-" + str(len(str(retention))), + ) + self.assertEqual(invalid.exception.code, "invalid_input") + + def test_refuses_missing_provenance_and_non_training_permission(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + missing = json.loads(json.dumps(upload)) + missing["manifest"]["provenance"].pop("grantIds") + with self.assertRaises(RuntimeFailure) as invalid: + runtime.register_taskset( + digest, + missing, + idempotency_key="missing-provenance", + ) + self.assertEqual(invalid.exception.code, "taskset_provenance_invalid") + + evaluation_only = json.loads(json.dumps(upload)) + evaluation_only["manifest"]["provenance"]["purposes"] = ["evaluation"] + directory = self._control("evaluation-only") + directory.mkdir() + (directory / "images").mkdir() + (directory / "taskset.json").write_text( + json.dumps(evaluation_only["manifest"]), + encoding="utf-8", + ) + (directory / "tasks.jsonl").write_text( + "".join(json.dumps(task) + "\n" for task in evaluation_only["tasks"]), + encoding="utf-8", + ) + for image in evaluation_only["images"]: + destination = directory / image["path"] + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(base64.b64decode(image["dataBase64"])) + evaluation_digest, evaluation_upload = taskset_upload(directory) + runtime.register_taskset( + evaluation_digest, + evaluation_upload, + idempotency_key="evaluation-only-upload", + ) + with self.assertRaises(RuntimeFailure) as forbidden: + runtime.submit( + backend="fixture", + max_updates=1, + max_seconds=5, + max_credits=1, + taskset_digest=f"sha256:{evaluation_digest}", + ) + self.assertEqual(forbidden.exception.code, "training_not_permitted") + + def test_deletion_removes_inference_retry_results_and_cached_taskset(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + runtime.register_taskset(digest, upload, idempotency_key="upload-for-cache-deletion") + finished = self._wait_for_job( + runtime, + runtime.submit( + backend="fixture", + max_updates=8, + max_seconds=5, + max_credits=8, + taskset_digest=f"sha256:{digest}", + )["id"], + ) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-retry-deletion", + ) + runtime.evaluate(self._evaluation(), idempotency_key="durable-inference") + inference_path = ( + self.state_dir + / "inference" + / ".idempotency" + / __import__("hashlib").sha256(b"durable-inference").hexdigest() + ) + self.assertTrue(inference_path.is_file()) + self.assertTrue(runtime._tasksets) + + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="delete retry result", + idempotency_key="delete-retry-result", + ) + + self.assertEqual(receipt["state"], "completed") + self.assertFalse(inference_path.exists()) + self.assertEqual(runtime._tasksets, {}) + self.assertEqual(runtime._verified_taskset_directories, {}) + + def test_restart_backfills_existing_tasksets_candidates_and_reports(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-backfill", + ) + runtime.close() + shutil.rmtree(self.state_dir / "privacy") + + reopened = LocalRuntime( + self.state_dir, + self.taskset_dir, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + disclosed = reopened.disclose({"exportId": EXPORT_ID}) + + self.assertEqual( + {item["kind"] for item in disclosed["items"]}, + {"taskset", "candidate", "report"}, + ) + self.assertEqual( + reopened.evaluate(self._evaluation())["candidateId"], + finished["candidateId"], + ) + + def test_restart_removes_legacy_inference_retries_without_provenance(self): + runtime = self._runtime() + runtime.close() + legacy = self.state_dir / "inference" / ".idempotency" / ("a" * 64) + legacy.write_text( + json.dumps( + { + "requestDigest": "sha256:" + "1" * 64, + "result": {RULE: {"probabilities": {"yes": 1, "no": 0}}}, + } + ), + encoding="utf-8", + ) + + reopened = LocalRuntime( + self.state_dir, + self.taskset_dir, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + + self.assertFalse(legacy.exists()) + + def test_upgrade_reconciles_backfilled_copies_against_completed_withdrawal(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + runtime.register_taskset(digest, upload, idempotency_key="legacy-upload") + finished = self._wait_for_job( + runtime, + runtime.submit( + backend="fixture", + max_updates=2, + max_seconds=5, + max_credits=2, + taskset_digest=f"sha256:{digest}", + )["id"], + ) + context = runtime._operation_context( + "legacy-completed-deletion", + max_seconds=30, + owner=RuntimeOwner("human", "privacy-operator"), + idempotency_key="legacy-completed-deletion", + ) + receipt = runtime.connectors.privacy.create_deletion( + { + "selectors": {"exportId": EXPORT_ID}, + "requestedBy": "privacy-operator", + "reason": "legacy completed withdrawal", + "retention": None, + }, + context, + ) + runtime.connectors.privacy.update_deletion( + receipt["id"], + { + **receipt, + "state": "completed", + "updatedAt": "2026-11-20T00:00:00Z", + "completedAt": "2026-11-20T00:00:00Z", + }, + expected_version=receipt["version"], + context=context, + ) + shutil.rmtree(self.state_dir / "privacy" / "copies") + (self.state_dir / "privacy" / "copies").mkdir(mode=0o700) + runtime.close() + + reopened = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + + self.assertFalse((self.state_dir / "tasksets" / digest).exists()) + self.assertFalse( + (self.state_dir / "candidates" / finished["candidateId"]).exists() + ) + + def test_deletion_api_derives_distinct_default_retry_keys_from_each_body(self): + runtime = self._runtime() + _server, _thread, base_url = self._serve(runtime) + headers = { + "Authorization": f"Bearer {TOKEN}", + "Content-Type": "application/json", + } + + first = self._http_json( + request.Request( + f"{base_url}/v1/deletions", + data=json.dumps( + { + "selectors": {"exportId": "export_missing_one"}, + "requestedBy": "privacy-operator", + "reason": "first withdrawal", + } + ).encode(), + headers=headers, + method="POST", + ) + ) + second = self._http_json( + request.Request( + f"{base_url}/v1/deletions", + data=json.dumps( + { + "selectors": {"exportId": "export_missing_two"}, + "requestedBy": "privacy-operator", + "reason": "second withdrawal", + } + ).encode(), + headers=headers, + method="POST", + ) + ) + + self.assertNotEqual(first["id"], second["id"]) + self.assertEqual(first["state"], "completed") + self.assertEqual(second["state"], "completed") + + def test_duplicate_deletion_returns_one_durable_receipt_across_restart(self): + runtime = self._runtime() + self._train(runtime) + first = runtime.request_deletion( + {"grantId": GRANT_ID}, + requested_by="privacy-operator", + reason="grant withdrawn", + idempotency_key="delete-grant", + ) + same_retry = runtime.request_deletion( + {"grantId": GRANT_ID}, + requested_by="privacy-operator", + reason="grant withdrawn", + idempotency_key="delete-grant", + ) + self.assertEqual(same_retry["id"], first["id"]) + completed = self._wait_for_deletion(runtime, first["id"]) + runtime.close() + + reopened = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": FixtureBackend()}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + duplicate = reopened.request_deletion( + {"grantId": GRANT_ID}, + requested_by="privacy-operator", + reason="grant withdrawn", + idempotency_key="delete-grant-after-restart", + ) + + self.assertEqual(duplicate["id"], first["id"]) + reopened_receipt = reopened.deletion(first["id"]) + self.assertEqual(reopened_receipt["id"], completed["id"]) + self.assertEqual(reopened_receipt["state"], "completed") + self.assertEqual(reopened_receipt["selectors"], completed["selectors"]) + self.assertEqual(reopened_receipt["items"], completed["items"]) + + def test_partial_connector_failure_stays_retryable_and_completes_after_restart(self): + runtime = self._runtime() + finished = self._train(runtime) + real_delete = runtime.connectors.candidates.delete + attempts = 0 + + def fail_once(candidate_id, context): + nonlocal attempts + attempts += 1 + if attempts == 1: + raise ConnectorError( + "candidate_delete_unavailable", + "Candidate storage is temporarily unavailable.", + category="unavailable", + retryable=True, + ) + return real_delete(candidate_id, context) + + with mock.patch.object( + runtime.connectors.candidates, "delete", side_effect=fail_once + ): + requested = runtime.request_deletion( + {"tasksetDigest": finished["tasksetDigest"]}, + requested_by="privacy-operator", + reason="source taskset deleted", + idempotency_key="delete-with-retry", + ) + pending = requested + + self.assertEqual(pending["state"], "pending") + self.assertTrue(pending["pendingItems"]) + self.assertTrue(all(item["retryable"] for item in pending["pendingItems"])) + self.assertNotIn("completedAt", pending) + runtime.close() + + reopened = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": FixtureBackend()}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + completed = self._wait_for_deletion(reopened, requested["id"]) + + self.assertEqual(completed["state"], "completed") + self.assertEqual(completed["pendingItems"], []) + self.assertGreaterEqual(attempts, 1) + + def test_pending_revocation_cancels_racing_work_and_blocks_new_uses(self): + started = self._control("deletion-job-started") + release = self._control("deletion-job-release") + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={ + "blocking": BlockingBackend(started, release), + "fixture": FixtureBackend(), + }, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(release.unlink, missing_ok=True) + self.addCleanup(runtime.close) + running = runtime.submit( + backend="blocking", max_updates=2, max_seconds=30, max_credits=2 + ) + digest, upload = taskset_upload(self.taskset_dir) + deadline = time.monotonic() + 5 + while not started.exists() and time.monotonic() < deadline: + time.sleep(0.01) + self.assertTrue(started.exists()) + + deletion = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="consent withdrawn during training", + idempotency_key="delete-racing-training", + ) + + with self.assertRaises(RuntimeFailure) as blocked: + runtime.submit( + backend="fixture", max_updates=1, max_seconds=5, max_credits=1 + ) + self.assertEqual(blocked.exception.code, "deletion_revoked") + for operation in ( + lambda: runtime.evaluate(self._evaluation()), + lambda: runtime.promote( + "candidate_" + "0" * 64, + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="racing-promotion", + ), + ): + with self.subTest(operation=operation): + with self.assertRaises(RuntimeFailure): + operation() + with self.assertRaises(RuntimeFailure) as upload_blocked: + runtime.register_taskset( + digest, + upload, + idempotency_key="upload-after-revocation", + ) + self.assertEqual(upload_blocked.exception.code, "deletion_revoked") + release.write_text("release", encoding="utf-8") + self.assertEqual(self._wait_for_job(runtime, running["id"])["state"], "cancelled") + receipt = self._wait_for_deletion(runtime, deletion["id"]) + self.assertEqual(receipt["state"], "completed") + + def test_withdrawal_cancels_in_flight_inference_before_receipt_completion(self): + started = self._control("deletion-inference-started") + release = self._control("deletion-inference-release") + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": BlockingInferenceBackend(started, release)}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(release.write_text, "release", encoding="utf-8") + self.addCleanup(runtime.close) + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-inference-withdrawal", + ) + failures = [] + + def evaluate(): + try: + runtime.evaluate(self._evaluation()) + except Exception as failure: + failures.append(failure) + + thread = threading.Thread(target=evaluate) + thread.start() + deadline = time.monotonic() + 5 + while not started.exists() and time.monotonic() < deadline: + time.sleep(0.01) + self.assertTrue(started.exists()) + + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="consent withdrawn during inference", + idempotency_key="delete-in-flight-inference", + ) + thread.join(5) + + self.assertFalse(thread.is_alive()) + receipt = self._wait_for_deletion(runtime, receipt["id"]) + self.assertEqual(receipt["state"], "completed") + self.assertEqual(len(failures), 1) + self.assertIsInstance(failures[0], RuntimeFailure) + self.assertEqual(failures[0].code, "deletion_revoked") + + def test_withdrawal_waits_for_in_flight_retry_inventory_before_completion(self): + started = self._control("retry-withdrawal-started") + release = self._control("retry-withdrawal-release") + calls = self._control("retry-withdrawal-calls") + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={ + "fixture": CountingBlockingInferenceBackend(started, release, calls) + }, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(release.write_text, "release", encoding="utf-8") + self.addCleanup(runtime.close) + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-retry-withdrawal", + ) + failures = [] + + def evaluate(): + try: + runtime.evaluate( + self._evaluation(), + idempotency_key="withdrawn-inference-retry", + ) + except Exception as failure: + failures.append(failure) + + thread = threading.Thread(target=evaluate) + thread.start() + deadline = time.monotonic() + 5 + while not started.exists() and time.monotonic() < deadline: + time.sleep(0.01) + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="withdraw during durable retry", + idempotency_key="delete-during-durable-retry", + ) + self.assertEqual(receipt["state"], "pending") + release.write_text("release", encoding="utf-8") + thread.join(5) + completed = self._wait_for_deletion(runtime, receipt["id"]) + + self.assertEqual(completed["state"], "completed") + self.assertEqual(len(failures), 1) + self.assertEqual(failures[0].code, "deletion_revoked") + remaining = list((self.state_dir / "inference" / ".idempotency").iterdir()) + self.assertEqual(remaining, []) + + def test_candidate_inventory_is_durable_before_candidate_bytes(self): + runtime = self._runtime() + real_put = runtime.connectors.privacy.put_copy + + def fail_candidate(record, context): + if record["kind"] == "candidate": + raise ConnectorError( + "privacy_unavailable", + "Privacy inventory is unavailable.", + category="unavailable", + retryable=True, + ) + return real_put(record, context) + + with mock.patch.object( + runtime.connectors.privacy, + "put_copy", + side_effect=fail_candidate, + ): + finished = self._wait_for_job( + runtime, + runtime.submit( + backend="fixture", + max_updates=2, + max_seconds=5, + max_credits=2, + )["id"], + ) + + self.assertEqual(finished["state"], "failed") + self.assertEqual( + list((self.state_dir / "candidates").glob("candidate_*")), + [], + ) + + def test_taskset_inventory_is_reserved_before_uploaded_bytes(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + + with mock.patch.object( + runtime.connectors.privacy, + "put_copy", + side_effect=ConnectorError( + "privacy_unavailable", + "Privacy inventory is unavailable.", + category="unavailable", + retryable=True, + ), + ), self.assertRaises(RuntimeFailure) as failed: + runtime.register_taskset( + digest, + upload, + idempotency_key="upload-without-inventory", + ) + + self.assertEqual(failed.exception.code, "privacy_unavailable") + self.assertFalse((self.state_dir / "tasksets" / digest).exists()) + + def test_inventory_failure_does_not_delete_a_preexisting_uploaded_taskset(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + runtime.register_taskset(digest, upload, idempotency_key="first-upload") + copy_id = next( + item["id"] + for item in runtime.disclose({"tasksetDigest": f"sha256:{digest}"})["items"] + if item["kind"] == "taskset" + ) + (self.state_dir / "privacy" / "copies" / f"{copy_id}.json").unlink() + + with mock.patch.object( + runtime.connectors.privacy, + "put_copy", + side_effect=ConnectorError( + "privacy_unavailable", + "Privacy inventory is unavailable.", + category="unavailable", + retryable=True, + ), + ), self.assertRaises(RuntimeFailure): + runtime.register_taskset( + digest, + upload, + idempotency_key="retry-existing-upload", + ) + + self.assertTrue((self.state_dir / "tasksets" / digest).is_dir()) + + def test_successful_taskset_retry_reactivates_failed_registration_inventory(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + real_register = runtime.connectors.tasksets.register + attempts = 0 + + def fail_once(bundle, context): + nonlocal attempts + attempts += 1 + if attempts == 1: + raise ConnectorError( + "taskset_unavailable", + "Taskset storage is unavailable.", + category="unavailable", + retryable=True, + ) + return real_register(bundle, context) + + with mock.patch.object( + runtime.connectors.tasksets, + "register", + side_effect=fail_once, + ): + with self.assertRaises(RuntimeFailure): + runtime.register_taskset( + digest, + upload, + idempotency_key="failed-taskset-registration", + ) + runtime.register_taskset( + digest, + upload, + idempotency_key="successful-taskset-registration", + ) + + disclosed = runtime.disclose({"tasksetDigest": f"sha256:{digest}"}) + uploaded = next( + item + for item in disclosed["items"] + if item["storage"]["reference"].startswith("uploaded-taskset:") + ) + self.assertEqual(uploaded["useState"], "active") + + def test_legacy_failed_registration_cannot_delete_verified_taskset(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + runtime.register_taskset(digest, upload, idempotency_key="verified-upload") + current = runtime.disclose({"tasksetDigest": f"sha256:{digest}"})["items"][0] + legacy = { + **current, + "id": "copy_" + + __import__("hashlib") + .sha256(f"taskset\0uploaded:taskset:{digest}".encode()) + .hexdigest()[:32], + "runtimeCopyId": f"uploaded:taskset:{digest}", + "sourceTrainingExportId": "export_stale_failed_upload", + "useState": "deleted", + "retention": { + "mode": "registration_failed", + "deletedAt": "2026-11-20T00:00:00Z", + }, + } + context = runtime._operation_context( + "legacy-registration", + max_seconds=30, + owner=RuntimeOwner("service", "test"), + ) + runtime.connectors.privacy.put_copy(legacy, context) + runtime.close() + + reopened = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(reopened.close) + receipt = reopened.request_deletion( + {"exportId": "export_stale_failed_upload"}, + requested_by="privacy-operator", + reason="withdraw stale failed upload", + idempotency_key="delete-stale-failed-upload", + ) + + self.assertEqual(receipt["state"], "completed") + self.assertTrue((self.state_dir / "tasksets" / digest).is_dir()) + retained = next(item for item in receipt["items"] if item["runtimeCopyId"] == legacy["id"]) + self.assertEqual(retained["state"], "retained") + + def test_mounting_an_uploaded_taskset_reconciles_operator_retention(self): + digest, upload = taskset_upload(self.taskset_dir) + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + runtime.register_taskset(digest, upload, idempotency_key="upload-before-mount") + runtime.close() + + mounted = LocalRuntime( + self.state_dir, + self.taskset_dir, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(mounted.close) + receipt = mounted.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="training consent withdrawn", + idempotency_key="delete-mounted-upload", + ) + + self.assertEqual(receipt["state"], "completed") + taskset_item = next( + item + for item in receipt["items"] + if item["kind"] == "taskset" and item["state"] == "retained" + ) + self.assertEqual(taskset_item["state"], "retained") + self.assertEqual( + taskset_item["retention"]["mode"], + "operator_managed", + ) + self.assertTrue((self.taskset_dir / "taskset.json").is_file()) + self.assertFalse((self.state_dir / "tasksets" / digest).exists()) + + def test_mount_after_completed_withdrawal_is_retained_and_never_reactivated(self): + runtime = LocalRuntime( + self.state_dir, + None, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="training consent withdrawn before mount", + idempotency_key="delete-before-mount", + ) + self.assertEqual(receipt["state"], "completed") + runtime.close() + + mounted = LocalRuntime( + self.state_dir, + self.taskset_dir, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(mounted.close) + disclosed = mounted.disclose({"exportId": EXPORT_ID}) + mounted_copy = next( + item + for item in disclosed["items"] + if item["storage"]["reference"].startswith("mounted-taskset:") + ) + + self.assertEqual(mounted_copy["useState"], "retained") + self.assertEqual(mounted_copy["retention"]["mode"], "operator_managed") + with self.assertRaises(RuntimeFailure) as blocked: + mounted.submit( + backend="fixture", + max_updates=1, + max_seconds=5, + max_credits=1, + ) + self.assertEqual(blocked.exception.code, "deletion_revoked") + + def test_concurrent_inference_retries_share_one_backend_execution(self): + started = self._control("retry-inference-started") + release = self._control("retry-inference-release") + calls = self._control("retry-inference-calls") + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={ + "fixture": CountingBlockingInferenceBackend(started, release, calls) + }, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(release.write_text, "release", encoding="utf-8") + self.addCleanup(runtime.close) + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-retry", + ) + responses = [] + failures = [] + + def evaluate(): + try: + responses.append( + runtime.evaluate( + self._evaluation(), + idempotency_key="same-inference", + ) + ) + except Exception as failure: + failures.append(failure) + + first = threading.Thread(target=evaluate) + second = threading.Thread(target=evaluate) + first.start() + deadline = time.monotonic() + 5 + while not started.exists() and time.monotonic() < deadline: + time.sleep(0.01) + self.assertTrue(started.exists()) + second.start() + time.sleep(0.05) + release.write_text("release", encoding="utf-8") + first.join(5) + second.join(5) + + self.assertFalse(failures) + self.assertEqual(len(responses), 2) + self.assertEqual(responses[0], responses[1]) + self.assertEqual(calls.read_text(encoding="utf-8").splitlines(), ["predict"]) + self.assertEqual(runtime._inference_idempotency_locks, {}) + + def test_inference_work_directory_failure_releases_retry_and_cancellation_state(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-work-failure", + ) + original_mkdir = Path.mkdir + + def fail_inference_work(path, *args, **kwargs): + if path.parent.name == "work" and path.name.startswith("inference_"): + raise OSError("disk unavailable") + return original_mkdir(path, *args, **kwargs) + + with mock.patch.object(Path, "mkdir", new=fail_inference_work): + with self.assertRaises(RuntimeFailure) as failed: + runtime.evaluate( + self._evaluation(), + idempotency_key="work-directory-failure", + ) + + self.assertEqual(failed.exception.code, "work_directory_unavailable") + self.assertEqual(runtime._inference_idempotency_locks, {}) + self.assertEqual(runtime._inference_cancellations, {}) + self.assertFalse( + any( + item["kind"] == "inference" + and item["useState"] == "registering" + for item in runtime.disclose({"exportId": EXPORT_ID})["items"] + ) + ) + + def test_inference_artifact_failure_removes_work_directory(self): + runtime = self._runtime() + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-before-artifact-failure", + ) + + with mock.patch.object( + runtime.local_connectors.candidates, + "artifact_path", + side_effect=ConnectorError( + "candidate_not_found", + "The candidate artifact was not found.", + category="not_found", + ), + ), self.assertRaises(RuntimeFailure): + runtime.evaluate( + self._evaluation(), + idempotency_key="artifact-failure", + ) + + self.assertEqual( + list((self.state_dir / "work").glob("inference_*")), + [], + ) + self.assertEqual(runtime._inference_idempotency_locks, {}) + self.assertEqual(runtime._inference_cancellations, {}) + + def test_invalid_inference_response_removes_uninventoried_retry_result(self): + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": InvalidInferenceResultBackend()}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-invalid-response", + ) + + with self.assertRaises(RuntimeFailure) as invalid: + runtime.evaluate( + self._evaluation(), + idempotency_key="invalid-response-retry", + ) + + self.assertEqual(invalid.exception.code, "inference_failed") + self.assertEqual( + list((self.state_dir / "inference" / ".idempotency").iterdir()), + [], + ) + + def test_failed_inference_cleanup_remains_inventoried_for_deletion_retry(self): + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": InvalidInferenceResultBackend()}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + finished = self._train(runtime) + runtime.promote( + finished["candidateId"], + promoted_by="privacy-operator", + expected_generation=0, + idempotency_key="promote-cleanup-failure", + ) + connector = runtime.inference_connectors["fixture"] + real_delete = connector.delete + failure = ConnectorError( + "inference_result_delete_unavailable", + "Retry storage is unavailable.", + category="unavailable", + retryable=True, + ) + + with mock.patch.object(connector, "delete", side_effect=failure): + with self.assertRaises(RuntimeFailure): + runtime.evaluate( + self._evaluation(), + idempotency_key="cleanup-failure-retry", + ) + disclosed = runtime.disclose({"exportId": EXPORT_ID}) + retry_copy = next( + item for item in disclosed["items"] if item["kind"] == "inference" + ) + self.assertEqual(retry_copy["useState"], "registering") + receipt = runtime.request_deletion( + {"exportId": EXPORT_ID}, + requested_by="privacy-operator", + reason="delete failed retry cleanup", + idempotency_key="delete-failed-retry-cleanup", + ) + self.assertEqual(receipt["state"], "pending") + + self.assertTrue(real_delete(retry_copy["storage"]["reference"], runtime._operation_context( + "cleanup-retry", + max_seconds=30, + owner=RuntimeOwner("service", "test"), + ))) + completed = self._wait_for_deletion(runtime, receipt["id"]) + self.assertEqual(completed["state"], "completed") + + def test_explicit_missing_mounted_taskset_fails_startup(self): + missing = self._control("missing-taskset") + + with self.assertRaises(TasksetError): + LocalRuntime( + self.state_dir, + missing, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + + def _runtime(self): + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={"fixture": FixtureBackend()}, + environment="test", + scope=RuntimeScope("tenant-one", "github/example"), + ) + self.addCleanup(runtime.close) + return runtime + + def _control(self, name): + return self.state_dir.with_name(f"{self.state_dir.name}-{name}") + + @staticmethod + def _aggregate_digest(values): + import hashlib + + return "sha256:" + hashlib.sha256( + json.dumps(values, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + def _train(self, runtime): + job = runtime.submit( + backend="fixture", max_updates=8, max_seconds=5, max_credits=8 + ) + finished = self._wait_for_job(runtime, job["id"]) + self.assertEqual(finished["state"], "succeeded") + return finished + + def _evaluation(self): + task = make_task(99, split="test", answer=True) + return { + "state": task["state"], + "images": [ + "data:image/png;base64," + base64.b64encode(PNG).decode("ascii") + ], + "questions": {RULE: task["questions"][RULE]}, + } + + def _wait_for_job(self, runtime, job_id): + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + job = runtime.job(job_id) + if job["state"] in {"succeeded", "failed", "cancelled"}: + return job + time.sleep(0.01) + self.fail(f"job {job_id} did not finish") + + def _wait_for_deletion(self, runtime, deletion_id, *, state="completed"): + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + receipt = runtime.deletion(deletion_id) + if receipt["state"] == state: + return receipt + if receipt["state"] == "failed": + self.fail(f"deletion {deletion_id} failed: {receipt}") + time.sleep(0.01) + self.fail(f"deletion {deletion_id} did not reach {state}") + + def _serve(self, runtime): + server = RuntimeHTTPServer(("127.0.0.1", 0), runtime, TOKEN) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + self.addCleanup(thread.join, 1) + self.addCleanup(server.server_close) + self.addCleanup(server.shutdown) + return server, thread, f"http://127.0.0.1:{server.server_address[1]}" + + def _http_json(self, http_request): + with request.urlopen(http_request) as response: + return json.load(response) + + def _http_error(self, http_request): + with self.assertRaises(error.HTTPError) as failed: + request.urlopen(http_request) + with failed.exception as response: + return response.code, json.load(response) + + def _assert_content_free(self, value): + serialized = json.dumps(value, sort_keys=True) + self.assertNotIn("Put a visible label", serialized) + self.assertNotIn("data:image", serialized) + self.assertNotIn(base64.b64encode(PNG).decode("ascii"), serialized) + for forbidden in ("questions", "labels", "tasks", "images", "evidence"): + self.assertNotIn(forbidden, self._nested_keys(value)) + + def _nested_keys(self, value): + if isinstance(value, dict): + return set(value).union( + *(self._nested_keys(item) for item in value.values()) + ) + if isinstance(value, list): + return set().union(*(self._nested_keys(item) for item in value)) + return set() + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/learning/src/decision/tasks.mjs b/packages/learning/src/decision/tasks.mjs index bbf238ea..3706969a 100644 --- a/packages/learning/src/decision/tasks.mjs +++ b/packages/learning/src/decision/tasks.mjs @@ -208,10 +208,7 @@ export function planDecisionTasks(bundle, builds) { .filter((record) => record.kind === "grant") .map((record) => record.id) .sort(compare), - reviewDigests: manifest.records - .filter((record) => record.kind === "memory") - .map((record) => record.digest) - .sort(compare), + reviewDigests: bundle.memories.map((memory) => memory.reviewDigest).sort(compare), evidenceDigests: manifest.records .filter((record) => ["collection", "episode", "event"].includes(record.kind)) .map((record) => record.digest) diff --git a/packages/learning/src/decision/taskset.mjs b/packages/learning/src/decision/taskset.mjs index e88a4dbb..7d2be7f4 100644 --- a/packages/learning/src/decision/taskset.mjs +++ b/packages/learning/src/decision/taskset.mjs @@ -3,6 +3,7 @@ import { mkdir, readFile, realpath, rm, stat, writeFile } from "node:fs/promises import path from "node:path" import { LearningError } from "../errors.mjs" import { oneLine } from "../recall.mjs" +import { canonicalJson } from "../canonical-json.mjs" import { brandCheckQuestions } from "./questions.mjs" export const TASK_SCHEMA = "replica.decision-task/v1" @@ -111,7 +112,12 @@ export async function writeDecisionTaskset( schemaVersion: TASKSET_SCHEMA, exportId: plan.exportId, scope: plan.scope, - provenance: plan.provenance, + provenance: { + ...plan.provenance, + evidenceDigest: `sha256:${createHash("sha256") + .update(canonicalJson(plan.provenance.evidenceDigests)) + .digest("hex")}`, + }, retention: { mode: "delete_on_withdrawal" }, createdAt, learnings: rules.length, diff --git a/packages/learning/test/decision.test.mjs b/packages/learning/test/decision.test.mjs index 31dca7e6..b22b55ba 100644 --- a/packages/learning/test/decision.test.mjs +++ b/packages/learning/test/decision.test.mjs @@ -452,6 +452,24 @@ describe("replica-learning decision", () => { tasksetManifest.provenance.grantIds, bundle.grants.map((grant) => grant.id).sort(), ) + assert.deepEqual(tasksetManifest.provenance.purposes, ["training"]) + assert.deepEqual( + tasksetManifest.provenance.reviewDigests, + bundle.memories.map((memory) => memory.reviewDigest).sort(), + ) + assert.deepEqual( + tasksetManifest.provenance.evidenceDigests, + bundle.manifest.records + .filter((record) => ["collection", "episode", "event"].includes(record.kind)) + .map((record) => record.digest) + .sort(), + ) + assert.equal( + tasksetManifest.provenance.evidenceDigest, + `sha256:${createHash("sha256") + .update(canonicalJson(tasksetManifest.provenance.evidenceDigests)) + .digest("hex")}`, + ) assert.deepEqual(tasksetManifest.retention, { mode: "delete_on_withdrawal" }) })