diff --git a/infrastructure/README.md b/infrastructure/README.md index 023ca851..60a2d57d 100644 --- a/infrastructure/README.md +++ b/infrastructure/README.md @@ -649,6 +649,41 @@ Reconcile the restored resources with a reviewed Terraform configuration and imp To move an existing installation, pause writes and workers, export PostgreSQL, copy artifacts with an object manifest, import both into the new installation, and verify counts/checksums and a representative operation before switching traffic. Keep source backups and a rollback window. No automatic migration between local and Azure installations is performed here. +## Optional Azure decision runtime + +Set `enable_decision_runtime = true` only after the customer has selected an Azure region with Container Apps serverless GPU support, obtained quota, and provisioned a manual scale-to-zero GPU Job plus a fixed-template control broker. Supply the job's resource ID, name, and managed identity object ID; the broker's HTTPS URL, resource ID, and custom invocation role; and the customer inference endpoint resource ID and custom inference-only role. The normal Hub PostgreSQL database stores decision job, lease, cancellation, idempotency, and promotion state; no second database is created. + +The optional path adds: + +- a private `decision-runtime` container in the existing keyless, versioned storage account; +- separate runtime and inference user-assigned identities, plus the supplied training job identity; +- coordinates for one customer-provisioned manual Container Apps Job on a T4 or A100 consumption GPU profile whose minimum count is zero; +- a module-owned, container-scoped Blob role that allows runtime and training reads, adds, and writes but no deletion, a broker-invocation grant for runtime, and the customer-supplied endpoint role for inference. + +Terraform exposes the decision identity and job coordinates only as non-secret outputs. Hub keeps only its existing identity and environment; deploy the connector bundle on separate customer-operated runtime hosts and attach the decision identities there. + +The pinned AzureRM provider does not yet accept Azure's serverless GPU workload profile types, so this module does not claim to provision the GPU job. Create it separately with a reviewed Azure deployment using a manual trigger, one execution replica, no automatic retry, a hard execution timeout, and zero idle GPU capacity. Put a broker in front of the job that accepts only the bounded Replica payload and starts or cancels that fixed template. The broker must reject image, command, environment, identity, registry, secret, and resource overrides. Do not grant Hub Container Apps Jobs Operator. Configure the connector's required cancellation-recovery hook to durably enqueue the execution ID before each direct cancellation attempt; a customer worker retries that cleanup after cancellation, deadline, or status loss cannot reach Azure. The job's immutable training image owns the model and GPU implementation. It must accept the connector's operation, scope, taskset, deadline, `maxUpdates`, `maxCredits`, and budget mode payload; write immutable adapter and report artifacts; stop before the deadline or hard credit limit; and return measured usage and cost when Azure supplies them. + +The storage account keeps normal 14-day soft-delete recovery and permits an authorized workflow to permanently delete soft-deleted versions. A decision-container lifecycle rule removes superseded versions after `decision_artifact_version_retention_days`, default 30. Terraform creates one exact runtime/training data role with Blob read, add, and write actions and no delete or permanent-delete actions. Use a separate short-lived withdrawal identity to recheck the affected export, permanently delete the current taskset or candidate Blobs and every version, and write the result to the customer's audit system. + +The inference role is intentionally not created here because its exact actions depend on the customer endpoint service. Define a custom role with only readiness and prediction actions, assign its ID through `decision_inference_role_definition_id`, and verify it cannot create, update, deploy, train, list credentials, or change endpoint access. The inference connector checks the expected promotion generation before every prediction and reports cold start separately. + +Exact authorized cloud walkthrough still required before production: + +1. Confirm the selected region advertises the chosen `Consumption-GPU-*` profile and grant enough GPU quota for the intended maximum concurrent jobs. +2. Build the reviewed training worker, pin its ACR image digest, and prove its result schema, cancellation, deadline, and hard-credit behavior against a non-production taskset. +3. Provision the manual Container Apps GPU Job with minimum GPU capacity zero, one execution replica, no automatic retry, and a hard timeout. Record its resource ID, name, and managed identity object ID. +4. Deploy the fixed-template broker, create an invocation-only role, and prove that its caller cannot read job secrets or override the image, command, environment, identity, registry, or resources. +5. Apply this Terraform module with the job and broker coordinates. Confirm the Blob container has no public or shared-key access, the module-owned data role omits delete and permanent-delete actions, runtime and training cannot delete a Blob or version, and each identity has only the documented grants. +6. Create the dedicated decision-runtime database role, then run `initialize_azure_postgres` through the existing migration role. Verify the `replica_decision_runtime` schema is inaccessible to Hub's application role, then exercise create, retry, lease expiry, cancellation, candidate persistence, promotion conflict, rollback, restart recovery, and consent-withdrawal deletion. +7. Upload a private taskset through managed identity, train one bounded candidate, and compare the job's reported usage and cost with Azure Monitor and Cost Management. +8. Deploy or select the customer inference endpoint, apply the custom invocation-only role, measure cold-start readiness, and prove a stale expected generation cannot score. +9. Exercise the dedicated withdrawal workflow and verify that the current taskset or candidate Blob, its soft-deleted copy, and every prior version are permanently unavailable while unrelated data remains. +10. Inspect Log Analytics and the configured OpenTelemetry/Pino sink to confirm signals contain operation IDs, bounded usage, cost, status, and hashed scope only—never screenshots, prompts, learnings, artifacts, tokens, or credential material. +11. Test retention, Blob version recovery, PostgreSQL point-in-time restore, identity revocation, endpoint outage, GPU quota exhaustion, and owned-resource teardown. + +Local Terraform and mocked connector tests do not perform this walkthrough and do not claim a live Azure deployment. + ## Installation removal Replica's setup wizard and deployment script do not remove installations. Earlier installer versions could leave private `.removal.json` files beside configurations. The wizard ignores these files and leaves them unchanged; they may contain installation details, so keep them private or delete them yourself after review. diff --git a/infrastructure/azure/application.tf b/infrastructure/azure/application.tf index d3fd8b00..1ca512f8 100644 --- a/infrastructure/azure/application.tf +++ b/infrastructure/azure/application.tf @@ -38,7 +38,6 @@ locals { try(trimspace(local.authentication_environment["HUB_APPLICATION_ORIGIN"]), "") != "" || contains(keys(var.runtime_secret_references), "HUB_APPLICATION_ORIGIN") ) - runtime_environment = merge({ NODE_ENV = "production" RUNTIME_MODE = "combined" diff --git a/infrastructure/azure/outputs.tf b/infrastructure/azure/outputs.tf index ef02b51b..d6bc0d83 100644 --- a/infrastructure/azure/outputs.tf +++ b/infrastructure/azure/outputs.tf @@ -1,25 +1,30 @@ output "installation" { description = "Non-secret release coordinates; no credentials are emitted." value = { - name = var.name - owner = var.owner - subscription_id = var.subscription_id - resource_group = azurerm_resource_group.hub.name - resource_group_id = azurerm_resource_group.hub.id - app_id = var.deploy_hub ? azurerm_container_app.hub[0].id : null - app_name = "${var.name}-hub" - setup_job_name = azurerm_container_app_job.setup.name - storage_account = azurerm_storage_account.artifacts.name - storage_account_id = azurerm_storage_account.artifacts.id - release_container = azurerm_storage_container.releases.name - release_blob = azurerm_storage_blob.release_lock.name - environment_id = azurerm_container_app_environment.hub.id - postgres_server_id = azurerm_postgresql_flexible_server.database.id - postgres_host = azurerm_postgresql_flexible_server.database.fqdn - managed_identity_id = azurerm_user_assigned_identity.hub.id - managed_identity_client_id = azurerm_user_assigned_identity.hub.client_id - setup_identity_id = azurerm_user_assigned_identity.setup.id - app_created = var.deploy_hub - url = var.deploy_hub ? "https://${azurerm_container_app.hub[0].ingress[0].fqdn}" : null + name = var.name + owner = var.owner + subscription_id = var.subscription_id + resource_group = azurerm_resource_group.hub.name + resource_group_id = azurerm_resource_group.hub.id + app_id = var.deploy_hub ? azurerm_container_app.hub[0].id : null + app_name = "${var.name}-hub" + setup_job_name = azurerm_container_app_job.setup.name + storage_account = azurerm_storage_account.artifacts.name + storage_account_id = azurerm_storage_account.artifacts.id + release_container = azurerm_storage_container.releases.name + release_blob = azurerm_storage_blob.release_lock.name + environment_id = azurerm_container_app_environment.hub.id + postgres_server_id = azurerm_postgresql_flexible_server.database.id + postgres_host = azurerm_postgresql_flexible_server.database.fqdn + managed_identity_id = azurerm_user_assigned_identity.hub.id + managed_identity_client_id = azurerm_user_assigned_identity.hub.client_id + setup_identity_id = azurerm_user_assigned_identity.setup.id + decision_storage_container = var.enable_decision_runtime ? azurerm_storage_container.decision[0].name : null + decision_training_job_id = var.enable_decision_runtime ? var.decision_training_job_id : null + decision_training_job_name = var.enable_decision_runtime ? var.decision_training_job_name : null + decision_runtime_client_id = var.enable_decision_runtime ? azurerm_user_assigned_identity.decision_runtime[0].client_id : null + decision_inference_client_id = var.enable_decision_runtime ? azurerm_user_assigned_identity.decision_inference[0].client_id : null + app_created = var.deploy_hub + url = var.deploy_hub ? "https://${azurerm_container_app.hub[0].ingress[0].fqdn}" : null } } diff --git a/infrastructure/azure/roles.tf b/infrastructure/azure/roles.tf index 8d610f1f..0cb717ab 100644 --- a/infrastructure/azure/roles.tf +++ b/infrastructure/azure/roles.tf @@ -19,6 +19,39 @@ resource "azurerm_user_assigned_identity" "setup" { tags = local.tags } +resource "azurerm_user_assigned_identity" "decision_runtime" { + count = var.enable_decision_runtime ? 1 : 0 + name = "${var.name}-decision-runtime" + resource_group_name = azurerm_resource_group.hub.name + location = var.location + tags = local.tags +} + +resource "azurerm_user_assigned_identity" "decision_inference" { + count = var.enable_decision_runtime ? 1 : 0 + name = "${var.name}-decision-inference" + resource_group_name = azurerm_resource_group.hub.name + location = var.location + tags = local.tags +} + +resource "azurerm_role_definition" "decision_artifact_writer" { + count = var.enable_decision_runtime ? 1 : 0 + name = "${var.name}-decision-artifact-writer" + scope = azurerm_storage_account.artifacts.id + description = "Read, create, and write Replica decision artifacts without Blob deletion." + + permissions { + data_actions = [ + "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action", + "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write", + ] + } + + assignable_scopes = [azurerm_storage_account.artifacts.id] +} + resource "azurerm_role_assignment" "runtime_secrets" { for_each = local.runtime_secret_scopes scope = each.value @@ -51,3 +84,31 @@ resource "azurerm_role_assignment" "setup_image_pull" { role_definition_name = "AcrPull" principal_id = azurerm_user_assigned_identity.setup.principal_id } + +resource "azurerm_role_assignment" "decision_runtime_artifacts" { + count = var.enable_decision_runtime ? 1 : 0 + scope = "${azurerm_storage_account.artifacts.id}/blobServices/default/containers/${azurerm_storage_container.decision[0].name}" + role_definition_id = azurerm_role_definition.decision_artifact_writer[0].role_definition_resource_id + principal_id = azurerm_user_assigned_identity.decision_runtime[0].principal_id +} + +resource "azurerm_role_assignment" "decision_training_artifacts" { + count = var.enable_decision_runtime ? 1 : 0 + scope = "${azurerm_storage_account.artifacts.id}/blobServices/default/containers/${azurerm_storage_container.decision[0].name}" + role_definition_id = azurerm_role_definition.decision_artifact_writer[0].role_definition_resource_id + principal_id = var.decision_training_principal_id +} + +resource "azurerm_role_assignment" "decision_training_control" { + count = var.enable_decision_runtime ? 1 : 0 + scope = var.decision_training_control_scope + role_definition_id = var.decision_training_control_role_definition_id + principal_id = azurerm_user_assigned_identity.decision_runtime[0].principal_id +} + +resource "azurerm_role_assignment" "decision_inference" { + count = var.enable_decision_runtime ? 1 : 0 + scope = var.decision_inference_scope + role_definition_id = var.decision_inference_role_definition_id + principal_id = azurerm_user_assigned_identity.decision_inference[0].principal_id +} diff --git a/infrastructure/azure/storage.tf b/infrastructure/azure/storage.tf index 29e0de14..8da0ee5c 100644 --- a/infrastructure/azure/storage.tf +++ b/infrastructure/azure/storage.tf @@ -14,14 +14,36 @@ resource "azurerm_storage_account" "artifacts" { blob_properties { versioning_enabled = true delete_retention_policy { - days = 14 + days = 14 + permanent_delete_enabled = var.enable_decision_runtime } + container_delete_retention_policy { days = 14 } } } +resource "azurerm_storage_management_policy" "artifacts" { + storage_account_id = azurerm_storage_account.artifacts.id + + rule { + name = "expire-decision-artifact-versions" + enabled = true + + filters { + prefix_match = ["decision-runtime/"] + blob_types = ["blockBlob"] + } + + actions { + version { + delete_after_days_since_creation = var.decision_artifact_version_retention_days + } + } + } +} + resource "azurerm_storage_container" "artifacts" { name = "artifacts" storage_account_id = azurerm_storage_account.artifacts.id @@ -36,6 +58,14 @@ resource "azurerm_storage_container" "releases" { depends_on = [azurerm_role_assignment.deployment_storage] } +resource "azurerm_storage_container" "decision" { + count = var.enable_decision_runtime ? 1 : 0 + name = "decision-runtime" + storage_account_id = azurerm_storage_account.artifacts.id + container_access_type = "private" + depends_on = [azurerm_role_assignment.deployment_storage] +} + resource "azurerm_storage_blob" "release_lock" { name = "deployment.lock" storage_account_name = azurerm_storage_account.artifacts.name diff --git a/infrastructure/azure/tests/deployment.tftest.hcl b/infrastructure/azure/tests/deployment.tftest.hcl index 05abee56..1010d622 100644 --- a/infrastructure/azure/tests/deployment.tftest.hcl +++ b/infrastructure/azure/tests/deployment.tftest.hcl @@ -15,6 +15,11 @@ mock_provider "azurerm" { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.App/containerApps/test-hub-hub" } } + mock_resource "azurerm_container_app_job" { + defaults = { + id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.App/jobs/test-job" + } + } mock_resource "azurerm_virtual_network" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.Network/virtualNetworks/test" @@ -48,6 +53,11 @@ mock_provider "azurerm" { principal_id = "00000000-0000-0000-0000-000000000002" } } + mock_resource "azurerm_role_definition" { + defaults = { + role_definition_resource_id = "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00000000-0000-0000-0000-000000000095" + } + } mock_resource "azurerm_log_analytics_workspace" { defaults = { id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.OperationalInsights/workspaces/test" @@ -97,6 +107,26 @@ override_resource { } } +override_resource { + target = azurerm_user_assigned_identity.decision_runtime + override_during = plan + values = { + id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.ManagedIdentity/userAssignedIdentities/decision-runtime" + client_id = "00000000-0000-0000-0000-000000000006" + principal_id = "00000000-0000-0000-0000-000000000006" + } +} + +override_resource { + target = azurerm_user_assigned_identity.decision_inference + override_during = plan + values = { + id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/test-hub/providers/Microsoft.ManagedIdentity/userAssignedIdentities/decision-inference" + client_id = "00000000-0000-0000-0000-000000000007" + principal_id = "00000000-0000-0000-0000-000000000007" + } +} + variables { subscription_id = "00000000-0000-0000-0000-000000000000" name = "test-hub" @@ -210,6 +240,7 @@ run "durable_state_requires_private_authenticated_access" { ) error_message = "PostgreSQL must be private, TLS-only, and retain recoverable backups." } + assert { condition = contains(coalesce(azurerm_subnet.database.service_endpoints, []), "Microsoft.Storage") error_message = "The database subnet must retain Microsoft.Storage for PostgreSQL WAL archival." @@ -218,6 +249,7 @@ run "durable_state_requires_private_authenticated_access" { condition = ( !azurerm_storage_account.artifacts.allow_nested_items_to_be_public && !azurerm_storage_account.artifacts.shared_access_key_enabled && + !azurerm_storage_account.artifacts.blob_properties[0].delete_retention_policy[0].permanent_delete_enabled && azurerm_storage_container.artifacts.container_access_type == "private" && azurerm_storage_account.artifacts.blob_properties[0].versioning_enabled && endswith(azurerm_role_assignment.artifacts.scope, "/blobServices/default/containers/artifacts") && @@ -228,6 +260,71 @@ run "durable_state_requires_private_authenticated_access" { } } +run "decision_runtime_uses_private_data_scale_to_zero_gpu_and_split_identities" { + command = plan + + variables { + deploy_hub = true + enable_decision_runtime = true + decision_training_job_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/customer/providers/Microsoft.App/jobs/customer-decision-training" + decision_training_job_name = "customer-decision-training" + decision_training_principal_id = "00000000-0000-0000-0000-000000000008" + decision_training_broker_url = "https://decision-broker.example.com" + decision_training_control_scope = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/customer/providers/Microsoft.App/containerApps/decision-broker" + decision_training_control_role_definition_id = "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00000000-0000-0000-0000-000000000098" + decision_inference_scope = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/customer/providers/Microsoft.MachineLearningServices/workspaces/customer/onlineEndpoints/decision" + decision_inference_role_definition_id = "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00000000-0000-0000-0000-000000000099" + } + + assert { + condition = ( + azurerm_storage_container.decision[0].container_access_type == "private" && + !azurerm_storage_account.artifacts.shared_access_key_enabled && + azurerm_storage_account.artifacts.blob_properties[0].versioning_enabled && + azurerm_storage_account.artifacts.blob_properties[0].delete_retention_policy[0].permanent_delete_enabled && + azurerm_storage_management_policy.artifacts.rule[0].actions[0].version[0].delete_after_days_since_creation == var.decision_artifact_version_retention_days && + azurerm_role_assignment.decision_runtime_artifacts[0].scope == "${azurerm_storage_account.artifacts.id}/blobServices/default/containers/decision-runtime" && + azurerm_role_assignment.decision_runtime_artifacts[0].role_definition_id == azurerm_role_definition.decision_artifact_writer[0].role_definition_resource_id && + azurerm_role_assignment.decision_training_artifacts[0].scope == "${azurerm_storage_account.artifacts.id}/blobServices/default/containers/decision-runtime" && + azurerm_role_assignment.decision_training_artifacts[0].role_definition_id == azurerm_role_definition.decision_artifact_writer[0].role_definition_resource_id && + !contains(azurerm_role_definition.decision_artifact_writer[0].permissions[0].data_actions, "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete") && + !contains(azurerm_role_definition.decision_artifact_writer[0].permissions[0].data_actions, "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/permanentDelete/action") + ) + error_message = "Decision tasksets, candidates, and reports must stay in the private versioned container through identity-based access." + } + assert { + condition = ( + azurerm_role_assignment.decision_training_control[0].scope == var.decision_training_control_scope && + azurerm_role_assignment.decision_training_control[0].role_definition_id == var.decision_training_control_role_definition_id && + output.installation.decision_training_job_name == var.decision_training_job_name + ) + error_message = "The reference path must invoke the fixed training job through the customer broker, not direct job operator access." + } + assert { + condition = ( + azurerm_container_app.hub[0].identity[0].identity_ids == toset([ + azurerm_user_assigned_identity.hub.id, + ]) && + !contains([for env in azurerm_container_app.hub[0].template[0].container[0].env : env.name], "AZURE_DECISION_TRAINING_BROKER") && + !contains([for env in azurerm_container_app.hub[0].template[0].container[0].env : env.name], "AZURE_DECISION_TRAINING_JOB") && + output.installation.decision_runtime_client_id == azurerm_user_assigned_identity.decision_runtime[0].client_id && + output.installation.decision_inference_client_id == azurerm_user_assigned_identity.decision_inference[0].client_id + ) + error_message = "Hub must not receive decision-runtime identities or broker coordinates; operators use the non-secret outputs on the separate runtime hosts." + } + assert { + condition = ( + azurerm_user_assigned_identity.decision_runtime[0].principal_id != var.decision_training_principal_id && + azurerm_user_assigned_identity.decision_runtime[0].principal_id != azurerm_user_assigned_identity.decision_inference[0].principal_id && + azurerm_role_assignment.decision_training_control[0].principal_id == azurerm_user_assigned_identity.decision_runtime[0].principal_id && + azurerm_role_assignment.decision_training_artifacts[0].principal_id == var.decision_training_principal_id && + azurerm_role_assignment.decision_inference[0].principal_id == azurerm_user_assigned_identity.decision_inference[0].principal_id && + azurerm_role_assignment.decision_inference[0].role_definition_id == var.decision_inference_role_definition_id + ) + error_message = "Runtime orchestration, GPU execution, and inference need separate least-privilege managed identities." + } +} + run "combined_release_uses_one_image_and_bounded_shutdown" { command = plan diff --git a/infrastructure/azure/variables.tf b/infrastructure/azure/variables.tf index 873a625a..0941b2d5 100644 --- a/infrastructure/azure/variables.tf +++ b/infrastructure/azure/variables.tf @@ -56,6 +56,111 @@ variable "acr_id" { description = "Optional existing ACR resource ID. No registry is created or deleted." } +variable "enable_decision_runtime" { + type = bool + default = false + description = "Provision the customer-operated Azure decision connector path." +} + +variable "decision_artifact_version_retention_days" { + type = number + default = 30 + description = "Maximum retention for superseded decision artifact versions; authorized withdrawal uses permanent deletion sooner." + + validation { + condition = var.decision_artifact_version_retention_days >= 1 && var.decision_artifact_version_retention_days <= 365 && floor(var.decision_artifact_version_retention_days) == var.decision_artifact_version_retention_days + error_message = "Decision artifact version retention must be an integer from 1 to 365 days." + } +} + +variable "decision_training_job_id" { + type = string + default = null + description = "Existing customer-operated scale-to-zero Container Apps GPU Job resource ID." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_job_id != null && can(regex("^/subscriptions/[^/]+/resourceGroups/[^/]+/providers/Microsoft\\.App/jobs/[^/]+$", var.decision_training_job_id))) + error_message = "Enable the decision runtime only with an existing Container Apps Job resource ID." + } +} + +variable "decision_training_job_name" { + type = string + default = null + description = "Existing customer-operated scale-to-zero Container Apps GPU Job name." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_job_name != null && can(regex("^[a-z][a-z0-9-]{1,30}[a-z0-9]$", var.decision_training_job_name))) + error_message = "Enable the decision runtime only with a valid Container Apps Job name." + } +} + +variable "decision_training_principal_id" { + type = string + default = null + description = "Object ID of the managed identity attached only to the customer training job." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_principal_id != null && can(regex("^[0-9a-fA-F-]{36}$", var.decision_training_principal_id))) + error_message = "Enable the decision runtime only with the training job managed identity object ID." + } +} + +variable "decision_training_broker_url" { + type = string + default = null + description = "Exact HTTPS URL of the customer broker that starts and cancels only the fixed training job template." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_broker_url != null && can(regex("^https://[a-z0-9.-]+(?::[1-9][0-9]{0,4})?$", var.decision_training_broker_url))) + error_message = "Enable the decision runtime only with an exact HTTPS training broker URL." + } +} + +variable "decision_training_control_scope" { + type = string + default = null + description = "Customer broker resource ID receiving the runtime identity invocation grant." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_control_scope != null && startswith(var.decision_training_control_scope, "/subscriptions/")) + error_message = "Enable the decision runtime only with the customer training broker resource ID." + } +} + +variable "decision_training_control_role_definition_id" { + type = string + default = null + description = "Customer-defined broker invocation role; it must not grant direct job reads, writes, secret listing, or execution overrides." + + validation { + condition = !var.enable_decision_runtime || (var.decision_training_control_role_definition_id != null && strcontains(var.decision_training_control_role_definition_id, "/providers/Microsoft.Authorization/roleDefinitions/")) + error_message = "Enable the decision runtime only with a broker invocation role definition ID." + } +} + +variable "decision_inference_scope" { + type = string + default = null + description = "Customer endpoint resource ID receiving the inference-only identity grant." + + validation { + condition = !var.enable_decision_runtime || (var.decision_inference_scope != null && startswith(var.decision_inference_scope, "/subscriptions/")) + error_message = "Enable the decision runtime only with a customer inference endpoint resource ID." + } +} + +variable "decision_inference_role_definition_id" { + type = string + default = null + description = "Customer-defined inference-only Azure role definition ID; it must not grant training or endpoint administration." + + validation { + condition = !var.enable_decision_runtime || (var.decision_inference_role_definition_id != null && strcontains(var.decision_inference_role_definition_id, "/providers/Microsoft.Authorization/roleDefinitions/")) + error_message = "Enable the decision runtime only with an inference-only role definition ID." + } +} + variable "postgres_admin_password" { type = string default = null @@ -150,6 +255,9 @@ locals { "CLIENT_DIRECTORY", "SHUTDOWN_TIMEOUT_MS", "ARTIFACT_STORAGE_PROVIDER", "ARTIFACT_STORAGE_DIRECTORY", "AZURE_STORAGE_ACCOUNT", "AZURE_STORAGE_CONTAINER", "AZURE_CLIENT_ID", "AZURE_KEY_VAULT_SECRET_REFERENCES", + "AZURE_DECISION_STORAGE_CONTAINER", "AZURE_DECISION_RUNTIME_CLIENT_ID", + "AZURE_DECISION_INFERENCE_CLIENT_ID", "AZURE_DECISION_TRAINING_BROKER", + "AZURE_DECISION_TRAINING_JOB", ] } diff --git a/packages/learning/environments/brand-check/README.md b/packages/learning/environments/brand-check/README.md index 2b628dae..b07f0080 100644 --- a/packages/learning/environments/brand-check/README.md +++ b/packages/learning/environments/brand-check/README.md @@ -361,6 +361,28 @@ Connector failures use stable codes plus one category: invalid input, not found, The single-node runtime preserves its HTTP API and lifecycle behavior while every infrastructure operation goes through the local bundle. Installed runtimes execute training and each inference request in a spawned process, so a backend and its configuration must remain serializable. Each inference request pays process-startup cost, and model state loaded inside one inference child is not retained for the next request. A future hosted bundle replaces the implementations, not the lifecycle records or routes, and must pass the same suite without provider-specific fields in the shared contracts. +### Run the Azure reference connectors + +[`azure_connectors.py`](azure_connectors.py) is the first hosted reference bundle. It remains customer-operated: the Blob account, PostgreSQL server, Container Apps environment, GPU quota, inference endpoint, identities, logs, and charges all stay in the customer's Azure subscription. Replica does not receive tasksets, candidates, reports, prompts, credentials, or telemetry. + +The mapping keeps the provider-neutral lifecycle unchanged: + +| Contract | Azure reference | +| --- | --- | +| Taskset and artifact | A private Azure Blob container with shared-key access disabled. Keys start with a repository scope hash, bytes are verified against SHA-256 digests, and conditional creates reject replacement. Storage versioning and soft delete support recovery, and a lifecycle rule expires superseded decision versions. The optional retention callback records customer policy. An authorized withdrawal purger calls `delete`, which permanently removes the current Blob and every version. | +| Job | Tables in the installation's existing PostgreSQL database hold the job record, compare-and-swap version, create idempotency key, durable cancellation, and expiring fenced lease. `initialize_azure_postgres` creates the dedicated `replica_decision_runtime` schema, revokes public access, and grants only the named decision-runtime database role. Run it from the existing migration boundary; do not create another database service or grant Hub's application role access. | +| Candidate and promotion | Candidate artifacts and reports use Blob; immutable descriptors and append-only promotion generations use PostgreSQL. Promotion and rollback compare the expected generation in the same transaction. | +| Training | `AzureContainerAppsTrainingConnector` starts one manual Azure job execution, passes the deadline and credit limits, polls a bounded execution, and cancels Azure work when the runtime is cancelled, expires, or loses status. Before each direct cancel attempt, the required recovery hook durably records the execution ID for broker cleanup; direct cancellation then retries three times as best-effort acceleration. The job result is authoritative for usage and cost when Azure supplies them. The transport can be replaced by Azure Machine Learning without changing the runtime contract. | +| Inference | `AzureEndpointInferenceConnector` calls only readiness and prediction operations on a customer endpoint. Every request names its expected promotion generation; startup or deployment drift is reported as retryable readiness or generation failure rather than scoring with another model. | +| Credential | `AzureManagedIdentityCredentialConnector` accepts only configured audiences and wraps short-lived workload-identity tokens. Its lifetime never exceeds the operation deadline, and it has no key or secret persistence path. | +| Observability | `AzureObservabilityConnector` emits usage and operational attributes to an injected Pino/OpenTelemetry bridge. It hashes tenant and repository names and allowlists provider metadata; screenshots, prompts, learning text, artifacts, and credential values cannot enter this signal. | + +The connector module deliberately takes narrow injected clients instead of adding Azure SDK dependencies to the environment. An embedding service may adapt the official Blob, identity, Container Apps Jobs, PostgreSQL, OpenTelemetry, and HTTP clients to those interfaces. Because training runs in a spawned worker, bundle composition also supplies a serializable managed-identity provider factory that constructs the SDK credential client inside that worker rather than serializing SDK client state. Tests use in-memory clients and need no Azure account. + +The Terraform reference under `infrastructure/azure` adds an optional private `decision-runtime` container and separate runtime and inference user-assigned identities. It returns their non-secret coordinates for the customer-operated runtime hosts; it does not attach them to Hub or inject broker and inference settings into Hub. A module-owned storage role gives the runtime and the customer GPU Job only decision-container Blob read, add, and write actions; it omits delete and permanent-delete actions. The runtime identity invokes a customer broker that starts and cancels one fixed job template; it never receives direct Container Apps Jobs Operator access because that role can list secrets and override images, commands, and environment variables at execution time. The job stays outside the module because the repository-pinned AzureRM provider does not yet accept Azure's serverless GPU workload profile types. The inference identity receives a customer-supplied custom role on the customer endpoint; that role must permit inference only, not training, deployment, or endpoint administration. + +Serverless GPU regions and quota are limited. A T4 profile is cheaper and smaller than A100, but either can have several minutes of image-pull and GPU allocation cold start after scaling to zero. The customer job must use a manual trigger, zero idle GPU capacity, one execution replica, no automatic retry, an immutable image, and a hard execution timeout. The job connector reports startup delay through job state and still applies the caller deadline. A customer that needs reserved capacity or Azure Machine Learning can replace only the job transport and inference transport. Azure's usage or Cost Management export remains the billing authority. + ### Trust, data, and credential boundaries - The operator chooses the host. The reference runtime never calls Replica or a cloud service. A later hosted implementation can keep the same job, candidate, promotion, and inference contracts. @@ -370,6 +392,8 @@ The single-node runtime preserves its HTTP API and lifecycle behavior while ever - The runtime token authorizes training, promotion, candidate reads, and inference. Keep it in a protected file or secret store and pass it through the environment, not the command line or image. - The service uses plain HTTP. Bind it to localhost as shown, or put customer-operated TLS and authentication in front of it before another host can connect. - A backend is trusted code. The reference runtime checks the taskset first and bounds time and updates, but a later model backend can read the screenshots and return candidate artifacts. +- The Azure inference identity must be separate from the training and runtime identities. Its customer-defined role may invoke the selected endpoint only. Do not assign Azure ML Contributor, Data Scientist, Container Apps Jobs Operator, Blob contributor, or another training/administration role to that identity. +- Do not give the runtime identity permanent-delete permission. Run withdrawal deletion through a separate customer workflow that rechecks the affected export, uses a dedicated short-lived identity, calls the connector's permanent version deletion, and records the result in the customer's audit system. ## Train d1-omni-600M diff --git a/packages/learning/environments/brand-check/__init__.py b/packages/learning/environments/brand-check/__init__.py index 4cd06675..ad1b2487 100644 --- a/packages/learning/environments/brand-check/__init__.py +++ b/packages/learning/environments/brand-check/__init__.py @@ -10,6 +10,15 @@ "BrandCheckAction", "BrandCheckEnv", "BrandCheckObservation", + "AzureConnectorBundle", + "AzureContainerAppsTrainingConnector", + "AzureEndpointInferenceConnector", + "AzureManagedIdentityCredentialConnector", + "AzureObservabilityConnector", + "AzurePostgresCandidateConnector", + "AzurePostgresJobConnector", + "AzureBlobArtifactConnector", + "AzureBlobTasksetConnector", "ConnectorConformanceFixture", "ConnectorError", "FixtureBackend", @@ -23,12 +32,23 @@ "TrainingResult", "connector_conformance", "create_local_connector_bundle", + "create_azure_connector_bundle", + "initialize_azure_postgres", "question_options", ] # The scoring modules use only the standard library. Importing the OpenEnv classes on first use # lets them, and their tests, run where openenv-core is not installed. _OPENENV_EXPORTS = { + "AzureBlobArtifactConnector": ".azure_connectors", + "AzureBlobTasksetConnector": ".azure_connectors", + "AzureConnectorBundle": ".azure_connectors", + "AzureContainerAppsTrainingConnector": ".azure_connectors", + "AzureEndpointInferenceConnector": ".azure_connectors", + "AzureManagedIdentityCredentialConnector": ".azure_connectors", + "AzureObservabilityConnector": ".azure_connectors", + "AzurePostgresCandidateConnector": ".azure_connectors", + "AzurePostgresJobConnector": ".azure_connectors", "BrandCheckAction": ".models", "BrandCheckObservation": ".models", "BrandCheckEnv": ".client", @@ -43,6 +63,8 @@ "TrainingResult": ".connectors", "RuntimeFailure": ".runtime", "create_local_connector_bundle": ".local_connectors", + "create_azure_connector_bundle": ".azure_connectors", + "initialize_azure_postgres": ".azure_connectors", } diff --git a/packages/learning/environments/brand-check/azure_connectors.py b/packages/learning/environments/brand-check/azure_connectors.py new file mode 100644 index 00000000..4f74432a --- /dev/null +++ b/packages/learning/environments/brand-check/azure_connectors.py @@ -0,0 +1,2398 @@ +"""Customer-operated Azure connectors for the Replica decision runtime.""" + +from __future__ import annotations + +import base64 +import gzip +import hashlib +import io +import json +import math +import os +import re +import shutil +import tarfile +import tempfile +import time +import uuid +from contextlib import contextmanager +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable, Iterator, Mapping, Protocol, Sequence + +from .connectors import ( + Artifact, + Cancellation, + Candidate, + ConnectorError, + ContentDigest, + Credential, + JobLease, + OperationContext, + Promotion, + RuntimeConnectors, + RuntimeOwner, + RuntimeScope, + TasksetArtifact, + TasksetBundle, + TrainingResult, + Usage, + check_context, +) +from .local_connectors import ( + _candidate_content_digest, + _digest, + _json_bytes, + _materialize_taskset_bundle, + taskset_digest, +) +from .taskset import MAX_IMAGE_BYTES, TasksetError, load_taskset + + +class AzureBlobContainer(Protocol): + """The subset of an Azure Blob container client used by this bundle.""" + + def upload_blob( + self, + name: str, + data: bytes, + *, + overwrite: bool = False, + metadata: Mapping[str, str] | None = None, + ) -> Any: ... + + def download_blob(self, name: str) -> Any: ... + + def list_blob_versions(self, name: str) -> Sequence[str]: ... + + def delete_blob( + self, + name: str, + *, + version_id: str | None = None, + permanent: bool = False, + ) -> Any: ... + + +class AzureTokenProvider(Protocol): + """A workload identity or managed identity token source.""" + + def get_token(self, audience: str) -> Any: ... + + +class AzureJobTransport(Protocol): + """A replaceable Azure Container Apps Jobs or Azure ML adapter.""" + + def start( + self, + job_name: str, + payload: Mapping[str, Any], + *, + bearer_token: str | None, + ) -> Mapping[str, Any]: ... + + def status( + self, + execution_id: str, + *, + bearer_token: str | None, + ) -> Mapping[str, Any]: ... + + def cancel( + self, + execution_id: str, + *, + bearer_token: str | None, + ) -> None: ... + + +class AzureEndpointTransport(Protocol): + """Inference-only endpoint that enforces scoped idempotency keys and request digests.""" + + def readiness(self, *, bearer_token: str | None) -> Mapping[str, Any]: ... + + def predict( + self, + request: Mapping[str, Any], + *, + bearer_token: str | None, + ) -> Mapping[str, Any]: ... + + +class AzureTelemetrySink(Protocol): + """A Pino/OpenTelemetry bridge owned by the embedding service.""" + + def emit(self, name: str, attributes: Mapping[str, Any]) -> None: ... + + +RetentionHook = Callable[[str, Mapping[str, str]], None] +CancellationRecoveryHook = Callable[ + [str, OperationContext], + None, +] +CredentialRefresh = Callable[[OperationContext], Credential | None] +TokenProviderFactory = Callable[[], AzureTokenProvider] +ConnectionFactory = Callable[[], Any] + +_TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"} +_ACTIVE_JOB_STATES = {"pending", "queued", "starting", "running"} +_SAFE_USAGE_METADATA = { + "azure.execution_id", + "azure.job_name", + "azure.region", + "azure.resource_id", + "azure.sku", + "cold_start", + "promotion_generation", + "provider_status", +} +_USAGE_KINDS = { + "cpu_seconds", + "credits", + "gpu_seconds", + "inference_seconds", + "input_tokens", + "job_cancellations", + "network_bytes", + "output_tokens", + "requests", + "storage_bytes", + "taskset_bytes", + "training_credits", + "training_seconds", +} +_USAGE_UNITS = {"bytes", "credits", "requests", "seconds", "tokens"} +_CURRENCY = re.compile(r"^[A-Z]{3}$") +_MAX_TASKSET_ARCHIVE_MEMBERS = 10_000 +_MAX_TASKSET_DESCRIPTOR_BYTES = 16 * 1024 * 1024 +_MAX_TASKSET_EXPANDED_BYTES = 180 * 1024 * 1024 +_MAX_TASKSET_ARCHIVE_BYTES = _MAX_TASKSET_EXPANDED_BYTES + 32 * 1024 * 1024 + + +def _scope_key(scope: RuntimeScope) -> str: + return hashlib.sha256( + f"{scope.tenant_id}\0{scope.repository}".encode("utf-8") + ).hexdigest() + + +def _scope_prefix(scope: RuntimeScope) -> str: + repository = "".join( + member if member.isalnum() or member in ".-_" else "-" + for member in scope.repository.lower() + ).strip("-") + return f"repositories/{repository[:96] or 'repository'}-{_scope_key(scope)[:16]}" + + +def _request_digest(value: Any) -> str: + return str(_digest(_json_bytes(value))) + + +def _json_value(value: Any) -> Any: + if isinstance(value, Mapping): + return {key: _json_value(member) for key, member in value.items()} + if isinstance(value, (tuple, list)): + return [_json_value(member) for member in value] + return value + + +def _download_bytes( + container: AzureBlobContainer, + name: str, + *, + maximum_bytes: int | None = None, +) -> bytes: + try: + response = container.download_blob(name) + if maximum_bytes is not None and hasattr(response, "chunks"): + chunks = [] + size = 0 + for chunk in response.chunks(): + size += len(chunk) + if size > maximum_bytes: + raise ConnectorError( + "azure_blob_oversized", + "The private Azure Blob exceeds its size limit.", + category="integrity", + ) + chunks.append(chunk) + data = b"".join(chunks) + elif maximum_bytes is not None and hasattr(response, "read"): + data = response.read(maximum_bytes + 1) + elif hasattr(response, "readall"): + data = response.readall() + elif isinstance(response, bytes): + data = response + else: + data = response.read() + except ConnectorError: + raise + except Exception as error: + if _is_missing(error): + raise ConnectorError( + "azure_blob_not_found", + "The private Azure Blob was not found.", + category="not_found", + ) from None + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + if not isinstance(data, bytes): + raise ConnectorError( + "azure_blob_invalid", + "Azure Blob Storage returned invalid content.", + category="integrity", + ) + if maximum_bytes is not None and len(data) > maximum_bytes: + raise ConnectorError( + "azure_blob_oversized", + "The private Azure Blob exceeds its size limit.", + category="integrity", + ) + return data + + +def _is_missing(error: Exception) -> bool: + return ( + getattr(error, "status_code", None) == 404 + or error.__class__.__name__ in {"ResourceNotFoundError", "BlobNotFound"} + ) + + +def _is_exists(error: Exception) -> bool: + return ( + getattr(error, "status_code", None) in (409, 412) + or error.__class__.__name__ + in {"ResourceExistsError", "ResourceModifiedError", "BlobAlreadyExists"} + ) + + +class _AzureScopedConnector: + def __init__(self, scope: RuntimeScope): + self.scope = scope + self.scope_key = _scope_key(scope) + self.prefix = _scope_prefix(scope) + + def _check(self, context: OperationContext, *, hide: bool = False) -> None: + check_context(context) + if context.scope != self.scope: + raise ConnectorError( + "scoped_record_not_found" if hide else "scope_unauthorized", + "The requested record was not found." + if hide + else "The operation scope does not match this Azure connector bundle.", + category="not_found" if hide else "unauthorized", + ) + + +class AzureBlobArtifactConnector(_AzureScopedConnector): + """Private, repository-scoped, immutable Azure Blob artifacts.""" + + def __init__( + self, + container: AzureBlobContainer, + scope: RuntimeScope, + *, + retention_hook: RetentionHook | None = None, + ): + super().__init__(scope) + self.container = container + self.retention_hook = retention_hook + + def put(self, artifact: Artifact, context: OperationContext) -> Artifact: + self._check(context) + if _digest(artifact.data) != artifact.digest: + raise ConnectorError( + "artifact_digest_mismatch", + "The artifact bytes do not match the claimed digest.", + category="integrity", + ) + name = self._name(artifact.digest) + metadata = { + "digest": artifact.digest.value, + "media_type_sha256": hashlib.sha256( + artifact.media_type.encode("utf-8") + ).hexdigest(), + "repository_scope": self.scope_key, + } + descriptor = _json_bytes( + { + "digest": str(artifact.digest), + "mediaType": artifact.media_type, + "metadata": artifact.metadata, + } + ) + try: + self.container.upload_blob( + name, + artifact.data, + overwrite=False, + metadata=metadata, + ) + except Exception as error: + if not _is_exists(error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + existing_data = _download_bytes(self.container, name) + if _digest(existing_data) != artifact.digest or existing_data != artifact.data: + raise ConnectorError( + "artifact_immutable", + "The artifact digest already identifies different content.", + category="integrity", + ) + try: + existing = self.get(artifact.digest, context) + except ConnectorError as stored_error: + if stored_error.category != "not_found": + raise + try: + self.container.upload_blob( + f"{name}.json", + descriptor, + overwrite=False, + metadata={"repository_scope": self.scope_key}, + ) + except Exception as descriptor_error: + if not _is_exists(descriptor_error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from descriptor_error + existing = self.get(artifact.digest, context) + if existing != artifact: + raise ConnectorError( + "artifact_immutable", + "The artifact digest already identifies different content.", + category="integrity", + ) + self._apply_retention(name, metadata) + return existing + try: + self.container.upload_blob( + f"{name}.json", + descriptor, + overwrite=False, + metadata={"repository_scope": self.scope_key}, + ) + except Exception as error: + if _is_exists(error): + existing = self.get(artifact.digest, context) + if existing == artifact: + self._apply_retention(name, metadata) + return existing + raise ConnectorError( + "artifact_immutable", + "The artifact digest already identifies different content.", + category="integrity", + ) from None + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + self._apply_retention(name, metadata) + return artifact + + def _apply_retention(self, name: str, metadata: Mapping[str, str]) -> None: + if self.retention_hook is None: + return + try: + self.retention_hook(name, metadata) + except Exception as error: + raise ConnectorError( + "retention_hook_failed", + "The artifact retention policy was not recorded.", + category="unavailable", + retryable=True, + ) from error + + def get(self, digest: ContentDigest, context: OperationContext) -> Artifact: + self._check(context, hide=True) + name = self._name(digest) + data = _download_bytes(self.container, name) + descriptor_bytes = _download_bytes(self.container, f"{name}.json") + try: + descriptor = json.loads(descriptor_bytes) + artifact = Artifact( + digest, + descriptor["mediaType"], + data, + descriptor.get("metadata", {}), + ) + except (KeyError, TypeError, ValueError, json.JSONDecodeError): + raise ConnectorError( + "artifact_damaged", + "The stored artifact descriptor is invalid.", + category="integrity", + ) from None + if descriptor.get("digest") != str(digest) or _digest(data) != digest: + raise ConnectorError( + "artifact_damaged", + "The stored artifact no longer matches its immutable identity.", + category="integrity", + ) + return artifact + + def delete(self, digest: ContentDigest, context: OperationContext) -> None: + """Permanently remove every Blob version for an authorized withdrawal.""" + self._check(context) + name = self._name(digest) + for target in (name, f"{name}.json"): + try: + versions = tuple(self.container.list_blob_versions(target)) + except Exception as error: + if _is_missing(error): + versions = () + else: + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + for version_id in versions: + check_context(context) + try: + self.container.delete_blob( + target, + version_id=version_id, + permanent=True, + ) + except Exception as error: + if not _is_missing(error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + try: + self.container.delete_blob(target, permanent=True) + except Exception as error: + if not _is_missing(error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + + def _name(self, digest: ContentDigest) -> str: + return ( + f"{self.prefix}/artifacts/{digest.algorithm}/" + f"{digest.value[:2]}/{digest.value}" + ) + + +class AzureBlobTasksetConnector(_AzureScopedConnector): + """Private immutable tasksets archived in repository-scoped Blob paths.""" + + def __init__( + self, + container: AzureBlobContainer, + scope: RuntimeScope, + *, + retention_hook: RetentionHook | None = None, + ): + super().__init__(scope) + self.container = container + self.retention_hook = retention_hook + self._cache_root = Path(tempfile.mkdtemp(prefix="replica-azure-tasksets-")) + + def close(self) -> None: + shutil.rmtree(self._cache_root, ignore_errors=True) + + def __del__(self) -> None: + self.close() + + def register( + self, bundle: TasksetBundle, context: OperationContext + ) -> TasksetArtifact: + self._check(context) + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + _materialize_taskset_bundle(root, bundle, context) + try: + taskset = load_taskset(root) + except TasksetError as error: + raise ConnectorError( + "taskset_invalid", + str(error), + category="invalid_input", + ) from None + digest = taskset_digest(taskset) + if bundle.digest is not None and bundle.digest != digest: + try: + _download_bytes(self.container, self._name(bundle.digest)) + except ConnectorError as error: + if error.category != "not_found": + raise + else: + raise ConnectorError( + "taskset_immutable", + "The taskset digest already identifies different content.", + category="conflict", + ) + raise ConnectorError( + "taskset_digest_mismatch", + "The taskset does not match the claimed digest.", + category="invalid_input", + ) + archive = self._archive(root) + if len(archive) > _MAX_TASKSET_ARCHIVE_BYTES: + raise ConnectorError( + "taskset_archive_too_large", + "The compressed taskset archive exceeds its size limit.", + category="invalid_input", + ) + request_digest = _request_digest( + { + "claimedDigest": str(bundle.digest) if bundle.digest else None, + "archiveDigest": str(_digest(archive)), + "scope": self.scope_key, + } + ) + idempotency_name = ( + f"{self.prefix}/tasksets/.idempotency/" + f"{hashlib.sha256(context.idempotency_key.encode()).hexdigest()}.json" + ) + retry_record = _json_bytes( + { + "requestDigest": request_digest, + "reference": f"taskset:{digest.value}", + } + ) + try: + self.container.upload_blob( + idempotency_name, + retry_record, + overwrite=False, + metadata={"repository_scope": self.scope_key}, + ) + except Exception as error: + if not _is_exists(error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + try: + retry = json.loads(_download_bytes(self.container, idempotency_name)) + except (TypeError, json.JSONDecodeError): + raise ConnectorError( + "taskset_idempotency_damaged", + "The taskset retry record is invalid.", + category="integrity", + ) from None + if not isinstance(retry, dict): + raise ConnectorError( + "taskset_idempotency_damaged", + "The taskset retry record is invalid.", + category="integrity", + ) + if ( + retry.get("requestDigest") != request_digest + or retry.get("reference") != f"taskset:{digest.value}" + ): + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + name = self._name(digest) + metadata = { + "digest": digest.value, + "repository_scope": self.scope_key, + "kind": "decision-taskset", + } + try: + self.container.upload_blob( + name, + archive, + overwrite=False, + metadata=metadata, + ) + except Exception as error: + if not _is_exists(error): + raise ConnectorError( + "azure_blob_unavailable", + "Azure Blob Storage is unavailable.", + category="unavailable", + retryable=True, + ) from error + existing = _download_bytes(self.container, name) + if existing != archive: + raise ConnectorError( + "taskset_immutable", + "The taskset digest already identifies different content.", + category="conflict", + ) + if self.retention_hook is not None: + try: + self.retention_hook(name, metadata) + except Exception as error: + raise ConnectorError( + "retention_hook_failed", + "The taskset retention policy was not recorded.", + category="unavailable", + retryable=True, + ) from error + return self.resolve(f"taskset:{digest.value}", context) + + def resolve(self, reference: str, context: OperationContext) -> TasksetArtifact: + self._check(context, hide=True) + try: + digest = ContentDigest("sha256", reference.removeprefix("taskset:")) + except ValueError: + raise ConnectorError( + "taskset_not_found", + "The taskset was not found.", + category="not_found", + ) from None + if reference != f"taskset:{digest.value}": + raise ConnectorError( + "taskset_not_found", + "The taskset was not found.", + category="not_found", + ) + root = self._cache_root / digest.value + if not root.is_dir(): + archive = _download_bytes( + self.container, + self._name(digest), + maximum_bytes=_MAX_TASKSET_ARCHIVE_BYTES, + ) + staging = self._cache_root / f".{digest.value}.{uuid.uuid4().hex}" + staging.mkdir(mode=0o700) + try: + with tarfile.open(fileobj=io.BytesIO(archive), mode="r:gz") as source: + _extract_taskset_archive(source, staging) + staged_taskset = load_taskset(staging) + if taskset_digest(staged_taskset) != digest: + raise TasksetError( + "The stored taskset no longer matches its immutable identity." + ) + try: + staging.rename(root) + except OSError: + if not root.is_dir(): + raise + except (tarfile.TarError, TasksetError, OSError): + raise ConnectorError( + "taskset_damaged", + "The stored taskset is invalid.", + category="integrity", + ) from None + finally: + shutil.rmtree(staging, ignore_errors=True) + try: + taskset = load_taskset(root) + except TasksetError: + raise ConnectorError( + "taskset_damaged", + "The stored taskset is invalid.", + category="integrity", + ) from None + if taskset_digest(taskset) != digest: + raise ConnectorError( + "taskset_damaged", + "The stored taskset no longer matches its immutable identity.", + category="integrity", + ) + return TasksetArtifact(reference, digest, taskset) + + def _archive(self, root: Path) -> bytes: + tar_buffer = io.BytesIO() + with tarfile.open(fileobj=tar_buffer, mode="w") as archive: + paths = tuple(path for path in sorted(root.rglob("*")) if path.is_file()) + if len(paths) > _MAX_TASKSET_ARCHIVE_MEMBERS: + raise ConnectorError( + "taskset_too_large", + "The taskset contains too many files.", + category="invalid_input", + ) + expanded = 0 + for path in paths: + relative = str(path.relative_to(root)) + size = path.stat().st_size + maximum_member_bytes = ( + _MAX_TASKSET_DESCRIPTOR_BYTES + if relative in ("taskset.json", "tasks.jsonl") + else MAX_IMAGE_BYTES + ) + if ( + size > maximum_member_bytes + or expanded + size > _MAX_TASKSET_EXPANDED_BYTES + ): + raise ConnectorError( + "taskset_too_large", + "The taskset exceeds the private archive size limit.", + category="invalid_input", + ) + data = path.read_bytes() + if len(data) != size: + raise ConnectorError( + "taskset_changed", + "The taskset changed while it was being archived.", + category="conflict", + ) + expanded += size + member = tarfile.TarInfo(relative) + member.size = size + member.mode = 0o400 + member.mtime = 0 + member.uid = 0 + member.gid = 0 + member.uname = "" + member.gname = "" + archive.addfile(member, io.BytesIO(data)) + return gzip.compress(tar_buffer.getvalue(), mtime=0) + + def _name(self, digest: ContentDigest) -> str: + return f"{self.prefix}/tasksets/{digest.value}/bundle.tar.gz" + + +class _AzurePostgresConnector(_AzureScopedConnector): + def __init__(self, connection_factory: ConnectionFactory, scope: RuntimeScope): + super().__init__(scope) + self.connection_factory = connection_factory + + @contextmanager + def _transaction(self) -> Iterator[Any]: + connection = self.connection_factory() + cursor = connection.cursor() + try: + yield cursor + connection.commit() + except Exception: + connection.rollback() + raise + finally: + cursor.close() + connection.close() + + def _execute(self, cursor: Any, sql: str, parameters: Sequence[Any]) -> Any: + cursor.execute(sql, tuple(parameters)) + return cursor + + +AZURE_POSTGRES_SCHEMA = """ +CREATE SCHEMA IF NOT EXISTS replica_decision_runtime; +CREATE TABLE IF NOT EXISTS replica_decision_runtime.replica_decision_jobs ( + scope_key text NOT NULL, + job_id text NOT NULL, + version bigint NOT NULL, + record jsonb NOT NULL, + create_idempotency_key text NOT NULL, + create_request_digest text NOT NULL, + lease_token text, + lease_owner_kind text, + lease_owner_id text, + lease_expires_at double precision, + cancellation_idempotency_key text, + PRIMARY KEY (scope_key, job_id), + UNIQUE (scope_key, create_idempotency_key) +); +CREATE TABLE IF NOT EXISTS replica_decision_runtime.replica_decision_candidates ( + scope_key text NOT NULL, + candidate_id text NOT NULL, + digest text NOT NULL, + descriptor jsonb NOT NULL, + PRIMARY KEY (scope_key, candidate_id) +); +CREATE TABLE IF NOT EXISTS replica_decision_runtime.replica_decision_promotions ( + scope_key text NOT NULL, + generation bigint NOT NULL, + candidate_id text NOT NULL, + previous_candidate_id text, + operation text NOT NULL, + owner_kind text NOT NULL, + owner_id text NOT NULL, + idempotency_key text NOT NULL, + request_digest text NOT NULL, + PRIMARY KEY (scope_key, generation), + UNIQUE (scope_key, idempotency_key) +); +CREATE TABLE IF NOT EXISTS replica_decision_runtime.replica_decision_idempotency ( + scope_key text NOT NULL, + operation text NOT NULL, + idempotency_key text NOT NULL, + request_digest text NOT NULL, + result jsonb NOT NULL, + PRIMARY KEY (scope_key, operation, idempotency_key) +); +""".strip() + + +def initialize_azure_postgres( + connection_factory: ConnectionFactory, + *, + runtime_role: str = "replica_decision_runtime", +) -> None: + """Install isolated decision tables and grant only the dedicated runtime role.""" + if re.fullmatch(r"[a-z_][a-z0-9_]{0,62}", runtime_role) is None: + raise ValueError("The decision runtime database role is not valid.") + connection = connection_factory() + cursor = connection.cursor() + try: + for statement in AZURE_POSTGRES_SCHEMA.split(";"): + if statement.strip(): + cursor.execute(statement) + quoted_role = f'"{runtime_role}"' + cursor.execute("REVOKE ALL ON SCHEMA replica_decision_runtime FROM PUBLIC") + cursor.execute( + f"GRANT USAGE ON SCHEMA replica_decision_runtime TO {quoted_role}" + ) + cursor.execute( + "GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA " + f"replica_decision_runtime TO {quoted_role}" + ) + connection.commit() + except Exception: + connection.rollback() + raise + finally: + cursor.close() + connection.close() + + +class AzurePostgresJobConnector(_AzurePostgresConnector): + """Durable PostgreSQL CAS, cancellation, idempotency, and lease state.""" + + def create( + self, job_id: str, record: Mapping[str, Any], context: OperationContext + ) -> Mapping[str, Any]: + self._check(context) + request_digest = _request_digest( + {"jobId": job_id, "record": record, "scope": self.scope_key} + ) + saved = dict(_json_value(record), version=1) + try: + with self._transaction() as cursor: + self._execute( + cursor, + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + ( + "replica-decision-create:" + f"{self.scope_key}:{context.idempotency_key}", + ), + ) + self._execute( + cursor, + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (f"replica-decision-job:{self.scope_key}:{job_id}",), + ) + self._execute( + cursor, + """ + SELECT job_id, create_request_digest, record, version + FROM replica_decision_runtime.replica_decision_jobs + WHERE scope_key = %s AND create_idempotency_key = %s + FOR UPDATE + """, + (self.scope_key, context.idempotency_key), + ) + retry = cursor.fetchone() + if retry is not None: + if retry[0] != job_id or retry[1] != request_digest: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + return saved + self._execute( + cursor, + """ + SELECT 1 FROM replica_decision_runtime.replica_decision_jobs + WHERE scope_key = %s AND job_id = %s + """, + (self.scope_key, job_id), + ) + if cursor.fetchone() is not None: + raise ConnectorError( + "job_exists", + "The job already exists.", + category="conflict", + ) + self._execute( + cursor, + """ + INSERT INTO replica_decision_runtime.replica_decision_jobs + (scope_key, job_id, version, record, + create_idempotency_key, create_request_digest) + VALUES (%s, %s, 1, %s::jsonb, %s, %s) + """, + ( + self.scope_key, + job_id, + json.dumps(_json_value(record)), + context.idempotency_key, + request_digest, + ), + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return saved + + def get(self, job_id: str, context: OperationContext) -> Mapping[str, Any]: + self._check(context, hide=True) + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + SELECT record, version FROM replica_decision_runtime.replica_decision_jobs + WHERE scope_key = %s AND job_id = %s + """, + (self.scope_key, job_id), + ) + row = cursor.fetchone() + except Exception as error: + raise _database_error(error) from error + if row is None: + raise ConnectorError( + "job_not_found", + "The job was not found.", + category="not_found", + ) + return dict(_loaded_json(row[0]), version=row[1]) + + def list( + self, states: Sequence[str], context: OperationContext + ) -> Sequence[Mapping[str, Any]]: + check_context(context) + if context.scope != self.scope: + return () + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + SELECT record, version FROM replica_decision_runtime.replica_decision_jobs + WHERE scope_key = %s + ORDER BY job_id + """, + (self.scope_key,), + ) + rows = cursor.fetchall() + except Exception as error: + raise _database_error(error) from error + selected = set(states) + return tuple( + dict(_loaded_json(record), version=version) + for record, version in rows + if _loaded_json(record).get("state") in selected + ) + + def update( + self, + job_id: str, + record: Mapping[str, Any], + *, + expected_version: int, + lease: JobLease, + context: OperationContext, + ) -> Mapping[str, Any]: + self._check(context) + saved = dict(_json_value(record), version=expected_version + 1) + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + UPDATE replica_decision_runtime.replica_decision_jobs + SET record = %s::jsonb, version = version + 1 + WHERE scope_key = %s AND job_id = %s AND version = %s + AND lease_token = %s AND lease_owner_kind = %s + AND lease_owner_id = %s + AND lease_expires_at > EXTRACT(EPOCH FROM clock_timestamp()) + AND COALESCE(record->>'state', '') <> 'cancelled' + """, + ( + json.dumps(_json_value(record)), + self.scope_key, + job_id, + expected_version, + lease.token, + lease.owner.kind, + lease.owner.id, + ), + ) + if cursor.rowcount != 1: + raise ConnectorError( + "job_version_or_lease_conflict", + "The job version or active worker lease changed.", + category="conflict", + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return saved + + def acquire( + self, job_id: str, *, ttl_seconds: float, context: OperationContext + ) -> JobLease: + self._check(context) + _check_ttl(ttl_seconds) + token = uuid.uuid4().hex + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + UPDATE replica_decision_runtime.replica_decision_jobs + SET lease_token = %s, lease_owner_kind = %s, lease_owner_id = %s, + lease_expires_at = EXTRACT(EPOCH FROM clock_timestamp()) + %s + WHERE scope_key = %s AND job_id = %s + AND COALESCE(record->>'state', '') <> 'cancelled' + AND ( + lease_token IS NULL + OR lease_expires_at <= EXTRACT(EPOCH FROM clock_timestamp()) + ) + """, + ( + token, + context.owner.kind, + context.owner.id, + ttl_seconds, + self.scope_key, + job_id, + ), + ) + if cursor.rowcount != 1: + raise ConnectorError( + "job_lease_conflict", + "Another worker holds the job lease.", + category="conflict", + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return JobLease( + job_id, + token, + context.owner, + time.monotonic() + ttl_seconds, + ) + + def renew( + self, lease: JobLease, *, ttl_seconds: float, context: OperationContext + ) -> JobLease: + self._check(context) + _check_ttl(ttl_seconds) + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + UPDATE replica_decision_runtime.replica_decision_jobs + SET lease_expires_at = EXTRACT(EPOCH FROM clock_timestamp()) + %s + WHERE scope_key = %s AND job_id = %s AND lease_token = %s + AND lease_owner_kind = %s AND lease_owner_id = %s + AND lease_expires_at > EXTRACT(EPOCH FROM clock_timestamp()) + """, + ( + ttl_seconds, + self.scope_key, + lease.job_id, + lease.token, + lease.owner.kind, + lease.owner.id, + ), + ) + if cursor.rowcount != 1: + raise ConnectorError( + "job_lease_lost", + "The job lease is no longer owned by this worker.", + category="conflict", + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return JobLease( + lease.job_id, + lease.token, + lease.owner, + time.monotonic() + ttl_seconds, + ) + + def release(self, lease: JobLease, context: OperationContext) -> None: + self._check(context) + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + UPDATE replica_decision_runtime.replica_decision_jobs + SET lease_token = NULL, lease_owner_kind = NULL, + lease_owner_id = NULL, lease_expires_at = NULL + WHERE scope_key = %s AND job_id = %s AND lease_token = %s + AND lease_owner_kind = %s AND lease_owner_id = %s + AND lease_expires_at > EXTRACT(EPOCH FROM clock_timestamp()) + """, + ( + self.scope_key, + lease.job_id, + lease.token, + lease.owner.kind, + lease.owner.id, + ), + ) + if cursor.rowcount != 1: + raise ConnectorError( + "job_lease_lost", + "The job lease is no longer owned by this worker.", + category="conflict", + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + + def cancel( + self, + job_id: str, + *, + expected_version: int, + context: OperationContext, + ) -> Mapping[str, Any]: + self._check(context) + request_digest = _request_digest( + { + "jobId": job_id, + "expectedVersion": expected_version, + "scope": self.scope_key, + } + ) + try: + with self._transaction() as cursor: + self._execute( + cursor, + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + ( + "replica-decision-cancel:" + f"{self.scope_key}:{context.idempotency_key}", + ), + ) + self._execute( + cursor, + """ + SELECT request_digest, result + FROM replica_decision_runtime.replica_decision_idempotency + WHERE scope_key = %s AND operation = 'cancel' + AND idempotency_key = %s + """, + (self.scope_key, context.idempotency_key), + ) + retry = cursor.fetchone() + if retry is not None: + if retry[0] != request_digest: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + return _loaded_json(retry[1]) + self._execute( + cursor, + """ + SELECT record, version + FROM replica_decision_runtime.replica_decision_jobs + WHERE scope_key = %s AND job_id = %s + FOR UPDATE + """, + (self.scope_key, job_id), + ) + row = cursor.fetchone() + if row is None: + raise ConnectorError( + "job_not_found", + "The job was not found.", + category="not_found", + ) + record, version = row + current = _loaded_json(record) + if version != expected_version: + raise ConnectorError( + "job_version_conflict", + "The job changed before cancellation.", + category="conflict", + ) + saved_record = dict(current, state="cancelled") + saved = dict(saved_record, version=expected_version + 1) + self._execute( + cursor, + """ + UPDATE replica_decision_runtime.replica_decision_jobs + SET record = %s::jsonb, version = version + 1, + cancellation_idempotency_key = %s + WHERE scope_key = %s AND job_id = %s AND version = %s + """, + ( + json.dumps(saved_record), + context.idempotency_key, + self.scope_key, + job_id, + expected_version, + ), + ) + self._execute( + cursor, + """ + INSERT INTO replica_decision_runtime.replica_decision_idempotency + (scope_key, operation, idempotency_key, request_digest, result) + VALUES (%s, 'cancel', %s, %s, %s::jsonb) + """, + ( + self.scope_key, + context.idempotency_key, + request_digest, + json.dumps(saved), + ), + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return saved + + +class AzurePostgresCandidateConnector(_AzurePostgresConnector): + """Blob-backed immutable candidates with PostgreSQL promotion generations.""" + + def __init__( + self, + connection_factory: ConnectionFactory, + artifacts: AzureBlobArtifactConnector, + scope: RuntimeScope, + ): + super().__init__(connection_factory, scope) + self.artifacts = artifacts + self._cache_root = Path(tempfile.mkdtemp(prefix="replica-azure-candidates-")) + + def close(self) -> None: + shutil.rmtree(self._cache_root, ignore_errors=True) + + def __del__(self) -> None: + self.close() + + def put(self, candidate: Candidate, context: OperationContext) -> Candidate: + self._check(context) + for artifact in candidate.artifacts: + if _digest(artifact.data) != artifact.digest: + raise ConnectorError( + "candidate_digest_mismatch", + "A candidate artifact does not match its digest.", + category="integrity", + ) + self.artifacts.put(artifact, context) + descriptor = { + "id": candidate.id, + "digest": str(candidate.digest), + "contentDigest": str( + _candidate_content_digest(candidate.manifest, candidate.artifacts) + ), + "manifest": _json_value(candidate.manifest), + "artifacts": [ + { + "digest": str(artifact.digest), + "mediaType": artifact.media_type, + "metadata": _json_value(artifact.metadata), + } + for artifact in candidate.artifacts + ], + } + try: + with self._transaction() as cursor: + self._execute( + cursor, + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + ( + "replica-decision-candidate:" + f"{self.scope_key}:{candidate.id}", + ), + ) + self._execute( + cursor, + """ + SELECT digest, descriptor FROM replica_decision_runtime.replica_decision_candidates + WHERE scope_key = %s AND candidate_id = %s + FOR UPDATE + """, + (self.scope_key, candidate.id), + ) + existing = cursor.fetchone() + if existing is not None: + if ( + existing[0] != str(candidate.digest) + or _loaded_json(existing[1]) != descriptor + ): + raise ConnectorError( + "candidate_immutable", + "The candidate ID already identifies different content.", + category="integrity", + ) + return candidate + self._execute( + cursor, + """ + INSERT INTO replica_decision_runtime.replica_decision_candidates + (scope_key, candidate_id, digest, descriptor) + VALUES (%s, %s, %s, %s::jsonb) + """, + ( + self.scope_key, + candidate.id, + str(candidate.digest), + json.dumps(descriptor), + ), + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return candidate + + def get(self, candidate_id: str, context: OperationContext) -> Candidate: + self._check(context, hide=True) + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + SELECT descriptor FROM replica_decision_runtime.replica_decision_candidates + WHERE scope_key = %s AND candidate_id = %s + """, + (self.scope_key, candidate_id), + ) + row = cursor.fetchone() + except Exception as error: + raise _database_error(error) from error + if row is None: + raise ConnectorError( + "candidate_not_found", + "The candidate was not found.", + category="not_found", + ) + descriptor = _loaded_json(row[0]) + try: + artifacts = tuple( + self.artifacts.get(_content_digest(item["digest"]), context) + for item in descriptor["artifacts"] + ) + candidate = Candidate( + descriptor["id"], + _content_digest(descriptor["digest"]), + descriptor["manifest"], + artifacts, + ) + except (KeyError, TypeError, ValueError): + raise ConnectorError( + "candidate_damaged", + "The candidate descriptor is invalid.", + category="integrity", + ) from None + if ( + candidate.id != candidate_id + or str(_candidate_content_digest(candidate.manifest, candidate.artifacts)) + != descriptor.get("contentDigest") + ): + raise ConnectorError( + "candidate_damaged", + "The candidate no longer matches its immutable identity.", + category="integrity", + ) + return candidate + + def promote( + self, + candidate_id: str, + *, + expected_generation: int, + operation: str, + context: OperationContext, + ) -> Promotion: + self._check(context) + if operation not in ("promote", "rollback"): + raise ConnectorError( + "promotion_operation_invalid", + "A promotion operation must be promote or rollback.", + category="invalid_input", + ) + request_digest = _request_digest( + { + "candidateId": candidate_id, + "expectedGeneration": expected_generation, + "operation": operation, + "owner": {"kind": context.owner.kind, "id": context.owner.id}, + "scope": self.scope_key, + } + ) + try: + with self._transaction() as cursor: + self._execute( + cursor, + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (f"replica-decision-promotion:{self.scope_key}",), + ) + self._execute( + cursor, + """ + SELECT generation, candidate_id, previous_candidate_id, + operation, owner_kind, owner_id, request_digest + FROM replica_decision_runtime.replica_decision_promotions + WHERE scope_key = %s AND idempotency_key = %s + FOR UPDATE + """, + (self.scope_key, context.idempotency_key), + ) + retry = cursor.fetchone() + if retry is not None: + if retry[6] != request_digest: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key was reused with different input.", + category="conflict", + ) + return _promotion_from_row(retry) + self._execute( + cursor, + """ + SELECT generation, candidate_id + FROM replica_decision_runtime.replica_decision_promotions + WHERE scope_key = %s + ORDER BY generation DESC LIMIT 1 + FOR UPDATE + """, + (self.scope_key,), + ) + current = cursor.fetchone() + generation = current[0] if current is not None else 0 + previous = current[1] if current is not None else None + if generation != expected_generation: + raise ConnectorError( + "promotion_generation_conflict", + "The promotion generation changed.", + category="conflict", + ) + self._execute( + cursor, + """ + SELECT 1 FROM replica_decision_runtime.replica_decision_candidates + WHERE scope_key = %s AND candidate_id = %s + """, + (self.scope_key, candidate_id), + ) + if cursor.fetchone() is None: + raise ConnectorError( + "candidate_not_found", + "The candidate was not found.", + category="not_found", + ) + promotion = Promotion( + generation + 1, + candidate_id, + previous, + operation, + context.owner, + ) + self._execute( + cursor, + """ + INSERT INTO replica_decision_runtime.replica_decision_promotions + (scope_key, generation, candidate_id, previous_candidate_id, + operation, owner_kind, owner_id, idempotency_key, request_digest) + VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s) + """, + ( + self.scope_key, + promotion.generation, + promotion.candidate_id, + promotion.previous_candidate_id, + promotion.operation, + promotion.owner.kind, + promotion.owner.id, + context.idempotency_key, + request_digest, + ), + ) + except ConnectorError: + raise + except Exception as error: + raise _database_error(error) from error + return promotion + + def current(self, context: OperationContext) -> Promotion | None: + check_context(context) + if context.scope != self.scope: + return None + try: + with self._transaction() as cursor: + self._execute( + cursor, + """ + SELECT generation, candidate_id, previous_candidate_id, + operation, owner_kind, owner_id, request_digest + FROM replica_decision_runtime.replica_decision_promotions + WHERE scope_key = %s + ORDER BY generation DESC LIMIT 1 + """, + (self.scope_key,), + ) + row = cursor.fetchone() + except Exception as error: + raise _database_error(error) from error + return None if row is None else _promotion_from_row(row) + + def artifact_path( + self, + candidate_id: str, + name: str, + context: OperationContext, + ) -> Path: + if ( + not isinstance(name, str) + or not name + or Path(name).name != name + or name in (".", "..") + ): + raise ConnectorError( + "candidate_artifact_name_invalid", + "Candidate artifact names must be single file names.", + category="invalid_input", + ) + candidate = self.get(candidate_id, context) + artifact = next( + ( + item + for item in candidate.artifacts + if item.metadata.get("name") == name + ), + None, + ) + if artifact is None: + raise ConnectorError( + "candidate_artifact_not_found", + "The candidate artifact was not found.", + category="not_found", + ) + directory = self._cache_root / hashlib.sha256( + candidate_id.encode("utf-8") + ).hexdigest() + directory.mkdir(parents=True, exist_ok=True) + path = directory / name + if not path.is_file(): + staging = directory / f".{name}.{uuid.uuid4().hex}" + staging.write_bytes(artifact.data) + os.chmod(staging, 0o400) + try: + staging.rename(path) + except OSError: + if not path.is_file(): + raise + finally: + staging.unlink(missing_ok=True) + if _digest(path.read_bytes()) != artifact.digest: + raise ConnectorError( + "candidate_cache_damaged", + "The cached candidate artifact is invalid.", + category="integrity", + ) + return path + + +class AzureContainerAppsTrainingConnector: + """One bounded, scale-to-zero Azure job execution per training request.""" + + name = "azure-container-apps-job" + production_eligible = True + + def __init__( + self, + transport: AzureJobTransport, + job_name: str, + *, + supports_hard_credit_budget: bool = True, + poll_interval_seconds: float = 2, + sleep: Callable[[float], None] = time.sleep, + cancellation_recovery_hook: CancellationRecoveryHook, + credential_refresh: CredentialRefresh | None = None, + ): + self.transport = transport + self.job_name = job_name + self.supports_hard_credit_budget = supports_hard_credit_budget + self.poll_interval_seconds = poll_interval_seconds + self.sleep = sleep + self.cancellation_recovery_hook = cancellation_recovery_hook + self.credential_refresh = credential_refresh + + def train( + self, + taskset: Any, + work_directory: Path, + *, + max_updates: int, + context: OperationContext, + credential: Credential | None, + ) -> TrainingResult: + check_context(context) + del work_directory + if context.budget_mode == "hard" and not self.supports_hard_credit_budget: + raise ConnectorError( + "hard_budget_unsupported", + "The Azure job connector cannot enforce a hard credit budget.", + category="invalid_input", + ) + remaining = context.deadline - time.monotonic() + payload = { + "operationId": context.operation_id, + "idempotencyKey": context.idempotency_key, + "scope": { + "tenantId": context.scope.tenant_id, + "repository": context.scope.repository, + }, + "taskset": { + "reference": f"taskset:{taskset_digest(taskset).value}", + "exportId": taskset.metadata.get("exportId"), + "taskCount": len(taskset.tasks), + }, + "limits": { + "maxUpdates": max_updates, + "deadlineEpoch": time.time() + max(0, remaining), + "maxCredits": context.max_credits, + "budgetMode": context.budget_mode, + }, + } + active_credential = self._credential(context, credential) + token = active_credential.value if active_credential is not None else None + try: + started = self.transport.start( + self.job_name, + payload, + bearer_token=token, + ) + if not isinstance(started, Mapping): + raise TypeError("invalid Azure execution") + execution_id = started["executionId"] + if not isinstance(execution_id, str) or not execution_id: + raise TypeError("invalid Azure execution ID") + except (KeyError, TypeError): + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an invalid training execution.", + category="integrity", + ) from None + except Exception as error: + raise ConnectorError( + "azure_training_unavailable", + "The Azure training job could not start.", + category="unavailable", + retryable=True, + ) from error + while True: + try: + check_context(context) + except ConnectorError: + self._cancel(execution_id, active_credential, context) + raise + try: + active_credential = self._credential(context, active_credential) + token = ( + active_credential.value + if active_credential is not None + else None + ) + status = self.transport.status( + execution_id, + bearer_token=token, + ) + except Exception as error: + try: + check_context(context) + except ConnectorError: + self._cancel(execution_id, active_credential, context) + raise + self._cancel(execution_id, active_credential, context) + raise ConnectorError( + "azure_training_unavailable", + "The Azure training job status is unavailable.", + category="unavailable", + retryable=True, + ) from error + try: + check_context(context) + except ConnectorError: + self._cancel(execution_id, active_credential, context) + raise + if not isinstance(status, Mapping): + self._cancel(execution_id, active_credential, context) + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an invalid training status.", + category="integrity", + ) + state = status.get("state") + if not isinstance(state, str): + self._cancel(execution_id, active_credential, context) + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an invalid training state.", + category="integrity", + ) + if state in _TERMINAL_JOB_STATES: + break + if state not in _ACTIVE_JOB_STATES: + self._cancel(execution_id, active_credential, context) + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an unsupported training state.", + category="integrity", + ) + self.sleep( + min( + self.poll_interval_seconds, + max(0, context.deadline - time.monotonic()), + ) + ) + if state == "cancelled": + raise ConnectorError( + "azure_training_cancelled", + "The Azure training job was cancelled.", + category="cancelled", + ) + if state != "succeeded": + raise ConnectorError( + "azure_training_failed", + "The Azure training job failed.", + category="internal", + ) + return _training_result(status, context, self.name) + + def _cancel( + self, + execution_id: str, + credential: Credential | None, + context: OperationContext, + ) -> None: + recovery_error: Exception | None = None + try: + self.cancellation_recovery_hook(execution_id, context) + except Exception as error: + recovery_error = error + for attempt in range(3): + try: + refreshed = self._credential(context, credential) + self.transport.cancel( + execution_id, + bearer_token=(refreshed.value if refreshed is not None else None), + ) + return + except Exception: + if attempt < 2: + self.sleep(0.1 * (attempt + 1)) + if recovery_error is not None: + raise ConnectorError( + "azure_training_cancellation_unrecorded", + "Azure training cancellation failed and could not be scheduled for recovery.", + category="internal", + ) from recovery_error + + def _credential( + self, + context: OperationContext, + current: Credential | None, + ) -> Credential | None: + now = time.monotonic() + if ( + current is not None + and current.expires_at > now + and current.expires_at >= min(context.deadline, now + 30) + ): + return current + if self.credential_refresh is None: + if current is None: + return None + raise ConnectorError( + "credential_refresh_required", + "The Azure training credential expires before the operation completes.", + category="unauthorized", + ) + refreshed = self.credential_refresh(context) + if refreshed is None: + return None + if ( + refreshed.scope != context.scope + or refreshed.operation_id != context.operation_id + or refreshed.expires_at <= time.monotonic() + ): + raise ConnectorError( + "credential_invalid", + "The refreshed Azure credential is not valid for this operation.", + category="unauthorized", + ) + return refreshed + + +class AzureEndpointInferenceConnector: + """Scoped inference through a customer endpoint, never a training/admin API.""" + + name = "azure-customer-endpoint" + + def __init__( + self, + transport: AzureEndpointTransport, + *, + endpoint_name: str, + expected_generation: int | Callable[[], int], + credential_audience: str = "inference", + ): + self.transport = transport + self.endpoint_name = endpoint_name + self.credential_audience = credential_audience + self.expected_generation = expected_generation + + def predict( + self, + adapter: Path, + request: Mapping[str, Any], + *, + context: OperationContext, + credential: Credential | None, + ) -> Mapping[str, Any]: + check_context(context) + if credential is not None and credential.audience != self.credential_audience: + raise ConnectorError( + "inference_credential_unauthorized", + "The credential is not scoped to the inference endpoint.", + category="unauthorized", + ) + expected_generation = ( + self.expected_generation() + if callable(self.expected_generation) + else self.expected_generation + ) + if ( + isinstance(expected_generation, bool) + or not isinstance(expected_generation, int) + or expected_generation < 1 + ): + raise ConnectorError( + "expected_generation_invalid", + "The expected inference generation is invalid.", + category="invalid_input", + ) + requested_generation = request.get("expectedGeneration") + if ( + isinstance(requested_generation, bool) + or not isinstance(requested_generation, int) + or requested_generation < 1 + ): + raise ConnectorError( + "expected_generation_required", + "Inference requires the promotion generation selected by the runtime.", + category="invalid_input", + ) + if requested_generation != expected_generation: + raise ConnectorError( + "inference_generation_conflict", + "The inference request expects another promotion generation.", + category="conflict", + ) + token = credential.value if credential is not None else None + try: + readiness = self.transport.readiness(bearer_token=token) + except Exception as error: + raise ConnectorError( + "azure_inference_unavailable", + "The customer inference endpoint is unavailable.", + category="unavailable", + retryable=True, + ) from error + if not isinstance(readiness, Mapping): + raise ConnectorError( + "azure_inference_response_invalid", + "The customer inference endpoint returned invalid readiness.", + category="integrity", + ) + generation = readiness.get("generation") + if generation != expected_generation: + raise ConnectorError( + "inference_generation_conflict", + "The inference endpoint serves another promotion generation.", + category="conflict", + retryable=readiness.get("state") in {"starting", "updating"}, + ) + if readiness.get("state") != "ready": + raise ConnectorError( + "inference_cold_start", + "The inference endpoint is not ready.", + category="unavailable", + retryable=True, + ) + check_context(context) + scoped_idempotency_key = "sha256:" + hashlib.sha256( + ( + _scope_key(context.scope) + + "\0" + + context.idempotency_key + ).encode("utf-8") + ).hexdigest() + request_identity = dict( + request, + endpoint=self.endpoint_name, + expectedGeneration=expected_generation, + ) + endpoint_request = dict( + request_identity, + operationId=context.operation_id, + idempotencyKey=scoped_idempotency_key, + ) + endpoint_request["requestDigest"] = _request_digest(request_identity) + try: + result = self.transport.predict( + endpoint_request, + bearer_token=token, + ) + except ConnectorError: + raise + except Exception as error: + raise ConnectorError( + "azure_inference_unavailable", + "The customer inference endpoint is unavailable.", + category="unavailable", + retryable=True, + ) from error + check_context(context) + if not isinstance(result, Mapping): + raise ConnectorError( + "azure_inference_response_invalid", + "The customer inference endpoint returned an invalid response.", + category="integrity", + ) + if result.get("generation") != expected_generation: + raise ConnectorError( + "inference_generation_conflict", + "The inference endpoint scored another promotion generation.", + category="conflict", + ) + answers = result.get("answers") + if not isinstance(answers, Mapping): + raise ConnectorError( + "azure_inference_response_invalid", + "The customer inference endpoint returned invalid answers.", + category="integrity", + ) + return dict(answers) + + +class AzureManagedIdentityCredentialConnector(_AzureScopedConnector): + """Short-lived workload identity credentials with no durable cloud keys.""" + + def __init__( + self, + provider: AzureTokenProvider, + scope: RuntimeScope, + *, + allowed_audiences: Sequence[str], + ): + super().__init__(scope) + self.provider = provider + self.allowed_audiences = frozenset(allowed_audiences) + + def acquire( + self, audience: str, context: OperationContext + ) -> Credential | None: + self._check(context) + if audience not in self.allowed_audiences: + raise ConnectorError( + "credential_audience_unauthorized", + "The requested credential audience is not allowed.", + category="unauthorized", + ) + try: + token = self.provider.get_token(audience) + value = token.token + expires_on = float(token.expires_on) + except Exception as error: + raise ConnectorError( + "managed_identity_unavailable", + "Azure managed identity could not mint a credential.", + category="unavailable", + retryable=True, + ) from error + lifetime = max(0, expires_on - time.time()) + expires_at = min(context.deadline, time.monotonic() + lifetime) + if not value or expires_at <= time.monotonic(): + raise ConnectorError( + "managed_identity_expired", + "Azure managed identity returned an expired credential.", + category="unavailable", + retryable=True, + ) + return Credential( + value, + expires_at, + audience, + context.scope, + context.operation_id, + ) + + +@dataclass(frozen=True) +class _TrainingCredentialRefresh: + provider_factory: TokenProviderFactory + scope: RuntimeScope + + def __call__(self, context: OperationContext) -> Credential | None: + refresh_context = OperationContext( + operation_id=context.operation_id, + idempotency_key=context.idempotency_key, + scope=context.scope, + owner=context.owner, + deadline=max(context.deadline, time.monotonic() + 300), + cancellation=Cancellation(), + max_credits=context.max_credits, + budget_mode=context.budget_mode, + ) + credentials = AzureManagedIdentityCredentialConnector( + self.provider_factory(), + self.scope, + allowed_audiences=("training",), + ) + return credentials.acquire("training", refresh_context) + + +class AzureObservabilityConnector(_AzureScopedConnector): + """Emit allowlisted usage and operation fields to Pino/OpenTelemetry bridges.""" + + def __init__(self, sink: AzureTelemetrySink, scope: RuntimeScope): + super().__init__(scope) + self.sink = sink + + def record(self, usage: Usage, context: OperationContext) -> None: + self._check(context) + if usage.scope != context.scope or usage.operation_id != context.operation_id: + raise ConnectorError( + "usage_scope_mismatch", + "Usage attribution does not match the operation.", + category="invalid_input", + ) + _validate_usage_fields(usage.kind, usage.unit, usage.currency) + metadata = { + key: value + for key, value in usage.metadata.items() + if key in _SAFE_USAGE_METADATA + and isinstance(value, (str, int, float, bool, type(None))) + } + attributes = { + "event": "replica.decision.usage", + "operation_id": usage.operation_id, + "tenant_id_hash": hashlib.sha256( + context.scope.tenant_id.encode("utf-8") + ).hexdigest()[:16], + "repository_hash": hashlib.sha256( + context.scope.repository.encode("utf-8") + ).hexdigest()[:16], + "connector": usage.connector, + "kind": usage.kind, + "quantity": usage.quantity, + "unit": usage.unit, + "cost": usage.cost, + "currency": usage.currency, + "source": usage.source, + **metadata, + } + self._emit("replica.decision.usage", attributes) + + def record_operation( + self, + name: str, + status: str, + context: OperationContext, + *, + metadata: Mapping[str, Any] | None = None, + ) -> None: + self._check(context) + safe_metadata = { + key: value + for key, value in (metadata or {}).items() + if key in _SAFE_USAGE_METADATA + and isinstance(value, (str, int, float, bool, type(None))) + } + self._emit( + "replica.decision.operation", + { + "event": "replica.decision.operation", + "name": name, + "status": status, + "operation_id": context.operation_id, + "tenant_id_hash": hashlib.sha256( + context.scope.tenant_id.encode("utf-8") + ).hexdigest()[:16], + "repository_hash": hashlib.sha256( + context.scope.repository.encode("utf-8") + ).hexdigest()[:16], + **safe_metadata, + }, + ) + + def _emit(self, name: str, attributes: Mapping[str, Any]) -> None: + try: + self.sink.emit(name, attributes) + except Exception as error: + raise ConnectorError( + "azure_observability_unavailable", + "The Azure observability sink is unavailable.", + category="unavailable", + retryable=True, + ) from error + + +@dataclass(frozen=True) +class AzureConnectorBundle: + """Azure composition plus the provider-neutral connector set.""" + + connectors: RuntimeConnectors + scope: RuntimeScope + postgres_schema: str = AZURE_POSTGRES_SCHEMA + + def close(self) -> None: + self.connectors.tasksets.close() + self.connectors.candidates.close() + + +def create_azure_connector_bundle( + *, + scope: RuntimeScope, + blob_container: AzureBlobContainer, + connection_factory: ConnectionFactory, + training: Mapping[str, AzureContainerAppsTrainingConnector], + inference: Mapping[str, AzureEndpointInferenceConnector], + token_provider: AzureTokenProvider, + training_token_provider_factory: TokenProviderFactory, + credential_audiences: Sequence[str], + telemetry_sink: AzureTelemetrySink, + retention_hook: RetentionHook | None = None, +) -> AzureConnectorBundle: + """Compose the first customer-operated Azure reference bundle.""" + artifacts = AzureBlobArtifactConnector( + blob_container, + scope, + retention_hook=retention_hook, + ) + credentials = AzureManagedIdentityCredentialConnector( + token_provider, + scope, + allowed_audiences=credential_audiences, + ) + + refresh_training_credential = _TrainingCredentialRefresh( + training_token_provider_factory, + scope, + ) + for connector in training.values(): + connector.credential_refresh = refresh_training_credential + + connectors = RuntimeConnectors( + tasksets=AzureBlobTasksetConnector( + blob_container, + scope, + retention_hook=retention_hook, + ), + artifacts=artifacts, + jobs=AzurePostgresJobConnector(connection_factory, scope), + candidates=AzurePostgresCandidateConnector( + connection_factory, + artifacts, + scope, + ), + training=training, + inference=inference, + credentials=credentials, + observability=AzureObservabilityConnector(telemetry_sink, scope), + ) + return AzureConnectorBundle(connectors, scope) + + +def _loaded_json(value: Any) -> dict[str, Any]: + if isinstance(value, str): + value = json.loads(value) + if not isinstance(value, dict): + raise ConnectorError( + "azure_postgres_state_damaged", + "The PostgreSQL decision state is invalid.", + category="integrity", + ) + return value + + +def _database_error(error: Exception) -> ConnectorError: + if isinstance(error, ConnectorError): + return error + return ConnectorError( + "azure_postgres_unavailable", + "The Replica PostgreSQL decision state is unavailable.", + category="unavailable", + retryable=True, + ) + + +def _check_ttl(ttl_seconds: float) -> None: + if not math.isfinite(ttl_seconds) or ttl_seconds <= 0: + raise ConnectorError( + "lease_ttl_invalid", + "A job lease needs a positive lifetime.", + category="invalid_input", + ) + + +def _content_digest(value: str) -> ContentDigest: + algorithm, separator, digest = value.partition(":") + if not separator: + raise ValueError("invalid digest") + return ContentDigest(algorithm, digest) + + +def _promotion_from_row(row: Sequence[Any]) -> Promotion: + return Promotion( + row[0], + row[1], + row[2], + row[3], + RuntimeOwner(row[4], row[5]), + ) + + +def _training_result( + status: Mapping[str, Any], + context: OperationContext, + connector: str, +) -> TrainingResult: + result = status.get("result") + if not isinstance(result, Mapping): + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an invalid training result.", + category="integrity", + ) + try: + adapter = base64.b64decode(result["adapterBase64"], validate=True) + usage_records = [] + for item in result.get("usage", ()): + _validate_usage_fields( + item["kind"], + item["unit"], + item.get("currency"), + ) + usage_records.append( + Usage( + operation_id=context.operation_id, + scope=context.scope, + connector=connector, + kind=item["kind"], + quantity=float(item["quantity"]), + unit=item["unit"], + cost=float(item["cost"]) if item.get("cost") is not None else None, + currency=item.get("currency"), + metadata={ + **{ + key: value + for key, value in item.get("metadata", {}).items() + if key in _SAFE_USAGE_METADATA + and not key.startswith("azure.") + }, + }, + source="azure", + ) + ) + usage = tuple(usage_records) + training = TrainingResult( + result["adapterName"], + adapter, + result["baselineReport"], + result["candidateReport"], + result.get("metadata", {}), + int(result["updatesCompleted"]), + usage, + ) + except (KeyError, TypeError, ValueError): + raise ConnectorError( + "azure_training_response_invalid", + "Azure returned an invalid training result.", + category="integrity", + ) from None + return training + + +def _validate_usage_fields( + kind: Any, + unit: Any, + currency: Any, +) -> None: + if ( + kind not in _USAGE_KINDS + or unit not in _USAGE_UNITS + or ( + currency is not None + and ( + not isinstance(currency, str) + or _CURRENCY.fullmatch(currency) is None + ) + ) + ): + raise ConnectorError( + "usage_field_invalid", + "Provider usage contains an unsupported kind, unit, or currency.", + category="invalid_input", + ) + + +def _extract_taskset_archive(source: tarfile.TarFile, root: Path) -> None: + expanded = 0 + members = 0 + for member in source: + members += 1 + if members > _MAX_TASKSET_ARCHIVE_MEMBERS: + raise tarfile.TarError("taskset archive has too many members") + target = (root / member.name).resolve() + maximum_member_bytes = ( + _MAX_TASKSET_DESCRIPTOR_BYTES + if member.name in ("taskset.json", "tasks.jsonl") + else MAX_IMAGE_BYTES + ) + if ( + member.islnk() + or member.issym() + or member.isdev() + or not member.isfile() + or member.size > maximum_member_bytes + or expanded + member.size > _MAX_TASKSET_EXPANDED_BYTES + or root.resolve() not in target.parents + ): + raise tarfile.TarError("unsafe taskset archive") + extracted = source.extractfile(member) + if extracted is None: + raise tarfile.TarError("missing taskset archive member") + data = extracted.read(member.size + 1) + if len(data) != member.size: + raise tarfile.TarError("taskset archive member size changed") + expanded += len(data) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(data) + os.chmod(target, 0o400) diff --git a/packages/learning/environments/brand-check/connector_conformance.py b/packages/learning/environments/brand-check/connector_conformance.py index 34be1f16..7b419fce 100644 --- a/packages/learning/environments/brand-check/connector_conformance.py +++ b/packages/learning/environments/brand-check/connector_conformance.py @@ -883,7 +883,7 @@ def run(case): ) prediction = fixture.connectors.inference[connector.name].predict( adapter, - {"questions": {}}, + {"questions": {}, "expectedGeneration": 1}, context=inference_context, credential=inference_credential, ) diff --git a/packages/learning/environments/brand-check/runtime.py b/packages/learning/environments/brand-check/runtime.py index 83417e47..75273e3b 100644 --- a/packages/learning/environments/brand-check/runtime.py +++ b/packages/learning/environments/brand-check/runtime.py @@ -721,11 +721,13 @@ def evaluate( else None ) project_request = getattr(connector, "project_request", _backend_request) + connector_request = dict(project_request(request)) + connector_request["expectedGeneration"] = promotion["generation"] answers = dict( _run_inference( connector, adapter, - project_request(request), + connector_request, work_directory=work, context=context, credential=credential, @@ -1761,6 +1763,7 @@ def _run_backend( ) -> TrainingResult: """Run backend code in a child that the runtime can stop at the job deadline.""" process_context = _process_context() + cancellation_event = process_context.Event() result_path = work_directory / "result.pickle" process = process_context.Process( target=_backend_process, @@ -1779,20 +1782,33 @@ def _run_backend( context.max_credits, context.budget_mode, credential, + cancellation_event, ), name=f"replica-{connector.name}-job", ) process.start() + timed_out = False try: while process.is_alive(): - context.cancellation.raise_if_requested() + if context.cancellation.requested: + cancellation_event.set() + process.join(5) + context.cancellation.raise_if_requested() remaining = context.deadline - time.monotonic() if remaining <= 0: + timed_out = True break process.join(min(0.05, remaining)) if process.is_alive(): - context.cancellation.cancel() - _stop_process(process) + cancellation_event.set() + process.join(5) + if timed_out: + if process.is_alive(): + _stop_process(process) + raise RuntimeFailure( + "job_timed_out", "The bounded training job reached maxSeconds." + ) + if time.monotonic() >= context.deadline: raise RuntimeFailure( "job_timed_out", "The bounded training job reached maxSeconds." ) @@ -1840,6 +1856,7 @@ def _backend_process( max_credits: float | None, budget_mode: str, credential: Credential | None, + cancellation_event: Any, ) -> None: """Load task data afresh and run one serializable connector in a spawned child process.""" try: @@ -1856,7 +1873,7 @@ def _backend_process( scope=scope, owner=owner, deadline=deadline, - cancellation=Cancellation(), + cancellation=Cancellation(cancellation_event), max_credits=max_credits, budget_mode=budget_mode, ) diff --git a/packages/learning/environments/brand-check/tests/test_azure_connectors.py b/packages/learning/environments/brand-check/tests/test_azure_connectors.py new file mode 100644 index 00000000..5d2c5781 --- /dev/null +++ b/packages/learning/environments/brand-check/tests/test_azure_connectors.py @@ -0,0 +1,2168 @@ +"""Contract coverage for the customer-operated Azure connector bundle.""" + +from __future__ import annotations + +import hashlib +import io +import json +import pickle +import tarfile +import tempfile +import threading +import time +import unittest +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +import replica_brand_check.azure_connectors as azure_module +from replica_brand_check.azure_connectors import ( + AzureBlobArtifactConnector, + AzureBlobTasksetConnector, + AzureContainerAppsTrainingConnector, + AzureEndpointInferenceConnector, + AzureManagedIdentityCredentialConnector, + AzureObservabilityConnector, + AzurePostgresCandidateConnector, + AzurePostgresJobConnector, + _extract_taskset_archive, + create_azure_connector_bundle, + initialize_azure_postgres, +) +from replica_brand_check.taskset import MAX_IMAGE_BYTES +from replica_brand_check.connector_conformance import ( + ConnectorConformanceFixture, + connector_conformance, +) +from replica_brand_check.connectors import ( + Artifact, + Cancellation, + Candidate, + ConnectorError, + ContentDigest, + Credential, + OperationContext, + RuntimeOwner, + RuntimeScope, + Usage, +) +from tests.support import make_task, taskset_bundle, write_taskset + + +_RESOURCES = [] + + +def _digest(data: bytes) -> ContentDigest: + return ContentDigest("sha256", hashlib.sha256(data).hexdigest()) + + +def _context( + scope: RuntimeScope, + operation_id: str, + *, + idempotency_key: str | None = None, + deadline: float | None = None, + cancellation: Cancellation | None = None, + max_credits: float | None = None, + budget_mode: str = "none", +) -> OperationContext: + return OperationContext( + operation_id=operation_id, + idempotency_key=idempotency_key or operation_id, + scope=scope, + owner=RuntimeOwner("worker", "azure-connector-test"), + deadline=deadline if deadline is not None else time.monotonic() + 30, + cancellation=cancellation or Cancellation(), + max_credits=max_credits, + budget_mode=budget_mode, + ) + + +def _artifact(data: bytes, **metadata) -> Artifact: + return Artifact( + digest=_digest(data), + media_type="application/octet-stream", + data=data, + metadata=metadata, + ) + + +def _candidate(name: str) -> Candidate: + artifact = _artifact(name.encode(), name="adapter.bin") + manifest = {"name": name, "artifact": str(artifact.digest)} + identity = hashlib.sha256() + identity.update( + ( + json.dumps(manifest, sort_keys=True, separators=(",", ":")) + "\n" + ).encode() + ) + identity.update(b"\0") + identity.update( + ( + json.dumps( + { + "digest": str(artifact.digest), + "mediaType": artifact.media_type, + "metadata": artifact.metadata, + }, + sort_keys=True, + separators=(",", ":"), + ) + + "\n" + ).encode() + ) + identity.update(b"\0") + identity.update(artifact.data) + return Candidate( + id=f"candidate_{name}", + digest=ContentDigest("sha256", identity.hexdigest()), + manifest=manifest, + artifacts=(artifact,), + ) + + +class _BlobDownload: + def __init__(self, value): + self._value = value + + def readall(self): + return self._value["data"] + + def chunks(self): + data = self._value["data"] + for offset in range(0, len(data), 64 * 1024): + yield data[offset : offset + 64 * 1024] + + +class ResourceExistsError(Exception): + pass + + +class ResourceNotFoundError(Exception): + pass + + +class _BlobService: + def __init__(self): + self.blobs = {} + self.versions = {} + self.stale_version_names = set() + + def upload_blob( + self, + name, + data, + *, + overwrite=False, + metadata=None, + **kwargs, + ): + del kwargs + if name in self.blobs and not overwrite: + raise ResourceExistsError(name) + if name in self.blobs: + self.versions.setdefault(name, []).append( + { + "id": f"version-{len(self.versions.get(name, [])) + 1}", + **self.blobs[name], + } + ) + self.blobs[name] = { + "data": bytes(data), + "metadata": dict(metadata or {}), + } + + def download_blob(self, name): + try: + return _BlobDownload(self.blobs[name]) + except KeyError: + raise ResourceNotFoundError(name) from None + + def list_blob_versions(self, name): + versions = [version["id"] for version in self.versions.get(name, ())] + if name in self.stale_version_names: + versions.insert(0, "already-deleted") + return tuple(versions) + + def delete_blob(self, name, *, version_id=None, permanent=False): + if not permanent: + raise AssertionError("withdrawal deletion must be permanent") + if version_id is not None: + versions = self.versions.get(name, []) + remaining = [ + version for version in versions if version["id"] != version_id + ] + if len(remaining) == len(versions): + raise ResourceNotFoundError(f"{name}?versionid={version_id}") + self.versions[name] = remaining + return + if self.blobs.pop(name, None) is None: + raise ResourceNotFoundError(name) + + +class _InterruptedBlobService(_BlobService): + def __init__(self): + super().__init__() + self.fail_descriptor_once = True + + def upload_blob(self, name, data, **kwargs): + if name.endswith(".json") and self.fail_descriptor_once: + self.fail_descriptor_once = False + raise RuntimeError("interrupted descriptor write") + return super().upload_blob(name, data, **kwargs) + + +class _Postgres: + """A transaction-shaped fake for the connector's PostgreSQL state procedures.""" + + def __init__(self): + self.jobs = {} + self.job_retries = {} + self.leases = {} + self.candidates = {} + self.promotions = {} + self.promotion_retries = {} + self.transactions = 0 + self.sql_jobs = {} + self.sql_candidates = {} + self.sql_promotions = [] + self.sql_idempotency = {} + + def __call__(self): + return self + + def cursor(self): + self.transactions += 1 + return _PostgresCursor(self) + + def commit(self): + pass + + def rollback(self): + pass + + def close(self): + pass + + @contextmanager + def transaction(self): + self.transactions += 1 + yield self + + def create_job(self, scope, job_id, record, idempotency_key): + retry_key = (scope, idempotency_key) + signature = (job_id, dict(record)) + existing_retry = self.job_retries.get(retry_key) + if existing_retry is not None: + if existing_retry["signature"] != signature: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key names different job input.", + category="conflict", + ) + return dict(existing_retry["result"]) + key = (scope, job_id) + if key in self.jobs: + raise ConnectorError( + "job_conflict", "The job already exists.", category="conflict" + ) + result = {**record, "id": job_id, "version": 1} + self.jobs[key] = result + self.job_retries[retry_key] = {"signature": signature, "result": result} + return dict(result) + + def get_job(self, scope, job_id): + try: + return dict(self.jobs[(scope, job_id)]) + except KeyError: + raise ConnectorError( + "job_not_found", "The job was not found.", category="not_found" + ) from None + + def list_jobs(self, scope, states): + return [ + dict(record) + for (record_scope, _), record in self.jobs.items() + if record_scope == scope and record["state"] in states + ] + + def update_job(self, scope, job_id, record, expected_version, lease_token): + current = self.get_job(scope, job_id) + lease = self.leases.get((scope, job_id)) + if ( + current["version"] != expected_version + or lease is None + or lease["token"] != lease_token + or lease["expires_at"] <= time.monotonic() + or current["state"] == "cancelled" + ): + raise ConnectorError( + "job_update_conflict", + "The job version or lease is stale.", + category="conflict", + ) + updated = {**record, "id": job_id, "version": expected_version + 1} + self.jobs[(scope, job_id)] = updated + return dict(updated) + + def cancel_job(self, scope, job_id, expected_version, idempotency_key): + retry_key = (scope, idempotency_key) + signature = (job_id, expected_version) + existing_retry = self.job_retries.get(retry_key) + if existing_retry is not None: + if existing_retry["signature"] != signature: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key names different cancellation input.", + category="conflict", + ) + return dict(existing_retry["result"]) + current = self.get_job(scope, job_id) + if current["version"] != expected_version: + raise ConnectorError( + "job_version_conflict", + "The job version is stale.", + category="conflict", + ) + cancelled = {**current, "state": "cancelled", "version": expected_version + 1} + self.jobs[(scope, job_id)] = cancelled + self.job_retries[retry_key] = { + "signature": signature, + "result": cancelled, + } + return dict(cancelled) + + def acquire_lease(self, scope, job_id, owner, ttl_seconds): + current = self.get_job(scope, job_id) + if current["state"] == "cancelled": + raise ConnectorError( + "job_cancelled", "A cancelled job cannot be leased.", category="conflict" + ) + key = (scope, job_id) + current_lease = self.leases.get(key) + if current_lease is not None and current_lease["expires_at"] > time.monotonic(): + raise ConnectorError( + "lease_conflict", "The job already has a lease.", category="conflict" + ) + lease = { + "token": f"lease-{len(self.leases) + self.transactions}", + "owner": owner, + "expires_at": time.monotonic() + ttl_seconds, + } + self.leases[key] = lease + return dict(lease) + + def renew_lease(self, scope, job_id, token, ttl_seconds): + lease = self.leases.get((scope, job_id)) + if ( + lease is None + or lease["token"] != token + or lease["expires_at"] <= time.monotonic() + ): + raise ConnectorError( + "lease_conflict", "The lease is stale.", category="conflict" + ) + lease = {**lease, "expires_at": time.monotonic() + ttl_seconds} + self.leases[(scope, job_id)] = lease + return dict(lease) + + def release_lease(self, scope, job_id, token): + lease = self.leases.get((scope, job_id)) + if lease is None or lease["token"] != token: + raise ConnectorError( + "lease_conflict", "The lease is stale.", category="conflict" + ) + del self.leases[(scope, job_id)] + + def put_candidate(self, scope, candidate): + key = (scope, candidate.id) + existing = self.candidates.get(key) + if existing is not None and existing != candidate: + raise ConnectorError( + "candidate_conflict", + "The candidate ID names different content.", + category="conflict", + ) + self.candidates[key] = candidate + return candidate + + def get_candidate(self, scope, candidate_id): + try: + return self.candidates[(scope, candidate_id)] + except KeyError: + raise ConnectorError( + "candidate_not_found", + "The candidate was not found.", + category="not_found", + ) from None + + def promote_candidate( + self, + scope, + candidate_id, + expected_generation, + operation, + owner, + idempotency_key, + ): + retry_key = (scope, idempotency_key) + signature = (candidate_id, expected_generation, operation, owner) + existing_retry = self.promotion_retries.get(retry_key) + if existing_retry is not None: + if existing_retry["signature"] != signature: + raise ConnectorError( + "idempotency_conflict", + "The idempotency key names a different promotion.", + category="conflict", + ) + return existing_retry["result"] + current = self.promotions.get(scope) + generation = 0 if current is None else current.generation + if generation != expected_generation: + raise ConnectorError( + "promotion_generation_conflict", + "The promotion generation is stale.", + category="conflict", + ) + self.get_candidate(scope, candidate_id) + result = SimpleNamespace( + generation=generation + 1, + candidate_id=candidate_id, + previous_candidate_id=None if current is None else current.candidate_id, + operation=operation, + owner=owner, + ) + self.promotions[scope] = result + self.promotion_retries[retry_key] = { + "signature": signature, + "result": result, + } + return result + + def current_promotion(self, scope): + return self.promotions.get(scope) + + +class _SchemaCursor: + def __init__(self, statements): + self.statements = statements + + def execute(self, sql, parameters=()): + self.statements.append((sql, tuple(parameters))) + + def close(self): + pass + + +class _SchemaConnection: + def __init__(self): + self.statements = [] + self.committed = False + + def cursor(self): + return _SchemaCursor(self.statements) + + def commit(self): + self.committed = True + + def rollback(self): + pass + + def close(self): + pass + + +class _PostgresCursor: + def __init__(self, database): + self.database = database + self.rowcount = 0 + self._one = None + self._many = [] + + def close(self): + pass + + def fetchone(self): + value = self._one + self._one = None + return value + + def fetchall(self): + return list(self._many) + + def execute(self, sql, parameters): + query = " ".join(sql.split()) + self.rowcount = 0 + self._one = None + self._many = [] + + if query.startswith("SELECT pg_advisory_xact_lock("): + return + if query.startswith("SELECT request_digest, result FROM replica_decision_runtime.replica_decision_idempotency"): + scope, key = parameters + row = self.database.sql_idempotency.get((scope, "cancel", key)) + if row is not None: + self._one = (row["request_digest"], row["result"]) + return + if query.startswith("INSERT INTO replica_decision_runtime.replica_decision_idempotency"): + scope, key, request_digest, result = parameters + self.database.sql_idempotency[(scope, "cancel", key)] = { + "request_digest": request_digest, + "result": result, + } + self.rowcount = 1 + return + if query.startswith("SELECT job_id, create_request_digest"): + scope, key = parameters + row = next( + ( + value + for (row_scope, _), value in self.database.sql_jobs.items() + if row_scope == scope and value["create_key"] == key + ), + None, + ) + if row is not None: + self._one = ( + row["job_id"], + row["create_digest"], + row["record"], + row["version"], + ) + return + if query.startswith("SELECT 1 FROM replica_decision_runtime.replica_decision_jobs"): + scope, job_id = parameters + if (scope, job_id) in self.database.sql_jobs: + self._one = (1,) + return + if query.startswith("INSERT INTO replica_decision_runtime.replica_decision_jobs"): + scope, job_id, record, create_key, create_digest = parameters + self.database.sql_jobs[(scope, job_id)] = { + "job_id": job_id, + "record": record, + "version": 1, + "create_key": create_key, + "create_digest": create_digest, + "lease_token": None, + "lease_owner_kind": None, + "lease_owner_id": None, + "lease_expires_at": None, + "cancellation_key": None, + } + self.rowcount = 1 + return + if query.startswith("SELECT record, version FROM replica_decision_runtime.replica_decision_jobs"): + if "job_id = %s" in query: + row = self.database.sql_jobs.get(tuple(parameters)) + if row is not None: + self._one = (row["record"], row["version"]) + else: + scope = parameters[0] + self._many = [ + (row["record"], row["version"]) + for (row_scope, _), row in sorted(self.database.sql_jobs.items()) + if row_scope == scope + ] + return + if ( + query.startswith("UPDATE replica_decision_runtime.replica_decision_jobs SET record = %s::jsonb") + and "lease_token = %s" in query + ): + ( + record, + scope, + job_id, + version, + token, + owner_kind, + owner_id, + ) = parameters + now = time.time() + row = self.database.sql_jobs.get((scope, job_id)) + if ( + row is not None + and row["version"] == version + and row["lease_token"] == token + and row["lease_owner_kind"] == owner_kind + and row["lease_owner_id"] == owner_id + and row["lease_expires_at"] > now + and json.loads(row["record"]).get("state") != "cancelled" + ): + row["record"] = record + row["version"] += 1 + self.rowcount = 1 + return + if query.startswith("UPDATE replica_decision_runtime.replica_decision_jobs SET lease_token = %s"): + token, owner_kind, owner_id, ttl_seconds, scope, job_id = parameters + now = time.time() + row = self.database.sql_jobs.get((scope, job_id)) + if ( + row is not None + and json.loads(row["record"]).get("state") != "cancelled" + and ( + row["lease_token"] is None + or row["lease_expires_at"] <= now + ) + ): + row.update( + lease_token=token, + lease_owner_kind=owner_kind, + lease_owner_id=owner_id, + lease_expires_at=now + ttl_seconds, + ) + self.rowcount = 1 + return + if query.startswith("UPDATE replica_decision_runtime.replica_decision_jobs SET lease_expires_at"): + ttl_seconds, scope, job_id, token, owner_kind, owner_id = parameters + now = time.time() + row = self.database.sql_jobs.get((scope, job_id)) + if ( + row is not None + and row["lease_token"] == token + and row["lease_owner_kind"] == owner_kind + and row["lease_owner_id"] == owner_id + and row["lease_expires_at"] > now + ): + row["lease_expires_at"] = now + ttl_seconds + self.rowcount = 1 + return + if query.startswith("UPDATE replica_decision_runtime.replica_decision_jobs SET lease_token = NULL"): + scope, job_id, token, owner_kind, owner_id = parameters + now = time.time() + row = self.database.sql_jobs.get((scope, job_id)) + if ( + row is not None + and row["lease_token"] == token + and row["lease_owner_kind"] == owner_kind + and row["lease_owner_id"] == owner_id + and row["lease_expires_at"] > now + ): + row.update( + lease_token=None, + lease_owner_kind=None, + lease_owner_id=None, + lease_expires_at=None, + ) + self.rowcount = 1 + return + if query.startswith("SELECT record, version, cancellation_idempotency_key"): + row = self.database.sql_jobs.get(tuple(parameters)) + if row is not None: + self._one = ( + row["record"], + row["version"], + row["cancellation_key"], + ) + return + if "cancellation_idempotency_key = %s" in query: + record, key, scope, job_id, version = parameters + row = self.database.sql_jobs.get((scope, job_id)) + if row is not None and row["version"] == version: + row["record"] = record + row["version"] += 1 + row["cancellation_key"] = key + self.rowcount = 1 + return + if query.startswith("SELECT digest, descriptor FROM replica_decision_runtime.replica_decision_candidates"): + row = self.database.sql_candidates.get(tuple(parameters)) + if row is not None: + self._one = (row["digest"], row["descriptor"]) + return + if query.startswith("INSERT INTO replica_decision_runtime.replica_decision_candidates"): + scope, candidate_id, digest, descriptor = parameters + self.database.sql_candidates[(scope, candidate_id)] = { + "digest": digest, + "descriptor": descriptor, + } + self.rowcount = 1 + return + if query.startswith("SELECT descriptor FROM replica_decision_runtime.replica_decision_candidates"): + row = self.database.sql_candidates.get(tuple(parameters)) + if row is not None: + self._one = (row["descriptor"],) + return + if "WHERE scope_key = %s AND idempotency_key = %s" in query: + scope, key = parameters + row = next( + ( + value + for value in self.database.sql_promotions + if value["scope"] == scope and value["idempotency_key"] == key + ), + None, + ) + if row is not None: + self._one = self._promotion_row(row) + return + if ( + query.startswith("SELECT generation, candidate_id") + and "ORDER BY generation DESC" in query + and "previous_candidate_id" not in query + ): + scope = parameters[0] + rows = [ + row for row in self.database.sql_promotions if row["scope"] == scope + ] + if rows: + current = max(rows, key=lambda row: row["generation"]) + self._one = (current["generation"], current["candidate_id"]) + return + if query.startswith("SELECT 1 FROM replica_decision_runtime.replica_decision_candidates"): + self._one = ( + (1,) + if tuple(parameters) in self.database.sql_candidates + else None + ) + return + if query.startswith("INSERT INTO replica_decision_runtime.replica_decision_promotions"): + ( + scope, + generation, + candidate_id, + previous, + operation, + owner_kind, + owner_id, + idempotency_key, + request_digest, + ) = parameters + self.database.sql_promotions.append( + { + "scope": scope, + "generation": generation, + "candidate_id": candidate_id, + "previous": previous, + "operation": operation, + "owner_kind": owner_kind, + "owner_id": owner_id, + "idempotency_key": idempotency_key, + "request_digest": request_digest, + } + ) + self.rowcount = 1 + return + if ( + query.startswith("SELECT generation, candidate_id, previous_candidate_id") + and "ORDER BY generation DESC" in query + ): + scope = parameters[0] + rows = [ + row for row in self.database.sql_promotions if row["scope"] == scope + ] + if rows: + self._one = self._promotion_row( + max(rows, key=lambda row: row["generation"]) + ) + return + raise AssertionError(f"Unexpected SQL: {query}") + + @staticmethod + def _promotion_row(row): + return ( + row["generation"], + row["candidate_id"], + row["previous"], + row["operation"], + row["owner_kind"], + row["owner_id"], + row["request_digest"], + ) + + +class _TrainingClient: + def __init__( + self, + cancellation=None, + fail_after_cancellation=False, + cancel_failures=0, + reported_credits=None, + ): + self.requests = [] + self.cancelled = [] + self.status_tokens = [] + self.cancel_tokens = [] + self._cancellation = cancellation + self._fail_after_cancellation = fail_after_cancellation + self._cancel_failures = cancel_failures + self._reported_credits = reported_credits + self._polls = 0 + + def start(self, job_name, request, *, bearer_token): + self.requests.append( + {"jobName": job_name, "payload": request, "bearerToken": bearer_token} + ) + return {"executionId": "training-1"} + + def status(self, job_id, *, bearer_token): + del job_id + self.status_tokens.append(bearer_token) + self._polls += 1 + if self._cancellation is not None and self._polls == 1: + self._cancellation.cancel() + if self._fail_after_cancellation: + raise RuntimeError("status connection closed") + return {"state": "running"} + return { + "state": "succeeded", + "result": { + "adapterName": "adapter.bin", + "adapterBase64": "YWRhcHRlcg==", + "baselineReport": {"meanReward": 0.5}, + "candidateReport": {"meanReward": 0.75}, + "metadata": {"scaleToZero": True}, + "updatesCompleted": self.requests[-1]["payload"]["limits"][ + "maxUpdates" + ], + "usage": [ + { + "kind": "gpu_seconds", + "quantity": 12, + "unit": "seconds", + "cost": 1.25, + "currency": "USD", + }, + { + "kind": "training_credits", + "quantity": ( + self._reported_credits + if self._reported_credits is not None + else min( + self.requests[-1]["payload"]["limits"]["maxUpdates"], + self.requests[-1]["payload"]["limits"]["maxCredits"] + or self.requests[-1]["payload"]["limits"]["maxUpdates"], + ) + ), + "unit": "credits", + "cost": 0, + "currency": "USD", + }, + ], + }, + } + + def cancel(self, job_id, *, bearer_token): + self.cancel_tokens.append(bearer_token) + self.cancelled.append(job_id) + if self._cancel_failures > 0: + self._cancel_failures -= 1 + raise RuntimeError("cancel unavailable") + + +class _StatusFailingTrainingClient(_TrainingClient): + def status(self, job_id, *, bearer_token): + del job_id, bearer_token + raise RuntimeError("status unavailable") + + +class _MalformedStatusTrainingClient(_TrainingClient): + def status(self, job_id, *, bearer_token): + del job_id, bearer_token + return [] + + +class _UnknownStatusTrainingClient(_TrainingClient): + def status(self, job_id, *, bearer_token): + del job_id, bearer_token + return {"state": "mystery"} + + +class _InferenceClient: + def __init__(self, readiness): + self.readiness_value = readiness + self.requests = [] + self.retries = {} + + def readiness(self, *, bearer_token): + del bearer_token + return dict(self.readiness_value) + + def predict(self, request, *, bearer_token): + key = request["idempotencyKey"] + previous = self.retries.get(key) + if previous is not None and previous != request["requestDigest"]: + raise ConnectorError( + "idempotency_conflict", + "The inference idempotency key was reused with different input.", + category="conflict", + ) + self.retries[key] = request["requestDigest"] + self.requests.append({"request": request, "bearerToken": bearer_token}) + answers = ( + {"learning-one": {"yes": 0.8, "no": 0.2}} + if request.get("questions") + else {"operationId": request["operationId"]} + ) + return { + "answers": answers, + "candidateId": "candidate_live", + "generation": request["expectedGeneration"], + "operationId": request.get("operationId"), + } + + +class _IdentityClient: + def __init__(self, expires_at): + self.expires_at = expires_at + self.audiences = [] + self.lock = threading.Lock() + + def get_token(self, audience): + self.audiences.append(audience) + return SimpleNamespace(token="short-lived-token", expires_on=self.expires_at) + + +class _IdentityFactory: + def __init__(self, expires_at): + self.expires_at = expires_at + + def __call__(self): + return _IdentityClient(self.expires_at) + + +class _TelemetrySink: + def __init__(self): + self.records = [] + + def emit(self, name, attributes): + self.records.append((name, attributes)) + + +class _FailingTelemetrySink: + def emit(self, name, attributes): + del name, attributes + raise RuntimeError("telemetry unavailable") + + +class AzureBlobArtifactConnectorTest(unittest.TestCase): + def test_scopes_digest_keys_and_refuses_immutable_content_replacement(self): + blobs = _BlobService() + retention = [] + first_scope = RuntimeScope("tenant-one", "github/acme/repo") + second_scope = RuntimeScope("tenant-one", "github/acme/other") + first = AzureBlobArtifactConnector( + blobs, + first_scope, + retention_hook=lambda name, metadata: retention.append((name, metadata)), + ) + second = AzureBlobArtifactConnector( + blobs, + second_scope, + retention_hook=lambda name, metadata: retention.append((name, metadata)), + ) + artifact = _artifact(b"private report", retention_class="training") + + self.assertEqual(first.put(artifact, _context(first_scope, "put-one")), artifact) + self.assertEqual(second.put(artifact, _context(second_scope, "put-two")), artifact) + self.assertEqual(len(blobs.blobs), 4) + self.assertEqual( + first.get(artifact.digest, _context(first_scope, "get-one")), + artifact, + ) + with self.assertRaises(ConnectorError) as changed: + first.put( + Artifact( + digest=artifact.digest, + media_type=artifact.media_type, + data=b"replacement", + metadata=artifact.metadata, + ), + _context(first_scope, "replace"), + ) + self.assertEqual(changed.exception.category, "integrity") + self.assertEqual(len(retention), 2) + primary_keys = [key for key in blobs.blobs if not key.endswith(".json")] + self.assertEqual(len(primary_keys), 2) + self.assertNotEqual(*primary_keys) + + def test_retry_repairs_metadata_after_an_interrupted_artifact_write(self): + blobs = _InterruptedBlobService() + scope = RuntimeScope("tenant-one", "github/acme/repo") + connector = AzureBlobArtifactConnector(blobs, scope) + artifact = _artifact(b"candidate report", kind="candidate-report") + + with self.assertRaises(ConnectorError) as interrupted: + connector.put(artifact, _context(scope, "interrupted-put")) + self.assertEqual(interrupted.exception.category, "unavailable") + self.assertEqual(len(blobs.blobs), 1) + + recovered = connector.put(artifact, _context(scope, "retry-put")) + + self.assertEqual(recovered, artifact) + self.assertEqual( + connector.get(artifact.digest, _context(scope, "read-recovered")), + artifact, + ) + self.assertEqual(len(blobs.blobs), 2) + + def test_retry_reapplies_retention_after_a_hook_failure(self): + blobs = _BlobService() + scope = RuntimeScope("tenant-one", "github/acme/repo") + attempts = [] + + def retention(name, metadata): + attempts.append((name, metadata)) + if len(attempts) == 1: + raise RuntimeError("retention service unavailable") + + connector = AzureBlobArtifactConnector( + blobs, + scope, + retention_hook=retention, + ) + artifact = _artifact(b"retained report", kind="candidate-report") + with self.assertRaises(ConnectorError) as failed: + connector.put(artifact, _context(scope, "retention-failed")) + self.assertEqual(failed.exception.code, "retention_hook_failed") + + self.assertEqual( + connector.put(artifact, _context(scope, "retention-retry")), + artifact, + ) + self.assertEqual(len(attempts), 2) + + def test_withdrawal_deletes_current_blobs_and_every_version(self): + blobs = _BlobService() + scope = RuntimeScope("tenant-one", "github/acme/repo") + connector = AzureBlobArtifactConnector(blobs, scope) + artifact = _artifact(b"withdrawn report", kind="candidate-report") + connector.put(artifact, _context(scope, "put-withdrawn")) + names = tuple(blobs.blobs) + for name in names: + blobs.upload_blob( + name, + blobs.blobs[name]["data"], + overwrite=True, + metadata=blobs.blobs[name]["metadata"], + ) + self.assertTrue(any(blobs.versions.values())) + + connector.delete(artifact.digest, _context(scope, "delete-withdrawn")) + + self.assertFalse(blobs.blobs) + self.assertFalse(any(blobs.versions.values())) + + def test_withdrawal_continues_after_a_stale_version_listing(self): + blobs = _BlobService() + scope = RuntimeScope("tenant-one", "github/acme/repo") + connector = AzureBlobArtifactConnector(blobs, scope) + artifact = _artifact(b"withdrawn report", kind="candidate-report") + connector.put(artifact, _context(scope, "put-stale-version")) + for name in tuple(blobs.blobs): + blobs.upload_blob( + name, + blobs.blobs[name]["data"], + overwrite=True, + metadata=blobs.blobs[name]["metadata"], + ) + blobs.stale_version_names.add(name) + + connector.delete(artifact.digest, _context(scope, "delete-stale-version")) + + self.assertFalse(blobs.blobs) + self.assertFalse(any(blobs.versions.values())) + + +class AzureBlobTasksetConnectorTest(unittest.TestCase): + def test_conflicting_retry_does_not_persist_the_losing_taskset(self): + first_directory = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + second_directory = write_taskset( + self, + [ + make_task(3, answer=False), + make_task(4, split="test", answer=False), + ], + ) + scope = RuntimeScope("tenant-one", "github/acme/repo") + blobs = _BlobService() + connector = AzureBlobTasksetConnector(blobs, scope) + self.addCleanup(connector.close) + retry_context = _context( + scope, + "taskset-winner", + idempotency_key="shared-taskset-request", + ) + connector.register(taskset_bundle(first_directory), retry_context) + winner_blobs = dict(blobs.blobs) + + with self.assertRaises(ConnectorError) as losing_request: + connector.register(taskset_bundle(second_directory), retry_context) + + self.assertEqual(losing_request.exception.code, "idempotency_conflict") + self.assertEqual(blobs.blobs, winner_blobs) + + def test_concurrent_resolution_atomically_publishes_one_cache(self): + directory = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + scope = RuntimeScope("tenant-one", "github/acme/repo") + blobs = _BlobService() + writer = AzureBlobTasksetConnector(blobs, scope) + registered = writer.register( + taskset_bundle(directory), + _context(scope, "register-concurrent"), + ) + writer.close() + connector = AzureBlobTasksetConnector(blobs, scope) + self.addCleanup(connector.close) + + with ThreadPoolExecutor(max_workers=2) as executor: + results = tuple( + executor.map( + lambda operation: connector.resolve( + registered.reference, + _context(scope, operation), + ), + ("resolve-one", "resolve-two"), + ) + ) + + self.assertEqual(results[0], results[1]) + self.assertEqual(results[0].reference, registered.reference) + self.assertEqual(results[0].digest, registered.digest) + + def test_archive_extraction_rejects_an_oversized_image_before_writing(self): + archive_bytes = io.BytesIO() + with tarfile.open(fileobj=archive_bytes, mode="w") as archive: + data = b"x" * (MAX_IMAGE_BYTES + 1) + member = tarfile.TarInfo("images/large.png") + member.size = len(data) + archive.addfile(member, io.BytesIO(data)) + archive_bytes.seek(0) + + with tempfile.TemporaryDirectory() as directory: + with tarfile.open(fileobj=archive_bytes, mode="r") as archive: + with self.assertRaises(tarfile.TarError): + _extract_taskset_archive(archive, Path(directory)) + self.assertEqual(list(Path(directory).rglob("*")), []) + + def test_resolution_rejects_an_oversized_blob_while_streaming_download(self): + scope = RuntimeScope("tenant-one", "github/acme/repo") + blobs = _BlobService() + connector = AzureBlobTasksetConnector(blobs, scope) + self.addCleanup(connector.close) + digest = ContentDigest("sha256", "a" * 64) + blobs.upload_blob(connector._name(digest), b"x" * 11) + + with mock.patch.object(azure_module, "_MAX_TASKSET_ARCHIVE_BYTES", 10): + with self.assertRaises(ConnectorError) as oversized: + connector.resolve( + f"taskset:{digest.value}", + _context(scope, "resolve-oversized"), + ) + + self.assertEqual(oversized.exception.code, "azure_blob_oversized") + + def test_registration_rejects_oversized_content_before_persisting_any_blob(self): + directory = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + scope = RuntimeScope("tenant-one", "github/acme/repo") + blobs = _BlobService() + connector = AzureBlobTasksetConnector(blobs, scope) + self.addCleanup(connector.close) + + with mock.patch.object(azure_module, "_MAX_TASKSET_EXPANDED_BYTES", 10): + with self.assertRaises(ConnectorError) as oversized: + connector.register( + taskset_bundle(directory), + _context(scope, "register-oversized"), + ) + + self.assertEqual(oversized.exception.code, "taskset_too_large") + self.assertFalse(blobs.blobs) + + def test_registration_rejects_an_oversized_archive_before_persistence(self): + directory = write_taskset( + self, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + scope = RuntimeScope("tenant-one", "github/acme/repo") + blobs = _BlobService() + connector = AzureBlobTasksetConnector(blobs, scope) + self.addCleanup(connector.close) + + with mock.patch.object(azure_module, "_MAX_TASKSET_ARCHIVE_BYTES", 10): + with self.assertRaises(ConnectorError) as oversized: + connector.register( + taskset_bundle(directory), + _context(scope, "register-archive-oversized"), + ) + + self.assertEqual(oversized.exception.code, "taskset_archive_too_large") + self.assertFalse(blobs.blobs) + + +class AzurePostgresConnectorTest(unittest.TestCase): + def setUp(self): + self.scope = RuntimeScope("tenant-one", "github/acme/repo") + self.database = _Postgres() + + def test_initialization_isolates_tables_in_a_dedicated_schema_and_role(self): + connection = _SchemaConnection() + + initialize_azure_postgres( + lambda: connection, + runtime_role="customer_decision_runtime", + ) + + sql = "\n".join(statement for statement, _ in connection.statements) + self.assertIn("CREATE SCHEMA IF NOT EXISTS replica_decision_runtime", sql) + self.assertIn( + "REVOKE ALL ON SCHEMA replica_decision_runtime FROM PUBLIC", + sql, + ) + self.assertIn( + 'GRANT USAGE ON SCHEMA replica_decision_runtime TO "customer_decision_runtime"', + sql, + ) + self.assertNotIn("replica_runtime", sql) + self.assertTrue(connection.committed) + + def test_job_transactions_preserve_idempotency_cas_lease_and_cancellation(self): + jobs = AzurePostgresJobConnector(self.database, self.scope) + create_context = _context( + self.scope, "create-job", idempotency_key="stable-job" + ) + created = jobs.create("job_one", {"state": "queued"}, create_context) + self.assertEqual( + jobs.create("job_one", {"state": "queued"}, create_context), + created, + ) + with self.assertRaises(ConnectorError) as reused: + jobs.create( + "job_two", + {"state": "queued"}, + _context(self.scope, "create-other", idempotency_key="stable-job"), + ) + self.assertEqual(reused.exception.category, "conflict") + + lease = jobs.acquire( + "job_one", + ttl_seconds=10, + context=_context(self.scope, "lease"), + ) + running = jobs.update( + "job_one", + {"state": "running"}, + expected_version=created["version"], + lease=lease, + context=_context(self.scope, "run"), + ) + self.assertEqual( + jobs.create("job_one", {"state": "queued"}, create_context), + created, + ) + with self.assertRaises(ConnectorError) as stale: + jobs.update( + "job_one", + {"state": "succeeded"}, + expected_version=created["version"], + lease=lease, + context=_context(self.scope, "stale"), + ) + self.assertEqual(stale.exception.category, "conflict") + + cancel_context = _context( + self.scope, "cancel", idempotency_key="stable-cancel" + ) + cancelled = jobs.cancel( + "job_one", + expected_version=running["version"], + context=cancel_context, + ) + self.assertEqual(cancelled["state"], "cancelled") + self.assertEqual( + jobs.cancel( + "job_one", + expected_version=running["version"], + context=cancel_context, + ), + cancelled, + ) + with self.assertRaises(ConnectorError) as changed_version: + jobs.cancel( + "job_one", + expected_version=cancelled["version"], + context=cancel_context, + ) + self.assertEqual(changed_version.exception.code, "idempotency_conflict") + second_job = jobs.create( + "job_two", + {"state": "queued"}, + _context(self.scope, "create-job-two"), + ) + with self.assertRaises(ConnectorError) as changed_job: + jobs.cancel( + "job_two", + expected_version=second_job["version"], + context=cancel_context, + ) + self.assertEqual(changed_job.exception.code, "idempotency_conflict") + with self.assertRaises(ConnectorError): + jobs.update( + "job_one", + {"state": "succeeded"}, + expected_version=cancelled["version"], + lease=lease, + context=_context(self.scope, "advance-cancelled"), + ) + self.assertGreaterEqual(self.database.transactions, 6) + + def test_candidate_promotion_is_generation_checked_and_exactly_retryable(self): + candidates = AzurePostgresCandidateConnector( + self.database, + AzureBlobArtifactConnector(_BlobService(), self.scope), + self.scope, + ) + first = candidates.put( + _candidate("one"), _context(self.scope, "candidate-one") + ) + second = candidates.put( + _candidate("two"), _context(self.scope, "candidate-two") + ) + promote_context = _context( + self.scope, "promote-one", idempotency_key="promote-one" + ) + promoted = candidates.promote( + first.id, + expected_generation=0, + operation="promote", + context=promote_context, + ) + self.assertEqual(promoted.generation, 1) + self.assertEqual( + candidates.promote( + first.id, + expected_generation=0, + operation="promote", + context=promote_context, + ), + promoted, + ) + with self.assertRaises(ConnectorError) as stale: + candidates.promote( + second.id, + expected_generation=0, + operation="promote", + context=_context(self.scope, "stale-promotion"), + ) + self.assertEqual(stale.exception.code, "promotion_generation_conflict") + self.assertEqual(candidates.current(_context(self.scope, "current")), promoted) + first_path = candidates.artifact_path( + first.id, + "adapter.bin", + _context(self.scope, "materialize-one"), + ) + second_path = candidates.artifact_path( + first.id, + "adapter.bin", + _context(self.scope, "materialize-two"), + ) + self.assertEqual(first_path, second_path) + self.assertEqual(first_path.read_bytes(), b"one") + cache_root = candidates._cache_root + with self.assertRaises(ConnectorError) as unsafe: + candidates.artifact_path( + first.id, + "../adapter.bin", + _context(self.scope, "materialize-unsafe"), + ) + self.assertEqual(unsafe.exception.code, "candidate_artifact_name_invalid") + candidates.close() + self.assertFalse(cache_root.exists()) + + +class AzureExecutionConnectorTest(unittest.TestCase): + def setUp(self): + self.scope = RuntimeScope("tenant-one", "github/acme/repo") + + def test_training_propagates_bounds_and_uses_authoritative_azure_usage(self): + client = _TrainingClient() + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + context = _context( + self.scope, + "train", + deadline=time.monotonic() + 15, + max_credits=7, + budget_mode="hard", + ) + credential = Credential( + "training-token", + context.deadline, + "training", + self.scope, + context.operation_id, + ) + + result = connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=credential, + ) + + request = client.requests[0] + self.assertEqual(request["jobName"], "replica-training") + self.assertEqual(request["payload"]["limits"]["maxUpdates"], 3) + self.assertEqual(request["payload"]["limits"]["maxCredits"], 7) + self.assertEqual(request["payload"]["limits"]["budgetMode"], "hard") + self.assertGreater(request["payload"]["limits"]["deadlineEpoch"], time.time()) + self.assertEqual(result.updates_completed, 3) + self.assertEqual(result.usage[0].source, "azure") + self.assertEqual(result.usage[0].cost, 1.25) + self.assertEqual(result.usage[0].currency, "USD") + + def test_training_returns_authoritative_usage_above_the_budget(self): + client = _TrainingClient(reported_credits=8) + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + context = _context( + self.scope, + "over-budget-usage", + max_credits=7, + budget_mode="hard", + ) + + result = connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual( + next(item.quantity for item in result.usage if item.kind == "training_credits"), + 8, + ) + + def test_training_refreshes_short_lived_credentials_before_polling(self): + client = _TrainingClient() + refreshed = [] + context = _context( + self.scope, + "refresh-training-token", + deadline=time.monotonic() + 300, + max_credits=7, + budget_mode="hard", + ) + initial = Credential( + "initial-token", + time.monotonic() + 1, + "training", + self.scope, + context.operation_id, + ) + + def refresh(selected): + credential = Credential( + f"refreshed-{len(refreshed) + 1}", + time.monotonic() + 120, + "training", + selected.scope, + selected.operation_id, + ) + refreshed.append(credential) + return credential + + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + credential_refresh=refresh, + ) + + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=initial, + ) + + self.assertEqual(client.requests[0]["bearerToken"], "refreshed-1") + self.assertEqual(client.status_tokens, ["refreshed-1"]) + self.assertEqual(len(refreshed), 1) + + def test_expired_cleanup_credential_refreshes_after_the_operation_deadline(self): + context = _context( + self.scope, + "refresh-after-deadline", + deadline=time.monotonic() - 10, + max_credits=7, + budget_mode="hard", + ) + expired = Credential( + "expired-token", + time.monotonic() - 5, + "training", + self.scope, + context.operation_id, + ) + refreshed = [] + + def refresh(selected): + credential = Credential( + "cleanup-token", + time.monotonic() + 120, + "training", + selected.scope, + selected.operation_id, + ) + refreshed.append(credential) + return credential + + connector = AzureContainerAppsTrainingConnector( + _TrainingClient(), + "replica-training", + cancellation_recovery_hook=lambda *_: None, + credential_refresh=refresh, + ) + + selected = connector._credential(context, expired) + + self.assertEqual(selected.value, "cleanup-token") + self.assertEqual(len(refreshed), 1) + + def test_unsupported_hard_budget_uses_provider_neutral_error_code(self): + connector = AzureContainerAppsTrainingConnector( + _TrainingClient(), + "replica-training", + supports_hard_credit_budget=False, + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + context = _context( + self.scope, + "unsupported-budget", + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as unsupported: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(unsupported.exception.code, "hard_budget_unsupported") + + def test_training_cancellation_cancels_the_remote_job(self): + cancellation = Cancellation() + client = _TrainingClient(cancellation) + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + context = _context( + self.scope, + "cancel-training", + cancellation=cancellation, + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as cancelled: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(cancelled.exception.category, "cancelled") + self.assertEqual(client.cancelled, ["training-1"]) + + def test_failed_status_poll_still_cancels_after_context_cancellation(self): + cancellation = Cancellation() + client = _TrainingClient(cancellation, fail_after_cancellation=True) + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + context = _context( + self.scope, + "cancel-failed-poll", + cancellation=cancellation, + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as cancelled: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(cancelled.exception.category, "cancelled") + self.assertEqual(client.cancelled, ["training-1"]) + + def test_failed_remote_cancellation_is_durably_scheduled(self): + cancellation = Cancellation() + client = _TrainingClient(cancellation, cancel_failures=3) + scheduled = [] + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda execution_id, context: scheduled.append( + (execution_id, context.operation_id) + ), + ) + context = _context( + self.scope, + "schedule-cancel-recovery", + cancellation=cancellation, + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as cancelled: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(cancelled.exception.category, "cancelled") + self.assertEqual(client.cancelled, ["training-1"] * 3) + self.assertEqual( + scheduled, + [("training-1", context.operation_id)], + ) + + def test_status_failure_schedules_recovery_before_abandoning_execution(self): + client = _StatusFailingTrainingClient() + scheduled = [] + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda execution_id, context: scheduled.append( + (execution_id, context.operation_id) + ), + ) + context = _context( + self.scope, + "status-failure-recovery", + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as unavailable: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(unavailable.exception.code, "azure_training_unavailable") + self.assertEqual(client.cancelled, ["training-1"]) + self.assertEqual(scheduled, [("training-1", context.operation_id)]) + + def test_malformed_status_schedules_cancellation_before_integrity_failure(self): + client = _MalformedStatusTrainingClient() + scheduled = [] + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda execution_id, context: scheduled.append( + (execution_id, context.operation_id) + ), + ) + context = _context( + self.scope, + "malformed-status", + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as invalid: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(invalid.exception.code, "azure_training_response_invalid") + self.assertEqual(client.cancelled, ["training-1"]) + self.assertEqual(scheduled, [("training-1", context.operation_id)]) + + def test_unknown_status_cancels_without_polling_until_the_deadline(self): + client = _UnknownStatusTrainingClient() + scheduled = [] + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda execution_id, context: scheduled.append( + (execution_id, context.operation_id) + ), + ) + context = _context( + self.scope, + "unknown-status", + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as invalid: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(invalid.exception.code, "azure_training_response_invalid") + self.assertEqual(client.cancelled, ["training-1"]) + self.assertEqual(scheduled, [("training-1", context.operation_id)]) + + def test_direct_cancellation_still_runs_when_recovery_scheduling_fails(self): + cancellation = Cancellation() + client = _TrainingClient(cancellation) + connector = AzureContainerAppsTrainingConnector( + client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: (_ for _ in ()).throw( + RuntimeError("queue unavailable") + ), + ) + context = _context( + self.scope, + "cancel-without-recovery-queue", + cancellation=cancellation, + max_credits=7, + budget_mode="hard", + ) + + with self.assertRaises(ConnectorError) as cancelled: + connector.train( + SimpleNamespace(metadata={"digest": "taskset-one"}, tasks=()), + Path("unused-work-directory"), + max_updates=3, + context=context, + credential=None, + ) + + self.assertEqual(cancelled.exception.category, "cancelled") + self.assertEqual(client.cancelled, ["training-1"]) + + def test_inference_checks_generation_and_reports_cold_start_readiness(self): + client = _InferenceClient( + { + "state": "starting", + "coldStart": True, + "generation": 4, + "retryAfterSeconds": 8, + } + ) + connector = AzureEndpointInferenceConnector( + client, + endpoint_name="replica-inference", + expected_generation=4, + ) + context = _context(self.scope, "predict") + credential = Credential( + "inference-token", + context.deadline, + "inference", + self.scope, + context.operation_id, + ) + + with self.assertRaises(ConnectorError) as cold: + connector.predict( + Path("candidate_live"), + {"expectedGeneration": 4, "questions": {}}, + context=context, + credential=credential, + ) + self.assertEqual(cold.exception.code, "inference_cold_start") + self.assertTrue(cold.exception.retryable) + self.assertFalse(client.requests) + + client.readiness_value = { + "state": "ready", + "coldStart": False, + "generation": 5, + } + with self.assertRaises(ConnectorError) as stale: + connector.predict( + Path("candidate_live"), + {"expectedGeneration": 4, "questions": {}}, + context=context, + credential=credential, + ) + self.assertEqual(stale.exception.code, "inference_generation_conflict") + self.assertFalse(client.requests) + + client.readiness_value["generation"] = 4 + response = connector.predict( + Path("candidate_live"), + {"expectedGeneration": 4, "questions": {"learning-one": {"type": "noul"}}}, + context=context, + credential=credential, + ) + self.assertEqual(response["learning-one"], {"yes": 0.8, "no": 0.2}) + self.assertNotIn("operationId", response) + self.assertEqual(len(client.requests), 1) + self.assertNotIn("training", client.requests[0]["request"]) + self.assertNotIn("admin", client.requests[0]["request"]) + self.assertEqual( + client.requests[0]["request"]["idempotencyKey"], + "sha256:" + + hashlib.sha256( + ( + hashlib.sha256( + f"{self.scope.tenant_id}\0{self.scope.repository}".encode() + ).hexdigest() + + "\0" + + context.idempotency_key + ).encode() + ).hexdigest(), + ) + self.assertEqual( + client.requests[0]["request"]["operationId"], + context.operation_id, + ) + self.assertRegex( + client.requests[0]["request"]["requestDigest"], + r"^sha256:[a-f0-9]{64}$", + ) + with self.assertRaises(ConnectorError) as conflict: + connector.predict( + Path("candidate_live"), + { + "expectedGeneration": 4, + "questions": {"another-learning": {"type": "noul"}}, + }, + context=context, + credential=credential, + ) + self.assertEqual(conflict.exception.code, "idempotency_conflict") + + with self.assertRaises(ConnectorError) as missing: + connector.predict( + Path("candidate_live"), + {"questions": {}}, + context=context, + credential=credential, + ) + self.assertEqual(missing.exception.code, "expected_generation_required") + + def test_inference_refuses_a_training_scoped_credential(self): + client = _InferenceClient( + {"state": "ready", "coldStart": False, "generation": 4} + ) + connector = AzureEndpointInferenceConnector( + client, + endpoint_name="replica-inference", + expected_generation=4, + ) + context = _context(self.scope, "predict-with-training-token") + training_credential = Credential( + "training-token", + context.deadline, + "training", + self.scope, + context.operation_id, + ) + + with self.assertRaises(ConnectorError) as refused: + connector.predict( + Path("candidate_live"), + {"expectedGeneration": 4, "questions": {}}, + context=context, + credential=training_credential, + ) + + self.assertEqual(refused.exception.category, "unauthorized") + self.assertFalse(client.requests) + + +class AzureIdentityAndObservabilityConnectorTest(unittest.TestCase): + def setUp(self): + self.scope = RuntimeScope("tenant-one", "github/acme/repo") + + def test_managed_identity_credential_is_deadline_bounded_and_not_retained(self): + deadline = time.monotonic() + 20 + identity = _IdentityClient(expires_at=time.time() + 600) + connector = AzureManagedIdentityCredentialConnector( + identity, + self.scope, + allowed_audiences=("https://replica-inference.azure.com/.default",), + ) + + credential = connector.acquire( + "https://replica-inference.azure.com/.default", + _context(self.scope, "credential", deadline=deadline), + ) + + self.assertEqual(credential.value, "short-lived-token") + self.assertLessEqual(credential.expires_at, deadline) + self.assertEqual( + identity.audiences, + ["https://replica-inference.azure.com/.default"], + ) + self.assertNotIn( + "short-lived-token", + repr(vars(connector)), + "The connector must not retain a durable copy of an acquired token.", + ) + + def test_telemetry_emits_only_allowlisted_operational_fields(self): + sink = _TelemetrySink() + connector = AzureObservabilityConnector(sink, self.scope) + context = _context(self.scope, "usage") + usage = Usage( + operation_id=context.operation_id, + scope=self.scope, + connector="azure-training", + kind="gpu_seconds", + quantity=12, + unit="seconds", + cost=1.25, + currency="USD", + source="azure", + metadata={ + "azure.region": "westus3", + "azure.execution_id": "training-1", + "screenshot": "raw-image-data", + "prompt": "private prompt", + "learnings": ["private learning"], + "credential": "secret-token", + }, + ) + + connector.record(usage, context) + + self.assertEqual(len(sink.records), 1) + event_name, record = sink.records[0] + self.assertEqual(event_name, "replica.decision.usage") + self.assertEqual(record["operation_id"], context.operation_id) + self.assertNotIn(self.scope.tenant_id, repr(record)) + self.assertNotIn(self.scope.repository, repr(record)) + self.assertEqual(record["azure.region"], "westus3") + self.assertEqual(record["azure.execution_id"], "training-1") + serialized = repr(record) + for secret in ( + "raw-image-data", + "private prompt", + "private learning", + "secret-token", + ): + self.assertNotIn(secret, serialized) + + def test_telemetry_rejects_free_form_usage_fields_before_emitting(self): + sink = _TelemetrySink() + connector = AzureObservabilityConnector(sink, self.scope) + context = _context(self.scope, "invalid-usage") + for label, kind, unit, currency in ( + ("kind", "raw_prompt_bytes", "bytes", None), + ("unit", "storage_bytes", "screenshots", None), + ("currency", "gpu_seconds", "seconds", "usd"), + ): + with self.subTest(field=label): + usage = Usage( + operation_id=context.operation_id, + scope=self.scope, + connector="azure-training", + kind=kind, + quantity=1, + unit=unit, + cost=1 if currency is not None else None, + currency=currency, + source="azure", + ) + with self.assertRaises(ConnectorError) as refused: + connector.record(usage, context) + self.assertEqual(refused.exception.category, "invalid_input") + self.assertFalse(sink.records) + + def test_telemetry_sink_failures_use_the_connector_error_contract(self): + connector = AzureObservabilityConnector( + _FailingTelemetrySink(), + self.scope, + ) + context = _context(self.scope, "failed-telemetry") + usage = Usage( + operation_id=context.operation_id, + scope=self.scope, + connector="azure-training", + kind="gpu_seconds", + quantity=1, + unit="seconds", + ) + + with self.assertRaises(ConnectorError) as usage_error: + connector.record(usage, context) + self.assertEqual(usage_error.exception.code, "azure_observability_unavailable") + self.assertTrue(usage_error.exception.retryable) + + with self.assertRaises(ConnectorError) as operation_error: + connector.record_operation("training", "failed", context) + self.assertEqual( + operation_error.exception.code, + "azure_observability_unavailable", + ) + + +class _CleanupSuite(unittest.TestSuite): + def run(self, result, debug=False): + try: + return super().run(result, debug) + finally: + while _RESOURCES: + _RESOURCES.pop().doCleanups() + + +def _conformance_fixture(): + case = unittest.TestCase() + directory = write_taskset( + case, + [ + make_task(1, answer=True), + make_task(2, split="test", answer=True), + ], + ) + scope = RuntimeScope("tenant-one", "github/acme/repo") + other_repository_scope = RuntimeScope("tenant-one", "github/acme/other") + other_tenant_scope = RuntimeScope("tenant-two", "github/acme/repo") + blobs = _BlobService() + database = _Postgres() + training_client = _TrainingClient() + training = AzureContainerAppsTrainingConnector( + training_client, + "replica-training", + poll_interval_seconds=0, + sleep=lambda _: None, + cancellation_recovery_hook=lambda *_: None, + ) + inference_client = _InferenceClient( + {"state": "ready", "coldStart": False, "generation": 1} + ) + inference = AzureEndpointInferenceConnector( + inference_client, + endpoint_name="replica-inference", + expected_generation=1, + ) + identity = _IdentityClient(expires_at=time.time() + 600) + telemetry = _TelemetrySink() + azure = create_azure_connector_bundle( + scope=scope, + blob_container=blobs, + connection_factory=database, + training={training.name: training}, + inference={training.name: inference}, + token_provider=identity, + training_token_provider_factory=_IdentityFactory(time.time() + 600), + credential_audiences=("training", "inference"), + telemetry_sink=telemetry, + ) + case.assertIsNotNone(training.credential_refresh) + pickle.dumps(training.credential_refresh) + bundle = taskset_bundle(directory) + registered = azure.connectors.tasksets.register( + bundle, + _context(scope, "azure-conformance-register"), + ) + + def usage(): + return tuple( + Usage( + operation_id=attributes["operation_id"], + scope=scope, + connector=attributes["connector"], + kind=attributes["kind"], + quantity=attributes["quantity"], + unit=attributes["unit"], + cost=attributes["cost"], + currency=attributes["currency"], + source=attributes["source"], + ) + for _, attributes in telemetry.records + ) + + _RESOURCES.append(case) + return ConnectorConformanceFixture( + connectors=azure.connectors, + scope=scope, + other_repository_scope=other_repository_scope, + other_tenant_scope=other_tenant_scope, + taskset_bundle=bundle, + taskset_digest=registered.digest, + taskset_reference=registered.reference, + usage=usage, + ) + + +def load_tests(loader, tests, pattern): + del loader, pattern + tests.addTests(connector_conformance("azure-reference", _conformance_fixture)) + return _CleanupSuite(tests) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/learning/environments/brand-check/tests/test_runtime.py b/packages/learning/environments/brand-check/tests/test_runtime.py index 56e4a7ac..fbaeab0a 100644 --- a/packages/learning/environments/brand-check/tests/test_runtime.py +++ b/packages/learning/environments/brand-check/tests/test_runtime.py @@ -146,6 +146,22 @@ def train(self, *args, **kwargs): return super().train(*args, **kwargs) +class CooperativeCancellationBackend(FixtureBackend): + """A hosted-style connector that records cancellation inside the worker.""" + + def __init__(self, started: Path, cancelled: Path): + self.started = started + self.cancelled = cancelled + + def train_with_context(self, *args, **kwargs): + context = kwargs["context"] + self.started.write_text("started", encoding="utf-8") + while not context.cancellation.requested: + time.sleep(0.01) + self.cancelled.write_text("cancelled", encoding="utf-8") + context.cancellation.raise_if_requested() + + class CapturingBackend(FixtureBackend): """A fixture backend that records the projected inference request.""" @@ -359,6 +375,7 @@ def test_backend_receives_no_free_form_task_or_question_text(self): "data:image/png;base64," + base64.b64encode(PNG).decode("ascii") ], "questions": {RULE: {"type": "noul"}}, + "expectedGeneration": 1, }, ) @@ -2462,6 +2479,65 @@ def test_http_cancels_a_running_job_without_publishing_a_stale_candidate(self): self.addCleanup(reopened.close) self.assertEqual(reopened.job(submitted["id"])["state"], "cancelled") + def test_running_worker_receives_cancellation_before_process_termination(self): + started = self.taskset_dir / ".cooperative-cancel-started" + cancelled = self.taskset_dir / ".cooperative-cancel-received" + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={ + "cooperative": CooperativeCancellationBackend(started, cancelled) + }, + environment="test", + ) + self.addCleanup(runtime.close) + submitted = runtime.submit( + backend="cooperative", + max_updates=1, + max_seconds=30, + max_credits=1, + ) + deadline = time.monotonic() + 5 + while not started.exists() and time.monotonic() < deadline: + time.sleep(0.01) + self.assertTrue(started.exists(), runtime.job(submitted["id"])) + + terminal = runtime.cancel( + submitted["id"], + idempotency_key="cancel-cooperative-worker", + ) + deadline = time.monotonic() + 5 + while not cancelled.exists() and time.monotonic() < deadline: + time.sleep(0.01) + + self.assertEqual(terminal["state"], "cancelled") + self.assertTrue(cancelled.exists()) + + def test_deadline_cleanup_remains_a_timeout_after_worker_cancellation(self): + started = self.taskset_dir / ".deadline-cancel-started" + cancelled = self.taskset_dir / ".deadline-cancel-received" + runtime = LocalRuntime( + self.state_dir, + self.taskset_dir, + backends={ + "cooperative": CooperativeCancellationBackend(started, cancelled) + }, + environment="test", + ) + self.addCleanup(runtime.close) + submitted = runtime.submit( + backend="cooperative", + max_updates=1, + max_seconds=1, + max_credits=1, + ) + + terminal = self._wait_for_job(runtime, submitted["id"]) + + self.assertEqual(terminal["state"], "failed") + self.assertEqual(terminal["error"]["code"], "job_timed_out") + self.assertTrue(cancelled.exists()) + def test_http_cancellation_is_safe_for_queued_retries_and_terminal_jobs(self): started = self.taskset_dir / ".queued-cancel-started" release = self.taskset_dir / ".queued-cancel-release"