Durable Workflow Rust SDK

durable-workflow is the first-party Rust client and worker SDK for Durable Workflow. Rust applications can start and inspect workflows, run workflow and activity handlers, and exchange typed Avro values with PHP, Python, and Rust workers through one durable runtime.

Use the SDK with a self-hosted Durable Workflow Server or a managed Durable Workflow Cloud namespace. Your workers remain ordinary Rust processes and scale independently from the runtime.

Install

Rust 1.86 or newer is required.

cargo add durable-workflow

Applications using Tokio entry points should also enable the Tokio features they need:

cargo add tokio --features macros,rt-multi-thread

Run the example

examples/hello_world.rs is a complete typed workflow and activity example. It starts a workflow, runs two activities, demonstrates retry and failure handling, waits for completion, and prints the result.

Start a bootstrapped self-hosted Server, then run the example from this repository:

DURABLE_WORKFLOW_SERVER_URL=http://127.0.0.1:8080 \
DURABLE_WORKFLOW_TOKEN=dev-token \
cargo run --example hello_world

Pass the Server origin without a trailing /api. TASK_QUEUE changes the default rust-workers queue, and GREETING_NAME changes the example input.

For a provisioned Cloud namespace, use the exact runtime URL and the separate client and worker credentials shown in Cloud:

DURABLE_WORKFLOW_RUNTIME_URL=https://cloud.durable-workflow.com/api/runtime/v1/namespaces/your-namespace-id \
DURABLE_WORKFLOW_RUNTIME_NAMESPACE=your.namespace \
DURABLE_WORKFLOW_CLIENT_TOKEN=your-client-token \
DURABLE_WORKFLOW_WORKER_TOKEN=your-worker-token \
cargo run --example hello_world

The namespace runtime URL is already complete. Do not append another /api.

Migrate from Rust SDK 3

Nexus calls add Error::NexusOperationFailed and CancellationCallKind::ServiceOperation. Both enums are now non_exhaustive. Include a fallback arm when matching either enum outside this crate so future error or cancellation kinds can be handled without another exhaustive-match migration. Handle a Nexus failure at its original durable call; a committed admission is separate from a synchronous completion.

WorkflowTask, ActivityTask and QueryTask retain optional, untrusted observability metadata. ActivityTask also exposes optional workflow, run and activity execution identities supplied by the runtime. When constructing task structs directly, initialize the new optional fields to None. Receiving these fields does not enable tracing, authorize work or invent missing correlation. Task structs remain directly constructible. Future additions to their public fields require another major release; extensible metadata inside the existing optional JSON fields does not require new task literal fields.

Existing workflow histories are unchanged. Preserve the workflow's recorded command sequence and verify representative histories before deployment. Nexus authoring requires explicit worker opt-in and an installed Server capability; ordinary worker registration does not enable it.

Core API

• Client starts, signals, queries, updates, cancels, terminates, describes, and awaits workflow executions.
• Worker registers workflow, activity, query and update handlers, declares workflow signals, and long-polls task queues.
• WorkflowContext provides durable activities, timers, conditions, child workflows, side effects, version markers, parallel operations, selection, sagas, message streams, memo, search attributes, and continue-as-new.
• Typed registration and result helpers preserve Serde request and result types over the fixed Avro Value protocol.
• Activity options cover retries, start-to-close, schedule-to-start, schedule-to-close, heartbeat timeouts, cancellation, and heartbeats.

Query execution authority

On a runtime advertising query_task_authority with worker protocol 1.20, Worker captures the original query claim before loading external inputs. It checks ownership without renewing the lease during loading and execution, before calling the handler, and before submitting its outcome. The original caller deadline and lease bound the entire operation. Expiry, shutdown or an unavailable observation drops the work without sending a completion or failure for it.

Use Worker::require_query_task_authority(true) to refuse registration unless worker-authenticated discovery advertises this support. Without this requirement, older runtimes and explicit protocol 1.19 keep compatibility: disclosed original timing supplies a local bound, while deliveries without timing retain legacy behavior. This path cannot observe live revocation and never calls the 1.20 query-status endpoint.

Async handlers must yield to Tokio to be cancelled promptly. Synchronous or CPU-bound callbacks cannot be interrupted while they block, but their late result is fenced before submission. Query handlers should avoid side effects. Direct Client completion and failure methods remain available for inspecting an original committed receipt.

Handler registration

Register each workflow and activity wire name once per Worker. Named query and update handlers are also scoped to their workflow type. Dynamic, typed, Avro and replayed adapters share these namespaces. Repeating a registration, including the same callback, is an error. Different Workers or handler kinds may reuse a name, and queries or updates on different workflow types may reuse a name.

Registration methods keep their existing signatures. A duplicate preserves the first handler and invalidates that Worker. Call worker.validate_registration()? after configuring it for a local check. register, run, run_until and run_once also reject it before contacting Server with Error::WorkerLoop and a duplicate_registration diagnostic. Local validation returns the typed DuplicateRegistrationError, exposing the handler kind, wire name, optional workflow scope, and both source locations and authoring adapters. It converts into the existing worker-error variant when used with ?, preserving exhaustive matches on the SDK's existing Error enum. Construct a new Worker to correct an invalid configuration. Choose the desired handler before registration instead of relying on replacement order.

After registering a workflow, declare every signal name it reads through wait_signal or signals before starting the worker:

# fn declare_signals(worker: &mut durable_workflow::Worker) -> durable_workflow::Result<()> {
worker.declare_workflow_signals("orders", &["finish", "changed"])?;
# Ok(())
# }

Server records those names and positional argument contracts when starting a run. Changing worker declarations does not change existing runs.

The SDK writes Avro payloads only. The fixed recursive Value schema preserves nulls, booleans, signed 64-bit integers, finite doubles, bytes, UTF-8 strings, lists, and string-keyed maps across official SDKs without customer-managed schemas or a registry.

Large payloads are uploaded automatically when the namespace advertises runtime storage. The SDK follows its inline threshold and upload limit, including batches that exceed the ordinary request limit. Uploads and downloads use the same runtime URL, namespace, and credential role, with size and SHA-256 verification. Client::builder(...).max_external_payload_bytes(...) limits unique downloaded bytes per response (64 MiB by default). Provider credentials are not needed; payload fetches never follow redirects.

Workers confirm and renew the exact workflow task claim while loading external inputs and paged history, then confirm it again before replay. Loading stops on lost authority or Worker shutdown without executing workflow code. Pending cooperative cancellation retains its original identity and cleanup deadline.

External activity inputs retain their original lease and execution deadline bounds while loading. Without the optional lease-renewal capability, protocol 1.20 observes the exact activity attempt before, during and after the transfer, and immediately before callback entry. Unconfirmed authority stops loading without invoking the callback. These observations do not renew the lease or record application heartbeats. Explicit protocol 1.19 uses the original local lease/deadline bounds and completion fencing; it cannot observe live revocation or ownership. Live authority supervision requires 1.20. Shutdown and dropping a poll stop its loading and observation together.

The optional Client::renew_activity_task_lease request is available only when actual runtime discovery advertises activity_lease_renewal=true. Pass the original task ID, attempt ID, lease owner and a positive request window of at most five seconds. Discovery and renewal share that window and use worker credentials. This request records no application progress. Validate the full original identity, granted lease and business deadlines before using its reply to extend local authority. Direct Client users manage their own renewal lifecycle.

When registration, cluster discovery or the actual poll advertises that boolean capability, Worker confirms and automatically maintains the original activity claim throughout input loading, the yielding callback and terminal publication. Missing or changed task, attempt, holder or run identity prevents callback entry. Lost authority, exhausted original timeouts or managed shutdown drop the callback future and fence retained callback Clients against late completion, failure, heartbeat or result upload. Retryable renewal errors grant no additional time. Older runtimes keep their existing path.

Runtime lease pings do not record application progress or satisfy a configured heartbeat timeout. Continue calling ctx.heartbeat(...) for real progress; only acknowledged same-attempt progress advances that recorded interval. Session-bound activities also require an acknowledged same-holder session grant and retain their original absolute TTL. Automatic maintenance requires callbacks to yield to Tokio; it cannot stop detached threads or reverse external side effects.

Completion still uses the runtime's attempt fence. Server 2.6.1 refuses superseded attempts but can accept an expired, unreclaimed attempt; see Workflow #769.

The Client's default 60-second HTTP timeout includes connection and the complete response body. Client::builder(...).timeout(Duration::from_secs(120)) selects a larger total budget before starting a Worker. Claimed input and history loading share that budget across all pages. Lease renewals do not restart it. Low-level cooperative polls retain their long-poll timeout plus five seconds, and explicit canonical cancellation refreshes retain their five-second budget.

Examples

Activity retry policies

ActivityRetryPolicy::new(max_attempts).backoff_intervals(...) preserves the complete explicit list, including entries that the attempt budget will not use. One attempt uses no retry delay. The server repeats the last entry when further retries are allowed, while the attempt budget still limits execution. Replay compares the complete stored list and the other known policy fields exactly; changing an unused entry is also a policy change. The default remains one attempt with an empty backoff list.

Local activities

Enable inline local execution with Worker::new(client, queue).local_activities(true). Register the callback with the ordinary activity registration methods, then call ctx.local_activity(...), local_activity_typed(...) or local_activity_avro_value(...) from workflow code. LocalActivityOptions sets bounded retries and start-to-close, schedule-to-close and heartbeat timeouts.

Local activities run in the workflow worker and bypass the activity queue. Server records their attempts, heartbeat details and terminal result. Committed results replay without running the callback. A worker lost before completion is acknowledged can execute the callback again, so external effects must be idempotent.

Callbacks must yield to Tokio. Timeout or lost workflow lease drops the async callback without requiring application heartbeats. Blocking work needs separate process supervision. Inline local execution and cooperative prepared local supervision use separate worker profiles. Registration rejects combining them. Database-generated execution and failure IDs become available after Server commits history. Use the failure kind, timeout kind and attempt number to branch during fresh execution, rather than testing whether an ID has been assigned.

Available since SDK 3.1.0. Default workers keep local execution disabled. Opted-in workers negotiate their local capability during registration.

Worker sessions

Enable Worker::worker_sessions(true) and declare the resource requirements that the worker actually satisfies with capabilities(...). Route a remote call with ctx.activity(...).in_worker_session(options) and use .typed::<Output>() or .avro_value() for its result. A parallel group's in_worker_session(...) applies the same routing to each activity leaf.

Server creates a session on the first admitted activity. After registration, worker.worker_session(options) also provides an explicit shared handle for create(), renew() and close(reason). ActivityContext::worker_session() exposes the current handle. Heartbeats renew the holder lease without extending the absolute TTL. Graceful worker shutdown drains activities and closes held sessions before deregistering.

lease_seconds(...) bounds holder authority and ttl_seconds(...) bounds the session's total lifetime. Reacquisition retains the original TTL deadline. Renew the handle explicitly while keeping an idle resource alive. Activity heartbeats also renew the lease. An async callback is dropped when its locally observed lease or TTL ends, even without application heartbeats. Callbacks must yield to Tokio.

Set max_concurrent_worker_sessions(...) to bound the worker's session registry and max_concurrent_activities(...) on the options to bound each session's activity concurrency. Uncreated or failed handles release their local slot when dropped. An admitted session keeps its slot while its holder lease is active.

Session memory is process-local. A replacement holder must rebuild its resources, and an interrupted activity can execute again. Use idempotency and attempt fencing for external side effects. Committed results replay from history without rebuilding resources or rerunning callbacks. Changing recorded session options fails replay. Local activities cannot use session routing.

The runnable session example uses a real process-local cache and prints its resource generation. Use Server 2.5.1 / Native 2.4.1 for session history and original TTL preservation. Session support starts with SDK 3.2.0.

Bounded sticky execution

Enable a durable-history cache explicitly on a Worker:

use durable_workflow::{Client, Result, StickyCacheOptions, Worker};
use std::time::Duration;

fn configure(client: Client) -> Result<Worker> {
    Worker::new(client, "rust-workers")
        .build_id("orders-v1")
        .sticky_cache(StickyCacheOptions::new(32)
            .max_history_bytes(8 * 1024 * 1024)
            .ttl(Duration::from_secs(60)))
}

Caching is disabled by default. A positive capacity opts in, and zero disables it. The defaults are 16 MiB of retained encoded history and a 300-second TTL. Set whole-second TTLs between 1 and 3600 seconds. Registration must confirm the worker, queue, namespace, build and sticky capability before it can poll with caching. Use Server 2.5.10 or newer. SDK support starts with 3.4.0.

The cache retains immutable wire histories, never live workflow instances or session resources. Replays decode a fresh snapshot. Entry and encoded-byte bounds use LRU eviction, and reads do not extend expiry. Decoding, replay, keys and cursor metadata use additional memory. Size the cache within your worker's memory budget.

A warm replay validates the inline prefix and fetches the retained tail cursor under the current task's lease and attempt. Missing, expired, changed or invalid entries fall back to complete durable history. Worker replacement needs no cache transfer. Committed side effects replay, and cancellation delivery remains governed by canonical Server history. Terminal completion and worker shutdown clear entries.

worker.sticky_cache_metrics() reports hits, misses, evictions, forced cold replays, retained entries and encoded history bytes. This reduces history fetching for warm runs. Measure CPU, memory and completion rate with your own workloads.

An explicit build_id(...) pins new runs to that deployment build. A different build cannot take an existing pinned run. Omit it to retain unversioned routing. The runnable example below prints its result and cache counters.

Deploying workflow patches

Use a stable change ID with ctx.patched(...) or ctx.get_version(...) to preserve deployment decisions through replay. An old unmarked history selects the legacy version -1 if the next durable operation was already recorded. New executions record one decision per change ID, even when code asks again.

Some older workers recorded a marker for every repeated call. Rust replays consistent markers at their original positions, retaining the recorded history and timestamps. Keep those call boundaries, or replace them with ctx.deprecate_patch(...), while such histories remain active. Replay consumes one matching historical marker per call and never skips an intervening activity, timer or side effect. Conflicting recorded versions and unsupported current ranges remain replay errors.

Recovering from Server outages

For a long-running service worker, enable .recover_transient_outages(true) on Worker. Retryable poll and worker-heartbeat failures then keep retrying with capped exponential backoff. A retried poll keeps its original request ID. run_until interrupts retry waits when shutdown arrives and settles any poll response already in flight.

The default preserves bounded retries. run_once remains bounded in either mode, and WorkerRetryPolicy.max_retries = 0 disables ordinary retries. Authentication, protocol, codec, handler and task settlement failures still return an error. Registration and deregistration are outside this recovery option. Keep a process supervisor for startup failures and process crashes.

Runnable examples

Example	Demonstrates
hello_world.rs	Typed worker, workflow, activities, retries, and completion
activity_options.rs	Activity retry and timeout policies
local_activities.rs	Explicit local execution, retries, durable timers and remote work
worker_sessions.rs	Typed activity routing, holder-local resources and graceful session close
sticky_execution.rs	Opt-in bounded history, signal replay and cache counters
condition_search_attributes.rs	Durable conditions and typed search attributes
continue_as_new.rs	Bounded histories and continue-as-new
parallel_saga.rs	Deterministic parallel work and saga compensation

Optional execution telemetry

Enable the telemetry Cargo feature and configure your application's OpenTelemetry provider, exporter, sampling and bounded export timeouts before constructing the worker. Configure the application's global W3C propagator before making admission calls or starting workers:

# #[cfg(feature = "telemetry")]
# fn main() -> Result<(), Box<dyn std::error::Error>> {
use durable_workflow::{Client, Worker};
use durable_workflow::telemetry::ExecutionTelemetry;
opentelemetry::global::set_text_map_propagator(
    opentelemetry_sdk::propagation::TraceContextPropagator::new(),
);
let client = Client::builder("http://localhost:8080").build()?;
let worker = Worker::new(client, "orders").telemetry(ExecutionTelemetry::new());
# Ok(())
# }
# #[cfg(not(feature = "telemetry"))]
# fn main() {}

The default build has no OpenTelemetry dependency. The optional feature needs the API and official W3C propagators supplied by the OpenTelemetry SDK crate; the application owns provider/exporter configuration and shutdown. If the application already uses a composite propagator, retain its other propagators and include W3C trace context. A missing propagator leaves task spans independent even when the Server supports carriers. The SDK does not replace the application's global configuration. Servers without the exact optional capability use independent task traces. When available, a valid run origin parents task spans; task-bound dispatch and validated command/outcome contexts become links. The original poll identity, generation and existing hydration, query and cooperative fences precede their use. Missing or insufficient authority omits propagation without changing business execution. Actual remote/local activity attempts have separate child spans. Use the configured Server/SDK tuple qualified in Workflow #773. Replay emits workflow work spans without pretending completed activities ran again. No span stays open during durable waiting.

Configure admission/command clients with client.with_telemetry(telemetry) and use existing client.cluster_info().await discovery during setup. Worker startup can perform one bounded optional discovery; its failure does not refuse business registration or add business retries. Only the exact supported worker_protocol.server_capabilities.observability_context object enables carriers. No mandatory protocol floor or Server version inference is added. Caller capture precedes HTTP transport. Worker child/continue commands capture their actual workflow scope before settlement and preserve business arguments and request keys.

Official W3C propagation validates the agreed byte limits. Baggage defaults empty; with_baggage_allowlist retains explicit keys only on run parents. Links never carry baggage. Envelopes are bounded to 4096 bytes and at most 32 links. Malformed metadata is ignored with fixed discarded/filtered span diagnostics. Use with_enabled(false) to disable spans and propagation explicitly.

Use telemetry::current_log_context() in a logging adapter, or wrap an existing OpenTelemetry logger provider in telemetry::ExecutionLoggerProvider before passing it to your application's logging bridge. It adds bounded namespace, worker, queue, task and available workflow/attempt IDs; trace/span IDs follow the current application child span. Request IDs and log messages remain owned by the application. The SDK installs no global logging bridge or provider. Tokio contexts attach during each poll and restore on yield, completion, panic and cancellation. Spawned application tasks must explicitly propagate their context when that is intended. Export failures do not change business results; applications must bound exporter I/O and drain their providers during shutdown. Abrupt process death may lose spans; telemetry is not exactly-once delivery.

Documentation

• Rust SDK landing page
• Generated API reference
• Rust SDK guide
• Self-hosted Server guide
• Cloud early access

Nexus service calls

Await a catalog operation from workflow code with Avro arguments:

let reply = ctx.call_nexus_service("billing", "invoices", "create", invoice).await?;
let receipt: Receipt = reply.result.deserialize()?;

The default synchronous call waits for its committed result. Cold replay resumes the original call without admitting it again. NexusServiceOptions selects a target namespace, idempotency key, async mode or acceptance-only waiting. Async and acceptance-only calls return the original admission, even when a terminal result appears later. Committed failures use Error::NexusOperationFailed. Calls currently run in the root workflow scope.

Enable Worker::nexus_cooperative_cancellation(true) for cooperative service waits. It enables ordinary cooperation and requires the installed Server to advertise protocol 1.20, both cancellation capabilities and a workflow_run binding. Cancellation reaches the original synchronous completion wait after owned target cleanup, preserving the original request and cleanup deadline. Use a cancellation shield for durable cleanup. See examples/nexus_cooperation.rs for caller and target workers. Ordinary cooperative workers keep their existing registration behavior when Nexus cooperation is disabled.

Compatibility

The SDK supports cooperative requests and bounded, replayable cleanup. Enable Worker::cooperative_cancellation(true) against a Server that advertises protocol 1.20 and the required capabilities. This profile supervises async Activity futures independently of application heartbeats. Independently cancellable scopes remain disabled. See the cancellation guide and v3 migration guide.

The crate publishes its supported Server and worker-protocol ranges in [package.metadata.durable-workflow] in Cargo.toml. Runtime capability manifests, not matching package version strings, determine protocol compatibility. Stable releases follow semantic versioning.

Development

cargo fmt --all --check
cargo test --locked --all-targets --all-features
cargo doc --locked --all-features --no-deps
cargo package --locked

The committed lockfile records the dependency graph used by CI and packaging. Fresh-consumer checks resolve dependencies independently to verify the published crate also builds for new consumers. Release verification runs tooling pinned to the workflow commit against a separate immutable source tag. For older tags that ignored the lockfile, it restores only the missing lock from the checksum-verified published crate, then requires the complete regenerated archive to match.

Replay and codec defects require a minimal regression fixture. See CONTRIBUTING.md for the corpus rules.

License

Durable Workflow Rust SDK is released under the MIT License.