Make externalized large-payload token self-describing (token v2)

[YunchuWang]

Summary

Makes the externalized large-payload token self-describing by embedding the storage account in it (token format v2). This is "Option A" for the dashboard-payload-access feature.

Today the token is blob:v1:<container>:<blobName> and does not carry the storage account — the account is implicit (whatever the worker's PayloadStore is currently configured with). That breaks two scenarios:

1. The DTS dashboard reads blobs directly from Storage using the signed-in user's own AAD token, so it needs the full blob URL — but the v1 token lacks the account.
2. If the customer repoints the store to a different account, old blobs (in the old account) become unresolvable because nothing records where each payload actually lives.

Fix: write the account into each token at upload time.

Token format

• v2 (new, written): blob:v2:<fullBlobUrl> — everything after blob:v2: is one complete absolute blob URI, e.g.
  • blob:v2:https://acct.blob.core.windows.net/mycontainer/<guid> (DNS style)
  • blob:v2:http://127.0.0.1:10000/devstoreaccount1/mycontainer/<guid> (Azurite path style)
  • BlobClient.Uri carries no SAS/key, so the token is non-secret and safe to persist.
• v1 (legacy, still read): blob:v1:<container>:<blobName> — account implicit = configured store.

Changes

• BlobPayloadStore.EncodeToken(Uri) emits v2 from the blob's absolute URI (blob.Uri) at upload time.
• DecodeToken parses both v2 (via BlobUriBuilder, handling DNS- and Azurite path-style URLs) and legacy v1.
• IsKnownPayloadToken recognizes both blob:v1: and blob:v2: so v2 tokens rehydrate.
• DownloadAsync honors the token's account:
  • Same account/container as the configured store → fast path via the existing container client (works with any auth).
  • Different account (post-repoint) → if identity auth (TokenCredential) is configured, read cross-account via a new BlobClient using the same retry options; otherwise throw a clear PayloadStorageException before any network call.
  • The download + gzip-decompress body is factored into a shared DownloadFromBlobAsync helper.
• Sample LargePayloadConsoleApp token check updated to also recognize blob:v2:.

Back-compat

Preview product; breaking existing blob:v1: payloads would be acceptable, but v1 read back-compat is kept anyway (cheap, avoids breaking in-flight orchestrations).

Caveat (by design)

Worker cross-account read only works with identity/managed-identity auth that has RBAC on both accounts. With connection-string/account-key auth the key is account-specific, so cross-account is impossible — the clear throw is the correct behavior. The dashboard (out of scope, different repo) is unaffected: it uses the user's own AAD token.

Tests

Added test/Grpc.IntegrationTests/BlobPayloadStoreTests.cs (no live Azure required):

• EncodeToken → v2 token contains container + blob; round-trips through DecodeToken.
• DecodeToken parses v2 DNS-style and Azurite path-style URLs.
• DecodeToken still parses legacy v1.
• IsKnownPayloadToken true for v1 + v2, false for a plain string.
• Cross-account guard: v2 token for a different account with no identity credential throws PayloadStorageException before any network call.

Full solution builds with 0 errors and no new analyzer warnings; the 6 new tests pass.

Coordination note — draft PR #758 (auto-purge)

Read-only inspection; not modified here.

PR #758 (yunchuwang-wangbill-blob-payload-autopurge-sdk) adds PayloadStore.DeleteAsync + BlobPayloadStore.DeleteAsync, which still use the v1-only DecodeToken/TokenPrefix and a v1-only IsKnownPayloadToken. DeleteExternalBlobActivity classifies the ArgumentException that the v1 decoder throws on an unrecognized prefix as a poison token → Discarded (acked so the backend clears the tombstone row).

If #758 merges on top of this change without reconciliation, every v2 token handed to DeleteAsync would fail the blob:v1: prefix check, throw ArgumentException, and be silently discarded without deleting the blob (payload leak) — and v2 tokens wouldn't be recognized as known. Whoever merges second should port #758's DeleteAsync/DecodeToken/IsKnownPayloadToken to the v2-aware versions (account-aware blob resolution, same fast-path/cross-account logic as DownloadAsync). There's also a mechanical merge conflict in BlobPayloadStore.cs since both branches edit the same token region.

---

Superseded/related (closed separately, not touched here): protobuf #77, this repo's #765, and a backend ADO PR.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:142

• In the v2 branch, blobUri is declared as Uri? from the DecodeToken tuple. With <Nullable>enable</Nullable>, passing it directly to the BlobClient constructor will produce nullable warnings and can mask unexpected nulls. Use a null-forgiving operator (or an explicit null check) here since v2 tokens always populate blobUri.

        else if (this.options.Credential != null)
        {
            // The payload lives in a different account (e.g. the store was repointed). Identity auth can still
            // read it as long as the credential has RBAC access to that account.
            blob = new BlobClient(blobUri, this.options.Credential, this.clientOptions);

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:200

• DecodeToken (v2) doesn't validate that the parsed URI actually contains both a container and blob name. For inputs like blob:v2:https://acct.blob.core.windows.net/container/ (missing blob) or blob:v2:https://acct.blob.core.windows.net/ (missing container), this returns empty names and later produces confusing failures. Fail fast with the same "Invalid external payload token format" exception used for other malformed tokens.

            BlobUriBuilder builder = new(blobUri);
            string container = builder.BlobContainerName;
            string name = builder.BlobName;
            builder.BlobName = string.Empty;
            Uri containerUri = builder.ToUri();

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:150

• This exception text says the payload lives in a different storage account, but the check (IsConfiguredContainer) actually compares the full container URI. If only the container differs (same account), this message will be misleading. Consider wording this as a mismatch with the configured payload store container (and avoid implying it's necessarily a different account).

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{containerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload reads " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot read another account.");

test/Grpc.IntegrationTests/BlobPayloadStoreTests.cs:97

• This assertion is coupled to the current exception wording. If the error message is updated to describe a configured-container mismatch (vs only "different storage account"), update this expectation accordingly so the test continues to validate the intended behavior.

        // Assert
        Assert.Contains("different storage account", ex.Message);
    }

[halspang]

Nothing major, mostly styling nits.

[halspang]

Nit: Can you fix the encoding on these strings, comments should be human readable.

[halspang]

Nit: Human readable.

[halspang]

More encoded strings. I'll stop commenting on them but please fix them everywhere.

[halspang]

Nit: With 4 items in the return, I think we've gotten to the point of having a DecodeTokenResult or something.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:150

• This exception message says the payload is in a different “storage account”, but the comparison is currently against the full container URI. If only the container differs (same account), this message is misleading. Adjust wording to reflect “container or account” (or similar) so the reported mismatch matches what was compared.

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{containerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload reads " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot read another account.");

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:201

• DecodeToken should reject v2 tokens that don’t actually include both a container name and blob name (e.g., a container-level URL or account root URL). Today those decode to empty container/blob values and will later fail in confusing ways (or produce misleading cross-account errors). Add validation after parsing the BlobUriBuilder fields.

            BlobUriBuilder builder = new(blobUri);
            string container = builder.BlobContainerName;
            string name = builder.BlobName;
            builder.BlobName = string.Empty;
            Uri containerUri = builder.ToUri();

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:137

• For v2 tokens, the code treats any mismatch in the configured container URI as a cross-account scenario. That means if the storage account stays the same but the configured ContainerName changes (or tokens reference a different container in the same account), DownloadAsync will either (a) incorrectly require identity auth or (b) throw, even though connection-string/account-key auth can typically read other containers within the same account. Consider comparing account separately from container, and for same-account/different-container resolve via the token’s container (e.g., create a BlobServiceClient with the configured connection string and then get the token’s container client) instead of failing fast.

This issue also appears in the following locations of the same file:

• line 146
• line 197

        if (this.IsConfiguredContainer(containerUri!))
        {
            // Same account and container as the configured store: reuse it (works with any auth mode).
            blob = this.containerClient.GetBlobClient(name);
        }

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:150

• This exception is thrown whenever the token's container URI doesn't match the configured container URI. That can happen due to a different container (not just a different storage account), so the message is currently misleading (it always says "different storage account"). Making the wording include container mismatch will make failures easier to understand without changing behavior.

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{decoded.ContainerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload reads " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot read another account.");

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:201

• DecodeToken() accepts any absolute URI after "blob:v2:" but doesn't validate that the parsed URI actually contains both a container and a blob name. For inputs like a service/account URI (or container-only URI), BlobUriBuilder will yield empty container/blob values and downstream code will throw confusing errors (or attempt to create a BlobClient with an empty name). Consider failing fast with the same "Invalid external payload token format" exception when container or blob name is empty.

            BlobUriBuilder builder = new(blobUri);
            string container = builder.BlobContainerName;
            string name = builder.BlobName;
            builder.BlobName = string.Empty;
            Uri containerUri = builder.ToUri();
            return new(true, container, name, blobUri, containerUri);

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:150

• The thrown PayloadStorageException message says the payload is in a “different storage account”, but this branch can also be reached when only the container differs (same account) because IsConfiguredContainer compares the full container URI. Adjust the message to describe the actual mismatch (token target vs configured container) and reserve the “different account” wording for when you’ve explicitly compared account identity.

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{decoded.ContainerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload reads " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot read another account.");

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:135

• DownloadAsync's v2 path currently treats any mismatch with the configured container URI as a cross-account scenario and (when not using TokenCredential) throws. This means a v2 token that points to the same storage account but a different container (e.g. ContainerName changed) becomes unreadable even though the configured credentials could still access it; it also makes the subsequent exception text misleading (it says “different storage account” even for container-only differences). Consider comparing storage account identity separately from container, and when the account matches, resolve the blob via a container client for decoded.Container (so v2 truly honors both account and container).

This issue also appears on line 146 of the same file.

        // v2 tokens are self-describing: honor the account and container encoded in the token.
        BlobClient blob;
        if (this.IsConfiguredContainer(decoded.ContainerUri!))
        {
            // Same account and container as the configured store: reuse it (works with any auth mode).