Http `REFUSED_STREAM` when uploading bundle files/artifacts leaves bundle in a broken state

[jmahlik]

Describe the issue

When uploading large amounts of wheels, the cli errors sporadically with http2 REFUSED_STREAM errors. When this happens the artifacts have been uploaded but the bundle files have not been updated to reference them. The old artifacts are deleted, so the bundle is left in a broken state.

It's difficult to reproduce since it's a transient http2 error. About one in 200-500 deploys hits this issue. It always appears on a GET request to /api/2.0/workspace/get-status. It may be an sdk issue with the retries on databricks.WorkspaceClient.

It seems like the deployment should be rolled back if it fails mid-deploy. Or the requests retried more gracefully. Maybe a configuration option for number of retries could be implemented?

Here's the edited output from a failed deployment.

Building project...
Building python_deps...

...

Lots of artifact uploads
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...

...

Uploading bundle files to /Workspace/<CENSORED>.bundle/files...
Error: Get "https://<CENSORED>.cloud.databricks.com/api/2.0/workspace/get-status?path=<CENSORED>.bundle%2Ffiles": stream error: stream ID 287; REFUSED_STREAM; received from peer

Configuration

Put a lot of wheels in the directory. An example is a requirements file with 100+ dependencies. With the build looking something like this.

  artifacts:
  project:
    type: whl
    build: |
      uv build . --wheel --out-dir dist
    path: .
    files:
      - source: ./dist/*.whl
  python_deps:
    type: whl
    path: .
    # TODO: uncomment if deps need updated, takes forever to plan
    build: |
      python -m pip download \
        -r requirements.txt \
        --no-deps \
        --only-binary :all: \
        --platform manylinux_2_28_x86_64 \
        --platform manylinux_2_27_x86_64 \
        --platform manylinux_2_17_x86_64 \
        --platform manylinux2014_x86_64 \
        --python-version 3.12 \
        --implementation cp \
        --abi none \
        --abi abi3 \
        --abi "cp312" \
        -d dist_deps
    files:
      - source: ./dist_deps/*

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. Get a stream error
3. The bundle is now out of sync with the artifacts

Expected Behavior

Roll back the deploy or retry the request.

Actual Behavior

Prod is broken, need to redeploy to fix.

OS and CLI version

Linux, tested on 0.298.0 and latest 1.14.1

Is this a regression?

No

Debug Logs

This happens in CI, haven't been able to reproduce it with debug logs.

[andrewnester]

We have a new feature called immutable folder which allows bundle to deploy to unique location and change the resources referencing the artifacts atomically, this would prevent this issue. Please give it a try.

#5254

[jmahlik]

We have a new feature called immutable folder which allows bundle to deploy to unique location and change the resources referencing the artifacts atomically, this would prevent this issue. Please give it a try.

#5254

It does sound like that would solve the problem. I just gave it a shot. Unfortunately, the bundle is too large in this case. Looks like the size limit is 100MB but the bundle is about 250MB.

databricks bundle deploy -t user

Building project...
Building python_deps...
Uploading immutable bundle snapshot...
Error: snapshot upload: Server received a request which exceeds maximum allowed content length. RequestSize(bytes): 264767245, Limit(bytes): 104857600 (400 BAD_REQUEST)

Endpoint: POST https://CENSORED/api/2.0/repos/snapshots
HTTP Status: 400 Bad Request
API error_code: BAD_REQUEST
API message: Server received a request which exceeds maximum allowed content length. RequestSize(bytes): 264767245, Limit(bytes): 104857600