Skip to content

acc: leave requests answered with a transient error out of cloud recordings - #7771

Open
denik wants to merge 3 commits into
mainfrom
denik/acc-proxy-drop-retried
Open

denik wants to merge 3 commits into
mainfrom
denik/acc-proxy-drop-retried

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

In cloud runs the acceptance proxy now records a request only once when the CLI repeats it after a transient error (408, 429 or 5xx). Requests are still written when they arrive, so the order is unchanged. The proxy remembers each attempt answered with a transient error. When an identical request (same method, path, query and body) arrives later, it removes the failed attempt from out.requests.txt before writing the repeat, and logs Dropped from out.requests.txt, repeated after a transient error: <method> <path>. A failed request that is not repeated stays recorded.

This applies only behind the proxy: cloud runs and local Proxy = true tests. With the local testserver, a 5xx only happens when a test injects one with fault.py, and those goldens show the failed attempt on purpose (create_504, create_transient_500, ...). They are unchanged.

This is the request-log counterpart of SuppressLines.warn_retry, which already keeps the retrying after ... warnings out of output.txt.

Judgment calls:

  • An attempt is dropped only when it is repeated, not on every 5xx. Otherwise a create that succeeded despite a 500, and that the recovery lookup then found, would lose the POST that created the resource. The file is also changed only while a request is recorded, not after a response has been sent, so a script that reads or removes the file right after the CLI exits cannot race with it.
  • Not covered: create-recovery lookups. When a Lakebase branch create gets a 504, the CLI reads the branch (404) before it repeats the create. The failed POST is dropped, but the extra GET stays, so that case can still flake (1 of the 5 failures below).

Why

Lakebase and UC endpoints answer with a 500/504 now and then. The CLI retries and the deploy succeeds, but the recorded request log had the request twice, and the out.requests*.json golden failed:

error snippet
error/3667 Warn: retrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main, then an extra "method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main" in out.requests.update.json
error/2330 retrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME], then an extra GET of the project in out.requests.destroy.json
error/3290 retrying after 500 Internal Server Error from PATCH /api/2.0/postgres/[MY_ENDPOINT_ID], then an extra PATCH in out.requests.update.json
error/3538 retrying after 504 Gateway Timeout from PATCH /api/2.1/unity-catalog/models/..., then an extra PATCH of the model

Tests: postgres_roles/update/DMS=true, postgres_branches/replace_existing/DMS=, postgres_endpoints/update_autoscaling/DMS=true, registered_models/comment_cleared/DMS=true.

Tests

  • New local selftest selftest/record_proxy_retry (Proxy = true, so requests go through the same proxy as in cloud runs):
    • It injects a 500 on a Lakebase create-operation poll, which the CLI retries. The recording has one poll, not two; without the change it has two.
    • It injects a 500 the CLI does not retry. The request stays recorded.
  • The local bundle/, selftest/ and cmd/ acceptance tests pass, except bundle/templates/pydabs/check-formatting, which needs ruff and fails on this machine without the change too.
  • The last commit asks CI to run selftest/record_cloud, postgres_roles/update and postgres_databases/basic on every cloud.

This pull request and its description were written by Isaac.

…rdings

On a real workspace the CLI repeats a request answered with a 5xx, 408 or
429, and the proxy recorded both attempts, so out.requests goldens failed
whenever the workspace had a hiccup. The proxy now removes the failed
attempt from out.requests.txt once its response arrives.

cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 18, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 04a240b67 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

10 tests: regression=0 unreleased=1 coverage=9 failing=0
test branch main (61f1fc5) latest (v1.26.0)
TestAccept/bundle/resources/postgres_databases/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/update/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/update/DMS=true ✅ ✅ ✅
TestAccept/selftest/record_cloud/basic ✅ ✅ ✅
TestAccept/selftest/record_cloud/error ✅ ✅ ✅
TestAccept/selftest/record_cloud/pipeline-crud ✅ ✅ ✅
TestAccept/selftest/record_cloud/volume-io ✅ ✅ ✅
TestAccept/selftest/record_cloud/workspace-file-io ✅ ✅ ✅
TestAccept/selftest/record_proxy_retry ✅ ✅ ❌
TestAccept/selftest/record_proxy_retry: branch ✅ | main (61f1fc5) ✅ | latest (v1.26.0) ❌

latest (v1.26.0):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== A request answered with a transient error and repeated by the CLI is recorded once\n\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...\nCreated postgres_projects.my_project\nFiles: 0 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n>>> print_requests.py --get //postgres\n{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/postgres/projects\",\n  \"q\": {\n    \"project_id\": \"test-pg-proj\"\n  },\n  \"body\": {\n    \"spec\": {\n      \"display_name\": \"Test Postgres Project\",\n      \"pg_version\": 16\n    }\n  }\n}\n{\n  \"method\": \"GET\",\n  \"path\": \"/api/2.0/postgres/projects/test-pg-proj/operations/[UUID]\"\n}\n\n=== A request answered with a transient error and not repeated stays recorded\n\n>>> musterr [CLI] current-user me\nError: Fault injected by test.\n\n>>> print_requests.py --get //scim\n{\n  \"method\": \"GET\",\n  \"path\": \"/api/2.0/preview/scim/v2/Me\"\n}\n"
        	            	actual  : "\n=== A request answered with a transient error and repeated by the CLI is recorded once\n\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...\nError: cannot create resources.postgres_projects.my_project: Fault injected by test. (500 INTERNAL_ERROR)\n\nEndpoint: GET [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj/operations/[UUID]?\nHTTP Status: 500 Internal Server Error\nAPI error_code: INTERNAL_ERROR\nAPI message: Fault injected by test.\n\nFiles: 0 uploaded, 0 deleted\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -5,35 +5,12 @@
        	            	 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        	            	-Created postgres_projects.my_project
        	            	+Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (500 INTERNAL_ERROR)
        	            	+
        	            	+Endpoint: GET [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj/operations/[UUID]?
        	            	+HTTP Status: 500 Internal Server Error
        	            	+API error_code: INTERNAL_ERROR
        	            	+API message: Fault injected by test.
        	            	+
        	            	 Files: 0 uploaded, 0 deleted
        	            	-Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	->>> print_requests.py --get //postgres
        	            	-{
        	            	-  "method": "POST",
        	            	-  "path": "/api/2.0/postgres/projects",
        	            	-  "q": {
        	            	-    "project_id": "test-pg-proj"
        	            	-  },
        	            	-  "body": {
        	            	-    "spec": {
        	            	-      "display_name": "Test Postgres Project",
        	            	-      "pg_version": 16
        	            	-    }
        	            	-  }
        	            	-}
        	            	-{
        	            	-  "method": "GET",
        	            	-  "path": "/api/2.0/postgres/projects/test-pg-proj/operations/[UUID]"
        	            	-}
        	            	+Exit code: 1
        	            	 
        	            	-=== A request answered with a transient error and not repeated stays recorded
        	            	-
        	            	->>> musterr [CLI] current-user me
        	            	-Error: Fault injected by test.
        	            	-
        	            	->>> print_requests.py --get //scim
        	            	-{
        	            	-  "method": "GET",
        	            	-  "path": "/api/2.0/preview/scim/v2/Me"
        	            	-}
        	            	-
        	Test:       	TestAccept/selftest/record_proxy_retry
        	Messages:   	selftest/record_proxy_retry/output.txt vs /tmp/TestAcceptselftestrecord_proxy_retry1308510464/002/output.txt
    acceptance_test.go:1178: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 8598757

Run: 44313665346

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 283 20 18:18
💚​ aws-windows-latest-4core-16gb 1 285 18 12:54
💚​ azure linux-2core-8gb 1 282 20 15:51
💚​ azure-windows-latest-4core-16gb 1 284 18 9:44
💚​ gcp linux-2core-8gb 1 283 20 15:25
💚​ gcp-windows-latest-4core-16gb 1 285 18 8:06
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 44 slowest tests (at least 2 minutes):
duration env testname
6:57 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
6:21 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/update/DMS=true
4:40 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
4:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/update/DMS=
4:28 aws-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/update/DMS=
4:15 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=true
3:56 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:55 gcp linux-2core-8gb TestExportDir
3:44 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/update/DMS=true
3:36 aws linux-2core-8gb TestAccept/bundle/resources/postgres_databases/basic/DMS=
3:34 aws-windows-latest-4core-16gb TestAccept
3:25 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=
3:17 aws linux-2core-8gb TestSyncFullFileSync
3:10 gcp-windows-latest-4core-16gb TestAccept
3:09 azure linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=
3:02 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
3:02 azure-windows-latest-4core-16gb TestAccept
3:01 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:58 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:57 aws linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=
2:52 azure linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=true
2:47 aws linux-2core-8gb TestSyncIncrementalFileSync
2:46 aws linux-2core-8gb TestAccept/bundle/resources/postgres_roles/update/DMS=true
2:40 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:33 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/update/DMS=
2:30 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:30 azure-windows-latest-4core-16gb TestLock
2:29 aws linux-2core-8gb TestAccept/bundle/resources/postgres_databases/basic/DMS=true
2:29 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:27 gcp linux-2core-8gb TestImportDirDoesNotOverwrite
2:25 azure linux-2core-8gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/source_scala
2:23 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:22 aws-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_databases/basic/DMS=
2:19 aws linux-2core-8gb TestImportDirWithOverwriteFlag
2:19 azure-windows-latest-4core-16gb TestExportDir
2:17 aws-windows-latest-4core-16gb TestImportDir
2:16 azure linux-2core-8gb TestAccept/bundle/resources/postgres_databases/basic/DMS=
2:15 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:14 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:10 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:09 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:06 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:02 gcp linux-2core-8gb TestFilerReadDir/workspace_files_extensions
2:01 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat

A request answered with a transient error is now removed from
out.requests.txt only once the same request is recorded again, so a
request the CLI does not repeat (e.g. a create that succeeded despite a
500 and was found by the recovery lookup) stays in the recording.

cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik marked this pull request as ready for review November 18, 2026 13:52
@denik
denik requested a review from a team as a code owner November 18, 2026 13:52
cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants