Skip to content

Commit d303706

Browse files
vikrantpuppalavikrantpuppala
authored
ci: add DBR LTS install check to catch ES-1960554-class regressions (#843)
* ci: add DBR LTS install check + fix pyarrow-compat it caught (ES-1960554) The thrift 0.23.0 bump (PR #796, shipped in 4.2.7) broke `pip install` on DBR LTS: thrift ships sdist-only and 0.23.0's setup.py calls sys.exit(0) on the build-success path, killing the PEP 517 backend before pip writes output.json. On the old setuptools shipped by DBR 14.3/15.4 LTS this is a hard install failure (SEV0 ES-1960554); 4.2.7 was yanked and reverted (#840). Our CI never caught it because every job installs via `poetry install` on a modern runner -- it never does a fresh `pip install` of the built wheel on an LTS toolchain, the real customer path that failed. CI check -------- Adds a PR check (gated to dependency changes) that builds the wheel and installs it INSIDE real DBR LTS clusters via the PECO workspace Jobs API (no PyPI publish) then runs a SELECT 1 smoke test. Matrix = supported LTS {13.3, 14.3, 15.4, 16.4, 17.3} x install target {base, pyarrow, kernel}. Auth is OAuth M2M as the PECO service principal throughout (driver -> workspace API and the notebook's connector -> warehouse smoke query); a PAT is warehouse-scoped and rejected by the workspace REST API. Older LTS ship an SDK too old for auth_type=oauth-m2m, so the smoke harness upgrades databricks-sdk. Per-run artifacts are cleaned up in a finally block. Connector fix (caught by the check) ----------------------------------- The check surfaced a real latent bug: a base install (no [pyarrow] extra) runs against a runtime's bundled pyarrow, and on DBR 13.3/14.3 that pyarrow predates the `promote_options` kwarg, so concat_table_chunks raised `TypeError: concat_tables() got an unexpected keyword argument 'promote_options'` on the Arrow result path. utils.py now falls back to the legacy `promote=True` (equivalent to promote_options="default") when the kwarg is unsupported, with a regression test. Validated end-to-end against the PECO workspace: green on thrift 0.22.0, and re-widening the pin to <0.24.0 fails on 14.3+15.4 with the exact output.json error -- a true guard, not a check that always passes. Also adds an incident-linked comment on the thrift pin so nobody re-widens it before the upstream fix (THRIFT-6067 / apache/thrift#3584) ships. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * test(e2e): authenticate e2e via OAuth M2M so staging tests match DATABRICKS_USER The e2e suite connected via a PAT (DATABRICKS_TOKEN). The Personal Staging Location tests PUT/GET/REMOVE against stage://tmp/<DATABRICKS_USER>/..., where DATABRICKS_USER is the PECO service principal (TEST_PECO_SP_ID). A personal stage is identity-scoped by design (there is even a test asserting you cannot touch another user's stage), so the connecting identity MUST equal DATABRICKS_USER. When DATABRICKS_TOKEN authenticates as a different identity, those tests fail with `PERMISSION_DENIED: <user> does not have access to Personal Stage`. Switch the e2e connection to OAuth M2M as the service principal via credentials_provider (conftest.auth_connect_kwargs), so the connecting identity IS the SP == DATABRICKS_USER. Falls back to the PAT when SP OAuth creds aren't set, so local PAT runs are unaffected. Wires DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET (TEST_PECO_SP_ID / TEST_PECO_SP_OAUTH_SECRET, already in azure-prod) into code-coverage.yml. Verified locally against the PECO workspace: all 9 staging_ingestion e2e tests pass via the real M2M path (including fails_to_modify_another_staging_user, which validates the identity scoping). Kernel e2e files are unchanged (they run in kernel-e2e.yml, ignored by code-coverage.yml). Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * test(e2e): add databricks-sdk dev dependency for OAuth M2M auth The e2e M2M auth (conftest.auth_connect_kwargs) imports databricks.sdk.core.oauth_service_principal, but databricks-sdk was not a project dependency, so `poetry install` in code-coverage.yml didn't provide it -- every e2e connection failed with `ModuleNotFoundError: No module named 'databricks.sdk'`. Add it to the dev group (test-only; not a runtime dep of the connector). CI's setup-poetry runs `poetry lock` before install, so the lockfile is regenerated on the runner. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Revert e2e OAuth M2M; use SP PAT instead (keep DBR LTS check on M2M) The azure-prod DATABRICKS_TOKEN is now a personal access token owned by the PECO service principal, so its identity matches DATABRICKS_USER. That fixes the Personal Staging Location tests (stage://tmp/<SP>/...) with a plain PAT, without the OAuth M2M machinery -- which also broke the retry/HTTP tests, since M2M makes a live token-endpoint call that those tests' urllib3 mocking intercepts. Reverts the e2e auth changes (conftest.auth_connect_kwargs + the consumer call sites + the databricks-sdk dev dep + the code-coverage.yml SP env) back to the plain access_token path. The DBR LTS install check keeps OAuth M2M: it hits the workspace Jobs/SCIM API (which rejects a warehouse-scoped PAT), is proven 15/15 green on M2M, and installs databricks-sdk itself in its own workflow. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> --------- Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
1 parent 815b341 commit d303706

6 files changed

Lines changed: 613 additions & 0 deletions

File tree

‎.github/workflows/dbr-lts-install.yml‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
name: DBR LTS Install
2+
3+
# Installs the CI-built connector wheel INSIDE real DBR LTS clusters and runs a
4+
# SELECT 1 smoke test. This reproduces the customer install path that broke in
5+
# ES-1960554 (thrift 0.23.0's sdist failing to build under the OLD setuptools
6+
# shipped by DBR LTS) -- something the poetry-install-based unit CI can't see,
7+
# because it never does a fresh `pip install` of the built artifact on an LTS
8+
# toolchain. See scripts/dbr_lts_install_check.py + scripts/dbr_lts_smoke_notebook.py.
9+
#
10+
# Trigger: pull_request, but the real matrix runs ONLY when dependency-affecting
11+
# files change (pyproject.toml / poetry.lock / this workflow / the two scripts).
12+
# Dependency changes are the only surface that can introduce this class of
13+
# failure, so unrelated PRs skip the (cluster-backed, slow) matrix entirely.
14+
# This is an informational check, not a required gate.
15+
#
16+
# External setup this workflow depends on (all ALREADY present in azure-prod):
17+
# - DATABRICKS_HOST / DATABRICKS_TOKEN (PECO service-principal PAT) and
18+
# TEST_PECO_WAREHOUSE_HTTP_PATH -- the three secrets referenced below.
19+
# - A writable UC Volume at CI_VOLUME (managed volume). The notebook-import
20+
# dir is derived from the token's own identity (no hardcoded principal).
21+
#
22+
# The supported-LTS matrix tracks the Databricks Runtime support lifecycle:
23+
# https://learn.microsoft.com/azure/databricks/release-notes/runtime/
24+
# 10.4 / 11.3 / 12.2 LTS are end-of-support and intentionally excluded. Revisit
25+
# when an LTS EOLs (13.3 -> Aug 22, 2026) or a new LTS GAs.
26+
27+
on:
28+
pull_request:
29+
30+
permissions:
31+
contents: read
32+
id-token: write
33+
34+
concurrency:
35+
group: dbr-lts-${{ github.ref }}
36+
cancel-in-progress: true
37+
38+
env:
39+
CI_VOLUME: /Volumes/peco/default/ci_wheels
40+
41+
jobs:
42+
# ───────────────────────────────────────────────────────────────
43+
# Detect whether dependency-affecting files changed on this PR.
44+
# ───────────────────────────────────────────────────────────────
45+
detect-changes:
46+
runs-on:
47+
group: databricks-protected-runner-group
48+
labels: linux-ubuntu-latest
49+
outputs:
50+
run_tests: ${{ steps.changed.outputs.run_tests }}
51+
steps:
52+
- name: Check out repo
53+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
54+
with:
55+
fetch-depth: 0
56+
- name: Detect dependency changes
57+
id: changed
58+
env:
59+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
60+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
61+
run: |
62+
CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")
63+
echo "Changed files:"; echo "$CHANGED"
64+
if echo "$CHANGED" | grep -qE "^(pyproject\.toml|poetry\.lock|\.github/workflows/dbr-lts-install\.yml|scripts/dbr_lts_install_check\.py|scripts/dbr_lts_smoke_notebook\.py)$"; then
65+
echo "run_tests=true" >> "$GITHUB_OUTPUT"
66+
echo "Dependency-affecting files changed — will run the LTS matrix"
67+
else
68+
echo "run_tests=false" >> "$GITHUB_OUTPUT"
69+
echo "No dependency changes — skipping the LTS matrix"
70+
fi
71+
72+
# ───────────────────────────────────────────────────────────────
73+
# Build the wheel once per matrix leg and install it on the target DBR
74+
# LTS runtime with the given extras.
75+
#
76+
# Matrix = supported LTS × install-target {base, pyarrow, kernel}. The
77+
# [kernel] extra is a no-op below Python 3.10; all current LTS are ≥3.10
78+
# so every leg is meaningful today. If a future LTS ships Python <3.10,
79+
# drop its kernel leg from `include`.
80+
# ───────────────────────────────────────────────────────────────
81+
lts-install:
82+
needs: detect-changes
83+
if: needs.detect-changes.outputs.run_tests == 'true'
84+
runs-on:
85+
group: databricks-protected-runner-group
86+
labels: linux-ubuntu-latest
87+
environment: azure-prod
88+
permissions:
89+
contents: read
90+
id-token: write
91+
strategy:
92+
fail-fast: false
93+
matrix:
94+
include:
95+
# 13.3 LTS (Python 3.10) — EOL Aug 22, 2026
96+
- { spark: "13.3.x-scala2.12", extras: "" }
97+
- { spark: "13.3.x-scala2.12", extras: "pyarrow" }
98+
- { spark: "13.3.x-scala2.12", extras: "kernel" }
99+
# 14.3 LTS (Python 3.10) — broke in ES-1960554
100+
- { spark: "14.3.x-scala2.12", extras: "" }
101+
- { spark: "14.3.x-scala2.12", extras: "pyarrow" }
102+
- { spark: "14.3.x-scala2.12", extras: "kernel" }
103+
# 15.4 LTS (Python 3.11) — broke in ES-1960554
104+
- { spark: "15.4.x-scala2.12", extras: "" }
105+
- { spark: "15.4.x-scala2.12", extras: "pyarrow" }
106+
- { spark: "15.4.x-scala2.12", extras: "kernel" }
107+
# 16.4 LTS (Python 3.12)
108+
- { spark: "16.4.x-scala2.12", extras: "" }
109+
- { spark: "16.4.x-scala2.12", extras: "pyarrow" }
110+
- { spark: "16.4.x-scala2.12", extras: "kernel" }
111+
# 17.3 LTS (Python 3.12)
112+
- { spark: "17.3.x-scala2.13", extras: "" }
113+
- { spark: "17.3.x-scala2.13", extras: "pyarrow" }
114+
- { spark: "17.3.x-scala2.13", extras: "kernel" }
115+
env:
116+
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
117+
# Driver -> workspace API (jobs/scim/files) uses OAuth M2M as the PECO
118+
# service principal. A plain PAT (DATABRICKS_TOKEN) is warehouse-scoped
119+
# and is rejected by the workspace REST API ("Invalid access token").
120+
DATABRICKS_CLIENT_ID: ${{ secrets.TEST_PECO_SP_ID }}
121+
DATABRICKS_CLIENT_SECRET: ${{ secrets.TEST_PECO_SP_OAUTH_SECRET }}
122+
DATABRICKS_HTTP_PATH: ${{ secrets.TEST_PECO_WAREHOUSE_HTTP_PATH }}
123+
steps:
124+
- name: Check out repository
125+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
126+
- name: Set up Python
127+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
128+
with:
129+
python-version: "3.11"
130+
- name: Configure JFrog pip index (OIDC)
131+
uses: ./.github/actions/setup-jfrog
132+
- name: Install build + driver tooling
133+
run: pip install "poetry==2.2.1" "databricks-sdk>=0.20,<1"
134+
- name: Build the connector wheel
135+
run: poetry build -f wheel
136+
- name: Install wheel on ${{ matrix.spark }} (extras=${{ matrix.extras || 'base' }})
137+
run: |
138+
WHEEL=$(ls dist/databricks_sql_connector-*.whl | head -1)
139+
echo "Built wheel: $WHEEL"
140+
python scripts/dbr_lts_install_check.py \
141+
--wheel "$WHEEL" \
142+
--smoke-notebook scripts/dbr_lts_smoke_notebook.py \
143+
--spark-version "${{ matrix.spark }}" \
144+
--extras "${{ matrix.extras }}" \
145+
--volume "${CI_VOLUME}" \
146+
--run-id "${GITHUB_RUN_ID}-${{ matrix.spark }}-${{ matrix.extras || 'base' }}" \
147+
--http-path "${DATABRICKS_HTTP_PATH}"

‎pyproject.toml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@ include = ["CHANGELOG.md"]
1010

1111
[tool.poetry.dependencies]
1212
python = "^3.8.0"
13+
# Keep the cap at <0.23.0. thrift ships sdist-only, and thrift 0.23.0's
14+
# setup.py calls sys.exit(0) on the build-success path, which kills the PEP
15+
# 517 build backend before pip writes output.json. On the OLD setuptools
16+
# shipped by DBR LTS (14.3 / 15.4) this is a hard `pip install` failure --
17+
# the SEV0 ES-1960554 incident (4.2.7 widened this to <0.24.0 and was yanked;
18+
# see PR #840). Do NOT re-widen until a fixed thrift (0.23.1 / 0.24.0, per
19+
# THRIFT-6067 / apache/thrift#3584) ships. The `DBR LTS Install` CI check
20+
# (.github/workflows/dbr-lts-install.yml) installs the built wheel on real DBR
21+
# LTS clusters and catches this regression when dependency pins change.
1322
thrift = "~=0.22.0"
1423
pandas = [
1524
{ version = ">=1.2.5,<4.0.0", python = ">=3.8,<3.13" },

0 commit comments

Comments
 (0)