CI Fix - Spark Failing Sometimes

[cmgoffena13]

Link: https://github.com/SQLMesh/sqlmesh/actions/runs/31642729110/job/94269004530

AI Notes

Failing Tests
None — Spark never ran. Job died during make install-dev.

What’s Wrong
uv pip install failed building dbt-core-experimental-parser==2.0.0b1 (pulled in via sqlmesh[dev] → dbt-core). Its build downloads a wheel from GitHub and got Remote end closed connection without response. Install/network flake, not a Spark bug.

Possible Fix

Rerun the job. If it keeps happening: pin/avoid that dbt-core path in CI, cache the wheel, or retry the download.

Adding retries in the Makefile so every CI job gets them.

Example

install-dev:
	@for i in 1 2 3; do \
		$(PIP) install -e ".[dev,web,slack,dlt,lsp]" ./examples/custom_materializations && exit 0; \
		echo "install-dev failed (attempt $$i/3), retrying..."; \
		sleep 5; \
	done; \
	exit 1

[cmgoffena13]

I just realized my approach is not platform agnostic, would fail on Windows. If this comes up again I'll look at how to implement retries probably through a python script. I think this is rare enough to close though.