Skip to content

Commit 3a6fff4

Browse files
feat: Add IBM Db2 engine adapter (#6030)
Signed-off-by: IBM Db2 Eco System <Hdm-dev-persona-db2-eco-system@ibm.com>
1 parent 263723f commit 3a6fff4

19 files changed

Lines changed: 2285 additions & 6 deletions

File tree

‎.github/scripts/install-prerequisites.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ ENGINE_DEPENDENCIES=""
1717

1818
if [ "$ENGINE" == "spark" ]; then
1919
ENGINE_DEPENDENCIES="default-jdk"
20+
elif [ "$ENGINE" == "db2" ]; then
21+
ENGINE_DEPENDENCIES="libxml2-dev build-essential"
2022
elif [ "$ENGINE" == "fabric" ]; then
2123
echo "Installing Microsoft package repository"
2224

‎.github/scripts/wait-for-db.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,20 @@ risingwave_ready() {
9090
probe_port 4566
9191
}
9292

93+
db2_ready() {
94+
probe_port 50001
95+
96+
echo "Waiting for Db2 to finish initialising (this can take 2-4 minutes)..."
97+
while true; do
98+
if docker exec db2 su - db2inst1 -c "db2 connect to TESTDB" > /dev/null 2>&1; then
99+
echo "Db2 is accepting connections"
100+
break
101+
fi
102+
echo "Db2 not yet ready; sleeping 15s..."
103+
sleep 15
104+
done
105+
}
106+
93107
echo "Waiting for $ENGINE to be ready..."
94108

95109
READINESS_FUNC="${ENGINE}_ready"

‎.github/workflows/db2.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: db2
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'sqlmesh/core/engine_adapter/db2.py'
9+
- 'tests/core/engine_adapter/test_db2.py'
10+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
11+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
12+
- '.github/workflows/db2.yaml'
13+
- 'pyproject.toml'
14+
pull_request:
15+
branches:
16+
- main
17+
paths:
18+
- 'sqlmesh/core/engine_adapter/db2.py'
19+
- 'tests/core/engine_adapter/test_db2.py'
20+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
21+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
22+
- '.github/workflows/db2.yaml'
23+
- 'pyproject.toml'
24+
25+
concurrency:
26+
group: db2-${{ github.event.pull_request.number || github.sha }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
jobs:
33+
db2-test:
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
env:
37+
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
38+
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
39+
UV: '1'
40+
steps:
41+
- uses: actions/checkout@v7
42+
- name: Set up Python
43+
uses: actions/setup-python@v7
44+
with:
45+
python-version: '3.12'
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@v7
48+
- name: Install SQLMesh dev dependencies
49+
run: |
50+
uv venv .venv
51+
source .venv/bin/activate
52+
make install-dev
53+
- name: Install OS-level dependencies
54+
run: ./.github/scripts/install-prerequisites.sh "db2"
55+
- name: Run tests
56+
run: |
57+
source .venv/bin/activate
58+
make db2-test
59+
- name: Upload test results
60+
uses: actions/upload-artifact@v7
61+
if: ${{ !cancelled() }}
62+
with:
63+
name: test-results-docker-db2
64+
path: test-results/
65+
retention-days: 7

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,9 @@ risingwave-test: engine-risingwave-up
222222

223223
starrocks-test: engine-starrocks-up
224224
pytest -n auto -m "starrocks" --reruns 3 --junitxml=test-results/junit-starrocks.xml
225+
226+
db2-test: engine-db2-up
227+
pytest -n auto -m "db2" --reruns 3 --junitxml=test-results/junit-db2.xml
225228

226229
#################
227230
# Cloud Engines #

‎docs/guides/connections.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ default_gateway: local_db
8484
* [BigQuery](../integrations/engines/bigquery.md)
8585
* [ClickHouse](../integrations/engines/clickhouse.md)
8686
* [Databricks](../integrations/engines/databricks.md)
87+
* [Db2](../integrations/engines/db2.md)
8788
* [DuckDB](../integrations/engines/duckdb.md)
8889
* [Fabric](../integrations/engines/fabric.md)
8990
* [MotherDuck](../integrations/engines/motherduck.md)

‎docs/integrations/engines/db2.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Db2
2+
3+
This page provides information about how to use SQLMesh with [IBM Db2](https://www.ibm.com/products/db2).
4+
5+
!!! info
6+
The Db2 engine adapter is a community contribution. Due to this, only limited community support is available.
7+
8+
This adapter depends on the external `db2-sqlglot-dialect` package, which pins its own SQLGlot version range. It isn't covered by SQLMesh's core CI and may temporarily break or fail to install after SQLMesh upgrades SQLGlot, until the plugin catches up.
9+
10+
## Local/Built-in Scheduler
11+
12+
**Engine Adapter Type**: `db2`
13+
14+
### Installation
15+
16+
```
17+
pip install "sqlmesh[db2]"
18+
```
19+
20+
### Connection options
21+
22+
| Option | Description | Type | Required |
23+
|---------------------|------------------------------------------------------------------------------------------------|:------:|:--------:|
24+
| `type` | Engine type name - must be `db2` | string | Y |
25+
| `host` | The hostname of the Db2 server | string | Y |
26+
| `port` | The port number of the Db2 server. Default: `50000` | int | N |
27+
| `database` | The name of the Db2 database to connect to | string | Y |
28+
| `username` | The username to use for authentication with the Db2 server | string | Y |
29+
| `password` | The password to use for authentication with the Db2 server | string | Y |
30+
| `db2_schema` | Sets `CURRENTSCHEMA` on the connection. Controls the default schema for unqualified references. Typically set to the same value as `username`. | string | Y |
31+
| `ssl` | Enable TLS/SSL encryption. Default: `false` | bool | N |
32+
| `connect_timeout` | The number of seconds to wait for the connection to the server. Default: `30` | int | N |
33+
| `concurrent_tasks` | Maximum number of tasks to run concurrently. Default: `4` | int | N |
34+
35+
## Important Notes
36+
37+
**State connection:** Db2 is **not supported** as a SQLMesh `state_connection`. Use DuckDB (recommended) or another supported engine for SQLMesh state storage:
38+
39+
```yaml linenums="1"
40+
gateways:
41+
db2:
42+
connection:
43+
type: db2
44+
host: localhost
45+
port: 50000
46+
database: TESTDB
47+
username: db2inst1
48+
password: your_password
49+
db2_schema: db2inst1
50+
state_connection:
51+
type: duckdb
52+
database: ./state/sqlmesh_state.db
53+
54+
default_gateway: db2
55+
56+
model_defaults:
57+
dialect: db2
58+
```
59+
60+
**Table naming:** Db2 rejects table names that start with an underscore (`_`). SQLMesh's default physical table naming convention can generate names beginning with `_`. To avoid this, set `physical_table_naming_convention` to `hash_md5` in your project config:
61+
62+
```yaml
63+
physical_table_naming_convention: hash_md5
64+
```
65+
66+
## Limitations
67+
68+
- **SCD Type 2 unsupported**: SCD Type 2 models are currently not supported on Db2.
69+
- **Single catalog only**: Db2 operates in single-catalog mode; cross-catalog queries are not supported.
70+
- **No inline column comments**: Column-level comments cannot be set inline during table creation.
71+
- **No atomic table replacement**: Db2 does not support `CREATE OR REPLACE TABLE`, so full model refreshes are not atomic. There is a brief window during which the table may be empty or partially populated.
72+
- **Identifier length**: Maximum identifier length is 128 characters.
73+
- **No `SELECT ... FOR UPDATE`**: Db2 does not support `SELECT ... FOR UPDATE` in the same way as OLTP databases; SQLMesh removes this clause when executing queries.
74+
75+
## Resources
76+
77+
- [IBM Db2 Documentation](https://www.ibm.com/docs/en/db2)
78+
- [IBM Db2 SQL Reference](https://www.ibm.com/docs/en/db2/11.5?topic=db2-sql)

‎docs/integrations/overview.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ SQLMesh supports the following execution engines for running SQLMesh projects (e
1616
* [BigQuery](./engines/bigquery.md) (bigquery)
1717
* [ClickHouse](./engines/clickhouse.md) (clickhouse)
1818
* [Databricks](./engines/databricks.md) (databricks)
19+
* [Db2](./engines/db2.md) (db2)
1920
* [DuckDB](./engines/duckdb.md) (duckdb)
2021
* [Fabric](./engines/fabric.md) (fabric)
2122
* [MotherDuck](./engines/motherduck.md) (motherduck)

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ nav:
8282
- integrations/engines/bigquery.md
8383
- integrations/engines/clickhouse.md
8484
- integrations/engines/databricks.md
85+
- integrations/engines/db2.md
8586
- integrations/engines/duckdb.md
8687
- integrations/engines/fabric.md
8788
- integrations/engines/motherduck.md

‎pyproject.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,10 @@ dev = [
112112
]
113113
dbt = ["dbt-core<2"]
114114
dlt = ["dlt"]
115+
db2 = [
116+
"ibm_db",
117+
"db2-sqlglot-dialect;python_version>=\"3.10\""
118+
]
115119
duckdb = []
116120
fabric = ["pyodbc>=5.0.0"]
117121
fabric-mssql-python = ["mssql-python>=1.1.0;python_version>=\"3.10\""]
@@ -270,6 +274,7 @@ markers = [
270274
"clickhouse: test for Clickhouse (standalone mode / cluster mode)",
271275
"clickhouse_cloud: test for Clickhouse (cloud mode)",
272276
"databricks: test for Databricks",
277+
"db2: test for Db2",
273278
"duckdb: test for DuckDB",
274279
"fabric: test for Fabric",
275280
"motherduck: test for MotherDuck",

‎sqlmesh/core/config/connection.py‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@
6060
# Nullable types are problematic
6161
"clickhouse",
6262
"starrocks",
63+
# Db2 rejects table names starting with underscore (_versions, _snapshots,
64+
# _environments, _intervals). Use a separate state_connection (e.g., DuckDB).
65+
"db2",
6366
}
6467
MOTHERDUCK_TOKEN_REGEX = re.compile(r"(\?|\&)(motherduck_token=)(\S*)")
6568
PASSWORD_REGEX = re.compile(r"(password=)(\S+)")
@@ -2602,6 +2605,93 @@ def _connection_factory(self) -> t.Callable:
26022605
BaseDuckDBConnectionConfig, # type: ignore[type-abstract]
26032606
}
26042607

2608+
2609+
class Db2ConnectionConfig(ConnectionConfig):
2610+
host: str
2611+
port: int = 50000
2612+
database: str
2613+
db2_schema: str
2614+
username: str
2615+
password: str
2616+
ssl: bool = False
2617+
ssl_cert: t.Optional[str] = None
2618+
ssl_key: t.Optional[str] = None
2619+
ssl_ca: t.Optional[str] = None
2620+
connect_timeout: int = 30
2621+
2622+
concurrent_tasks: int = 4
2623+
register_comments: bool = True
2624+
pre_ping: bool = True
2625+
2626+
type_: t.Literal["db2"] = Field(alias="type", default="db2")
2627+
DIALECT: t.ClassVar[t.Literal["db2"]] = "db2"
2628+
DISPLAY_NAME: t.ClassVar[t.Literal["Db2"]] = "Db2"
2629+
DISPLAY_ORDER: t.ClassVar[t.Literal[19]] = 19
2630+
2631+
_engine_import_validator = _get_engine_import_validator("ibm_db", "db2")
2632+
2633+
@property
2634+
def _connection_kwargs_keys(self) -> t.Set[str]:
2635+
return {
2636+
"host",
2637+
"port",
2638+
"database",
2639+
"db2_schema",
2640+
"username",
2641+
"password",
2642+
}
2643+
2644+
@property
2645+
def _engine_adapter(self) -> t.Type[EngineAdapter]:
2646+
adapter = getattr(engine_adapter, "Db2EngineAdapter", None)
2647+
if adapter is None:
2648+
raise ConfigError(
2649+
"The Db2 engine adapter requires Python 3.10+ and the 'db2' extra. "
2650+
"Please install it with: pip install 'sqlmesh[db2]'"
2651+
)
2652+
return t.cast(t.Type[EngineAdapter], adapter)
2653+
2654+
def get_catalog(self) -> t.Optional[str]:
2655+
"""Db2 stores catalog names in uppercase; normalise here so the default_catalog
2656+
passed to the adapter matches what get_current_catalog() returns at runtime."""
2657+
catalog = super().get_catalog()
2658+
return catalog.upper() if catalog else None
2659+
2660+
@property
2661+
def _connection_factory(self) -> t.Callable:
2662+
import ibm_db_dbi # type: ignore
2663+
2664+
ssl = self.ssl
2665+
ssl_cert = self.ssl_cert
2666+
ssl_key = self.ssl_key
2667+
ssl_ca = self.ssl_ca
2668+
connect_timeout = self.connect_timeout
2669+
2670+
def connect_db2(**kwargs: t.Any) -> t.Any:
2671+
conn_str_parts = [
2672+
f"DATABASE={kwargs['database']}",
2673+
f"HOSTNAME={kwargs['host']}",
2674+
f"PORT={kwargs['port']}",
2675+
"PROTOCOL=TCPIP",
2676+
f"UID={kwargs['username']}",
2677+
f"PWD={kwargs['password']}",
2678+
f"CURRENTSCHEMA={kwargs['db2_schema']}",
2679+
f"CONNECTTIMEOUT={connect_timeout}",
2680+
]
2681+
if ssl:
2682+
conn_str_parts.append("SECURITY=SSL")
2683+
if ssl_cert:
2684+
conn_str_parts.append(f"SSLClientCertificate={ssl_cert}")
2685+
if ssl_key:
2686+
conn_str_parts.append(f"SSLClientKey={ssl_key}")
2687+
if ssl_ca:
2688+
conn_str_parts.append(f"SSLServerCertificate={ssl_ca}")
2689+
conn_str = ";".join(conn_str_parts) + ";"
2690+
return ibm_db_dbi.connect(conn_str, "", "")
2691+
2692+
return connect_db2
2693+
2694+
26052695
CONNECTION_CONFIG_TO_TYPE = {
26062696
# Map all subclasses of ConnectionConfig to the value of their `type_` field.
26072697
tpe.all_field_infos()["type_"].default: tpe

0 commit comments

Comments
 (0)