Skip to content

Commit 6f2b797

Browse files
EhabEaseeEhabEasee
authored
Merge branch 'main' into feat/clustered-by-auto-none
2 parents 4f29141 + 991a327 commit 6f2b797

46 files changed

Lines changed: 2558 additions & 224 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,13 @@ install-dev-dbt-%:
4949
$(MAKE) install-dev; \
5050
if [ "$$version" = "1.6.0" ]; then \
5151
echo "Applying overrides for dbt 1.6.0"; \
52-
$(PIP) install 'pydantic>=2.0.0' 'google-cloud-bigquery==3.30.0' 'databricks-sdk==0.28.0' --reinstall; \
52+
$(PIP) install 'pydantic>=2.0.0' 'google-cloud-bigquery==3.30.0' 'databricks-sdk==0.28.0' \
53+
'pyOpenSSL>=24.0.0' --reinstall; \
5354
fi; \
5455
if [ "$$version" = "1.7.0" ]; then \
5556
echo "Applying overrides for dbt 1.7.0"; \
56-
$(PIP) install 'databricks-sdk==0.28.0' --reinstall; \
57+
$(PIP) install 'databricks-sdk==0.28.0' \
58+
'pyOpenSSL>=24.0.0' --reinstall; \
5759
fi; \
5860
if [ "$$version" = "1.5.0" ]; then \
5961
echo "Applying overrides for dbt 1.5.0"; \

‎docs/concepts/models/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ This table lists each engine's support for `TABLE` and `VIEW` object comments:
184184
| DuckDB <=0.9 | N | N |
185185
| DuckDB >=0.10 | Y | Y |
186186
| MySQL | Y | Y |
187-
| MSSQL | N | N |
187+
| MSSQL | Y | Y |
188188
| Postgres | Y | Y |
189189
| GCP Postgres | Y | Y |
190190
| Redshift | Y | N |

‎docs/concepts/models/python_models.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,33 @@ def entrypoint(
369369
)
370370
```
371371

372+
Blueprint variables can also be used as **column names and column types** in the `columns` dictionary. For example, if each blueprint produces a model with a different set of column names and types, both can be parameterized using the same `@{variable}` syntax:
373+
374+
```python linenums="1"
375+
import pandas as pd
376+
from sqlmesh import ExecutionContext, model
377+
378+
@model(
379+
"@{customer}.metrics",
380+
kind="FULL",
381+
blueprints=[
382+
{"customer": "customer1", "primary_metric": "revenue", "primary_type": "int", "secondary_metric": "cost", "secondary_type": "double"},
383+
{"customer": "customer2", "primary_metric": "sales", "primary_type": "text", "secondary_metric": "profit", "secondary_type": "double"},
384+
],
385+
columns={
386+
"@{primary_metric}": "@{primary_type}",
387+
"@{secondary_metric}": "@{secondary_type}",
388+
},
389+
)
390+
def entrypoint(context: ExecutionContext, **kwargs) -> pd.DataFrame:
391+
return pd.DataFrame({
392+
context.blueprint_var("primary_metric"): [1],
393+
context.blueprint_var("secondary_metric"): [1.5],
394+
})
395+
```
396+
397+
Global variables (defined in the project config) can also be used as column names and types in the same way.
398+
372399
Note the use of curly brace syntax `@{customer}` in the model name above. It is used to ensure SQLMesh can combine the macro variable into the model name identifier correctly - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
373400

374401
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints="@gen_blueprints()"`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.

‎docs/integrations/dlt.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,12 @@ This will create the configuration file and directories, which are found in all
2828

2929
SQLMesh will also automatically generate models to ingest data from the pipeline incrementally. Incremental loading is ideal for large datasets where recomputing entire tables is resource-intensive. In this case utilizing the [`INCREMENTAL_BY_TIME_RANGE` model kind](../concepts/models/model_kinds.md#incremental_by_time_range). However, these model definitions can be customized to meet your specific project needs.
3030

31-
#### Specify the path to the pipelines directory
31+
#### Specify the path to the pipelines working directory
3232

33-
The default location for dlt pipelines is `~/.dlt/pipelines/<pipeline_name>`. If your pipelines are in a [different directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify the path explicitly:
33+
The default location for dlt pipeline working state is `~/.dlt/pipelines/<pipeline_name>`. If dlt stores your pipeline state in a [different pipelines working directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify that directory explicitly. This should be the directory where dlt stores pipeline state, not the directory containing your pipeline scripts:
3434

3535
```bash
36-
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-directory> dialect
36+
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-working-directory> dialect
3737
```
3838

3939
### Generating models on demand
@@ -58,10 +58,10 @@ sqlmesh dlt_refresh <pipeline-name> --force
5858
sqlmesh dlt_refresh <pipeline-name> --table <dlt-table>
5959
```
6060

61-
- **Provide the explicit path to the pipelines directory** (using `--dlt-path`):
61+
- **Provide the explicit path to the pipelines working directory** (using `--dlt-path`):
6262

6363
```bash
64-
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-directory>
64+
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-working-directory>
6565
```
6666

6767
#### Configuration

‎docs/integrations/engines/azuresql.md‎

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,41 @@ pip install "sqlmesh[azuresql]"
1414
```
1515
pip install "sqlmesh[azuresql-odbc]"
1616
```
17+
Set `driver: "pyodbc"` in your connection options.
18+
19+
20+
#### Python Driver (Official Microsoft driver for Azure SQL):
21+
See [`mssql-python`](https://pypi.org/project/mssql-python/) for more information.
22+
23+
```
24+
pip install "sqlmesh[azuresql-mssql-python]"
25+
```
26+
27+
Set `driver: "mssql-python"` in your connection options. This driver supports
28+
[Entra ID auth](https://github.com/microsoft/mssql-python/wiki/Microsoft-Entra-ID-support),
29+
for detailed connection options see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database).
30+
31+
!!! note
32+
The `mssql-python` driver [requires](https://pypi.org/project/mssql-python/) `python >= 3.10`.
33+
1734

1835
### Connection options
1936

20-
| Option | Description | Type | Required |
21-
| ----------------- | ---------------------------------------------------------------- | :----------: | :------: |
22-
| `type` | Engine type name - must be `azuresql` | string | Y |
23-
| `host` | The hostname of the Azure SQL server | string | Y |
24-
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
25-
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
26-
| `port` | The port number of the Azure SQL server | int | N |
27-
| `database` | The target database | string | N |
28-
| `charset` | The character set used for the connection | string | N |
29-
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
30-
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
31-
| `appname` | The application name to use for the connection | string | N |
32-
| `conn_properties` | The list of connection properties | list[string] | N |
33-
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
34-
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
35-
| `driver_name` | The driver name to use for the connection. E.g., *ODBC Driver 18 for SQL Server* | string | N |
36-
| `odbc_properties` | The dict of ODBC connection properties. E.g., authentication: ActiveDirectoryServicePrincipal. See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |
37+
| Option | Description | Type | Required |
38+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
39+
| `type` | Engine type name - must be `azuresql` | string | Y |
40+
| `host` | The hostname of the Azure SQL server | string | Y |
41+
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
42+
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
43+
| `port` | The port number of the Azure SQL server | int | N |
44+
| `database` | The target database | string | N |
45+
| `charset` | The character set used for the connection | string | N |
46+
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
47+
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
48+
| `login_attempts` | The number of reconnection attempts before failing. Default: 1 <br><br>*This option only applies to the `mssql-python` driver. | int | N |
49+
| `appname` | The application name to use for the connection | string | N |
50+
| `conn_properties` | The list of connection properties | list[string] | N |
51+
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
52+
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
53+
| `driver_name` | The driver name to use for the connection (e.g., *ODBC Driver 18 for SQL Server*). | string | N |
54+
| `odbc_properties` | The dict of ODBC connection properties (e.g., *authentication: ActiveDirectoryServicePrincipal*). See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16).<br><br>*For the `mssql-python` driver, please see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database). | dict | N |

0 commit comments

Comments
 (0)