Skip to content

feat(driver): cached statement executors, insert lastrowid, and Cloud SQL connector retries - #930

Merged
cofin merged 5 commits into
mainfrom
feat/issues-927-929
Nov 21, 2026
Merged

cofin merged 5 commits into
mainfrom
feat/issues-927-929

Conversation

@cofin

@cofin cofin commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Summary

Let external adapters run cached statements through one compiled function, preserve last_inserted_id across cached INSERT execution paths, and update the AsyncPG Cloud SQL and AlloyDB connector wrappers to accept asyncpg.create_pool connection kwargs and retry transient failures with bounded exponential backoff.

Key Changes

  • Driver (sqlspec/driver/__init__.py, sqlspec/driver/_common.py, sqlspec/driver/_sync.py, sqlspec/driver/_async.py, sqlspec/core/result/_base.py):

    • Add set_cached_statement_executor(executor, *, state=None, owner=None) on SyncDriverAdapterBase and AsyncDriverAdapterBase, and export the SyncCachedStatementExecutor and AsyncCachedStatementExecutor protocols from sqlspec.driver. Passing None removes the executor. When owner is given and a subclass overrides one of the watched driver methods, the call returns False and leaves the current executor and state unchanged.
    • Add resolve_last_inserted_id(cursor) on CommonDriverAttributesMixin.
    • Register resolve_last_inserted_id in _CACHE_HIT_HOOKS and guard cache_hit_hooks_unchanged() against non-type arguments and non-subclasses.
    • Populate last_inserted_id on DMLResult in _execute_cache_hit_with_cursor for cached INSERT operations affecting at least one row across both the direct cursor and dispatch_execute fallback paths.
    • Expand the docstring for SQLResult.create_cached_select().
  • MySQL, pymssql & Sybase adapters (sqlspec/adapters/{pymysql,aiomysql,asyncmy,mysqlconnector}/core.py, sqlspec/adapters/{pymssql,sybase}/driver.py):

    • Reject bool values in resolve_rowcount() and normalize_lastrowid() across all four MySQL adapters.
    • Pass normalize_lastrowid(cursor) into DMLResult inside execute_cache_hit() (and execute_cache_hit_async()) on successful cached INSERT queries with affected_rows > 0.
    • Override resolve_last_inserted_id() to return None in PymssqlDriver and SybaseDriver to match dispatch_execute().
  • AsyncPG Cloud SQL & AlloyDB connectors (sqlspec/adapters/asyncpg/config.py, sqlspec/adapters/asyncpg/core.py):

    • Update _AsyncpgCloudSqlConnector.__call__ and _AsyncpgAlloydbConnector.__call__ to accept *args, **kwargs from asyncpg.create_pool, stripping connector-managed keys (ssl, direct_tls, host, port, dsn, credentials, instance selectors) and None values while forwarding runtime options such as loop, connection_class, record_class, statement_cache_size, command_timeout, timeout, and server_settings.
    • Forward remaining options from AsyncpgConfig.create_connection() to managed connector wrappers while keeping custom zero-arg connect callbacks unchanged.
    • Add connector_retry_count (3), connector_retry_delay (0.5), and connector_retry_max_delay (5.0) to AsyncpgDriverFeatures with eager validation in _validate_connector_config() and shared DEFAULT_CONNECTOR_* constants in core.py.
    • Add compiled is_transient_connector_error() and calculate_connector_backoff_seconds() helpers using sys.modules lookups (no eager optional imports) and cycle-/depth-bounded causal traversal so transient control-plane and transport errors retry with jittered backoff while permanent auth, permission, TLS/certificate, configuration, and PostgreSQL errors fail immediately.
  • PyMySQL Cloud SQL connector (sqlspec/adapters/pymysql/config.py, sqlspec/adapters/pymysql/core.py):

    • Retry transient Cloud SQL connector failures in the pool's connection factory with the same connector_retry_count, connector_retry_delay, and connector_retry_max_delay settings, validated when the config is built.
    • Add adapter-local is_transient_connector_error() and calculate_connector_backoff_seconds() helpers; MySQL, authentication, and configuration errors are raised on the first attempt.
  • Documentation (docs/databases/postgres/asyncpg.md, docs/databases/mysql.md, docs/reference/driver/, docs/reference/core/results.md, docs/changelog.md):

    • Regenerate the AsyncpgDriverFeatures and PyMysqlDriverFeatures configuration tables.
    • Document set_cached_statement_executor, the executor protocols, resolve_last_inserted_id, and SQLResult.create_cached_select.
    • Add the v0.73.0 changelog section.
  • Release (pyproject.toml, uv.lock):

    • Bump package version from 0.72.2 to 0.73.0.
  • Tests (tests/unit/driver/test_query_cache.py, tests/unit/adapters/test_asyncpg/test_cloud_connectors.py, tests/unit/adapters/test_pymysql/test_cloud_sql_connector.py, tests/unit/adapters/test_{pymysql,aiomysql,asyncmy,mysqlconnector}/test_cache_hit_executor.py, tests/integration/adapters/_shared/_driver_type_system.py):

    • Add unit tests covering driver exports, setting and removing a cached statement executor, executors that return None, observability routing, subclass override detection, generic sync/async cached INSERT last_inserted_id paths, and pymysql/aiomysql/asyncmy/mysqlconnector cache-hit row IDs.
    • Add real asyncpg.create_pool integration tests for both Cloud SQL and AlloyDB wrappers, reserved kwarg filtering, retry recovery/exhaustion, fail-fast classification matrices, connector retry config validation, and subprocess import isolation.
    • Add PyMySQL retry recovery, retry limit, fail-fast, and settings validation tests.
    • Register the three retry keys in DRIVER_FEATURE_CONSUMED_KEYS for asyncpg and pymysql.

Closes #927
Closes #928
Closes #929

Expose a public registration surface for external adapters that implement single-call compiled cache-hit executors, and preserve `last_inserted_id` across both generic driver cache-hit fallback paths and the four MySQL adapter compiled executors.

- Export `SyncCacheHitExecutor`, `AsyncCacheHitExecutor`, and `cache_hit_hooks_unchanged` from `sqlspec.driver`.
- Add `cache_hit_executor`, `cache_hit_state`, `clear_cache_hit_executor`, and `resolve_last_inserted_id` on `CommonDriverAttributesMixin`, and typed `register_cache_hit_executor` methods on `SyncDriverAdapterBase` and `AsyncDriverAdapterBase`.
- Preserve `last_inserted_id` on `DMLResult` for cached `INSERT` executions with `affected_rows > 0` in `_execute_cache_hit_with_cursor` and across `pymysql`, `aiomysql`, `asyncmy`, and `mysqlconnector` (`normalize_lastrowid` and `resolve_rowcount` also reject `bool` values).
- Override `resolve_last_inserted_id` to return `None` in `PymssqlDriver` and `SybaseDriver` to match `dispatch_execute`.
- Document the public cache-hit registration API and `SQLResult.create_cached_select` in reference docs.

Closes #927
Closes #928
…r errors

Update the Cloud SQL and AlloyDB asyncpg connector wrappers to accept arbitrary `asyncpg.create_pool` positional and keyword arguments, filter connector-managed and `None` options, and retry transient control-plane and transport failures with bounded jittered backoff.

- Accept `*args, **kwargs` in `_AsyncpgCloudSqlConnector.__call__` and `_AsyncpgAlloydbConnector.__call__`, stripping connector-owned keys (`ssl`, `direct_tls`, `host`, `port`, `dsn`, credentials, instance selectors) and `None` values while preserving runtime options (`loop`, `connection_class`, `record_class`, `statement_cache_size`, `command_timeout`, `timeout`, `server_settings`).
- Forward non-connector options from `AsyncpgConfig.create_connection()` to managed connector wrappers while keeping custom zero-arg `connect` callbacks unchanged.
- Add `connector_max_retries`, `connector_retry_backoff_seconds`, and `connector_retry_max_backoff_seconds` to `AsyncpgDriverFeatures` with eager validation and shared default constants (`3`, `0.5`, `5.0`) in `sqlspec/adapters/asyncpg/core.py`.
- Add compiled `is_transient_connector_error` and `calculate_connector_backoff_seconds` helpers in `sqlspec/adapters/asyncpg/core.py` using `sys.modules` lookups and cycle-/depth-bounded causal traversal to classify transient vs. permanent errors without eager optional imports.
- Regenerate `docs/databases/postgres/asyncpg.md` configuration tables and update driver feature honesty tests.

Closes #929
Bump package version from 0.72.2 to 0.73.0 and add the v0.73.0 release section to `docs/changelog.md`.
@codecov-commenter

codecov-commenter commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.24561% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.70%. Comparing base (28cdd34) to head (f1c11d9).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sqlspec/adapters/pymysql/core.py 95.65% 3 Missing and 3 partials ⚠️
sqlspec/adapters/asyncpg/core.py 98.54% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
+ Coverage   84.65%   84.70%   +0.05%     
==========================================
  Files         629      629              
  Lines      142422   142845     +423     
  Branches    30813    30912      +99     
==========================================
+ Hits       120564   120999     +435     
+ Misses      15222    15217       -5     
+ Partials     6636     6629       -7     
Flag Coverage Δ
cloud-adbc-storage 29.76% <16.44%> (-0.04%) ⬇️
integration 56.54% <26.09%> (-0.10%) ⬇️
mssql-db2 30.68% <16.88%> (-0.05%) ⬇️
mysql-mariadb-embedded 38.01% <25.43%> (-0.05%) ⬇️
oracle 30.06% <16.44%> (-0.04%) ⬇️
postgres-cockroach 39.72% <18.64%> (-0.07%) ⬇️
py3.11 84.54% <98.24%> (+0.05%) ⬆️
py3.12 84.56% <98.24%> (+0.05%) ⬆️
py3.13 84.55% <98.24%> (+0.05%) ⬆️
py3.14 84.30% <98.22%> (+0.06%) ⬆️
unit 82.00% <98.02%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/adapters/aiomysql/core.py 81.25% <100.00%> (+0.51%) ⬆️
sqlspec/adapters/asyncmy/core.py 81.68% <100.00%> (+0.51%) ⬆️
sqlspec/adapters/asyncpg/config.py 94.27% <100.00%> (+3.41%) ⬆️
sqlspec/adapters/mysqlconnector/core.py 84.42% <100.00%> (+0.11%) ⬆️
sqlspec/adapters/pymssql/driver.py 93.09% <100.00%> (+0.07%) ⬆️
sqlspec/adapters/pymysql/config.py 93.95% <100.00%> (+1.72%) ⬆️
sqlspec/adapters/sybase/driver.py 65.66% <100.00%> (+0.39%) ⬆️
sqlspec/core/result/_base.py 80.33% <ø> (ø)
sqlspec/driver/__init__.py 62.50% <ø> (ø)
sqlspec/driver/_async.py 83.00% <100.00%> (+0.06%) ⬆️
... and 4 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… retry settings

Reduce the new public surface to one method and two protocols, and align the AsyncPG connector retry settings with the existing connection retry names.

- Rename `SyncCacheHitExecutor` and `AsyncCacheHitExecutor` to `SyncCachedStatementExecutor` and `AsyncCachedStatementExecutor`, and `register_cache_hit_executor()` to `set_cached_statement_executor()`.
- Passing `None` to `set_cached_statement_executor()` removes the executor, replacing `clear_cache_hit_executor()`.
- Remove the `cache_hit_executor` and `cache_hit_state` properties and stop exporting `cache_hit_hooks_unchanged` from `sqlspec.driver`.
- Leave the current executor and state unchanged when the `owner` check fails.
- Rename `connector_max_retries`, `connector_retry_backoff_seconds`, and `connector_retry_max_backoff_seconds` to `connector_retry_count`, `connector_retry_delay`, and `connector_retry_max_delay`.
@cofin cofin changed the title feat(driver): cache-hit executor API, insert lastrowid, and asyncpg connector retries feat(driver): cached statement executors, insert lastrowid, and asyncpg connector retries Nov 21, 2026
Retry transient Cloud SQL connector failures in the PyMySQL connection factory, matching the AsyncPG connector settings.

- Add `connector_retry_count`, `connector_retry_delay`, and `connector_retry_max_delay` to `PyMysqlDriverFeatures`, validated when the config is built.
- Add adapter-local `is_transient_connector_error` and `calculate_connector_backoff_seconds` helpers in `sqlspec/adapters/pymysql/core.py`; MySQL, authentication, and configuration errors are raised on the first attempt.
- Write the AsyncPG and PyMySQL retry loops without an unreachable trailing raise.
- Replace the AsyncPG pool test connection with a stub that sets no private asyncpg attributes, and cover status shapes, implicit exception context, missing optional modules, and uninitialized connectors.
- Regenerate the `PyMysqlDriverFeatures` configuration table.
@cofin cofin changed the title feat(driver): cached statement executors, insert lastrowid, and asyncpg connector retries feat(driver): cached statement executors, insert lastrowid, and Cloud SQL connector retries Nov 21, 2026
@cofin
cofin merged commit 2d2f676 into main Nov 21, 2026
81 of 83 checks passed
@cofin
cofin deleted the feat/issues-927-929 branch November 21, 2026 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants