Skip to content

14654 Add Redshift Data Connections driver#14830

Open
softwarenerd wants to merge 6 commits into
mainfrom
14654-data-connections-add-redshift-connection-for-internal-testing
Open

14654 Add Redshift Data Connections driver#14830
softwarenerd wants to merge 6 commits into
mainfrom
14654-data-connections-add-redshift-connection-for-internal-testing

Conversation

@softwarenerd

@softwarenerd softwarenerd commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a Redshift data connection driver (positron-data-driver-redshift). Redshift speaks the PostgreSQL wire protocol, so the driver is built on pg and shares structure with the PostgreSQL driver, with additional handling for the ways a serverless warehouse differs from a local Postgres: cold-start connects, idle-dropped sockets, and a per-statement latency floor.

What's included

  • Redshift driver: user/password connection (host accepts a full host:port/database endpoint paste), SSL on by default, schema/table/view/column browsing, and generated connection code for redshift_connector (Python) and DBI/RPostgres (R).
  • Cross-database browsing: on RA3/Serverless clusters the tree gains a top-level Databases group and browses every database in the namespace over one connection via the SVV_ALL_* catalog views and three-part references, detected at runtime.
  • Connection resilience (also applied to the PostgreSQL driver): TCP keepalive so idle sockets survive, an idle-error guard so a dropped socket can't crash the extension host, transparent reconnect-and-retry when a query hits a dead connection, and bounded connect-retry with backoff so a resuming serverless workgroup is waited out instead of failing.
  • Data Explorer summary performance: column profiling is computed in ~3 batched statements per request (one scalar-aggregate scan covering null counts, summary stats, and an exact PERCENTILE_CONT median; one UNION ALL for all histograms; one UNION ALL for all frequency tables) instead of the previous ~4-5 statements per column. Redundant per-column null-count and quantile round-trips are eliminated.
  • Pass coalescing: the summary panel re-requests profiles on layout churn, so the RPC handler now runs at most one pass per dataset at a time and cancels superseded passes at statement boundaries, so a burst of requests can't stack statements on the single connection.
  • Type-safe value rendering and graceful degradation: frequency-table values are rendered to text per Redshift type (booleans via CASE, SUPER via JSON_SERIALIZE, VARBYTE via TO_HEX, spatial via ST_AsText, else cast), and the histogram/frequency batches degrade to a missing sparkline rather than failing the whole pass if a value type still can't be rendered.

Why the profiling rework

On the test cluster (8 RPUs, serverless), every statement carries roughly a 1-second fixed cost regardless of data size, so the original design (a separate scan per column per metric, run serially over a single connection) took 60+ seconds and timed out even on tiny tables (a 370K-row, 19-column table; a 365-row, 8-column table). The bottleneck was statement count and per-statement overhead, not scan time, so the fix was to collapse each request to a handful of set-based statements and stop redundant/overlapping work rather than to sample or approximate. The frontend's windowed chunking (PROFILE_CHUNK_SIZE) is left unchanged and honored.

Testing

  • Unit tests for the driver (connection lifecycle, cross-database detection, schema browsing, endpoint parsing), the reconnecting client (keepalive config, reconnect-and-retry, coalesced reconnect, connect-retry vs terminal-error), and the table view (batched scalar/histogram/frequency statements, boolean rendering, graceful degradation).
  • Manually validated against a live Redshift Serverless workgroup: connect through a cold start, browse single-database and cross-database trees, and open multiple tables in the Data Explorer with the summary sidebar. A diagnostic "Redshift Data Explorer" output channel logs the per-pass query timeline.

Known limitations / follow-ups

  • The equivalent Data Explorer profiling rework has not yet been ported to the PostgreSQL driver (its per-statement floor is far lower, so it is less urgent).
  • Frequency tables are batched per chunk, so an unrenderable column type drops that chunk's frequency tables (not just the one column) rather than isolating to the single column; the summary still renders.
  • SSL currently connects without server-certificate verification (the Redshift CA is not yet bundled); IAM and Okta auth mechanisms are stubbed for later.

@github-actions

Copy link
Copy Markdown

E2E Tests 🚀
This PR will run tests tagged with: @:critical @:connections

Why these tags?
Tag Source
@:critical Always runs (required)
@:connections Changed files

More on automatic tags from changed files.

Warning

This PR touches a Positron directory that isn't mapped in test-tag-paths-map.json: extensions/positron-data-driver-redshift/. Add an entry (an e2e tag or [] for no coverage) so future changes are tagged automatically.

readme  valid tags

@softwarenerd softwarenerd changed the title 14654 data connections add redshift connection for internal testing 14654 Add Amazon Redshift Data Connections driver Jul 10, 2026
@softwarenerd softwarenerd requested a review from timtmok July 10, 2026 21:48
@softwarenerd softwarenerd changed the title 14654 Add Amazon Redshift Data Connections driver 14654 Add Redshift Data Connections driver Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data Connections: Add Redshift connection for internal testing

1 participant