14654 Add Redshift Data Connections driver#14830
Open
softwarenerd wants to merge 6 commits into
Open
Conversation
…and SVV_ALL_* catalog views
…edshift" to "Redshift"
…hift and PostgreSQL drivers
… coalescing, type-safe frequency rendering, and graceful degradation to fix Data Explorer summary timeouts
|
E2E Tests 🚀 Why these tags?
More on automatic tags from changed files. Warning This PR touches a Positron directory that isn't mapped in test-tag-paths-map.json: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Redshift data connection driver (
positron-data-driver-redshift). Redshift speaks the PostgreSQL wire protocol, so the driver is built onpgand shares structure with the PostgreSQL driver, with additional handling for the ways a serverless warehouse differs from a local Postgres: cold-start connects, idle-dropped sockets, and a per-statement latency floor.What's included
host:port/databaseendpoint paste), SSL on by default, schema/table/view/column browsing, and generated connection code forredshift_connector(Python) and DBI/RPostgres (R).SVV_ALL_*catalog views and three-part references, detected at runtime.PERCENTILE_CONTmedian; oneUNION ALLfor all histograms; oneUNION ALLfor all frequency tables) instead of the previous ~4-5 statements per column. Redundant per-column null-count and quantile round-trips are eliminated.CASE,SUPERviaJSON_SERIALIZE,VARBYTEviaTO_HEX, spatial viaST_AsText, else cast), and the histogram/frequency batches degrade to a missing sparkline rather than failing the whole pass if a value type still can't be rendered.Why the profiling rework
On the test cluster (8 RPUs, serverless), every statement carries roughly a 1-second fixed cost regardless of data size, so the original design (a separate scan per column per metric, run serially over a single connection) took 60+ seconds and timed out even on tiny tables (a 370K-row, 19-column table; a 365-row, 8-column table). The bottleneck was statement count and per-statement overhead, not scan time, so the fix was to collapse each request to a handful of set-based statements and stop redundant/overlapping work rather than to sample or approximate. The frontend's windowed chunking (
PROFILE_CHUNK_SIZE) is left unchanged and honored.Testing
Known limitations / follow-ups