Add end-to-end Parquet VARIANT support

[xiangfu0]

Summary

Add an end-to-end VARIANT user journey for Apache Pinot:

• define a single-value VARIANT dimension retained in Pinot's raw forward index
• ingest top-level, non-repeated Apache Parquet VARIANT(1) values from unshredded and shredded files
• optionally materialize frequently queried scalar paths during ingestion while retaining the original value
• expose Spark-style parse, extraction, type, existence, null, and JSON-rendering functions
• carry Variant through Pinot schemas, segments, query plans, responses, DDL, Java/JDBC clients, and both query engines
• provide a packaged batch quickstart plus integration and mixed-version compatibility coverage

Raw Variant values are deliberately opaque to operations that require equality, hashing, or ordering. Queries must extract a typed scalar before comparing, grouping, joining, sorting, using set operations, or applying general aggregates. COUNT(raw_variant) remains supported.

The persisted format, null semantics, ownership boundaries, and rollout contract are documented in pinot-spi/VARIANT_DESIGN.md.

End-to-end flow

Define a table column with "dataType": "VARIANT", storage null handling enabled, dictionary disabled, and a raw forward index. The native Parquet reader reconstructs VARIANT(1) metadata/value pairs—including supported shredded layouts—into Pinot's versioned PVAR envelope.

Frequently queried fields can be materialized with ingestion transforms while the full Variant remains available:

SET enableNullHandling=true;

SELECT eventType, COUNT(*)
FROM variantEvents
GROUP BY eventType
ORDER BY eventType;

SELECT
  eventId,
  variant_get(payload, '$.user.id', 'STRING') AS userId,
  variant_get(payload, '$.amount', 'DOUBLE') AS amount
FROM variantEvents
WHERE eventType = 'checkout'
ORDER BY eventId;

SELECT eventId, variantToJson(payload)
FROM variantEvents
ORDER BY eventId;

Supported scope

• top-level, non-repeated Parquet VARIANT(1) columns
• single-value Pinot VARIANT dimensions without dictionaries
• raw forward-index retention and optional scalar materialization
• storage null handling enabled by schema or table configuration
• query-time Variant functions with SET enableNullHandling=true
• single-stage and multi-stage query execution

Nested/repeated Variant columns, streaming ingestion, and quoted path keys are outside the initial scope.

A JSON index is intentionally not accepted on a VARIANT column. The existing JSON index expects Pinot JSON text and its query rewrite semantics; treating the binary PVAR envelope as JSON would be incorrect. Indexing Variant paths should be introduced separately with an explicit physical/index contract. Materialized scalar columns are the supported indexed path for this PR.

The automatic Parquet reader selection preserves existing Avro-metadata precedence. Files containing Avro metadata must explicitly select the native Parquet reader to ingest Variant.

Production safety

• centralizes PVAR framing and validation in VariantEnvelope and freezes version-1 bytes with golden tests
• assigns protobuf wire value 24 to Variant while preserving UUID/UUID_ARRAY at 22/23
• rejects unsupported schemas, table/index configurations, query null-handling modes, and raw Variant operations
• validates Parquet decimal precision, scale, bounds, hostile exponents, malformed rows, and buffer ownership
• handles array-backed, direct, sliced, and read-only Parquet buffers without changing source positions or limits
• compiles constant paths/types once and reuses cursors in ingestion, single-stage, and multi-stage execution
• distinguishes SQL null, encoded Variant null, the string "null", missing paths, and conversion failures across query and client response formats
• keeps Parquet column/schema dependencies isolated to the input plugin
• preserves producer-defined physical object-value ordering, including valid non-monotonic field offsets, by sizing each selected subtree from its own self-delimiting encoding
• uses an allocation-free first-field/binary lookup for wide objects with an interoperability-safe fallback for parquet-java UTF-16 and Arrow-RS/spec UTF-8 key orderings

Rolling upgrades

Variant remains an explicit full-fleet activation feature: upgrade controllers, brokers, servers, clients, and external ingestion jobs before registering a Variant schema, and do not roll back while Variant tables are active.

This remains one gated PR because splitting wire/schema registration from ingestion and query guards would create mergeable partial-activation states. The layered commits remain independently reviewable while activation stays fail-closed until every layer is present.

Compatibility coverage verifies:

• all-old components reject the unknown wire type deterministically
• a new broker with old servers fails with upgrade guidance
• a mixed old/new server fleet rejects the whole query without returning partial rows
• an all-new fleet succeeds
• an old broker with upgraded servers remains queryable for the verified path
• capability checks, readiness retries, expected-error matching, and rollback are bounded and deterministic

Quickstart

./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar
build/bin/quick-start-variant-batch.sh

The quickstart registers variantEvents, ingests the committed five-row Parquet fixture, uploads the segment, and runs materialized-field, nested-extraction, JSON-rendering, and null-semantics queries. Its resources live under pinot-tools/src/main/resources/examples/batch/variantEvents.

Verification

• 23 VariantTypeTest scenarios across both query engines in the full integration-test reactor, including an externally encoded Parquet object with non-monotonic field offsets
• 72 focused VariantUtilsTest cases covering every Variant physical encoding, wide-object thresholds, parquet-java/Arrow-RS Unicode ordering, malformed values, and reusable cursors
• 18 focused native-Parquet reader scenarios, including byte-for-byte preservation of unshredded non-monotonic objects
• all-old, new/old, mixed-server, all-new, old-broker/new-server, and rollback compatibility flows
• fresh post-rebase JDK 25 reactors covering Variant utilities, both query engines, aggregation, segment and table validation, the full Variant integration suite, and the upstream grouping-set regression
• spotless, checkstyle, license formatting, and license checks across every affected module
• clean eight-commit history on current upstream/master
• git diff --check

Performance benchmark

A 2,000,000-row workstation benchmark compared identical logical events represented as JSON text and Parquet VARIANT(1), then ingested into raw Pinot columns with ZSTD compression. A third JSON scenario added Pinot's JSON index and is reported separately because it is an index-assisted comparison, not a format-only comparison. The reproducible harness is preserved in the benchmark snapshot.

Ingestion and storage

Representation	Median build	Throughput	Source Parquet	Pinot segment	Result
JSON raw	32.942 s	60,760 rows/s	285.52 MiB	294.16 MiB	Baseline
VARIANT raw	14.315 s	139,714 rows/s	279.18 MiB	298.89 MiB	2.29× faster ingestion
JSON + JSON index	194.873 s	10,263 rows/s	285.52 MiB	1,253.70 MiB	Index-assisted

Compared with raw JSON, the VARIANT source was 2.22% smaller and its Pinot segment was 1.61% larger. Adding the JSON index made the JSON segment 4.26× the size of the raw JSON segment.

Query latency

Each value is the median of four round-level p50 values. Every round used two warmups and 10 measured sequential executions.

Workload	JSON p50	VARIANT p50	Result
Materialized eventType count (control)	0.326 ms	0.349 ms	Effectively neutral
Nested numeric sum	2,674.153 ms	592.766 ms	VARIANT 4.49× faster
Nested tier filter	2,704.314 ms	681.794 ms	VARIANT 3.98× faster
Four-path numeric aggregate	9,037.254 ms	1,051.095 ms	VARIANT 8.60× faster
Selective country filter	JSON index: 6.553 ms	VARIANT raw scan: 645.687 ms	JSON index 98.69× faster; not format-only

The raw comparison used four Parquet files/Pinot segments and four ingestion rounds with alternating JSON/VARIANT build order. Source generation was excluded from ingestion timing, and every query result was checked against independently generated ground truth.

These are paired, sequential, warm-cache, single-stage measurements from an Apple M2 Max workstation with 32 GB RAM, macOS 26.5.2, and Temurin 25 on AC power. They are directional workstation evidence, not distributed production-cluster throughput: network transfer, multi-stage execution, concurrent clients, distributed scheduling, and production hardware were outside the run.

Wide-object navigation microbenchmark

A focused JMH A/B compared the pre-fix PR head (ae60e3de73) with the interoperability fix (bc5658c3a3) on JDK 25. Each case used one fork, two 500 ms warmups, three 500 ms measurements, flat integer values, and -prof gc.

Fields	Lookup	Before	After	Change
32	first	32.3 ns/op	37.8 ns/op	+17.1%
32	middle	235.1 ns/op	77.4 ns/op	67.1% faster
32	last	424.4 ns/op	189.3 ns/op	55.4% faster
32	missing	296.0 ns/op	314.4 ns/op	+6.2%
100	first	34.6 ns/op	40.5 ns/op	+17.1%
100	middle	633.3 ns/op	71.3 ns/op	88.7% faster
100	last	1,226.0 ns/op	266.5 ns/op	78.3% faster
100	missing	797.8 ns/op	860.7 ns/op	+7.9%

The first-field probe avoids the roughly 5.8× regression found in the initial binary-search implementation. Missing keys retain a small cost because a failed parquet-java-order binary search must fall back to byte equality for external producer ordering. Both versions are effectively allocation-free (normalized values below 0.02 B/op, no GC). These short microbenchmarks are directional; the large middle/last improvements are clear, while smaller deltas have wider uncertainty.

Review areas

This PR changes public SPI and query-wire behavior. The main ownership reviews are:

• SPI, schema, and query-wire compatibility
• Parquet and input-format ingestion
• single-stage and multi-stage query semantics
• Java, JDBC, Arrow, JSON, and gRPC response behavior
• rolling-upgrade activation and rollback

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.13847% with 436 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.81%. Comparing base (5e914c9) to head (2bf0462).

Files with missing lines	Patch %	Lines
...va/org/apache/pinot/common/utils/VariantUtils.java	77.84%	110 Missing and 100 partials ⚠️
...n/inputformat/parquet/ParquetVariantConverter.java	81.75%	14 Missing and 13 partials ⚠️
...ransform/function/VariantGetTransformFunction.java	77.35%	12 Missing and 12 partials ⚠️
...uery/runtime/operator/operands/VariantOperand.java	82.95%	5 Missing and 10 partials ⚠️
...not/common/evaluator/InbuiltFunctionEvaluator.java	82.35%	4 Missing and 8 partials ⚠️
...pinot/query/runtime/operator/HashJoinOperator.java	47.82%	9 Missing and 3 partials ⚠️
...n/inputformat/parquet/ParquetAvroRecordReader.java	75.00%	9 Missing and 1 partial ⚠️
...java/org/apache/pinot/spi/utils/PinotDataType.java	59.09%	8 Missing and 1 partial ⚠️
...java/org/apache/pinot/common/utils/DataSchema.java	70.37%	4 Missing and 4 partials ⚠️
...er/validation/TypeCapabilityValidationVisitor.java	89.47%	1 Missing and 7 partials ⚠️
... and 36 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19101      +/-   ##
============================================
+ Coverage     67.55%   67.81%   +0.26%     
- Complexity     1430     1512      +82     
============================================
  Files          3486     3501      +15     
  Lines        224100   226435    +2335     
  Branches      35370    35808     +438     
============================================
+ Hits         151392   153564    +2172     
+ Misses        60678    60599      -79     
- Partials      12030    12272     +242     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	?
java-25	67.81% <82.13%> (+0.26%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.81% <82.13%> (+0.26%)	⬆️
unittests	67.81% <82.13%> (+0.26%)	⬆️
unittests1	57.90% <78.41%> (+0.26%)	⬆️
unittests2	39.42% <35.84%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds end-to-end support for the VARIANT logical type across Pinot ingestion (notably Parquet VARIANT(1)), query planning/execution (single-stage + multi-stage), response encoders, and Java/JDBC client surfaces, while explicitly rejecting unsupported raw-VARIANT operations (e.g., ordering/grouping/join keys) with actionable errors.

Changes:

• Introduces VARIANT as a first-class Pinot data type, including schema/DDL handling, wire/proto support, and null-sentinel semantics.
• Adds Parquet VARIANT(1) ingestion support via the native Parquet reader and includes a packaged “variant batch” quickstart with resources and CI coverage.
• Adds query-time VARIANT functions (parse/get/existence/type/json rendering) and enforces operator-level restrictions where raw VARIANT lacks equality/hash/ordering semantics.

Reviewed changes

Copilot reviewed 174 out of 175 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pom.xml	Adds managed dependency for org.apache.parquet:parquet-variant.
pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/QuickStartTest.java	Maps new quickstart command strings to VariantQuickStart.
pinot-tools/src/main/resources/examples/batch/variantEvents/*	Adds VARIANT batch quickstart schema/table config, ingestion spec, and documentation.
pinot-tools/pom.xml	Adds packaged quick-start-variant-batch program entry.
pinot-sql-ddl/src/main/, src/test/	Adds DDL compile/reverse/roundtrip support for VARIANT and validates default emission rules.
pinot-spi/src/main/, src/test/	Adds PinotDataType.VARIANT conversion/ingestion mapping and schema validation updates.
pinot-segment-spi/src/main/**	Handles VARIANT default-null sentinel persisted as empty string in segment metadata.
pinot-segment-local/src/main/, src/test/	Adds VARIANT validation/sanitization rules; disables min/max, ordering, dictionary creation where unsupported; adds targeted tests.
pinot-query-planner/src/main/, src/test/	Adds VARIANT type mapping (Calcite/plan nodes), plan validation, constant-folding guardrails, and serde coverage.
pinot-query-runtime/src/main/, src/test/	Enforces VARIANT restrictions at runtime operators (sort/window/set/group-by/etc.) and adds operator tests.
pinot-core/src/main/, src/test/	Enforces VARIANT restrictions in single-stage operators/aggregations/distinct/order-by and adds tests.
pinot-common/src/main/, src/test/	Adds VARIANT to encoders, datablock equality, proto enum, function canonicalization, and introduces scalar VariantFunctions.
pinot-clients/pinot-jdbc-client/src/main/, src/test/	Treats VARIANT as canonical JSON text (VARCHAR) while preserving encoded-variant-null vs SQL-null behavior.
pinot-clients/pinot-java-client/src/main/, src/test/	Preserves legacy getString() null behavior while distinguishing VARIANT nulls.
pinot-plugins/pinot-input-format/pinot-parquet/src/main/**	Enhances Parquet reader selection/metadata helpers and initializes native reader with schema-bound converters for VARIANT.
pinot-controller/src/main/, src/test/	Ensures stored-vs-compiled schema default comparisons handle VARIANT byte[] defaults by content.
pinot-compatibility-verifier/**	Adds file-contains operation and VARIANT mixed-version suite coverage including old-broker/new-servers phase.
.github/workflows/scripts/.pinot_quickstart.sh	Adds CI validation for the packaged VARIANT batch quickstart.

[Copilot]

Pull request overview

Copilot reviewed 174 out of 175 changed files in this pull request and generated no new comments.

Suppressed comments (1)

pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:81

• The validation uses supportsOrdering() but the error text always mentions raw VARIANT. supportsOrdering() is also false for other non-orderable logical types (e.g. OBJECT / array types), so this can surface a misleading VARIANT-specific message when the real unsupported type is something else.

Consider tailoring the message based on the actual column type (keep the VARIANT guidance when type == VARIANT, otherwise emit a generic "ORDER BY does not support values of type X" message).

[xiangfu0]

Reviewed the full VARIANT PR (checked out the branch; ran domain-focused passes over SPI/envelope, Parquet ingestion, the VariantUtils codec, single-stage + multi-stage enforcement, schema/table validation, and wire/client/compat). Overall this is very carefully engineered — the PVAR envelope, the four null-state handling, and the defense-in-depth opacity guards all check out, and I found no data-corruption or crash bugs in the core paths.

Two items I'd suggest fixing before merge (inline: an OBJECT-comparison regression and a partial-upsert validation gap), plus a few low-severity polish notes.

Verified-correct highlights: proto appends VARIANT=24 with no renumbering and ColumnDataType is name-serialized (no ordinal()-indexed tables exist repo-wide, so the mid-enum insertion is safe); Parquet buffer ownership / shredded-vs-unshredded selection / reader precedence / reinit safety; codec bounds + per-row cursor reset + thread-confinement; and the full set of schema/table restrictions (dictionary / secondary indexes / PK / partition / sorted / star-tree (default and explicit) / metrics-agg / upsert-comparison all rejected, min/max never published, never sorted).

[xiangfu0]

Follow-up on my review: all four findings are now fixed and merged into this PR (head is now 8b38248f).

Severity	Finding	Fix
Medium	Comparison / IN / IS DISTINCT FROM guards rejected any non-orderable/non-equatable type (OBJECT, arrays, MAP) with a VARIANT-specific error, narrowing existing behavior	Guards now reject only raw VARIANT, in both engines: FilterOperand, TransformOperandFactory, BinaryOperatorTransformFunction, InTransformFunction
Medium	VARIANT OVERWRITE-only partial-upsert check only covered listed columns; unlisted columns use defaultPartialUpsertStrategy (and a custom partialUpsertMergerClass), which could silently apply INCREMENT/APPEND/UNION/IGNORE to a PVAR envelope	TableConfigUtils now validates the effective strategy for every VARIANT column and rejects a custom merger on VARIANT columns
Low	Opacity errors said "raw VARIANT" for any non-orderable type	Messages now name the actual type; raw-VARIANT wording + variantGet guidance kept only for VARIANT (VariantTypeValidationVisitor, SortOperator, SortedMailboxReceiveOperator, OrderByComparatorFactory)
Low	Planner did not validate GROUP BY keys (runtime did)	VariantTypeValidationVisitor.validateAggregateInputs now validates AggregateNode group keys

Regression tests were added for each. spotless/checkstyle/license are clean on all four affected modules, and every affected unit-test class passes locally on JDK 25 (VariantTableConfigValidationTest, FilterOperandTest, VariantTypeValidationVisitorTest, and the pinot-core comparison/IN/ORDER BY suites).

On the current CI run, the Linter and all VARIANT-exercising tests pass. The three red checks are pre-existing flakes unrelated to these changes and to VARIANT: FilteredAggregationsTest.testFilterVsCase (Unit Set 1, same as before this PR) and testGeneratedQueries/testConcurrentQueries random-query flakes in the two integration sets (no VariantType* failures). A re-run should clear them.