Add Spark HBase connector classes to the HBase image so server-side column pushdown works

[razvan]

Summary

Predicate (column) pushdown from the Spark HBase connector currently fails at runtime because the HBase RegionServers do not have the connector classes on their classpath. To evaluate a pushed-down filter, the RegionServer must deserialize and instantiate org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter (plus its generated protobuf class), which ships only in the Spark image, not the HBase image. We should make these classes available on the HBase RegionServer classpath so pushdown works end-to-end.

Background

While extending the hbase-connector integration test in spark-k8s-operator with filtered queries, pushdown failed. After fixing two Spark-image packaging issues (shipping hbase-spark-protocol-shaded and aligning the protobuf toolchain), the filter serialized correctly on the Spark side but was then rejected by the RegionServer:

org.apache.hadoop.hbase.DoNotRetryIOException:
  java.lang.ClassNotFoundException: org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1612)
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:589)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2604)

This is inherent to how the hbase-spark connector implements pushdown: the custom filter is shipped in the RPC and evaluated server-side on the RegionServer.

Root cause

The Spark HBase connector JARs (hbase-spark, hbase-spark-protocol-shaded) are built and shipped only in the spark-k8s image. The HBase image has no knowledge of the connector, so RSRpcServices cannot load SparkSQLPushDownFilter when it tries to reconstruct the pushed-down filter.

Impact

This is a performance optimization, not a correctness fix. Spark pushdown is best-effort: when the connector does not push a predicate, Spark keeps it in the plan and applies it in the executors after reading. Results are identical whether the filter runs server-side or Spark-side — there is no scenario where disabling pushdown returns wrong or extra rows.

The flag only gates the server-side value filter (SparkSQLPushDownFilter) for predicates on non-row-key columns. With it disabled:

• More data crosses the wire (RegionServer → executors), since HBase returns all rows in the scanned range and Spark discards non-matching ones — the dominant cost for a selective filter on a large table.
• RegionServers do less useful filtering per byte scanned; executors spend more CPU/memory filtering rows HBase could have dropped at the source.

Two more impactful optimizations are not affected, because they are implemented separately from SparkSQLPushDownFilter:

• Row-key range pushdown — row-key predicates are still turned into scan start/stop bounds (or a point Get), so row-key access is not degraded. This is usually the biggest lever for HBase.
• Column (family/qualifier) projection — only the columns in the catalog/projection are requested.

So the degradation is narrow: it affects queries filtering on non-row-key column values with low selectivity relative to the row-key range scanned, on large tables. For typical row-key-driven access the difference is negligible — which is relevant for prioritizing this issue.

Proposed change

Make the connector filter classes available on the HBase RegionServer classpath, e.g. by adding the following JARs (built in spark-k8s/hbase-connectors/) to the HBase image's classpath (/stackable/hbase/lib/ or via HBASE_CLASSPATH):

• hbase-spark-*.jar (contains SparkSQLPushDownFilter)
• hbase-spark-protocol-shaded-*.jar (contains the generated SparkFilterProtos)

Filters are evaluated on RegionServers, so those are the role that needs the classes.

Open questions / considerations

• Layering: the HBase image would gain a dependency on a Spark-specific connector, which is arguably a layering violation. Alternatives worth weighing: an opt-in overlay/config, a documented "bring your own JAR" volume mount, or an operator-level option rather than baking it into the base image.
• Version compatibility: the connector's protobuf-generated classes must be binary-compatible with the RegionServer's hbase-thirdparty (shaded protobuf). Note the connector is currently built against a different hbase-thirdparty than some HBase versions ship (e.g. the Spark image bundles hbase 2.6.4/thirdparty 4.1.12, HBase 2.6.6 ships 4.1.13). This coupling needs an explicit, maintained mapping.
• Connector/Spark version coupling: the connector is tied to a Spark version. One HBase image may face multiple connector builds; decide which (if any) is bundled.
• Scope: which HBase versions and which roles (RegionServers; likely not masters). Confirm whether Phoenix or other consumers are affected.
• Footprint & security: adding connector JARs increases image size and attack surface for all HBase users, most of whom don't use Spark pushdown.

Current workaround

The hbase-connector kuttl test in spark-k8s-operator disables server-side pushdown via hbase.spark.pushdown.columnfilter=false, so WHERE clauses are evaluated Spark-side over a full scan. This keeps the test green but does not exercise true server-side pushdown. When this issue is resolved, that option should be removed and the test should assert pushdown works with the default (true).

Acceptance criteria

• A Spark job reading from HBase with default pushdown (hbase.spark.pushdown.columnfilter=true) and a WHERE filter succeeds against the SDP HBase image (no ClassNotFoundException on the RegionServer).
• The hbase-connector integration test is updated to exercise pushdown with the default setting and passes.
• The connector↔hbase-thirdparty version compatibility is documented and enforced in the build config.