Introduce version-specific behavior in Spark expressions

[andygrove]

Is your feature request related to a problem or challenge?

Many Spark expressions have different behavior across Spark versions. This is especially the case when comparing Spark 3.x and 4.x where there are many breaking changes.

I think it is important to start addressing this in the Spark-compatible expressions in DataFusion.

fwiw, the approach we take in Comet is that each expression has a getSupportLevel method that can return Compatible, Incompatible(reason), or Unsupported. These methods are context-aware based on Spark version, Spark configuration (e.g. ANSI mode enabled/disabled), and the specific arguments being passed to the expression. Here is an example:

override def getSupportLevel(expr: Reverse): SupportLevel = {
  if (containsBinary(expr.child.dataType)) {
    Incompatible(Some("reverse on array containing binary is not supported"))
  } else {
    Compatible(None)
  }
}

We also have a shim layer where we can implement different code per Spark version. This was necessary for Comet because there are API changes in Spark between versions and we need to compile per-version. It is simpler for DataFusion because we can just pass the Spark version (or some other flags) into the expression constructor.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[andygrove]

@shehabgamin Would be good to get your thoughts on this

[shehabgamin]

@andygrove This makes sense to me and would be valuable. I also wonder whether we should make the relevant Spark configuration available to all UDFs so they can make the same kind of context-aware compatibility decisions.

More broadly, I like the idea of a getSupportLevel mechanism where each Spark-compatible expression can report whether it is compatible, incompatible with a clear reason, or simply unsupported, based on Spark version, Spark configuration, and the specific arguments involved.

It is also worth thinking about whether compatibility should always be strictly version-locked. If a newer Spark version adds functionality to the same UDF, we might want to allow older versions to benefit from that functionality as well when it is safe to do so. That could even be exposed as a configuration option, letting users choose between strict version fidelity and newer function behavior. For example, a user targeting an older Spark version could still run a Spark SQL query that uses newer behavior for an existing function, rather than being restricted to the exact historical semantics of that older version.