[INTERNAL_ERROR] The "collect" action failed.

[andygrove]

Describe the bug

awslabs ran into the following error when testing a recent 0.15.0 snapshot release (0.15.0-preview-1 tag). For some reason, there is no more detail provided in the stack trace, making this difficult to debug.

Execution 'q14b-v4.0' failed: [INTERNAL_ERROR] The "collect" action failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.:
org.apache.spark.SparkException$.internalError(SparkException.scala:107)
org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:536)
org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:548)
org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:4321)
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201)
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108)
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
org.apache.spark.sql.Dataset.withAction(Dataset.scala:4321)
org.apache.spark.sql.Dataset.collect(Dataset.scala:3575)

This issue only happens on specific queries in TPC-DS:

q14a-v4.0
q14b-v4.0
q31-v4.0
q47-v4.0
q57-v4.0

The issue does not happen with 0.14.0 release.

Steps to reproduce

No response

Expected behavior

No response

Additional context

/**
   * Converts asserts, null pointer exceptions to internal errors.
   */
  private[sql] def toInternalError(msg: String, e: Throwable): Throwable = e match {
    case e @ (_: java.lang.NullPointerException | _: java.lang.AssertionError) =>
      SparkException.internalError(
        msg + " You hit a bug in Spark or the Spark plugins you use. Please, report this bug " +
          "to the corresponding communities or vendors, and provide the full stack trace.",
        e)
    case e: Throwable =>
      e
  }

[andygrove]

The issue also happens in the 0.14.1-rc1 release.

In the 0.15.0 snapshot release it happens for scan=auto as well as scan=native_iceberg_compat.

[comphead]

Thats frustrating the stacktrace is so short without having a root cause. Is there any steps to reproduce locally?

[andygrove]

I set up spark-sql-perf locally and generated the partitioned 1TB data set and ran q14a-4.0 and cannot reproduce the issue so far

edit: I am testing single node spark standalone mode with data locally, so not the same as awslabs setup of EKS/S3 and multiple executors

[andygrove]

I forgot to mention earlier that setting spark.comet.dppFallback.enabled=false is a workaround for the issue, but obviously results in poor performance because Comet does not implement DPP yet so will scan all partitions.

The issue is almost certainly related to the recent PR that changed DPP fallback behavior: #3879

[andygrove]

I am now looking at https://github.com/awslabs/data-on-eks to see if there are clues there to any differences between the way the benchmark is run by awslabs vs the way I am currently running it