[ahshahid]

… which otherwise prevents re-use of exchange for in memory cached plans

What changes were proposed in this pull request?

The canonicalization of TableCacheQueryStageExec uses the default, instead of the original InMemoryTableScanExec, as result two TableCacheQueryStageExec which hold canonically same InMemoryTableScanExec , do not match

Why are the changes needed?

Affects performance as it prevents re-use of exchange.

Does this PR introduce any user-facing change?

No

How was this patch tested?

added bug test. will add some more tests.

Was this patch authored or co-authored using generative AI tooling?

No

[ulysses-you]

thank you @ahshahid , this fix looks correct to me, some comments.

[ulysses-you]

can we move the following code from ExchangeQueryStageExec to QueryStageExec ?

val _canonicalized: SparkPlan

override def doCanonicalize(): SparkPlan = _canonicalized

[ahshahid]

done.. this required handling in ResultQueryStageExec too.. I have done the change , but test for ResultQueryStageExec not there.. Not sure if the canonicalized plan would ever be used in that class, as per my understanding , it represents a final evaluated stage.

[ahshahid]

Thank you Xiduo You, for review. Will implement the feedback and update pr in a day.. Regards Asif A democracy of barbarians is worse than A dictatorship

…

On Wed, Aug 26, 2026, 1:15 AM Xiduo You ***@***.***> wrote: ***@***.**** commented on this pull request. thank you @ahshahid <https://github.com/ahshahid> , this fix looks correct to me, some comments. ------------------------------ In sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/TableCacheQueryStageExecSuite.scala <#58003 (comment)>: > + +class TableCacheQueryStageExecSuite extends SharedSparkSession with AdaptiveSparkPlanHelper { + import testImplicits._ + + test("SPARK-58767: re-use of exchange involving cached plans ") { + withTempPaths(3) { paths => + // to be safe use 3 data paths so that the 3 tables are not in any situation canonicalized + // same. + val data = Seq((1, 1), (2, 2), (3, 3), (4, 4), (6, 6), (5, 5)) + paths.zipWithIndex.foreach{case(path, indx) => + val tableNumber = indx + 1 + spark.createDataFrame(data).toDF(s"c1_$tableNumber", s"c2_$tableNumber").write.mode( + SaveMode.Overwrite).parquet(path.getAbsolutePath)} + withTempView("v1") { + withTable("t1", "t2", "t3") { + spark.sql( please wrap with withCache() ------------------------------ In sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala <#58003 (comment)>: > @@ -307,6 +308,8 @@ case class TableCacheQueryStageExec( } } + override def doCanonicalize(): SparkPlan = _canonicalized can we move the following code from ExchangeQueryStageExec to QueryStageExec ? val _canonicalized: SparkPlan override def doCanonicalize(): SparkPlan = _canonicalized ------------------------------ In sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/TableCacheQueryStageExecSuite.scala <#58003 (comment)>: > + ) + + spark.table("t1").where($"c1_1" > 0).createTempView("v1") + spark.catalog.cacheTable("v1") + val al1 = spark.table("v1").as("al1") + val al2 = spark.table("v1").as("al2") + val t2 = spark.table("t2") + val t3 = spark.table("t3") + val j1 = t2.where($"c2_2" > 0).join(t3, t2.col("c1_2") === t3.col("c1_3")) + assert(j1.collect().nonEmpty) + val j2 = j1.join( + broadcast(al1.where($"al1.c1_1" > 0)), $"al1.c1_1" === t2.col("c1_2")) + assert(j2.collect().nonEmpty) + val df = j2 + .join(broadcast(al2.where($"al2.c1_1" > 0)), $"al2.c1_1" === t3.col("c1_3")) + assert(df.collect().nonEmpty) can we use checkAnswer for more accurate result ? — Reply to this email directly, view it on GitHub <#58003?email_source=notifications&email_token=AC6XG2HEG62R4XTDESXQJ5D5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#pullrequestreview-5027924642>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC6XG2CBSG5XWO7VBBUNX6T5L2MAJAVCNFSNUABEKJSXA33TNF2G64TZHMYTOMJWGU3DKOB3JFZXG5LFHM2TCNBYGEZDSOBZHCQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AC6XG2BTKOGU5XEBKYLYTVT5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AC6XG2FGPGHAB7MD76XNDKT5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[ulysses-you]

For ResultQueryStageExec, we should not make canonicalized change , the semantics of ResultQueryStageExec is plan + resultHandler.

[ahshahid]

ResultQueryStageExec extends QueryStageExec. So if the abstract fields of _canonicalize is moved to QueryStageExec, what do you have in mind to override it in ResultQueryStageExec?

override def doCanonicalize(): SparkPlan = {
ResultQueryStageExec(-1, _canonicalized, _canonicalized, resultHandler)
}
?

[ahshahid]

@ulysses-you ..I suppose the overriding doCanonicalize in ResultQueryStageExec as in the PR , should work as right canonicalized code. From what I understand of what you implied of "plan + resultHandler", looks like ResultQueryStageExec , also had an issue, which should be addressed by this PR...
Pls lmk your thoughts.

[ahshahid]

ResultQuerystageExec extends the abstract class where you asked to move the fields... A democracy of barbarians is worse than A dictatorship

…

On Thu, Aug 27, 2026, 10:11 PM Xiduo You ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala <#58003 (comment)>: > @@ -942,7 +943,8 @@ case class AdaptiveSparkPlanExec( optimizeQueryStage(plan, isFinalStage = true), postStageCreationRules(supportsColumnar), "AQE Post Stage Creation") - val resultStage = ResultQueryStageExec(currentStageId, optimizedRootPlan, resultHandler) + val resultStage = ResultQueryStageExec(currentStageId, optimizedRootPlan, plan.canonicalized, For ResultQueryStageExec, we should not make canonicalized change , the semantics of ResultQueryStageExec is plan + resultHandler. — Reply to this email directly, view it on GitHub <#58003?email_source=notifications&email_token=AC6XG2H7TOX3VEPBHYHX4DL5MEIBHA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBUHAYDSNJWG42KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#pullrequestreview-5048095674>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC6XG2GEPJVQAIPKUJ227PT5MEIBHAVCNFSNUABEKJSXA33TNF2G64TZHMYTOMJWGU3DKOB3JFZXG5LFHM2TCNBYGEZDSOBZHCQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AC6XG2FWLBNIATCLO3MYO535MEIBHA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBUHAYDSNJWG42KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AC6XG2C54REIJ4LUD755I5T5MEIBHA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBUHAYDSNJWG42KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>