[GLUTEN-11524][VL][DOC] Update gpu documentation

[marin-ma]

Update gpu documentation regarding dynamic execution, hybrid execution, gpu concurrency and async shuffle read.

Related issue: #11524

[Copilot]

Pull request overview

Updates VeloxGPU.md to expand the GPU documentation around stage-level execution mode selection, hybrid CPU/GPU scheduling concepts, GPU concurrency, and async shuffle read tuning for the Velox backend.

Changes:

• Reworks “Dynamic Execution” to describe AQE-stage execution-mode adjustment.
• Adds new sections for “CPU/GPU Hybrid Execution” and performance tuning (GPU concurrency + async shuffle reader).
• Renumbers downstream sections to account for the new content.

Suppressed comments (2)

docs/get-started/VeloxGPU.md:114

• The properties snippet uses spark.gluten.sql.columnar.hybridExecution.* settings that do not exist in the codebase (they only appear in this doc). This will fail for users who copy/paste the config.

spark.gluten.auto.adjustStageResource.enabled = true
spark.gluten.sql.columnar.hybridExecution.enabled = true

# Step 3 – tell Spark about the GPU resource on each executor
spark.gluten.sql.columnar.hybridExecution.gpuResource.amountPerTask = 0.1 # fractional: 10 concurrent GPU tasks/executor

docs/get-started/VeloxGPU.md:127

• These table rows document spark.gluten.sql.columnar.hybridExecution.* and spark.gluten.sql.columnar.gpu.onlyOffloadJoinStage, but neither key exists anywhere in the codebase (only in this markdown). The table should avoid listing non-existent configuration keys.

| `spark.gluten.sql.columnar.hybridExecution.enabled` | `true` | Enable CPU/GPU hybrid execution. Stages are scheduled to CPU or GPU nodes based on their execution mode. Requires AQE (`spark.sql.adaptive.enabled=true`). |
| `spark.gluten.sql.columnar.hybridExecution.gpuResource.name` | `gpu` | The Spark custom-resource name for GPU. Must match `spark.executor.resource.<name>.*` and `spark.task.resource.<name>.*`. |
| `spark.gluten.sql.columnar.hybridExecution.cpuResource.name` | `cpu` | The Spark custom-resource name for CPU. Must match `spark.executor.resource.<name>.*` and `spark.task.resource.<name>.*` for CPU-stage scheduling to take effect. |
| `spark.gluten.sql.columnar.hybridExecution.gpuResource.amountPerTask` | `0.1` | Fractional GPU resource amount per task. Controls how many GPU tasks can run concurrently on a single executor (e.g. `0.1` → 10 tasks share 1 GPU). |
| `spark.gluten.sql.columnar.gpu.onlyOffloadJoinStage` | `true` | When `true`, only stages that contain a join operator are offloaded to GPU. All other stages execute on CPU. Useful for workloads where only join-heavy stages benefit from GPU acceleration. |

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (6)

docs/get-started/VeloxGPU.md:172

• In the Spark worker properties example, the discoveryScript line is missing an '='. Without it, Spark won't parse the property correctly.

```properties
spark.worker.resource.gpu.amount             = 1
spark.worker.resource.gpu.discoveryScript    /path/to/getGpuResources.sh

**docs/get-started/VeloxGPU.md:142**
* The CPU resource discovery script example echoes JSON without quoting. In bash, unquoted braces/quotes can be subject to brace expansion / word splitting and can produce invalid JSON output for Spark's resource discovery.

#!/usr/bin/env bash
echo {"name": "cpu", "addresses":["0"]}

**docs/get-started/VeloxGPU.md:154**
* The GPU resource discovery script echoes JSON without quoting. This can break the JSON output (and therefore Spark resource discovery). Quote the JSON and interpolate $ADDRS safely.

#!/usr/bin/env bash
ADDRS=$(nvidia-smi --query-gpu=index --format=csv,noheader
| sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/","/g')
echo {"name": "gpu", "addresses":["$ADDRS"]}

**docs/get-started/VeloxGPU.md:166**
* In the Spark worker properties example, the discoveryScript line is missing an '='. Without it, Spark won't parse the property correctly.

This issue also appears on line 169 of the same file.

spark.worker.resource.cpu.amount = 1
spark.worker.resource.cpu.discoveryScript /path/to/getCpuResources.sh

docs/get-started/VeloxGPU.md:205

• The async shuffle reader example sets gpuAsyncShuffleReader.maxPrefetchBytes, but the actual config key (and the one used elsewhere in this doc) is spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.maxPrefetchBytes. As written, the example won't have any effect.

spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.enabled = true
spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.threadPoolSize = 8
gpuAsyncShuffleReader.maxPrefetchBytes = 2GB

docs/get-started/VeloxGPU.md:274

• The link text says "Gluten GPU Issue #9098" but the URL points to issue #8851. Either update the URL to 9098 or adjust the displayed issue number so they match.

2. [Gluten GPU Issue #9098](https://github.com/apache/gluten/issues/8851) - Development tracker.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (6)

docs/get-started/VeloxGPU.md:76

• §7 describes dynamic execution via AQE, but it never explicitly states that AQE must be enabled; without spark.sql.adaptive.enabled=true, AdjustStageExecutionMode won’t run (it returns the plan unchanged outside adaptive context). Add a short prerequisite note here so users don’t misconfigure GPU offload.

Gluten uses Spark's Adaptive Query Execution (AQE) framework to evaluate each stage
independently at runtime and select the appropriate execution mode (CPU or GPU).

docs/get-started/VeloxGPU.md:99

• Hybrid execution is implemented by applying per-stage ResourceProfiles (via ApplyResourceProfileExec). In practice this usually requires executor dynamic allocation and a cluster manager that supports multiple resource profiles (the existing stage-level resource adjustment doc calls out YARN/Kubernetes + dynamic allocation). Consider adding that prerequisite here to avoid users expecting it to work in unsupported modes.

With hybrid execution enabled, GPU stages identified in §7 are assigned a dedicated GPU
resource profile via `GlutenAutoAdjustStageResourceProfile`. Spark then schedules those
tasks only on executors that advertise a GPU resource. Scan stages and other non-GPU stages
continue to run on regular CPU executors.

docs/get-started/VeloxGPU.md:192

• In the “Recommended Spark application settings” block, spark.worker.resource.cpu.discoveryScript is a worker-side setting (configured on the standalone worker via --properties-file / worker conf), not an application-side setting like spark.executor.resource.*. As written, this mixes scopes and can mislead users into passing worker configs to spark-submit.

# Register CPU resource to Spark's default resource profile
spark.executor.resource.cpu.amount = 1
spark.worker.resource.cpu.discoveryScript = /path/to/getCpuResources.sh

docs/get-started/VeloxGPU.md:143

• The CPU resource discovery script can be simplified by emitting single-quoted JSON, avoiding the escaped quotes and making it easier to copy/paste safely.

echo {\"name\": \"cpu\", \"addresses\":[\"0\"]}

docs/get-started/VeloxGPU.md:155

• In the GPU discovery script, prefer $(...) over legacy backticks and use printf with proper quoting; this avoids word-splitting surprises and is easier to read/copy.

ADDRS=`nvidia-smi --query-gpu=index --format=csv,noheader | sed -e ':a' -e 'N' -e'$!ba' -e 's/\n/","/g'`
echo {\"name\": \"gpu\", \"addresses\":[\"$ADDRS\"]}

docs/get-started/VeloxGPU.md:203

• Minor formatting: add a space before the parenthetical in the comment so it reads cleanly ("read (recommended …)") in the copied configuration snippet.

# Enable async shuffle read(recommended for all GPU execution)