From dedf527269342b830f559f9cdb24828ba75003e3 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Thu, 20 Aug 2026 20:43:14 +0000 Subject: [PATCH] docs(eval-skill): add NVFP4 model-card sampling reference Add references/nvfp4-modelcard-sampling.md: the published temperature, top_p and max generation length for the 2026 NVFP4 checkpoints under huggingface.co/nvidia that disclose them, grouped by model family, and make Step 3 of the evaluation skill cross-check it before writing eval params. Config generation currently rederives sampling params per run from a single model card, which is slow and silently wrong in two ways. First, most NVIDIA NVFP4 cards paste a TensorRT-LLM quickstart containing SamplingParams(temperature=0.8, top_p=0.95) -- byte-identical across Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Phi-4, Qwen2.5-VL and six Qwen3 repos -- which is boilerplate, not an eval setting, yet it is the most prominent temperature= in the card. Second, when a card is silent the skill falls back to a generic 65536/16384 default even though a sibling in the same family publishes an exact value. All 69 NVFP4 checkpoints in the org were read; the table lists the 28 published in 2026 whose cards disclose usable settings, so absence means "read the card" rather than "not yet checked". Rows carry provenance -- 23 "eval" where the card ties the values to its accuracy table, 5 "rec" where it recommends them without that tie -- and the lookup procedure resolves per field so a "rec" row or an unpublished field has defined behavior. Per-task exceptions are captured (GLM-5.2 GPQA 100000 vs 64000; Qwen3.5-397B-V2 tau2 128000; Qwen3.6 SciCode temperature 0.6; Kimi-K3 uncapped for Terminal-Bench). Cross-check rule: exact model listed -> use its row; absent -> use the nearest same-family rows; card conflicts with the family pattern -> card wins, but surface the discrepancy. Generations do break the pattern, so the priors are scoped to what the table actually shows -- DeepSeek is explicitly carved out of the 1.0/0.95 house default because both its rows use top_p=1.0. The file ends with a refresh recipe covering the two extraction traps found while building it: cards that publish the cap only as a footnote under the accuracy table ("*Max OSL for evals can be as high as 64K"), and DeepSeek stating sampling in its usage block rather than a "Benchmarked with" sentence. Neither is reachable from the obvious grep. Not changelog-worthy: agent-skill documentation, no user-facing API or behavior change. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Chenjie Luo --- plugins/modelopt/skills/evaluation/SKILL.md | 9 +- .../references/model-card-research.md | 15 +- .../references/nvfp4-modelcard-sampling.md | 149 ++++++++++++++++++ 3 files changed, 169 insertions(+), 4 deletions(-) create mode 100644 plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index e144c56873f..f6a5b69f8a9 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -274,13 +274,16 @@ nemo_evaluator_config: Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere. +**Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` / `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's "Lookup" section. + #### `max_new_tokens` — mandatory model-card lookup 1. **Fetch the HF model card before writing the value.** Not optional. 2. Scan for any `max_tokens` / `max_new_tokens` / "output length" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment. -3. If the card is genuinely silent after a thorough read, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user. -4. **Forbidden:** writing `max_new_tokens: ` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence. -5. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence. +3. **Consult `references/nvfp4-modelcard-sampling.md` as a reference.** Listed and in agreement → proceed with confidence. Listed and different → **the card wins**; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its `max_num_tokens` column records the card's *headline* cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row. +4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user. +5. **Forbidden:** writing `max_new_tokens: ` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence. +6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence. #### Quantization-aware benchmark defaults diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index d007a6dccd0..80db5cce28b 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -3,6 +3,17 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it carefully, the FULL text, the devil is in the details. Extract ALL relevant configurations: - Sampling params (`temperature`, `top_p`) + - **Only trust a sentence that ties the values to the benchmarks** ("Benchmarked + with…", "…were evaluated with…", "We evaluate the model using…") or a + "Recommended Sampling" row. The `SamplingParams(temperature=0.8, top_p=0.95)` + in a card's TensorRT-LLM/vLLM quickstart snippet is boilerplate copied + verbatim across unrelated models — **never** read eval settings out of it. + - **Then cross-check `nvfp4-modelcard-sampling.md`** — published + `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints + under `huggingface.co/nvidia` that disclose them, grouped by family. + Required for any NVFP4 checkpoint or same-family sibling; it is also the + best source of a default when the card is silent. Pre-2026 releases are out + of scope there — read their cards. - Context length (`deployment.extra_args: "--max-model-len "`) - **Output length (`max_new_tokens`) — mandatory extraction.** Scan the card for any `max_tokens` / `max_new_tokens` / "output length" @@ -11,7 +22,9 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca apply at the top level (no per-task overrides). If the card is genuinely silent on output length, note that explicitly and fall back to the generic default (64K reasoning / 16K non-reasoning) — never write a - config with "card not yet checked" + generic default. See SKILL.md + config with "card not yet checked" + generic default. Check + `nvfp4-modelcard-sampling.md` for the model or its family before falling + back; a same-family published cap beats the generic default. See SKILL.md Step 3 "`max_new_tokens` — pick a single top-level value" for the full rule. - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed) diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md new file mode 100644 index 00000000000..f1f91d80a60 --- /dev/null +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -0,0 +1,149 @@ +# NVFP4 Model-Card Sampling Reference + +Published `temperature` / `top_p` / max generation length for the **2026 NVFP4 +checkpoints under [huggingface.co/nvidia](https://huggingface.co/nvidia/models) +whose cards disclose them** — 25 rows, collected 2026-08-20. All 69 NVFP4 +checkpoints in the org were read; absent are those published before 2026-01-01, +those whose cards disclose nothing usable, and **speculative-decoding variants +(`-DSpark`, `-DFlash`)** — spec decoding is verified against the target and does +not change its output distribution, so those checkpoints share their base +checkpoint's row. A miss here means "read the card", not "not yet checked". + +Use it to reproduce a published NVFP4 number, and as the cross-check when a card +is silent or ambiguous. It does not replace reading the card — see +`model-card-research.md`. + +## Lookup + +**The card is the source of truth; this table is a reference, not a constraint.** +Use it to confirm what you read, to fill a gap when the card is silent, and as a +sanity check when you are unsure — never to override a value the card states. + +1. **Exact row, resolved per field.** `eval` → use it, cite the row. `rec` → use + it, but note in the config comment that it is recommended sampling, not a + stated eval setting; if a same-family `eval` row disagrees, surface both. + `—` → that field is unpublished; resolve **it alone** via step 2. + `max_num_tokens` is the card's *headline* cap — where a note names a higher + per-task cap (GLM-5.2 GPQA `100000`, Qwen3.5-397B-V2 τ²-Telecom `128000`, + Kimi-K3 uncapped for Terminal-Bench) and that task is in your suite, SKILL.md + Step 3's take-the-highest rule governs the single top-level value, not the + column. +2. **No row** (new or unreleased variant, non-NVIDIA baseline, pre-2026) → take + the nearest same-family rows as the expected value. +3. **Card vs. table.** Agree → proceed. Card silent + family consistent → adopt + the family value and cite this file in a line comment; that beats SKILL.md + Step 3's generic 65536 / 16384. **Card disagrees → the card wins**, but + surface it — defaults shift between generations, so a mismatch means re-read, + not auto-correct. +4. **Baseline and candidate share one setting.** Cards report both precisions + measured under the single setting listed; use the NVFP4 row for both. + +**Per-task sampling is precedent, not mandate.** Some notes record a +benchmark-specific `temperature` / `top_p` (Qwen3.6 SciCode `0.6`; Qwen3.6-27B +τ²-Bench Telecom `0.0` / `top_p=1.0`; Kimi-K3 `top_p=1.0` agentic). Engineers do +tune sampling per benchmark, so **follow the card you are working from** and use +these as the cross-check. Where the two disagree, **escalate to the user on a +regime change, not a nudge** — greedy (`temperature ≤ 0.1` or `top_p ≤ 1e-4`) +versus sampled flips the regime and materially moves both score and variance; +`0.95` vs `1.0` does not. NEL accepts per-task `temperature` / `top_p` under +`evaluation.tasks.*.nemo_evaluator_config`; only `max_new_tokens` is barred +(SKILL.md Step 3). + +> **Never take sampling from a card's quickstart snippet.** +> `SamplingParams(temperature=0.8, top_p=0.95)` and `max_tokens=32` are +> boilerplate, repeated verbatim across unrelated models. Only *Benchmarked +> with…* / *evaluated with…* / *We evaluate the model using…* sentences, +> "Recommended Sampling" rows, and footnotes under the accuracy table count. + +`provenance` — **`eval`** (20 rows): card ties the values to its accuracy table, +authoritative. **`rec`** (5 rows): card recommends them for inference without +that tie. `max_num_tokens` is the max generation length, i.e. +`nemo_evaluator_config.config.params.max_new_tokens`. + +| Model card ID | temp | top_p | max_num_tokens | prov | notes | +| --- | --- | --- | --- | --- | --- | +| `nvidia/DeepSeek-V4-Flash-NVFP4` | 1.0 | **1.0** | 384000 | eval | `top_p=1.0`, unlike every other row here | +| `nvidia/Qwen3.6-35B-A3B-NVFP4` | 1.0 | 0.95 | 131072 | eval | SciCode used `temperature=0.6` | +| `nvidia/Qwen3.6-27B-NVFP4` | 1.0 | 0.95 | 81920 | eval | SciCode `0.6`; τ²-Bench Telecom `0.0` / `top_p=1.0` | +| `nvidia/Qwen3.5-397B-A17B-NVFP4-V2` | 0.6 | 0.95 | 64000 | eval | τ²-Bench Telecom used `128000` | +| `nvidia/Qwen3.5-397B-A17B-NVFP4` | 0.6 | 0.95 | 64000 | eval | | +| `nvidia/Qwen3.5-122B-A10B-NVFP4` | 0.6 | 0.95 | 64000 | eval | | +| `nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4` | **0.0** | **1.0e-05** | 16384 | eval | greedy — instruct variant | +| `nvidia/GLM-5.2-NVFP4` | 1.0 | 0.95 | 64000 | eval | GPQA Diamond used `100000` | +| `nvidia/GLM-5.1-NVFP4` | 1.0 | 0.95 | 64000 | eval | benchmarked on `vllm/vllm-openai:v0.19.1` | +| `nvidia/GLM-5-NVFP4` | 1.0 | 0.95 | 131072 | eval | | +| `nvidia/GLM-4.7-NVFP4` | 1.0 | 0.95 | 131072 | eval | | +| `nvidia/Kimi-K3-NVFP4` | 1.0 | 0.95 | 65536 | eval | **uncapped for Terminal-Bench**; card also recommends `top_p=1.0` agentic, `n=1`, `presence_penalty=0`, `frequency_penalty=0` | +| `nvidia/Kimi-K2.7-Code-NVFP4` | 1.0 | 0.95 | 64000 | eval | | +| `nvidia/Kimi-K2.6-NVFP4` | 1.0 | 0.95 | 128000 | eval | | +| `nvidia/MiniMax-M3-NVFP4` | 1.0 | 0.95 | 65536 | eval | baseline is native MXFP8 | +| `nvidia/MiniMax-M2.5-NVFP4` | 1.0 | 0.95 | 64000 | eval | | +| `nvidia/Gemma-4-31B-IT-NVFP4` | 1.0 | 0.95 | 131072 | eval | | +| `nvidia/Gemma-4-26B-A4B-NVFP4` | 1.0 | 0.95 | 131072 | eval | | +| `nvidia/diffusiongemma-26B-A4B-it-NVFP4` | upstream | upstream | `null` (uncapped) | eval | defers to `google/diffusiongemma-26B-A4B-it`; diffusion decoding, serve with `--override-generation-config '{"max_new_tokens": null}'`. Uncapped is deliberate — do **not** substitute a numeric fallback | +| `nvidia/Ising-Calibration-1.5-31B-NVFP4` | 0.2 | — | 8192 zero-shot / 32767 ICL | rec | domain model (Gemma-4-31B derivative) | +| `nvidia/Mistral-Medium-3.5-128B-NVFP4` | 0.7 | 0.95 | — | eval | benchmarked with `reasoning_effort="high"` | +| `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` | 1.0 | 0.95 | — | rec | its spec-decode siblings' cards say *Benchmarked with* these same values; eval recipes live in NeMo Gym, client examples use `max_tokens=16000` | +| `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` | 1.0 | 0.95 | — | rec | card: use across **all** tasks and serving backends | +| `nvidia/NVIDIA-Nemotron-Labs-3-Elastic-30B-A3B-NVFP4` | 1.0 | 1.0 | — | rec | reasoning tasks | +| `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` | 0.6 think / 0.2 instruct | 0.95 think / — | 20480 think / 1024 instruct | rec | think adds `reasoning_budget=16384`, `grace_period=1024`; instruct sets `top_k=1` | + +## Priors (verify against the card) + +- **`1.0 / 0.95` is the house default** — GLM 4.7–5.2, Kimi K2.6–K3, MiniMax + M2.5–M3, Gemma 4, Nemotron 3/3.5, Qwen3.6. Best guess when a recent card is + silent. +- **DeepSeek is carved out of it** — its row uses `top_p=1.0`, not `0.95`. + Never carry the house default onto an unlisted DeepSeek variant. +- **Qwen splits by variant** — thinking `0.6 / 0.95` at Qwen3.5, raised to `1.0` + at Qwen3.6; instruct/coder near-greedy `0 / 1e-5` with `16384`. +- **GLM** — `1.0 / 0.95` throughout; cap fell from 131072 (4.7, 5) to 64000 + (5.1, 5.2). +- **Caps cluster at 64000 / 65536 / 81920 / 128000 / 131072**, 64000 most + common. `16384` appears only with greedy instruct Qwen; DeepSeek-V4-Flash's + `384000` is a long-context outlier. +- **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench + Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench + (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`, so when a card + lists two caps **take the maximum** as the single top-level value and note the + split in a comment. + +## Refreshing + +Built from the HF API, verified to match the website pagination page for page +(918 repos across `p=0..31`, identical NVFP4 sets). + +```bash +curl -s "https://huggingface.co/api/models?author=nvidia&limit=1000" -o all.json +python3 -c " +import json, re +KEEP = re.compile(r'-NVFP4(-V\d+|-QAD)?\$', re.I) # target checkpoints only +for m in json.load(open('all.json')): + if KEEP.search(m['id']) and m.get('createdAt', '') >= '2026-01-01': + print(m['id']) +" > ids.txt + +mkdir -p cards +# 404 = repo ships no card; 401 = gated, fetch with 'hf download README.md' +# (never interpolate the HF token into a curl argument) +while read id; do curl -sfL "https://huggingface.co/$id/raw/main/README.md" \ + -o "cards/${id//\//_}.md"; done < ids.txt + +grep -ihnE "benchmark(ed|ing) (parameters|with)|were evaluated with|we evaluate the model using|evaluation settings" cards/*.md +grep -ihnE "max OSL|for evals?|including benchmarking" cards/*.md +``` + +The second grep is **not optional**: some cards give the cap only as a footnote +under the accuracy table (*"\*Max OSL for evals can be as high as 64K"*), and +DeepSeek states sampling in its `## Input:` usage block — neither is reachable +from the first. + +`KEEP` matches a **target checkpoint's** name shape — `…-NVFP4`, plus the +`-V2`-style revision and `-QAD` (a quantization recipe, so still a target). It +therefore drops, by construction, every repo class that would only pollute the +table: `-DSpark` / `-DFlash` speculative-decoding variants (verified against the +target, so identical accuracy — they duplicate the base row), `-Eagle3` draft +heads, and `-MLPerf-Inference-Closed-*` submission snapshots (which ship no +card). `re.I` matters: DeepSeek spells its revisions `-v2`, not `-V2`. The one +blind spot is a repo named `-FP4-*` whose `hf_quant_config.json` says NVFP4 — +rare, and none currently in scope; check the `fp4` tag if you need certainty.