[None][fix] Clamp very small non-zero temperature in SamplingParams

[CodersAcademy006]

Description

Fixes #15715.

SamplingParams only validated temperature >= 0, so a positive but tiny value like 1e-12 passed validation unchanged and was forwarded to the sampling backend. There, logits / temperature overflows to inf/nan in float16/bfloat16 (a typical logit ~10 divided by 1e-12 is ~1e13, far past fp16's ~65504 max), producing undefined sampling behavior.

This clamps 0 < temperature < MIN_SAMPLING_TEMPERATURE (1e-2) up to that floor, mirroring the guard vLLM uses. Behavior for the other cases is unchanged:

input	result
temperature = 0.0	kept 0.0 (greedy)
0 < temperature < 1e-2	clamped to 1e-2 (with a warning)
temperature >= 1e-2	unchanged
temperature < 0	still rejected

Test Coverage

Adds tests/unittest/llmapi/test_sampling_params_temperature.py covering the clamp, the greedy (0.0) passthrough, the boundary/normal passthrough, and the negative-rejection path.

PR Checklist

• PR title follows [JIRA/NVBugs/GitHub issue/None][type] Summary format
• Test cases added
• Commit is signed off (DCO)

cc @chfeng-cs — thanks for the detailed report and repro.

Summary by CodeRabbit

• Bug Fixes

  • Very small positive temperature values are now automatically raised to a safe minimum, improving sampling stability.
  • Zero temperature remains unchanged for greedy sampling.
  • Negative temperature values continue to be rejected.

• Tests

  • Added coverage for minimum, zero, normal, and invalid temperature values.

[coderabbitai]

📝 Walkthrough

Walkthrough

SamplingParams now clamps non-zero temperatures below 1e-2, preserves zero and valid temperatures, and retains rejection of negative values. Unit tests cover the new clamp and boundary cases.

Changes

Sampling temperature validation

Layer / File(s)	Summary
Temperature clamp and validation coverage tensorrt_llm/sampling_params.py, tests/unittest/llmapi/test_sampling_params_temperature.py	Adds MIN_SAMPLING_TEMPERATURE, clamps tiny positive values during validation with a warning, and tests zero, threshold, normal, tiny positive, and negative temperatures.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and accurately summarizes the main change: clamping tiny non-zero sampling temperatures.
Description check	✅ Passed	The description follows the template and includes the issue, solution, test coverage, and checklist items.
Linked Issues check	✅ Passed	The code and tests address #15715 by clamping tiny positive temperatures, preserving zero and normal values, and rejecting negatives.
Out of Scope Changes check	✅ Passed	The changes are limited to the requested validation fix and corresponding unit tests, with no obvious unrelated additions.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

tests/unittest/llmapi/test_sampling_params_temperature.py (2)

7-11: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Verify the warning side effect.

This test confirms clamping but not the required warning emission. Add an assertion using the repository’s logger-capture mechanism that the warning is emitted before/while clamping.

As per path instructions, test coverage should explicitly validate the requested behavior.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/llmapi/test_sampling_params_temperature.py` around lines 7 -
11, Add logger-capture assertions to test_tiny_temperature_is_clamped so it
verifies that clamping a temperature below MIN_SAMPLING_TEMPERATURE emits the
required warning. Keep the existing clamped-value assertion and use the
repository’s established mechanism to inspect the warning message and severity.

Source: Path instructions

---

14-15: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert greedy decoding, not only zero preservation.

The test name promises greedy behavior, but it only checks that temperature remains 0.0. Also assert the resulting SamplingParams instance is recognized as greedy, or exercise the public path that consumes that state.

As per path instructions, test coverage should explicitly validate the requested behavior.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/llmapi/test_sampling_params_temperature.py` around lines 14 -
15, Update test_zero_temperature_stays_greedy to assert that the resulting
SamplingParams instance is recognized as greedy, in addition to preserving the
temperature value; use the class’s existing public greedy-state indicator or
consumer path.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/llmapi/test_sampling_params_temperature.py`:
- Around line 7-11: Add logger-capture assertions to
test_tiny_temperature_is_clamped so it verifies that clamping a temperature
below MIN_SAMPLING_TEMPERATURE emits the required warning. Keep the existing
clamped-value assertion and use the repository’s established mechanism to
inspect the warning message and severity.
- Around line 14-15: Update test_zero_temperature_stays_greedy to assert that
the resulting SamplingParams instance is recognized as greedy, in addition to
preserving the temperature value; use the class’s existing public greedy-state
indicator or consumer path.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c9743aff-1472-41d1-b275-e296c385ee74

📥 Commits

Reviewing files that changed from the base of the PR and between 5e77882 and 6c08736.

📒 Files selected for processing (2)

• tensorrt_llm/sampling_params.py
• tests/unittest/llmapi/test_sampling_params_temperature.py

[CodersAcademy006]

Noting there are two earlier PRs for this same issue — #15716 (from the reporter @chfeng-cs) and #16222 (@lonexreb) — so maintainers can pick whichever they prefer. This one aims to be the most complete of the three:

• clamps 0 < temperature < 1e-2 up to the floor (mirrors vLLM), leaving temperature = 0 greedy;
• additionally rejects non-finite temperature (inf/nan) up front — the old >= 0 check let those through (nan < 0 is False, inf is unclamped), and they also produce undefined sampling;
• documents the behavior on the temperature docstring;
• uses logger.debug (not warning) to avoid per-request log spam on the serving hot path;
• adds a parametrized test matrix covering unset / greedy / tiny / just-below-floor / boundary / normal, plus negative and non-finite rejection.

Happy to consolidate into one of the existing PRs instead if that's cleaner for review.