Repository navigation
metrics test-units - #376
Merged
Merged
Conversation
kevssim
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR information
This PR adds unit tests for
src/twinkle/metric/: 77 new test functions (91 → 168) across six files undertests/metric/:test_grpo_clip_sampler.py(new, 27): GRPO / CISPO / GSPO clip decisions, sampler reconciliation, top-k high-KL records, temperature rescaling, multi-microbatch cursor and input edge cases.test_metrics.py(66 → 96): Accuracyseqstrategy, LossMetric sum reduction and per-channel aggregation, TrainMetric, DPOMetric raggedref_logps, EmbeddingMetric, GRPO old-logps row mismatch, reporting helpers.test_ppo_metric.py(16 → 18): PPOValueMetric explained variance / clip / advantage stats, and PPOMetric parity with GRPOMetric.test_reporting.py(4 → 15): JSONL append and flush, timeouts and non-fatal backend failures, SwanLab prefix.test_throughput.py(5 → 9): variable-length batches and padded width, invalid input rejection, window throughput and memory reduction.test_types.py(new, 3): MetricRecord immutability and stage/status validation.Two supporting source fixes:
GRPOMetricdid not advance its cursors when a microbatch was dropped. The dropped microbatch still owns rows in the flattened old / adv / sampler lists, so later microbatches were aligned against the wrong old logps and the global sample index in high-KL records shifted; emptylogpsalso raisedTypeErrorvialen(None). Both skip cases now advancecursor/_gsi_cursorby the microbatch's logps row count beforecontinue, and emptylogpsreturns early.TrainMetricnow guards the reported speed against a zero-length timing window. Coarse platform clocks (Windows) can return the same timestamp twice, andstep delta / 0raisedZeroDivisionErrorinside metric calculation; a zero window now reports0.0, leavingitersand elapsed time unchanged.PR information
本 PR 为
src/twinkle/metric/补充单元测试,新增 77 个测试函数(91 → 168),分布在tests/metric/下 6 个文件:test_grpo_clip_sampler.py(新增 27 个):GRPO / CISPO / GSPO 的 clip 判定、与采样端对账、top-k 高 KL 记录、温度重标定、多 microbatch 游标与输入边界处理。test_metrics.py(66 → 96):Accuracy 的seq策略、LossMetric 的 sum 归约与分通道聚合、TrainMetric、DPOMetric 的 raggedref_logps、EmbeddingMetric、GRPO old-logps 行数不匹配、上报辅助函数。test_ppo_metric.py(16 → 18):PPOValueMetric 的 explained variance / clip / advantage 统计,以及 PPOMetric 与 GRPOMetric 的一致性。test_reporting.py(4 → 15):JSONL 追加写与 flush、超时与后端失败不致命、SwanLab 前缀。test_throughput.py(5 → 9):变长 batch 与 padded width、非法输入报错、窗口吞吐与显存归约。test_types.py(新增 3 个):MetricRecord 不可变与 stage/status 校验。同时包含两处配套的源码修复:
GRPOMetric在跳过 microbatch 时游标不前进:被丢弃的 microbatch 仍占用 old / adv / sampler 展平列表中的行,此前会让后续 microbatch 对齐到错误的 old logps,并使高 KL 记录中的全局样本序号偏移;空logps也会因len(None)抛TypeError。现在两种情况都先按该 microbatch 的 logps 行数推进cursor/_gsi_cursor再跳过,空logps直接返回。TrainMetric在计算上报 speed 时对零长度时间窗做保护:Windows 等平台的时钟粒度较粗,连续两次取时间可能得到相同值,step 增量 / 0会抛ZeroDivisionError使指标计算中断;现在时间窗为 0 时 speed 记为0.0,iters与耗时不受影响。