Skip to content

metrics test-units - #376

Merged
kevssim merged 1 commit into
modelscope:dev-swift-v5from
doctorMcy:test-units-mertics
Nov 12, 2026
Merged

kevssim merged 1 commit into
modelscope:dev-swift-v5from
doctorMcy:test-units-mertics

Conversation

@doctorMcy

Copy link
Copy Markdown

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

This PR adds unit tests for src/twinkle/metric/: 77 new test functions (91 → 168) across six files under tests/metric/:

  • test_grpo_clip_sampler.py (new, 27): GRPO / CISPO / GSPO clip decisions, sampler reconciliation, top-k high-KL records, temperature rescaling, multi-microbatch cursor and input edge cases.
  • test_metrics.py (66 → 96): Accuracy seq strategy, LossMetric sum reduction and per-channel aggregation, TrainMetric, DPOMetric ragged ref_logps, EmbeddingMetric, GRPO old-logps row mismatch, reporting helpers.
  • test_ppo_metric.py (16 → 18): PPOValueMetric explained variance / clip / advantage stats, and PPOMetric parity with GRPOMetric.
  • test_reporting.py (4 → 15): JSONL append and flush, timeouts and non-fatal backend failures, SwanLab prefix.
  • test_throughput.py (5 → 9): variable-length batches and padded width, invalid input rejection, window throughput and memory reduction.
  • test_types.py (new, 3): MetricRecord immutability and stage/status validation.

Two supporting source fixes:

  • GRPOMetric did not advance its cursors when a microbatch was dropped. The dropped microbatch still owns rows in the flattened old / adv / sampler lists, so later microbatches were aligned against the wrong old logps and the global sample index in high-KL records shifted; empty logps also raised TypeError via len(None). Both skip cases now advance cursor / _gsi_cursor by the microbatch's logps row count before continue, and empty logps returns early.
  • TrainMetric now guards the reported speed against a zero-length timing window. Coarse platform clocks (Windows) can return the same timestamp twice, and step delta / 0 raised ZeroDivisionError inside metric calculation; a zero window now reports 0.0, leaving iters and elapsed time unchanged.

PR information

本 PR 为 src/twinkle/metric/ 补充单元测试,新增 77 个测试函数(91 → 168),分布在 tests/metric/ 下 6 个文件:

  • test_grpo_clip_sampler.py(新增 27 个):GRPO / CISPO / GSPO 的 clip 判定、与采样端对账、top-k 高 KL 记录、温度重标定、多 microbatch 游标与输入边界处理。
  • test_metrics.py(66 → 96):Accuracy 的 seq 策略、LossMetric 的 sum 归约与分通道聚合、TrainMetric、DPOMetric 的 ragged ref_logps、EmbeddingMetric、GRPO old-logps 行数不匹配、上报辅助函数。
  • test_ppo_metric.py(16 → 18):PPOValueMetric 的 explained variance / clip / advantage 统计,以及 PPOMetric 与 GRPOMetric 的一致性。
  • test_reporting.py(4 → 15):JSONL 追加写与 flush、超时与后端失败不致命、SwanLab 前缀。
  • test_throughput.py(5 → 9):变长 batch 与 padded width、非法输入报错、窗口吞吐与显存归约。
  • test_types.py(新增 3 个):MetricRecord 不可变与 stage/status 校验。

同时包含两处配套的源码修复:

  • GRPOMetric 在跳过 microbatch 时游标不前进:被丢弃的 microbatch 仍占用 old / adv / sampler 展平列表中的行,此前会让后续 microbatch 对齐到错误的 old logps,并使高 KL 记录中的全局样本序号偏移;空 logps 也会因 len(None) 抛 TypeError。现在两种情况都先按该 microbatch 的 logps 行数推进 cursor / _gsi_cursor 再跳过,空 logps 直接返回。
  • TrainMetric 在计算上报 speed 时对零长度时间窗做保护:Windows 等平台的时钟粒度较粗,连续两次取时间可能得到相同值,step 增量 / 0 会抛 ZeroDivisionError 使指标计算中断;现在时间窗为 0 时 speed 记为 0.0,iters 与耗时不受影响。

@kevssim
kevssim merged commit d5ff1b5 into modelscope:dev-swift-v5 Nov 12, 2026
0 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants