Problem
HarmScorerEvaluator._compute_metrics deliberately reports an undefined t-test as NaN (pyrit/score/scorer_evaluation/scorer_evaluator.py:711, and test_compute_harm_metrics_partial_agreement in tests/unit/score/test_scorer_evaluator.py pins np.isnan(metrics.t_statistic)). That is the right in-memory signal.
The serialization boundary does not carry it. ScorerMetrics.to_json (pyrit/score/scorer_evaluation/scorer_metrics.py:87) and both registry writers (scorer_metrics_io.py:330 for the atomic rewrite, :513 for the append) call json.dumps with its default allow_nan=True, so a metrics row whose statistic is undefined is written with bare NaN tokens.
Two consequences, both silent:
- The JSONL stops being JSON. Python's own reader takes it back, so nothing looks wrong from inside PyRIT, but RFC 8259 has no such literal:
JSON.parse in JavaScript rejects the whole document, as do the .NET and Go standard libraries. pyrit/datasets/scorer_evals/**/*_metrics.jsonl are files people read with other tools and in CI.
- Readers that do get through misread it.
NaN < 0.05 is False, so "the test was undefined" is indistinguishable from "not significant", and any mean or sum over that column silently becomes NaN.
Reproduction
Measured on 7b533109, a HarmScorerMetrics whose t_statistic and p_value are NaN (the shape _compute_metrics returns for a constant non-zero bias):
to_json(): "t_statistic": NaN, "p_value": NaN
python json.loads(): succeeds
p_value < 0.05: False
node JSON.parse(): rc=1 (Unexpected token)
grep -c NaN pyrit/datasets/scorer_evals is empty today, so no shipped file is affected yet — this bites the first evaluation that hits the degenerate case and writes the registry.
Expected
null at the serialization boundary, since that is JSON's "no value", with the field types widened to float | None so the absence is visible in the dataclass too, and allow_nan=False on the writes so any future non-finite value fails loudly instead of producing a document other parsers reject. In memory nothing changes: NaN stays what _compute_metrics reports.
One consequence to decide on: doc/code/scoring/4_scorer_metrics.py:184 formats p_value with :.4f, which raises TypeError on None. The page works today only because it prints nan. I would rather not touch that file while #2860 is editing the same notebook pair — a guard there (or here) is a two-line follow-up either way.
Problem
HarmScorerEvaluator._compute_metricsdeliberately reports an undefined t-test asNaN(pyrit/score/scorer_evaluation/scorer_evaluator.py:711, andtest_compute_harm_metrics_partial_agreementintests/unit/score/test_scorer_evaluator.pypinsnp.isnan(metrics.t_statistic)). That is the right in-memory signal.The serialization boundary does not carry it.
ScorerMetrics.to_json(pyrit/score/scorer_evaluation/scorer_metrics.py:87) and both registry writers (scorer_metrics_io.py:330for the atomic rewrite,:513for the append) calljson.dumpswith its defaultallow_nan=True, so a metrics row whose statistic is undefined is written with bareNaNtokens.Two consequences, both silent:
JSON.parsein JavaScript rejects the whole document, as do the .NET and Go standard libraries.pyrit/datasets/scorer_evals/**/*_metrics.jsonlare files people read with other tools and in CI.NaN < 0.05isFalse, so "the test was undefined" is indistinguishable from "not significant", and any mean or sum over that column silently becomesNaN.Reproduction
Measured on
7b533109, aHarmScorerMetricswhoset_statisticandp_valueareNaN(the shape_compute_metricsreturns for a constant non-zero bias):grep -c NaN pyrit/datasets/scorer_evalsis empty today, so no shipped file is affected yet — this bites the first evaluation that hits the degenerate case and writes the registry.Expected
nullat the serialization boundary, since that is JSON's "no value", with the field types widened tofloat | Noneso the absence is visible in the dataclass too, andallow_nan=Falseon the writes so any future non-finite value fails loudly instead of producing a document other parsers reject. In memory nothing changes:NaNstays what_compute_metricsreports.One consequence to decide on:
doc/code/scoring/4_scorer_metrics.py:184formatsp_valuewith:.4f, which raisesTypeErroronNone. The page works today only because it printsnan. I would rather not touch that file while #2860 is editing the same notebook pair — a guard there (or here) is a two-line follow-up either way.