Skip to content

Undefined t-statistic is written as bare NaN, making the metrics registry invalid JSON #2875

Description

@feiiiiii5

Problem

HarmScorerEvaluator._compute_metrics deliberately reports an undefined t-test as NaN (pyrit/score/scorer_evaluation/scorer_evaluator.py:711, and test_compute_harm_metrics_partial_agreement in tests/unit/score/test_scorer_evaluator.py pins np.isnan(metrics.t_statistic)). That is the right in-memory signal.

The serialization boundary does not carry it. ScorerMetrics.to_json (pyrit/score/scorer_evaluation/scorer_metrics.py:87) and both registry writers (scorer_metrics_io.py:330 for the atomic rewrite, :513 for the append) call json.dumps with its default allow_nan=True, so a metrics row whose statistic is undefined is written with bare NaN tokens.

Two consequences, both silent:

  1. The JSONL stops being JSON. Python's own reader takes it back, so nothing looks wrong from inside PyRIT, but RFC 8259 has no such literal: JSON.parse in JavaScript rejects the whole document, as do the .NET and Go standard libraries. pyrit/datasets/scorer_evals/**/*_metrics.jsonl are files people read with other tools and in CI.
  2. Readers that do get through misread it. NaN < 0.05 is False, so "the test was undefined" is indistinguishable from "not significant", and any mean or sum over that column silently becomes NaN.

Reproduction

Measured on 7b533109, a HarmScorerMetrics whose t_statistic and p_value are NaN (the shape _compute_metrics returns for a constant non-zero bias):

to_json():            "t_statistic": NaN, "p_value": NaN
python json.loads():  succeeds
p_value < 0.05:       False
node JSON.parse():    rc=1  (Unexpected token)

grep -c NaN pyrit/datasets/scorer_evals is empty today, so no shipped file is affected yet — this bites the first evaluation that hits the degenerate case and writes the registry.

Expected

null at the serialization boundary, since that is JSON's "no value", with the field types widened to float | None so the absence is visible in the dataclass too, and allow_nan=False on the writes so any future non-finite value fails loudly instead of producing a document other parsers reject. In memory nothing changes: NaN stays what _compute_metrics reports.

One consequence to decide on: doc/code/scoring/4_scorer_metrics.py:184 formats p_value with :.4f, which raises TypeError on None. The page works today only because it prints nan. I would rather not touch that file while #2860 is editing the same notebook pair — a guard there (or here) is a two-line follow-up either way.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions