Skip to content

Add a 'single' ordering to JaggedFeatureProcessor for per-sample attributes - #1461

Merged
jhnwu3 merged 7 commits into
masterfrom
single-ordering
Nov 16, 2026
Merged

jhnwu3 merged 7 commits into
masterfrom
single-ordering

Conversation

@Logiquo

@Logiquo Logiquo commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Based on master, after the rename in #1415, the bridges in #1416 and the label timeline in #1429.

Adds a "single" ordering to JaggedFeatureProcessor for attributes a sample has exactly once, such as a patient's sex, insurance and age. A field can hold them on their own, or as the outermost level above the patient's visits. There are no new types: the input is the usual Records with columns of length 1, and the output keeps the packed layout that models already read. The legacy static fields' bridges, RNN_V2 and the YAIB tasks use it too, and RNN_V2 now reads the per-step label's timeline.

  • Semantics: a "single" level holds exactly one record, the sample's own. It has no order and no time, and needs tokens or values. Only the outermost level can be "single": below it, a record's one child would hold nothing its parent couldn't.
  • Input: Records whose columns have length 1, e.g. {"tokens": [("F", "Medicare")], "values": [[71.0]], "children": [visits]}.
  • Output:
    • The "single" level's tensors are [1, ...] per sample and [B, ...] per batch.
    • collate() still appends the batch offsets, here 0 .. B. So batch_schema() keeps its form, and SequenceEmbeddingModel embeds "single" levels with no change.
    • Below it, a batch is packed exactly as without the "single" level: (L - 1, "offsets") equals that processor's batch offsets.
  • Checks:
    • __init__ rejects "single" below the outermost level, and a "single" level with neither tokens nor values.
    • As the level has no time, resample (which needs "series") and along-time imputation (forward_fill, backward_fill, linear) are rejected too. Along-time imputation now needs "sequence" or "series" instead of anything but "set", which changes nothing for the existing orderings.
    • process() raises if the level's columns don't have length 1.
    • validate() checks for one record per sample: a sample has one record at that level, and a batch's offsets step by 1.
  • Statistics: imputation and normalisation are fitted over every sample's record, as on any level.
  • Docs:
    • The API page gets an example section, "5. Per-Sample Attributes".
    • "single" is added to the glossary, the LevelConfig and output-role tables, and the design rationale.
    • The mapping table gets two rows, and the page gets two Records construction examples.
    • The Records, LevelConfig and JaggedFeatureProcessor docstrings describe it, with a LevelConfig doctest.

Bridges

The bridges from #1416 that made each sample a set of one record now make it a "single" level:

  • TensorProcessor when the tensor's first axis has a fixed length (a varying first axis is still one record per row, a sequence or a set);
  • ImageProcessor, and so DicomProcessor;
  • AudioProcessor.

Only the metadata changes (levels, ordering()). The converted batches are the same, since their offsets were already 0 .. B.

RNN_V2

  • Encoding: a level runs the RNN only if it is a "sequence" or a "series". A "single" level is pooled like a "set", so each sample's one record (its embedding and the summary of its children) is passed on as the field's summary, without a one-step RNN.
  • Per-step labels: the model now reads the label processor's timeline (Require a timeline on the timeseries label processors #1429), instead of requiring a "series" top level on every field.
    • A field on the timeline needs a "series" top level and a causal RNN, as before: each label uses its top-level state at or before the label's time. A field whose top level has no time ("single", "set" or "sequence") is rejected, and the error says to take it off the timeline. bidirectional=True is rejected on it.
    • Every other field holds for the whole sample. It is encoded whole, whatever its levels, and every label record of the sample gets its summary. That covers a lone "single" level (static values, or a bridged legacy tensor), a "single" level above visits, a "set" or "sequence" top level, and a "series", which is no longer cut at label times. Such a field may be bidirectional.
    • A timeline name that isn't an input field of the model is rejected.
  • Docs: the RNN_V2 docstring and its row in docs/api/models.rst give a "single" level as the form of static values, replacing "a series with one record at t = 0". The docstring example's age field is now a "single" level, and the per-step paragraph describes the timeline rules above.

YAIB tasks

The YAIB tasks on MIMIC-III, MIMIC-IV and eICU emit static as a "single" level: {"values": [[age, sex, height, weight]]}, with no time, through _static_record (formerly _static_series). Imputation, normalisation and the missing-value mask are unchanged. static is off the labels' timeline (["dynamic"]), so with RNN_V2 every hour's prediction gets its embedding, without a one-step RNN. The docstrings, the comment in the examples/yaib_*_rnn_v2.py scripts and the ICUAcuityMIMIC4 schema comment say so.

Tests

  • TestSingle (test_sequence_processor_v2.py):
    • Bad configurations are rejected: "single" below the outermost level, no tokens or values, resample, and along-time imputation.
    • schema(), batch_schema() and ordering(), for a "single" level alone and above two levels.
    • process() reads a list of one record and rejects 0 or 2.
    • collate() stacks one record per sample, and the levels below match the same processor without the "single" level.
    • Mean imputation, the missing indicator and z-scores use statistics from every sample.
    • validate() rejects a sample or a batch without exactly one record per sample.
  • TestStorage.test_dataloader_stacks_a_single_level: a "single" field goes through create_sample_dataset and get_dataloader.
  • TestCodeMapping: a one-to-many code mapping is rejected on a "single" level, since that needs a level-0 "set".
  • test_unified_bridge.py: the fixed-shape tensor, image and audio bridges report {0: "single"}. Their converted batches still pass validate(batched=True) in the existing decoder.
  • test_rnn_v2.py:
    • TestStatics.test_a_single_level_is_passed_on_without_an_rnn: the level's encoder is an Identity, and its summary is its records' embedding.
    • TestStatics.test_a_single_level_above_visits: only the visits level runs an RNN. The summary is the "single" record's embedding followed by the visits' summary, which is zeros for a sample without visits.
    • TestStatics.test_every_label_record_sees_the_single_record: with per-step labels, each label record gets its sample's summary.
    • TestStatics.test_a_series_off_the_timeline_is_not_cut_at_label_times: a series off the timeline, with its record after every label, is seen by every label.
    • TestPerStep.test_a_timeline_field_needs_a_series_top_level: a lone "single" level, and one above a "series", are rejected on the timeline.
    • TestPerStep.test_a_field_off_the_timeline_is_encoded_whole: off the timeline, a lone "single", a "single" above a "series" and a "sequence" top level are accepted, and each label record gets its sample's summary.
    • TestPerStep.test_timeline_fields_must_be_inputs, test_no_bidirectional_on_the_timeline and test_bidirectional_off_the_timeline.
    • TestLegacyFields.test_per_step_label_with_a_legacy_tensor and test_per_step_label_on_a_legacy_sequence: rejected on the timeline, and working beside a timed field off it.
  • test_yaib_mimic3.py, test_yaib_mimic4.py, test_yaib_eicu.py: static has no time, processes to (values, mask), and still matches the previous TensorProcessor preprocessing.
  • Runs:
    • Latest commit: test_rnn_v2, test_per_step_labels, test_unified_bridge, test_sequence_processor_v2 and the three test_yaib files: 350 tests pass, and every line the PR adds to rnn.py runs. tools/check_pr_rules.py passes, and pyright reports no new errors on rnn.py, test_rnn_v2 and the test_yaib files.
    • Earlier commits: python -W error::FutureWarning -m pytest on test_unified_bridge, test_rnn_v2, test_sequence_embedding, test_sequence_processor_v2, test_dicom_processor (with pydicom), test_image_processor and test_per_step_labels: 320 tests pass. Every line of the RNN_V2 section of rnn.py runs in test_rnn_v2 and test_unified_bridge.
    • The doctests of jagged_processor, unified_bridge, sequence_embedding and RNN_V2 pass.
    • The "single" examples on the docs page run and print the outputs in their comments.
    • tools/check_pr_rules.py --base upstream/master --head HEAD passes. Pyright reports the same errors on the changed files as on master, and ruff finds nothing new in the added lines.

A 'single' level holds exactly one record, the sample's own, such as a
patient's sex, insurance and age, alone or above their visits. Its Records
columns have length 1, so tasks keep using the Records type; process()
raises on any other length, and validate() checks one record per sample.
Only the outermost level can be single, it needs tokens or values, and it
has no order or time, so resample and the forward_fill, backward_fill and
linear imputations are rejected (that check now requires 'sequence' or
'series' instead of only excluding 'set'). The output layout is unchanged:
[1, ...] per sample, [B, ...] per batch, and collate() still appends the
batch offsets, here 0 .. B, so models reading V2 batches need no change.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Test Coverage

Total: 83.63% (37318 / 44622 lines)

Modified file Coverage Lines
pyhealth/models/rnn.py 93.46% 343 / 367
pyhealth/processors/jagged_processor.py 100.00% 707 / 707
pyhealth/processors/unified_bridge.py 99.20% 371 / 374
pyhealth/tasks/acuity_mimic4.py 97.92% 141 / 144
pyhealth/tasks/yaib_eicu.py 91.98% 642 / 698
pyhealth/tasks/yaib_mimic3.py 92.96% 581 / 625
pyhealth/tasks/yaib_mimic4.py 93.16% 531 / 570

Workflow run · 95376a8

The bridges of fixed-shape TensorProcessor fields, ImageProcessor (and
DicomProcessor) and AudioProcessor now describe each sample's one record
as a 'single' level instead of a 'set' of one record. The batches they
produce are unchanged: their offsets were already 0 .. B.

RNN_V2 runs its RNN only over 'sequence' and 'series' levels; a 'single'
level, like a 'set', is pooled, which passes each sample's one record on.
With per-step labels, a field that is only a 'single' level (static
values, such as a bridged legacy tensor) is accepted, and every label
record of a sample uses its record; a 'single' level above other levels
is rejected, as every step would see all their records. The docs now
give a 'single' level as the form of static values.
RNN_V2 now reads the per-step label's timeline instead of inferring it
from each field's structure:

- A field on the timeline needs a 'series' top level and a causal RNN,
  and each label uses its state at or before the label's time. A field
  without time is rejected there, with an error saying to take it off
  the timeline.
- Every other field holds for the whole sample: it is encoded whole,
  whatever its levels (a lone 'single', a 'single' above other levels,
  a 'set' or 'sequence' top level, or a series, which is no longer cut
  at label times), and every label record gets its summary. Such a
  field may be bidirectional.
- A timeline name that is not an input field of the model is rejected.

The YAIB tests and examples follow the tasks' static field, now a
'single' level without time.
@Logiquo
Logiquo marked this pull request as ready for review November 15, 2026 23:26
@Logiquo

Logiquo commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@Logiquo Logiquo added the status: automatic The PR will go through automatic review process. label Nov 15, 2026
@Logiquo

Logiquo commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review

suggestion: approve
decision: need change

Thank you for this contribution. It adds a "single" ordering to JaggedFeatureProcessor for per-sample attributes. The legacy bridges, RNN_V2 (which now reads the label timeline) and the YAIB static fields use it. The description is detailed, the docs match the code, and the tests cover the new branches.

The change looks ready to merge from my reading of the diff.

Coverage is above the bar. The whole repository is at 83.63%, against 60%. Every changed file is above 80%: rnn.py 93%, jagged_processor.py 100%, unified_bridge.py 99%, and the task files 92–98%.

Why this can't be approved automatically: the change touches shared code. JaggedFeatureProcessor is the base processor that the V2 models, the legacy-to-V2 bridges and the YAIB tasks all build on. It gets a new constructor check, a new validate() rule, and a tighter rule for along-time imputation. The TensorProcessor, ImageProcessor and AudioProcessor bridges now report "single" from ordering() and levels where they reported "set". RNN_V2's encoding rule and its handling of per-step labels also change, so downstream behaviour is different. A maintainer will make the final call.

Optional suggestions:

  • A series field that is left off the label timeline is now encoded whole, so every label sees records that come after its time. A short warning in the RNN_V2 docstring about mis-specified timelines would help users avoid silently leaking future data.

  • Warning: this PR can't be approved automatically because it touches the core part of PyHealth (base classes or shared code). To stay on automatic review, back out that part, or ask a maintainer to review it, then comment /review. If the next review finds this again, the PR switches to status: manual.

Reviewed commit 95376a8

@jhnwu3 jhnwu3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@jhnwu3
jhnwu3 merged commit e7548dc into master Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: automatic The PR will go through automatic review process.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants