Repository navigation
Add a 'single' ordering to JaggedFeatureProcessor for per-sample attributes - #1461
Conversation
A 'single' level holds exactly one record, the sample's own, such as a patient's sex, insurance and age, alone or above their visits. Its Records columns have length 1, so tasks keep using the Records type; process() raises on any other length, and validate() checks one record per sample. Only the outermost level can be single, it needs tokens or values, and it has no order or time, so resample and the forward_fill, backward_fill and linear imputations are rejected (that check now requires 'sequence' or 'series' instead of only excluding 'set'). The output layout is unchanged: [1, ...] per sample, [B, ...] per batch, and collate() still appends the batch offsets, here 0 .. B, so models reading V2 batches need no change.
Test CoverageTotal: 83.63% (37318 / 44622 lines)
|
The bridges of fixed-shape TensorProcessor fields, ImageProcessor (and DicomProcessor) and AudioProcessor now describe each sample's one record as a 'single' level instead of a 'set' of one record. The batches they produce are unchanged: their offsets were already 0 .. B. RNN_V2 runs its RNN only over 'sequence' and 'series' levels; a 'single' level, like a 'set', is pooled, which passes each sample's one record on. With per-step labels, a field that is only a 'single' level (static values, such as a bridged legacy tensor) is accepted, and every label record of a sample uses its record; a 'single' level above other levels is rejected, as every step would see all their records. The docs now give a 'single' level as the form of static values.
RNN_V2 now reads the per-step label's timeline instead of inferring it from each field's structure: - A field on the timeline needs a 'series' top level and a causal RNN, and each label uses its state at or before the label's time. A field without time is rejected there, with an error saying to take it off the timeline. - Every other field holds for the whole sample: it is encoded whole, whatever its levels (a lone 'single', a 'single' above other levels, a 'set' or 'sequence' top level, or a series, which is no longer cut at label times), and every label record gets its summary. Such a field may be bidirectional. - A timeline name that is not an input field of the model is rejected. The YAIB tests and examples follow the tasks' static field, now a 'single' level without time.
|
/review |
|
/review |
There was a problem hiding this comment.
AI Review
suggestion: approve
decision: need change
Thank you for this contribution. It adds a "single" ordering to JaggedFeatureProcessor for per-sample attributes. The legacy bridges, RNN_V2 (which now reads the label timeline) and the YAIB static fields use it. The description is detailed, the docs match the code, and the tests cover the new branches.
The change looks ready to merge from my reading of the diff.
Coverage is above the bar. The whole repository is at 83.63%, against 60%. Every changed file is above 80%: rnn.py 93%, jagged_processor.py 100%, unified_bridge.py 99%, and the task files 92–98%.
Why this can't be approved automatically: the change touches shared code. JaggedFeatureProcessor is the base processor that the V2 models, the legacy-to-V2 bridges and the YAIB tasks all build on. It gets a new constructor check, a new validate() rule, and a tighter rule for along-time imputation. The TensorProcessor, ImageProcessor and AudioProcessor bridges now report "single" from ordering() and levels where they reported "set". RNN_V2's encoding rule and its handling of per-step labels also change, so downstream behaviour is different. A maintainer will make the final call.
Optional suggestions:
-
A series field that is left off the label timeline is now encoded whole, so every label sees records that come after its time. A short warning in the RNN_V2 docstring about mis-specified timelines would help users avoid silently leaking future data.
-
Warning: this PR can't be approved automatically because it touches the core part of PyHealth (base classes or shared code). To stay on automatic review, back out that part, or ask a maintainer to review it, then comment
/review. If the next review finds this again, the PR switches tostatus: manual.
Reviewed commit 95376a8
Based on master, after the rename in #1415, the bridges in #1416 and the label timeline in #1429.
Adds a
"single"ordering toJaggedFeatureProcessorfor attributes a sample has exactly once, such as a patient's sex, insurance and age. A field can hold them on their own, or as the outermost level above the patient's visits. There are no new types: the input is the usualRecordswith columns of length 1, and the output keeps the packed layout that models already read. The legacy static fields' bridges,RNN_V2and the YAIB tasks use it too, andRNN_V2now reads the per-step label'stimeline."single"level holds exactly one record, the sample's own. It has no order and no time, and needstokensorvalues. Only the outermost level can be"single": below it, a record's one child would hold nothing its parent couldn't.Recordswhose columns have length 1, e.g.{"tokens": [("F", "Medicare")], "values": [[71.0]], "children": [visits]}."single"level's tensors are[1, ...]per sample and[B, ...]per batch.collate()still appends the batch offsets, here0 .. B. Sobatch_schema()keeps its form, andSequenceEmbeddingModelembeds"single"levels with no change."single"level:(L - 1, "offsets")equals that processor's batch offsets.__init__rejects"single"below the outermost level, and a"single"level with neithertokensnorvalues.resample(which needs"series") and along-time imputation (forward_fill,backward_fill,linear) are rejected too. Along-time imputation now needs"sequence"or"series"instead of anything but"set", which changes nothing for the existing orderings.process()raises if the level's columns don't have length 1.validate()checks for one record per sample: a sample has one record at that level, and a batch's offsets step by 1."single"is added to the glossary, theLevelConfigand output-role tables, and the design rationale.Recordsconstruction examples.Records,LevelConfigandJaggedFeatureProcessordocstrings describe it, with aLevelConfigdoctest.Bridges
The bridges from #1416 that made each sample a
setof one record now make it a"single"level:TensorProcessorwhen the tensor's first axis has a fixed length (a varying first axis is still one record per row, asequenceor aset);ImageProcessor, and soDicomProcessor;AudioProcessor.Only the metadata changes (
levels,ordering()). The converted batches are the same, since their offsets were already0 .. B.RNN_V2
"sequence"or a"series". A"single"level is pooled like a"set", so each sample's one record (its embedding and the summary of its children) is passed on as the field's summary, without a one-step RNN.timeline(Require a timeline on the timeseries label processors #1429), instead of requiring a"series"top level on every field."series"top level and a causal RNN, as before: each label uses its top-level state at or before the label's time. A field whose top level has no time ("single","set"or"sequence") is rejected, and the error says to take it off the timeline.bidirectional=Trueis rejected on it."single"level (static values, or a bridged legacytensor), a"single"level above visits, a"set"or"sequence"top level, and a"series", which is no longer cut at label times. Such a field may be bidirectional.RNN_V2docstring and its row indocs/api/models.rstgive a"single"level as the form of static values, replacing "a series with one record at t = 0". The docstring example'sagefield is now a"single"level, and the per-step paragraph describes the timeline rules above.YAIB tasks
The YAIB tasks on MIMIC-III, MIMIC-IV and eICU emit
staticas a"single"level:{"values": [[age, sex, height, weight]]}, with no time, through_static_record(formerly_static_series). Imputation, normalisation and the missing-value mask are unchanged.staticis off the labels' timeline (["dynamic"]), so withRNN_V2every hour's prediction gets its embedding, without a one-step RNN. The docstrings, the comment in theexamples/yaib_*_rnn_v2.pyscripts and theICUAcuityMIMIC4schema comment say so.Tests
TestSingle(test_sequence_processor_v2.py):"single"below the outermost level, notokensorvalues,resample, and along-time imputation.schema(),batch_schema()andordering(), for a"single"level alone and above two levels.process()reads a list of one record and rejects 0 or 2.collate()stacks one record per sample, and the levels below match the same processor without the"single"level.validate()rejects a sample or a batch without exactly one record per sample.TestStorage.test_dataloader_stacks_a_single_level: a"single"field goes throughcreate_sample_datasetandget_dataloader.TestCodeMapping: a one-to-many code mapping is rejected on a"single"level, since that needs a level-0"set".test_unified_bridge.py: the fixed-shape tensor, image and audio bridges report{0: "single"}. Their converted batches still passvalidate(batched=True)in the existing decoder.test_rnn_v2.py:TestStatics.test_a_single_level_is_passed_on_without_an_rnn: the level's encoder is anIdentity, and its summary is its records' embedding.TestStatics.test_a_single_level_above_visits: only the visits level runs an RNN. The summary is the"single"record's embedding followed by the visits' summary, which is zeros for a sample without visits.TestStatics.test_every_label_record_sees_the_single_record: with per-step labels, each label record gets its sample's summary.TestStatics.test_a_series_off_the_timeline_is_not_cut_at_label_times: a series off the timeline, with its record after every label, is seen by every label.TestPerStep.test_a_timeline_field_needs_a_series_top_level: a lone"single"level, and one above a"series", are rejected on the timeline.TestPerStep.test_a_field_off_the_timeline_is_encoded_whole: off the timeline, a lone"single", a"single"above a"series"and a"sequence"top level are accepted, and each label record gets its sample's summary.TestPerStep.test_timeline_fields_must_be_inputs,test_no_bidirectional_on_the_timelineandtest_bidirectional_off_the_timeline.TestLegacyFields.test_per_step_label_with_a_legacy_tensorandtest_per_step_label_on_a_legacy_sequence: rejected on the timeline, and working beside a timed field off it.test_yaib_mimic3.py,test_yaib_mimic4.py,test_yaib_eicu.py:statichas notime, processes to(values, mask), and still matches the previousTensorProcessorpreprocessing.rnn.pyruns.tools/check_pr_rules.pypasses, and pyright reports no new errors onrnn.py, test_rnn_v2 and the test_yaib files.python -W error::FutureWarning -m pyteston test_unified_bridge, test_rnn_v2, test_sequence_embedding, test_sequence_processor_v2, test_dicom_processor (with pydicom), test_image_processor and test_per_step_labels: 320 tests pass. Every line of theRNN_V2section ofrnn.pyruns in test_rnn_v2 and test_unified_bridge.jagged_processor,unified_bridge,sequence_embeddingandRNN_V2pass."single"examples on the docs page run and print the outputs in their comments.tools/check_pr_rules.py --base upstream/master --head HEADpasses. Pyright reports the same errors on the changed files as on master, and ruff finds nothing new in the added lines.