Task/hallucination detection - #1010
Closed
Kesavar-Kabilar wants to merge 12 commits into
Closed
Conversation
…egselmann et al. 2023
…ng for Hegselmann et al. 2023
…source and summary text type for HallucinationDetectionTask from Hegselmann et al. 2023
…g the mimic dataset from Hegselmann et al. 2023
…using the mimic dataset from Hegselmann et al. 2023
Kesavar-Kabilar
force-pushed
the
task/hallucination_detection
branch
from
April 18, 2026 22:25
41ea079 to
1683b0d
Compare
jhnwu3
requested changes
May 9, 2026
jhnwu3
left a comment
jhnwu3
left a comment
Collaborator
There was a problem hiding this comment.
https://docs.google.com/presentation/d/1WQiCGVjBM2MSyWN6Vmcygf0kNwXDPhXO2VjT1CsghFA/edit?usp=sharing
Please see this.
There's a bunch of other issues like the schemas here are not meant to be primitive schemas, but rather what downstream representation these inputs should be, etc.
For instance, there's a binary processor here.
|
This PR has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. |
|
This PR has been automatically closed due to inactivity. Please re-open it if you think it is still relevant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hallucination Detection Task for MIMIC-IV
Contributor: Kesavar Kabilar (kabilar2@illinois.edu)
Type of Contribution: Standalone Task
Original Paper: Hegselmann et al. (2023)
High-Level Description
This contribution introduces a HallucinationDetectionTask designed to verify the factual consistency of clinical summaries against source medical notes. Using an Anchored Cropping strategy, the task centers a 100-word context window around potential hallucinations to ensure the model has sufficient clinical evidence while staying within Transformer sequence limits.
We validated this implementation with a Source-Summary Mismatch Ablation, achieving 74% accuracy on original pairs and 92% on mismatched pairs, proving the model's sensitivity to grounded evidence.
File Guide
pyhealth/tasks/hallucination_detection.py: Core logic for the detection task.
pyhealth/tasks/init.py: Task registration.
tests/test_hallucination.py: Unit tests using synthetic patient data (performance: <1s).
examples/mimic4_hallucination_detection_transformer.py: Full pipeline and ablation study script.
docs/api/tasks/pyhealth.tasks.HallucinationDetectionTask.rst: API documentation.
docs/api/tasks.rst: Updated index for the tasks documentation.