-
Notifications
You must be signed in to change notification settings - Fork 7.1k
[Data] Progress Reporting Refactor 2 #59629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Data] Progress Reporting Refactor 2 #59629
Conversation
Signed-off-by: kyuds <[email protected]>
Signed-off-by: kyuds <[email protected]>
Signed-off-by: kyuds <[email protected]>
Signed-off-by: Daniel Shin <[email protected]>
Signed-off-by: Daniel Shin <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request refactors the progress bar system in Ray Data by introducing abstract base classes, BaseProgressBar and BaseExecutionProgressManager, to standardize progress reporting. Common progress bar logic, including block_until_complete and fetch_until_complete methods, along with utility functions like _extract_num_rows and truncate_operator_name, are moved into new dedicated _internal/progress modules. Existing progress bar implementations (ProgressBar, RichExecutionProgressManager, TqdmExecutionProgressManager) are updated to inherit from these new base classes, and type hints across various files are adjusted to reflect these changes. A review comment identified a bug in BaseProgressBar.fetch_until_complete where cancellation could lead to a KeyError due to an incomplete ref_to_result dictionary, suggesting that a RuntimeError should be raised instead of breaking the loop to handle cancellation explicitly.
| with _canceled_threads_lock: | ||
| if t in _canceled_threads: | ||
| break |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The break statement here will cause a KeyError on line 87 if the operation is cancelled, because ref_to_result will not contain all the refs. This is a bug. To handle cancellation correctly, you should raise an exception. This will make the cancellation explicit to the caller and prevent the unhandled exception.
| with _canceled_threads_lock: | |
| if t in _canceled_threads: | |
| break | |
| with _canceled_threads_lock: | |
| if t in _canceled_threads: | |
| raise RuntimeError("The fetch operation was cancelled.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is as per original implementation of progress bars...
Signed-off-by: Daniel Shin <[email protected]>
bveeramani
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice
Description
Follow up to #59350
Related issues
N/A
Additional information
N/A