[mypyc] Move non-captured generator state to private frames - #21991
Merged
Merged
Conversation
Persistent generator bindings previously lived in the closure environment even when no nested function could access them. This made private generator state part of a potentially shared object and prevented private-frame optimizations from applying to it. Keep captured bindings in the closure environment, but put noncaptured arguments, locals, and nested-function references on the private generator frame. Apply the same routing to deleted error-overlap locals, which need attribute storage eagerly for their definedness bitmap. Update nested environment links to follow the selected storage object and add coverage for mixed captured and private state.
Generator locals stored as attributes exposed their internal generated names in invalid-deletion diagnostics. Strip the generated attribute prefix when reporting an invalid deletion so the diagnostic names the source variable instead of its internal storage slot.
Generator arguments were processed by add_args_to_env and then again by add_args_to_generator_frame at both generator setup sites. Each pass walked every argument and repeated the generator-environment classification; the environment pass also computed bitmap argument counts that it did not use. Consolidate both paths in add_generator_args, which assigns each argument to either the closure environment or the private generator frame in one pass. Have environment finalization expose its instance without independently adding generator arguments. This preserves the existing storage and lifetime rules while removing duplicate IR-building work.
Source-named slots on generator frames that can be inherited may collide with slots introduced by another frame class. Give private frame slots a class-qualified prefix so each declaring class has its own namespace. Use a distinct prefix for generator-frame and closure-environment storage, and teach generated diagnostics to recover the original source name from either form.
The generator frame split left several redundant routing checks behind. Generator arguments no longer pass through add_args_to_env, variables listed in a function's free-variable set are already known to require environment storage, and only nested function objects can still choose between the closure environment and private generator frame. Rename the routing predicate to describe capture instead of its storage destination, use it for both generator and ordinary-function lifetime decisions, and remove the add_vars_to_env helper closure. Also state the separate-environment invariant directly and undo ordering and type-assertion leftovers from an abandoned completed-generator optimization.
Final generator frame classes cannot be inherited and are also the classes eligible for merging with their environments, so their private source-named attributes cannot collide with inherited or closure attributes. Use the plain generator-frame prefix for final classes while retaining class qualification for non-final frames. Continue recognizing both forms when recovering source names for diagnostics.
p-sawicki
approved these changes
Sep 15, 2026
Comment on lines
+2215
to
2216
| del value # E: "value" cannot be deleted \ | ||
| # N: Using "__deletable__ = ['<attr>']" in the class body enables "del obj.<attr>" |
Collaborator
There was a problem hiding this comment.
i think it'd make sense to keep the TODO comment because the note doesn't apply in this case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some generator-owned persistent state previously lived in the closure environment whenever a generator needed a separate environment class, even when nested functions could not access that state. This was potentially quite inefficient.
Move non-captured arguments, nested-function objects, and a small subset of locals to the private generator frame. Most non-argument locals were already stored there.
This mainly affects generators and coroutines that contain a nested function or a lambda, or are themselves nested. Simple generators whose frame and environment were already merged are unchanged.
Performance impact:
On free-threaded builds, moved fields use non-atomic, borrow-friendly attribute accesses instead of atomic environment accesses, avoiding relatively slow synchronized operations and increfs/decrefs. Captured variables remain in the shared environment and continue to use atomic access.
GIL builds also benefit from borrowing frame reads. Savings in a generator method that accessed
selfnine times and contained a nested helper: