Use PyModExport and PyABIInfo APIs in pymodule implementation #5753

ngoldbaum · 2026-01-26T21:48:54Z

The first two commits fix some errors from #5746 that were missed in review.

The bulk of the changes are in the third commit, which updates the pymodule implementation to use the new PEP 793 APIs.

After that there are some commits for issues I encountered in CI.

Implementation notes

My original goal was to avoid defining PyModuleDef objects directly on the 3.15 and newer. It turns out the inittab tests need the legacy init hook, so I still need to unconditionally define a PyModuleDef. When we add support for a opaque PyObject ABI, we'll need to disable append_to_inittab on those builds.

I also had to work around a limitation of using both the PyModuleDef and slots, see this discourse post for more context. I'm asking in the discourse post whether it makes sense to relax the check in CPython that I'm working around by defining both SLOTS and SLOTS_MINIMAL in the pymodule macro implementation.

With the new APIs in 3.15, I'm then able to set up modules and submodules without any reference to a module object.

I replaced the const generic parameter on PyModuleSlots and PyModuleSlotsBuilder because we statically know how many slots PyO3 knows how to define, so it's not necessary for users of the API to keep track. See the logic defining the MAX_SLOTS variable and how it's used in the implementation.

ngoldbaum · 2026-01-26T21:50:55Z

src/impl_/pymodule.rs

+        #[cfg(not(Py_3_15))]
+        {
+            unsafe { *self.ffi_def.get() }.m_slots
+        }


I thought it was easier to read if I repeat myself a bit in the Py_3_15 and not(Py_3_15) blocks. Let me know if anyone would prefer to see me collapse this a bit.

src/impl_/pymodule.rs

guide/src/python-from-rust/calling-existing-code.md

src/impl_/pymodule.rs

codspeed-hq · 2026-01-27T17:06:32Z

Merging this PR will not alter performance

✅ 99 untouched benchmarks
⏩ 1 skipped benchmark¹

_{Comparing ngoldbaum:moduledef-hide-details (9b62a4e) with main (12d57fe)}

1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

ngoldbaum · 2026-01-27T17:29:46Z

Tests are passing now but I don't understand why the benchmark is failing or why the code coverage test is failing. Are the 3.15 test runners not uploading coverage reports, perhaps?

I'm going to add a changelog entry and mark this ready for review.

ngoldbaum · 2026-01-27T18:11:06Z

@bschoenmaeckers do you happen to have any context about why my changes to type initialization are impacting the benchmark you added last week?

bschoenmaeckers · 2026-01-27T18:16:30Z

@bschoenmaeckers do you happen to have any context about why my changes to type initialization are impacting the benchmark you added last week?

It has been jumping up and down since rust 1.93. I'm not sure what is going on. So it it safe to ignore for now

ngoldbaum · 2026-01-30T15:23:39Z

src/impl_/pymodule.rs

+            .with_doc(DOC)
+            .build();
+
+        static SLOTS_MINIMAL: PyModuleSlots = PyModuleSlotsBuilder::new()


Petr opened python/cpython#144340, which will hopefully allow us to avoid this dance in 3.15a6, due out the week after next. I'll try to make sure that gets merged.

I guess that would remove the need for SLOTS_MINIMAL in the macros as well (and makes my comment about PEP 820 irrelevant)?

Yup, see this discourse post and replies for some background on this. I was also unhappy with the ugliness of SLOTS_MINIMAL and Petr's PR lets me delete it.

The CPython PR is in so we can wait to merge this PR until the next alpha is out and github actions picks it up. Should be tomorrow or the day after. I’ll also delete the SLOTS_MINIMAL hack.

davidhewitt

Thanks, this looks great to me! Just a few small suggestions.

davidhewitt · 2026-02-06T13:33:28Z

pyo3-macros-backend/src/module.rs

        });
+        result.extend(quote! {


I think you can do it in a single quote! invocation.

Suggested change

});

result.extend(quote! {

davidhewitt · 2026-02-06T13:34:43Z

pyo3-macros-backend/src/module.rs

+            // The full slots, used for the PyModExport initializaiton
+            static SLOTS: impl_::PyModuleSlots = impl_::PyModuleSlotsBuilder::new()
                .with_mod_exec(__pyo3_module_exec)
                .with_gil_used(#gil_used)
+                .with_name(__PYO3_NAME)
+                .with_doc(#doc)
                .build();


This already makes me want slots reuse from PEP 820 😂

src/impl_/pymodule.rs

davidhewitt · 2026-02-06T13:40:22Z

src/impl_/pymodule.rs

        assert!(
-            self.len < N,
+            self.len < MAX_SLOTS,
            "N must be greater than the number of slots pushed"
        );


the assert failed message probably wants to mention MAX_SLOTS instead of N. I also wonder if we can / should move this check or a variant of it to self.push()?

davidhewitt · 2026-02-06T13:42:10Z

src/impl_/pymodule.rs

+            .with_doc(DOC)
+            .build();
+
+        static SLOTS_MINIMAL: PyModuleSlots = PyModuleSlotsBuilder::new()


I guess that would remove the need for SLOTS_MINIMAL in the macros as well (and makes my comment about PEP 820 irrelevant)?

davidhewitt · 2026-02-06T13:43:18Z

src/impl_/pymodule.rs

With other test removed.

Suggested change

fn test_module_slots_builder_overflow() {

src/impl_/pymodule.rs

davidhewitt · 2026-02-06T13:45:15Z

src/impl_/pymodule.rs

+        PyModuleSlotsBuilder::new()
+            .with_mod_exec(module_exec)
+            .with_mod_exec(module_exec)
+            .with_mod_exec(module_exec)
+            .with_mod_exec(module_exec)
            .with_mod_exec(module_exec)
            .with_mod_exec(module_exec)
            .build();


To avoid needing to maintain in the future this probably works:

Suggested change

PyModuleSlotsBuilder::new()

.with_mod_exec(module_exec)

.with_mod_exec(module_exec)

.with_mod_exec(module_exec)

.with_mod_exec(module_exec)

.with_mod_exec(module_exec)

.with_mod_exec(module_exec)

.build();

let mut builder = PyModuleSlotsBuilder::new();

for _ in 0..MAX_SLOTS + 1 {

builder = builder.with_mod_exec(module_exec);

}

builder.build();

ngoldbaum · 2026-02-06T21:40:44Z

I think I addressed all the comments. I also noticed that we weren't actually using the PyABIInfo slot in the pymodule macro and have fixed that in 3bd805a.

As a consequence, we will start getting nicer errors if someone tries to import an extension built using the wrong ABI. Here's what I see when I run a Python test using 3.15t then switch to 3.15 using with pyenv and re-run them without clearing the cargo build cache:

ImportError while importing test module '/Users/goldbaum/Documents/pyo3/pytests/tests/test_enums.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.15-dev/lib/python3.15/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/test_enums.py:2: in <module>
    from pyo3_pytests import enums
.nox/test/lib/python3.15/site-packages/pyo3_pytests/__init__.py:1: in <module>
    from .pyo3_pytests import *
E   ImportError: pyo3_pytests.pyo3_pytests: only compatible with free-threaded CPython

A bit nicer than a segfault!

ngoldbaum added CI-build-full free-threading labels Jan 26, 2026

ngoldbaum commented Jan 26, 2026

View reviewed changes

ngoldbaum mentioned this pull request Jan 26, 2026

Implement support for PEP 793 #5644

Open

ngoldbaum commented Jan 26, 2026

View reviewed changes

guide/src/python-from-rust/calling-existing-code.md Outdated Show resolved Hide resolved

src/impl_/pymodule.rs Outdated Show resolved Hide resolved

ngoldbaum commented Jan 26, 2026

View reviewed changes

src/impl_/pymodule.rs Outdated Show resolved Hide resolved

ngoldbaum added the CI-no-fail-fast If one job fails, allow the rest to keep testing label Jan 26, 2026

ngoldbaum marked this pull request as ready for review January 27, 2026 17:31

ngoldbaum commented Jan 30, 2026

View reviewed changes

ngoldbaum mentioned this pull request Jan 30, 2026

Fix missing return types in PyModExport API wrappers #5764

Merged

davidhewitt reviewed Feb 6, 2026

View reviewed changes

ngoldbaum mentioned this pull request Feb 6, 2026

gh-140550: allow slots that repeat information from PyModuleDef python/cpython#144340

Merged

ngoldbaum added 13 commits February 6, 2026 13:32

Use PyModExport and PyABIInfo APIs in pymodule implementation

3975c85

Add PyModExport function

7b032a2

DNM: temporarily disable append_to_inittab doctest

a7375d1

fix issues seen on older pythons in CI

6816e2f

fix incorrect ModuleDef setup on 3.15

c289ea5

Expose both the PyInit and PyModExport initialization hooks

2392337

fix clippy

957d8d3

add changelog entry

add822c

avoid unnecessary extra quote! use

b9138e7

rename MAX_SLOTS as MAX_SLOTS_WITH_TRAILING_NULL

29e1791

Move MAX_SLOTS check to push

9e3d495

Apply David's suggestions for tests

35fd975

Always include abi info slot in PyModule initialization

3bd805a

ngoldbaum force-pushed the moduledef-hide-details branch from 03d6398 to 3bd805a Compare February 6, 2026 21:32

ngoldbaum added 2 commits February 6, 2026 14:43

only unwind panics if panic = "unwind"

b8d2191

appease clippy

9b62a4e

ngoldbaum mentioned this pull request Feb 8, 2026

Explore support for the opaque PyObject ABI #5786

Open

Use PyModExport and PyABIInfo APIs in pymodule implementation #5753

Are you sure you want to change the base?

Use PyModExport and PyABIInfo APIs in pymodule implementation #5753

Conversation

ngoldbaum commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation notes

Uh oh!

ngoldbaum Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed-hq bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

ngoldbaum commented Jan 27, 2026

Uh oh!

ngoldbaum commented Jan 27, 2026

Uh oh!

bschoenmaeckers commented Jan 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ngoldbaum commented Jan 26, 2026 •

edited

Loading

ngoldbaum Jan 26, 2026 •

edited

Loading

codspeed-hq bot commented Jan 27, 2026 •

edited

Loading

ngoldbaum commented Feb 6, 2026 •

edited

Loading