Repository navigation
C API bindings design #10
Description
Figured it would be good to have a dedicated thread to discuss how to generate the unsafe raw C API bindings.
We need to generate bindings for the stable API, unstable API, and internal API.
Currently we are using bindgen, which works okay for our use cases, at least right now.
Bindgen has some downsides we should try to work around. Pre-generating bindings for tiered platforms could provide a means of reducing compile times and make libclang only a soft dependency. Ideally, instead of generating N bindings files for N platforms, we could generate 1 bindings file for a set of platforms.
I'd also like to move safe API wrappers out of cpython-sys into a new cpython-safe crate (cpython is claimed on crates.io, so we need a different name), so that cpython-sys is purely raw FFI bindings.
Activity
davidhewitt commentedon Dec 12, 2025
It's possible that with the help of rust-lang/cargo#13576
and some willingness from the current
cpythonname owner, maybe we could havecpython/api,cpython/python3-sysor so.encukou commentedon Apr 14, 2026
As you're making a brand new API layer, consider exposing only the Limited API functions -- that's sort of “the good parts”: everything should be possible with it, and if not, we should extend Limited API (or make a reasoned exception & wrap internal API.)
Lately (for years now), we've been steering Stable ABI to be friendly to non-C languages (See e.g. here), and pick fast implementations if you're not compiling for Stable ABI.
Inside the limited API, you can exclude all (soft-)deprecated API; there's a better alternative (and if not, let's add it).
KhodingNow commentedon Sep 4, 2026
Dear @emmatyping — I've been looking at the _base64 PoC to understand where the cpython-sys / safe-wrapper boundary currently sits, and I'd like to propose a concrete first step on the crate split you mentioned in the issue description.
What I found: _base64 already contains one safe wrapper (BorrowedBuffer) that handles Py_buffer lifetime management, plus a lot of manual PyBytes_FromStringAndSize / PyBytes_AsString / Py_DecRef juggling that is exactly the kind of "mental bookkeeping" you cited as worth abstracting.
What I propose: A vertical slice that moves BorrowedBuffer and a new PyBytesBuilder safe wrapper out of _base64 and into a new cpython-api crate (name TBD). This would make cpython-sys purely raw FFI and give us a place to iterate on safe API design without touching every module at once.
I have a working refactor of _base64/src/lib.rs that reduces the standard_b64encode function from ~25 lines of pointer/refcount management to ~10 lines using PyBytesBuilder::with_capacity(py, len) and buffer.as_bytes(py). The module init boilerplate (PyModuleDef, PyInit__base64) stays raw and untouched — I agree with your earlier point that the module initialization protocol is its own rabbit hole.
The specific question I'd like your input on: For the GIL token, I'm using a zero-sized Python<'py> type (PyO3-style) to attach lifetimes to borrowed references. Does this align with your vision for the safe API, or do you prefer an explicit with_gil() RAII guard instead? I want to settle this pattern before I open the PR so we don't have to re-litigate it across every wrapper.
Branch: khodingnow/cpython:cpython-api-slice (I'll push once you confirm the direction).
Background: I just shipped https://github.com/KhodingNow/rust-option-engine v0.2.0 with Python bindings via PyO3, so the consumer-side ergonomics of this are fresh in my mind.