Process-Bound Python Environments

Since version 2.2.0

Philosophy

In Erlang, processes are the fundamental unit of isolation. Each process has its own heap, mailbox, and lifecycle. When a process crashes, it takes its state with it and can be restarted clean by a supervisor.

erlang_python extends this philosophy to Python: each Erlang process gets its own isolated Python environment. Variables, imports, and objects defined in one process are invisible to others, even when using the same Python context.

This design enables:

Clean restarts: Resetting Python state = terminating the Erlang process
Fault isolation: A corrupted Python state crashes only its owning process
Supervision: Standard OTP supervisors can manage Python-backed actors
Actor model: Build stateful Python services that behave like gen_servers

How It Works

When you call py:exec/eval/call, the library:

Looks up a process-local environment keyed by {ContextPid, InterpreterId}
Creates one if it doesn't exist (a Python dict inside the interpreter)
Uses that dict as the namespace for execution
Cleans up automatically when the Erlang process exits

%% Process A
spawn(fun() ->
    Ctx = py:context(1),
    ok = py:exec(Ctx, <<"state = 'hello'">>),
    {ok, <<"hello">>} = py:eval(Ctx, <<"state">>)
end).

%% Process B - same context, isolated state
spawn(fun() ->
    Ctx = py:context(1),
    %% state is undefined here - different process
    {error, _} = py:eval(Ctx, <<"state">>)
end).

OWN_GIL Mode

OWN_GIL contexts (Python 3.12+) provide true parallel execution with dedicated pthreads. Process-bound environments work with OWN_GIL, allowing multiple Erlang processes to share a single OWN_GIL context while maintaining isolated Python namespaces.

Explicit Environment Creation

For OWN_GIL contexts, you can explicitly create and manage environments:

%% Create an OWN_GIL context
{ok, Ctx} = py_context:start_link(1, owngil),

%% Create a process-local environment
{ok, Env} = py_context:create_local_env(Ctx),

%% Get the NIF reference for low-level operations
CtxRef = py_context:get_nif_ref(Ctx),

%% Execute code in the isolated environment
ok = py_nif:context_exec(CtxRef, <<"
class MyService:
    def __init__(self):
        self.counter = 0
    def increment(self):
        self.counter += 1
        return self.counter

service = MyService()
">>, Env),

%% Call functions in the environment
{ok, 1} = py_nif:context_eval(CtxRef, <<"service.increment()">>, #{}, Env),
{ok, 2} = py_nif:context_eval(CtxRef, <<"service.increment()">>, #{}, Env).

Sharing Context, Isolating State

Multiple Erlang processes can share an OWN_GIL context while maintaining isolated namespaces:

%% Shared OWN_GIL context
{ok, Ctx} = py_context:start_link(1, owngil),
CtxRef = py_context:get_nif_ref(Ctx),

%% Process A - its own namespace
spawn(fun() ->
    {ok, EnvA} = py_context:create_local_env(Ctx),
    ok = py_nif:context_exec(CtxRef, <<"x = 'from A'">>, EnvA),
    {ok, <<"from A">>} = py_nif:context_eval(CtxRef, <<"x">>, #{}, EnvA)
end),

%% Process B - separate namespace, same context
spawn(fun() ->
    {ok, EnvB} = py_context:create_local_env(Ctx),
    ok = py_nif:context_exec(CtxRef, <<"x = 'from B'">>, EnvB),
    {ok, <<"from B">>} = py_nif:context_eval(CtxRef, <<"x">>, #{}, EnvB)
end).
%% Both execute in parallel on the same OWN_GIL thread, but with isolated state

When to Use Explicit vs Implicit Environments

Approach	API	Use Case
Implicit	`py:exec/eval/call`	Simple cases, automatic management
Explicit	`create_local_env` + `py_nif:context_*`	OWN_GIL, fine-grained control, multiple envs per process

Use implicit (py:exec) when:

Using worker or subinterp modes
One environment per process is sufficient
You want automatic lifecycle management

Use explicit (create_local_env) when:

Using OWN_GIL mode for parallel execution
Need multiple environments in a single process
Want to pass environments between processes
Need direct NIF-level control

Event Loop Environments

The event loop API also supports per-process namespaces. Each Erlang process gets an isolated namespace within the event loop, allowing you to define functions and state that persist across async task calls.

Defining Functions for Async Tasks

%% Get the event loop reference
{ok, Loop} = py_event_loop:get_loop(),
LoopRef = py_event_loop:get_nif_ref(Loop),

%% Define a function in this process's namespace
ok = py_nif:event_loop_exec(LoopRef, <<"
import asyncio

async def my_async_function(x):
    await asyncio.sleep(0.1)
    return x * 2

counter = 0

async def increment_and_get():
    global counter
    counter += 1
    return counter
">>),

%% Call the function via create_task - uses __main__ module
{ok, Ref} = py_event_loop:create_task(Loop, '__main__', my_async_function, [21]),
{ok, 42} = py_event_loop:await(Ref),

%% State persists across calls
{ok, Ref1} = py_event_loop:create_task(Loop, '__main__', increment_and_get, []),
{ok, 1} = py_event_loop:await(Ref1),
{ok, Ref2} = py_event_loop:create_task(Loop, '__main__', increment_and_get, []),
{ok, 2} = py_event_loop:await(Ref2).

Evaluating Expressions

%% Evaluate expressions in the process's namespace
{ok, 42} = py_nif:event_loop_eval(LoopRef, <<"21 * 2">>),

%% Access variables defined via exec
ok = py_nif:event_loop_exec(LoopRef, <<"result = 'computed'">>),
{ok, <<"computed">>} = py_nif:event_loop_eval(LoopRef, <<"result">>).

Process Isolation

Different Erlang processes have isolated event loop namespaces:

{ok, Loop} = py_event_loop:get_loop(),
LoopRef = py_event_loop:get_nif_ref(Loop),

%% Process A defines x
spawn(fun() ->
    ok = py_nif:event_loop_exec(LoopRef, <<"x = 'A'">>),
    {ok, <<"A">>} = py_nif:event_loop_eval(LoopRef, <<"x">>)
end),

%% Process B has its own x
spawn(fun() ->
    ok = py_nif:event_loop_exec(LoopRef, <<"x = 'B'">>),
    {ok, <<"B">>} = py_nif:event_loop_eval(LoopRef, <<"x">>)
end).

Cleanup

Event loop namespaces are automatically cleaned up when the Erlang process exits. The event loop monitors each process that creates a namespace and removes it on process termination.

Automatic Env Reuse with py:exec

Functions defined via py:exec(Ctx, Code) can be called directly using the async task API (py_event_loop:run/3,4, create_task/3,4, spawn_task/3,4). The process-local environment is automatically detected and used for function lookup.

%% Create a context and define an async function
Ctx = py:context(1),
ok = py:exec(Ctx, <<"
import asyncio

async def process_data(items):
    results = []
    for item in items:
        await asyncio.sleep(0.001)
        results.append(item * 2)
    return results
">>),

%% Call it directly - env is reused automatically
{ok, [2,4,6]} = py_event_loop:run('__main__', process_data, [[1,2,3]]).

%% Also works with create_task and spawn_task
Ref = py_event_loop:create_task('__main__', process_data, [[4,5,6]]),
{ok, [8,10,12]} = py_event_loop:await(Ref).

%% Fire-and-forget
ok = py_event_loop:spawn_task('__main__', process_data, [[7,8,9]]).

This eliminates the need to manually pass environment references when calling functions defined in the process-local namespace.

Building Python Actors

The process-bound model enables a pattern we call "Python actors" - Erlang processes that encapsulate Python state and expose it through message passing.

Basic Actor Pattern

-module(py_counter).
-behaviour(gen_server).

-export([start_link/0, increment/1, decrement/1, get/1]).
-export([init/1, handle_call/3, handle_cast/2]).

start_link() ->
    gen_server:start_link(?MODULE, [], []).

increment(Pid) -> gen_server:call(Pid, increment).
decrement(Pid) -> gen_server:call(Pid, decrement).
get(Pid) -> gen_server:call(Pid, get).

init([]) ->
    Ctx = py:context(),
    ok = py:exec(Ctx, <<"
class Counter:
    def __init__(self):
        self.value = 0
    def increment(self):
        self.value += 1
        return self.value
    def decrement(self):
        self.value -= 1
        return self.value
    def get(self):
        return self.value

counter = Counter()
">>),
    {ok, #{ctx => Ctx}}.

handle_call(increment, _From, #{ctx := Ctx} = State) ->
    {ok, Value} = py:eval(Ctx, <<"counter.increment()">>),
    {reply, Value, State};

handle_call(decrement, _From, #{ctx := Ctx} = State) ->
    {ok, Value} = py:eval(Ctx, <<"counter.decrement()">>),
    {reply, Value, State};

handle_call(get, _From, #{ctx := Ctx} = State) ->
    {ok, Value} = py:eval(Ctx, <<"counter.get()">>),
    {reply, Value, State}.

handle_cast(_Msg, State) ->
    {noreply, State}.

Usage:

{ok, Counter} = py_counter:start_link(),
1 = py_counter:increment(Counter),
2 = py_counter:increment(Counter),
1 = py_counter:decrement(Counter),
1 = py_counter:get(Counter).

Reset via Process Termination

Following Erlang's "let it crash" philosophy, resetting Python state is simple:

%% Supervise the Python actor
init([]) ->
    Children = [
        #{id => py_worker,
          start => {py_worker, start_link, []},
          restart => permanent}
    ],
    {ok, {#{strategy => one_for_one}, Children}}.

%% To reset: just terminate and let supervisor restart
reset_worker(Sup) ->
    ok = supervisor:terminate_child(Sup, py_worker),
    {ok, _} = supervisor:restart_child(Sup, py_worker).

No need to manually clear variables, reload modules, or reset interpreter state. The new process starts with a fresh Python environment.

Stateful ML Pipeline

-module(ml_predictor).
-behaviour(gen_server).

-export([start_link/1, predict/2]).
-export([init/1, handle_call/3, terminate/2]).

start_link(ModelPath) ->
    gen_server:start_link(?MODULE, ModelPath, []).

predict(Pid, Features) ->
    gen_server:call(Pid, {predict, Features}).

init(ModelPath) ->
    Ctx = py:context(),
    %% Define functions and load model - stored in process-bound environment
    ok = py:exec(Ctx, <<"
import pickle

_model = None

def load_model(path):
    global _model
    with open(path, 'rb') as f:
        _model = pickle.load(f)
    return True

def predict(features):
    return _model.predict([features]).tolist()[0]
">>),
    %% Load model - it's stored in _model global within this process's env
    {ok, true} = py:call(Ctx, '__main__', load_model, [ModelPath]),
    {ok, #{ctx => Ctx}}.

handle_call({predict, Features}, _From, #{ctx := Ctx} = State) ->
    {ok, Result} = py:call(Ctx, '__main__', predict, [Features]),
    {reply, {ok, Result}, State}.

terminate(_Reason, _State) ->
    %% Python environment automatically cleaned up
    ok.

Advantages

Aspect	Benefit
Isolation	Processes cannot interfere with each other's Python state
Cleanup	No resource leaks - process death = environment cleanup
Restart	Fresh state by terminating process (no manual reset logic)
Supervision	OTP supervisors manage Python actors like any other process
Debugging	Process dictionary inspection shows environment reference
Memory	Each process's Python memory counted separately

Trade-offs

Aspect	Consideration
Memory overhead	Each process has separate Python dict; no sharing
Startup cost	Environment created on first call per process
No shared state	State sharing requires explicit message passing or ETS
Module caching	Imported modules cached at interpreter level, not per-process

When to Use

Good fit:

Stateful services (sessions, connections, workflows)
Actor-style Python components
Isolated workers that may need reset
Per-request processing with state accumulation
Supervised Python services

Consider alternatives when:

Sharing state between many processes (use ETS or message passing)
State must survive process restarts (use external storage)
Memory is constrained (many processes = many environments)
Truly stateless operations (environment overhead unnecessary)

Comparison with Other Models

vs. Global Interpreter State

Traditional Python embedding shares state globally. Any code can modify any variable. Isolation requires explicit namespace management.

With process-bound environments:

%% Each process is automatically isolated
spawn(fun() -> py:exec(Ctx, <<"x = 1">>) end),
spawn(fun() -> py:exec(Ctx, <<"x = 2">>) end).
%% No conflict - different environments

vs. Multiple Interpreters

Some systems create separate Python interpreters per "session". This provides isolation but:

High memory cost per interpreter
GIL contention in multi-interpreter setups
Complex lifecycle management

Process-bound environments use a single interpreter (or subinterpreter pool) but isolate at the namespace level - lightweight and efficient.

vs. Stateless Lambda-Style

Some systems treat Python as pure functions with no state between calls:

%% Stateless style - no persistence
py:call(math, sqrt, [16]).

Process-bound environments allow both stateless and stateful patterns in the same system.

Technical Details

Environments are stored as NIF resources with the following lifecycle:

Creation: First py:exec/eval/call in a process allocates an environment
Storage: Reference kept in process dictionary under py_local_env
Usage: Each call uses the environment as local namespace
Cleanup: NIF resource destructor runs when process terminates

For subinterpreters, environments are created inside the target interpreter using its memory allocator - critical for memory safety.

Interpreter ID Validation

Each py_env_resource_t stores the Python interpreter ID (interp_id) when created. For OWN_GIL contexts, before any operation using a process-local env, the system validates that the env belongs to the current interpreter:

PyInterpreterState *current_interp = PyInterpreterState_Get();
if (penv->interp_id != PyInterpreterState_GetID(current_interp)) {
    return {error, env_wrong_interpreter};
}

This prevents:

Using an env from a destroyed interpreter (dangling pointer)
Using an env created for a different OWN_GIL context
Memory corruption from cross-interpreter dict access

Cleanup Safety

For the main interpreter (interp_id == 0), the destructor acquires the GIL and decrefs the Python dicts normally.

For subinterpreters, the destructor skips Py_DECREF because:

PyGILState_Ensure cannot safely acquire a subinterpreter's GIL
The Python objects will be freed when the subinterpreter is destroyed via Py_EndInterpreter

This design prioritizes safety over avoiding minor memory leaks during edge cases.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Process-Bound Python Environments

Philosophy

How It Works

OWN_GIL Mode

Explicit Environment Creation

Sharing Context, Isolating State

When to Use Explicit vs Implicit Environments

Event Loop Environments

Defining Functions for Async Tasks

Evaluating Expressions

Process Isolation

Cleanup

Automatic Env Reuse with py:exec

Building Python Actors

Basic Actor Pattern

Reset via Process Termination

Stateful ML Pipeline

Advantages

Trade-offs

When to Use

Comparison with Other Models

vs. Global Interpreter State

vs. Multiple Interpreters

vs. Stateless Lambda-Style

Technical Details

Interpreter ID Validation

Cleanup Safety

See Also

Uh oh!

FilesExpand file tree

process-bound-envs.md

Latest commit

History

process-bound-envs.md

File metadata and controls

Process-Bound Python Environments

Philosophy

How It Works

OWN_GIL Mode

Explicit Environment Creation

Sharing Context, Isolating State

When to Use Explicit vs Implicit Environments

Event Loop Environments

Defining Functions for Async Tasks

Evaluating Expressions

Process Isolation

Cleanup

Automatic Env Reuse with py:exec

Building Python Actors

Basic Actor Pattern

Reset via Process Termination

Stateful ML Pipeline

Advantages

Trade-offs

When to Use

Comparison with Other Models

vs. Global Interpreter State

vs. Multiple Interpreters

vs. Stateless Lambda-Style

Technical Details

Interpreter ID Validation

Cleanup Safety

See Also