Repository navigation
varlink: multiplexing #232
Description
Sometimes people want to speak Varlink over a single stream, for example RS232 or USB4STREAM where opening a second stream might not be cheap, simple or even possible. For that we should provide the option to do multiplexing of multiple connections over a single one, so that multiple method calls can be active simultaneously.
Activity
mvo5 commentedon Sep 11, 2026
If/when we do this I think it would be good to consider:
a) unless we are okay with breaking the "procol-upgrade" case this needs to be something "outside" of the varlink itself (after the upgrade we are not even speaking json anymore so doing it inside e.g. a header wouldn't work) so it needs to be able to mltiplex frames of bytes
b) for regular networks http2 would be the natural choice as it already supports multiplexing (a bit too complext for RS232 though)
With that in mind, maybe we just make suggestions? Like "yamux-shaped" (exact recomendation TBD, rs232 probably makes it even more low-level actually) if you want to keep it simple, h2 if you have a http stack anyway already.
poettering commentedon Sep 11, 2026
Here's lennart's proposal:
Let's not add this to Varlink's protocol itself. Instead, let's formalize the "bridge" concept from the original varlink design (mentioned at various places on varlink.org, but not implemented in systemd), and let's then add multiplexing support to that.
Specifically:
The Varlink "bridge" as originally designed is a single socket that many services can be reached by: it simply routes method calls to a set of sockets previously registered for a specific interface name. from the client side there's hence little effective difference from talking directly to the target service or via the bridge. it's kinda like the existing varlink http bridge, just without the http part.
(The bridge is described briefly in the FAQ on https://varlink.org/FAQ and on the screenshots page https://varlink.org/Screenshots – but nothing more formal)
My idea would be to:
First, make the bridge part of the varlink spec, i.e. official.
Second, introduce an explicit multiplex interface that the bridge would implement. org.varlink.Multiplex or so. This interface has primarily one call that does a protocol upgrade to a simple binary protocol implement very basic round robin multiplexed stream scheduling. Each stream would itself be a bridge connection, and thus can be used all other registered services (these "second" level bridge connections probably shouldn't implement the multiplex protocol though, to refuse ridiculous nesting).
Why binary protocol for the multiplexing instead of wrapping messages in outer JSON (as i originally envisioned)? Mostly because that wouldn't work nicely if the inner streams are upgraded too because then we'd either have to base64 the upgraded traffic and that'd suck, given that after upgrading any data can be sent.
strawman proposal how the multiplexing protocol could look like: the stream would be used to deliver a series of frames. Frames would have a tiny 64 bit header:
flags would have:
0xFFwould be a marker for the protocol, which is supposed to be different from all bytes valid in regular varlink json messages, and also different from the NUL separator.The
sizewould be a 16bit size of the following frame.the
stream IDwould be 32bit id the client can pick freely and is used to identify and distinguish multiple streams. any further message in either direction with the same ID belongs to the same stream.The flags field has two defined bits (all others must be zero): syn and fin. If only "syn" is set then the frame is the beginning of a new stream, identified by the id field. If there's already a stream with the same id it is implicitly disconnected, and a new one begins. If only "fin" is set, than the frame data is the last data of the stream. if neither syn nor fin are set, then the frame is a regular frame in the stream indicated by id. if both "sync" and "fin" are set, this acts as indicator to terminate the multiplexing, ending all leftover streams.
frame boundaries are only relevant to allow multiplexing. the frames of each stream are supposed to be joined as if there was no boundaries in between. varlink messages will be separated by NUL bytes inside the frames. if multiple connections are established each side should probably round robin schedule frames and put limits on the frame size, but this is an implementation detail.
Using this: if you want to make use of the multiplexing logic for a project, for example over rs232, you'd just invoke an instance of the bridge on /dev/ttyS0. It would then immediately react to regular Varlink. After a org.varlink.Multiplex.Start() call it would switch to the multiplexed protocol, and now the client can send many varlink requests in parallel.
Varlink libraries could even implement the multiplexer natively on their server side: optionally react to the multiplexing calls inside the library and offer multiplexing this way. they could also implement this client side and mostly hide it away: try the multiplex call as part of connection setup, if desired.
Benefits of this design:
specifically for systemd we'd probably implement bridge, multiplexer and also the "resolver" (see varlink.org) in the same binary i guess, and offer it on the same socket.
poettering commentedon Sep 11, 2026
(the framing is very similar to @chergert's ideas, btw. i didn't do the ack/fail flagging for stream setup though. we probably should add that though, simply so that we can refuse allocating too many streams over a single connection)
mvo5 commentedon Sep 11, 2026
So I am working on a relay/bridge idea for varlink that is mostly inspired by providing a way to make the varlink-http-bridge accessible behind a NAT/firewall setup, i.e. a strict setup that only allows https outbound (c.f. https://github.com/systemd/varlink-http-bridge/compare/main...mvo5:varlink-http-bridge:relayd?expand=1). So I guess its a slightly different/broader issue I'm trying to deal with (its actually more a slightly specialized reverse proxy for end-to-end encrypted stream thinking about it)
It does requires to multiplex multiple bytes streams over a single connection (that happen to be varlink connections) too. In this setup using http2 is the natural shape (as everything else is build around it already and it gives us everything above plus flow-control and back-pressure). I guess that is okay and orthogonal to this proposal/idea?
For rs232 the multiplexing described above sounds fine to me.
chergert commentedon Sep 11, 2026
Despite their similarities on the surface, I think that service aggregation like a bridge and muxing should remain separate concepts.
The varlink-glib prototype was useful in figuring out some muxing semantics that this more compact framing would benefit from. For example. stream creation needs to have an ACK/rejection path so the receiver can enforce limits before handing off a stream. Additionally, stream identifiers need to ensure they won't collide (see the even/odd semantics in varlink-glib) and reject rather than closing an existing stream.
I would consider changing varlink-glib's 16-byte header to 8-byte, but it would be a shame if we lost the ability to frame messages up to 256kb. That tends to be the sweet spot in bulk transfer (which I want to do over USB for OS provisioning and data synchronization). Additionally, losing the OPEN/ACK/REJECT/DATA/FIN-or-CLOSE would be a real loss (but using flags for this is fine).
We should also define if close is directional (half-close) and how unknown or late frames are handled. And what should happen in terms of fairness between streams (byte accounting, message accounting, etc).
Without per-stream flow control, a slow consumer forces an implementation either to buffer data without bound or to stop reading from the shared transport, thereby stalling unrelated streams.
Making this a general upgrade protocol rather than a bridge-private facility lets ordinary services, serial transports, reverse tunnels, and bridges share one implementation. The bridge then remains the policy/routing layer on top.
In terms of the RS232 reset, it will not function as described unless your framing is in-tact but I suspect that is exactly the situation where it is most useful. The 8 reset bytes could get consumed as data.
jhenstridge commentedon Sep 22, 2026
Are people really wanting to open a second stream, or do they want to issue multiple method calls without requiring a complete response to the first before a reply to the second can be sent?
This might be the case where multiple streams are being used because one is tied up by an endless stream of
continues=truereplies. If it was possible issue more method calls on such a stream and have their replies interleave with the existing continuation replies, then that second stream might not be necessary.chergert commentedon Sep 22, 2026
I would say "sort of".
There are two major things that come into play if you allow multiple calls on the same connection.
This is an issue on serial/USB if you want to do a bulk data transfer and the ability to do anything at a higher priority (such as a cancel request).