misc: add arca-networking code#33
Conversation
There was a problem hiding this comment.
I think we could use some existing crate (like embedded-io) that provides no_std io traits so that we are not redoing some of the things...
|
|
||
| --- | ||
|
|
||
| ## 3. Stream layer (`data-pipe`) |
There was a problem hiding this comment.
I feel like the more ergonomic way of handling this asyncness would be that for each direction of the write ---> read of the pipe, there needs to be an event fd like file descriptor for the writer to notify the reader that there are new data readable, instead of doing the current yield_now.
This file descriptor would be set up differently depends on whether the pipe is going into Arca or leaving Arca, but I could imagine the Linux monitor that relays from Arca is implemented as a bunch of tokio::spawn of loops triggered by the ready event fd.
There was a problem hiding this comment.
For sending notifications into Arca the Linux end can write to an EventFd to trigger an interrupt in Arca (which will reschedule any kernel thread that had called kthread::wifi() to deschedule itself). For sending notifications out of Arca the Arca end can make a hypercall, which could notify a condition variable in Rust.
| } | ||
|
|
||
| /// Bytes available to write. Called by the producer. | ||
| pub fn free_space(&self, capacity: u64) -> u64 { |
There was a problem hiding this comment.
I would change "used_space ==> readable_len" and "free_space ==> writable_len"
| --- | ||
|
|
||
| ## 4. Message catalog | ||
|
|
There was a problem hiding this comment.
I feel like we should put in more layers of abstractions here. The control pipe messaging channel should be only about that the Arca side would like to create a new pair of bidirectional pipes, and then there is a separate protocol for negotiating what this newly-created pair of pipes is for (e.g. this could be by the first frame/message sent from the Arca side on this pipe). So it could be something like
-
Control Pipe: Arca -> Linux: NewStreamRequest; Linux -> Arca: DataStreamInfo
-
Data Pipe:
- First Frame from Arca to Linux: Protocol (0 = FileSystem, 1= TCPListener, 2 = TCPConnection, etc.). We can also piggy back the actual protocol payload here (like the Endpoint for TCP Listener)
- If it's a pipe for file system: Arca -> Linux: ReadFileRequest { path }; Linux -> Arca: FileData { payload }
- If it's a pipe for listening: Arca first creates a new stream via control pipe; Arca -> Linux: AcceptConnection (with the new pipe id as payload); Linux -> Arca: IncomingConnection (also setup the relaying on the monitor side)
Then, on the monitor side, the implementation would be something like for each kind of pipe, we implement the logic for handling the pipe-specific protocol.
| per-connection byte relay: | ||
|
|
||
| ```rust | ||
| loop { |
There was a problem hiding this comment.
I think pump_once and the relaying should probably be independent tokio::spawn if possible...
| └── PROTOCOL.md # this file | ||
| ``` | ||
|
|
||
| The Arca-facing public surface is intentionally small: |
There was a problem hiding this comment.
If we do split the control frame into more layers of abstractions, Arca kernel should only care about open/read/write/close of a stream. The actual protocol implementation would be part of the user procedures.
| │ └── src/ | ||
| │ ├── protocol.rs # wire types + payload encodings | ||
| │ ├── message.rs # ControlRequest / ControlReply enums + to_frame / try_from | ||
| │ ├── codec.rs # read_frame / write_frame / FrameReadBuf |
There was a problem hiding this comment.
Everything under control/ could be made no_std if we move codec.rs to somewhere else. I think codec.rs should be together with monitor as the Linux code, and control should just be a no_std crate shared by Arca side/user procedures and Linux side that handles messages.
| len: MAX_FRAME_PAYLOAD + 1, | ||
| }); | ||
| } | ||
| match transport.read(&mut self.storage[self.len..]) { |
There was a problem hiding this comment.
ummm you should be able to know the payload_len after reading the header, then I think calculating how much is left in the frame would be better than consume_prefix? And if we simplify the control pipe to just opening new streams, the payload len for the control pipe should be a constant number.
| }) | ||
| } | ||
|
|
||
| fn write_u32(out: &mut [u8], v: u32) -> usize { |
There was a problem hiding this comment.
These functions should be renamed as serialize_...?
| 4 | ||
| } | ||
|
|
||
| fn write_endpoint(out: &mut [u8], ep: &Endpoint) -> usize { |
There was a problem hiding this comment.
Also it's probably cleaner if these write/read (or indeed serialize/deserialize ) are part of the struct impl
| /// | ||
| /// `listener_id == 0` means "outbound connection, no listener was involved". | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq)] | ||
| pub struct ConnectionReady { |
There was a problem hiding this comment.
Should this be in message.rs?
| @@ -0,0 +1,215 @@ | |||
| //! Wire-protocol data types for the **single control pipe** between Arca | |||
There was a problem hiding this comment.
I'm not sure why the current content of protocol.rs is in here...Like the type of messages should probably be defined in message.rs and here we only have things more like helper methods like a Frame that accepts a generic Message type, or commonly used structs (Endpoint, etc.) and the serialize/deserialize methods for those. That feels more like a message_util.rs to me (?)
2 participants
Draft PR for commenting/discussing purpose for now.