WIT: Import/Export inconsistencies

[Game4Freak]

This is from the point of view of using component composition for system design where dynamic linking / composing components is key.

Issue: Asymmetrical import/export resolution

Transitive import/export dependencies are resolved inconsistently for imports and exports:

• Imports dependencies are resolved as additional imports.
• Exports dependencies are resolved as additional imports.

This causes worlds with symmetrical definitions to be resolved asymmetrical.
While both worlds look like they would compose perfectly together, once resolved they don't and instead leave a gap behind.

Example:

/// Shared types
interface types {
    type start = string;
    type end = string;

    record path {
        start: start,
        end: end
    }

    resource path-graph {
        constructor();
        push: func(path: path);
        get-ends: func(start: start) -> list<end>;
        get-starts: func(end: end) -> list<start>;
    }
}

/// Validate provided graphs for correctness
interface validation {
    use types.{start, end, path-graph};

    validate: func(graph: path-graph) -> bool;
    validate-path: func(start: start, end: end) -> bool;
}

/// Component that want to use validation
world service {
    import validation;
}

/// Component that provides validation
world validator {
    export validation;
}

This would resolve into these worlds on the components:

/// Component that want to use validation
world service {
    import types;
    import validation;
}

/// Component that provides validation
world validator {
    import types;
    export validation;
}

While the validation interface can be composed, the types interface cannot be composed because it is imported by both components.
This would require an explicit export or a third component or the host to provide these types.

The obvious solution would be to update the resolution to be symmetrical:

• Imports dependencies are resolved as additional imports.
• Exports dependencies are resolved as additional exports.

But this isn't a valuable solution either as it forces the exporter to implement the types even tho it has no use to do so.

Solution: Fix inconsistent differentiation between values and signatures (interfaces)

For the sake of less confusion I will call the classic interfaces/traits "signatures"

Wit types can be split into 2 categories.
These categories have different uses cases for wit itself.

Please correct me if I'm wrong

Category	type	Compilation	Runtime
Values	string, u32, record, variant, ...	✅	❌
Signatures	resource, function	✅	✅

For the runtime, Values don't really matter as the actual implementation is added automatically and doesn't need to be imported or exported.
For the runtime, Signatures matter as the actual implementation is not added automatically and needs to be either imported or implemented to be exported.

So, looking back at the example:

/// Shared types
interface types {
    type start = string;      // <--- import/export doesn't matter during runtime
    type end = string;        // <--- import/export doesn't matter during runtime

    record path {             // <--- import/export doesn't matter during runtime
        start: start,
        end: end
    }

    resource path-graph {     // <--- ❗ import/export does matter during runtime
        constructor();
        push: func(path: path);
        get-ends: func(start: start) -> list<end>;
        get-starts: func(end: end) -> list<start>;
    }
}

This insight alone doesn't change anything as we also need to rethink the import/export behaviour.

• Values don't need import / export schemantic at all.
  The same definitions can be used for both imports and exports.
  It's same for generated code via bindgen (kind of relates to Split 'use' inside worlds into 'use import' and 'use export' #308).
• Signatures require import / export schemantic.
  Whenever a signature is taken into scope it needs to be defined if it's an export or import.

This requires changes to the way entries are referenced via the use directive.

• use to reference a value
• use import to import a signature reference
• use export to export a signature reference
  That's the relation to Split 'use' inside worlds into 'use import' and 'use export' #308

Looking at the example again:

/// Validate provided graphs for correctness
interface validation {
    use types.{start, end};             
    
    // use types.{path-graph};
    use import types.{path-graph};      // <--- The validator only wants the signature to validate the graph and doesn't provide the implementation.

    validate: func(graph: path-graph) -> bool;
    validate-path: func(start: start, end: end) -> bool;
}

/// Component that want to use validation
world service {                         // <--- Has to export the path-graph
    import validation;
}

/// Component that provides validation
world validator {                       // <--- Will import the path-graph
    export validation;
}

This would resolve into these worlds on the components:

/// Component that want to use validation
world service {
    use types.{start, end, path};
    use export types.{path-graph};
    import validation;
}

/// Component that provides validation
world validator {
    use types.{start, end, path};
    use import types.{path-graph};
    export validation;
}

Summary

With these changes interfaces are more expressive in what they require / provide and are resolved symmetrical.

I just want to share my point of view from using wit and it's interfaces as contracts in general where a symmetrical definition is important.
I'm open for any discussion :)

Benefits

• Symmetrical worlds
• Symmetrical import/export resolution
• More expressive interfaces
• Easier component composition
• No export/import type mismatches in bindgen for value types
• Split 'use' inside worlds into 'use import' and 'use export' #308: Same behaviour for use in worlds and interfaces

Changes to WIT

• Add export keyword to use directive (world and interface)
• Add import keyword to use directive (world and interface)
• Require export / import on use for resources and functions

Changes to bindgen (optional)

• Remove different export / import handling for wit types of the Values category (string, u32, record, variant...)

Handling breaking changes

Make use syntactical suguar for use import on resources and functions.

use types;

translates to

use types.{start, end, path};
use import types.{path-graph};       // <--- Same behaviour

World resolution

Export Side

// Before
import types;

// After
use types.{start, end, path};
use import types.{path-graph};       // <--- Same behaviour

Import Side

// Before
import types;

// After
use types.{start, end, path};
use export types.{path-graph};       // <--- Breaking change

Could be feature gated via adding it by default as import aswell and defing the export to reference the import

use import types.{path-graph};
use export types.{path-graph} with { types = import types };

Resulting in

// Before
import types;

// After
use types.{start, end, path};
use import types.{path-graph};       // <--- Same behaviour
use export types.{path-graph} with { types = import types };

Example uses in WASI

wasi:http@0.3

interface types {
    resource request {}
    resource response {}
    variant error-code {}
}

interface handler {
  use types.{error-code};
  use import types.{request, response};    // <--- As a request handler i only need the signature of request and response

  handle: async func(request: request) -> result<response, error-code>;
}

wasi:keyvalue@0.2

interface types {
    resource bucket {}
    variant error {}
}

interface store {
    use types.{error};
    use export types.{bucket};             // <--- As a store i need to provide the implementation of the bucket signature

    open: func(identifier: string) -> result<bucket, error>;
}

[lukewagner]

Hi! Thanks for the thoughtful writeup. Agreed that there are some expressivity gaps that we need to close here with WIT, and I continue to think #308, which you mention above, is a good idea (it's just a matter of getting some time to get it implemented).

So #308 adds use import/use export for use within world definitions, and there I think they have a pretty clear meaning b/c they're simply filling some missing local naming expressivity (in particular when an import and export have the same name).

However, I don't understand what use import/use export would mean inside an interface definition since interfaces are atomic fragments of functionality that can be imported or exported and, when exported, their uses can be resolved to imports or exports as decided independently by each world that imports or exports the interface. For example, wasi:http/middleware wants the use types.{request,response,error-code}s inside both the imported and exported wasi:http/handler interfaces to resolve to a single import of wasi:http/types. But if I wanted to build a component that implements WASI HTTP in terms of WASI sockets, then I'll want my component to export wasi:http/handler and wasi:http/types, and thus the definition of the handler interface can't know whether its contained use types.{request,response,error-code} resolves to an import or an export in isolation.

I tried to understand the desired behavior you wanted from your examples but when I look at the two example WIT snippets that immediately precede the "Summary" section, it looks like the service and validator components are cyclicly importing each others' resource types, which the raw component model (e.g., in WAT) doesn't allow at all (it raises Very Hard questions, particularly concerning the type theory of resource types) and so we can't express at the WIT level either.

[Game4Freak]

Thanks for the response :)
Probably was a bit too much at once. I will try to explain my core issue for this.
Also I have no issue with scrapping the whole proposal I made if there are better ways.

This is how I would understand WIT if I look at it from the behaviour of languages like rust, c#, typescript ...
I will use rust for comparison

A interface is like a trait on the component itself.
import <interface> means I just use the trait.
export <interface> means I need to provide the trait implementation. I don't have to implement it myself but I have to provide it.

A resource is very similar to a interface.
A resource is like a trait on values.
Using a resource inside a function's input parameter means the caller has to provide the trait implementation and the function just uses it.
Using a resource inside a function's return means the function has to provide the trait implementation and the caller just uses it.
This translates to the importer of the function has to provide the input resource implementation and the exporter has to provide the output resource implementation.

interface handler {
    resource request {}
    resource response {}

    handle: func(request: request) -> response;
}

world service {
    export handler;
}

world server {
    import handler;
}

Would according to this logic translate to

world service {
    import handler.{request};
    export handler.{handle, response];
}

world server {
    import handler.{handle, response};
    export handler.{request};
}

This is just the pure behaviour I would expect from interfaces and resources.
This is also symmetrical wich means that the interface is always fulfilled from both sides (export and import).
For me this is very important for component composition and the problem I want to tackle with this issue.

A worlds is like a config / code that handles how the interface (traits) are fulfilled.
When I import the handler interface into my world it forces me to provide the request resource. Then in the world I define how I want to provide it. Either as pure export = I implement it myself. Or as another import linked to the export = I just pass it through.
From the interface point of view it doesn't matter how I provide it, I just have to provide it.

This is obviously cyclic between the components wich can hit limits in some cases.
Lazy linking like the wasmtime linker doesn't directly have a issue with this but something like wac probably does.
To break the cycle the server world could simple define that it needs to import handler.{request} aswell and that the imported handler.{request} should be used for export handler.{request}.
This way the cycle between the components can be optimized away as both worlds import the same hander.{request} from somewhere else.

The current asymmetrical behaviour of interfaces is probably valid due to limits etc. but something nobody would expect when working with it as we are all used to interface / traits to be symmetrical in pretty much any programming language.
If the interfaces behave more as we expect it, it will cause things like cyclic components by default (can be an issue in some cases) but overall composition is way easier as interfaces are always fulfilled and wo don't have any magic imports.
Also it's way easier for people to understand and use as it's something we are all used to.

For me this goes kind of into 2 directions with WIT.

• Either WIT is mean to be more low level like WAT wich makes the current behaviour to prevent limits totally valid.
  But then the component-model ecosystem is missing a more high level interface definition (like a superset of wit) to behave as the average developer would expect and to allow for easier composition and easier systems design.
• Or WIT itself should be that high level interface definition but in this case it needs some work to behave as such (atleast from my point of view)

The potential of component composition with the component model is insane but right now it's very hard to work with as it's missing some rules (like always forcing someone to provide the types instead of having to get them magically from somewhere else).

[Game4Freak]

The use import and use export was just a try from me to get these symmetrical rules into WIT using patterns already present or in other proposals.
Also to make it behave more as we would usually expect as the current behaviour is confusing if you don't know the quirks.
I would need a interface like this for it to behave in the way we are used to from other languages:

interface imports {
    resource request {}
}
interface handler {             // On export:
    use imports.{request};  // Will be resolved to import
    resource response {};    // Will be resolved to export
    handle: func(request: request) -> response
}

Because the dependency definition and resolution is not explicit enough, it really matters how you define and compose your interfaces to get the desired behaviour for the interface itself. This seems very counterintuitive that this has such a high impact on behaviour (kinda like meta programming in wit itself :D). It took me some time to figure it out what was going on when I just wanted to compose 2 worlds with the wasi:http/handler interface.

[lukewagner]

Just focusing on this bit (which transitively propagates into the rest of the comment and so might help clear things up):

Using a resource inside a function's input parameter means the caller has to provide the trait implementation and the function just uses it.
Using a resource inside a function's return means the function has to provide the trait implementation and the caller just uses it.

So this bit isn't quite right: the component responsible for implementing a resource type is independent of how that resource type is used in function parameters or results. In particular, if a component imports a resource type T, it can use T (owned or borrowed) handles in any (import|export) function (parameter|result). If a component defines a fresh resource type and export it as U, the component can then use U handles in any export function (parameter|result), but not in imports (due to the aforementioned acyclicy). Since resource types are generative, I think of resource types as special second-class values that are generated when instantiating the component that defines them.

[Game4Freak]

Thanks again, it was more of a theortical talk but I can see that going into the interfaces to resolve the issue is not a valuable option due to the strict acyclicy rules.

But the core issues (see top comment) is still the same:

world a {
    import wasi:http/handler@0.3.0;
}
world b {
    export wasi:http/handler@0.3.0;
}

world b will NOT satisfy world a in the current state.
This makes composition harder than it needs to be.

This is due to the mentioned inconsistency for the dependency resolution:

Transitive import/export dependencies are resolved inconsistently for imports and exports:

• Imports dependencies are resolved as additional imports.
• Exports dependencies are resolved as additional imports.

[lukewagner]

I think this scenario you've outlined does compose (e.g., using tools like wac plug today), and it behaves like you'd expect, allowing a component targeting b to handle HTTP requests without having to also implement the raw HTTP request and response resource types (and all the methods on these resource types defined by wasi:http/types). In particular, when I compose a component targeting a with a component targeting b, the new composed parent component (that contains instances of a and b as children) will import only one implementation of wasi:http/types and share its types and methods with both the a and b instances, so that the two child components can pass handles to requests and responses between each other without any conversions.

Now if b wants to implement wasi:http/types (for some more-rare, more-advanced virtualization scenario), it could write:

world b2 {
  export wasi:http/types;
  export wasi:http/handler;
}

and now the WIT resolution rules will resolve the use types.{request,response} inside wasi:http/handler to the exported wasi:http/types and there will be no implicit imports. Furthermore, a component targeting b2 could also be plugged into an a component (which doesn't need to change or care about where it got its imported wasi:http/types).

[Game4Freak]

Yeah but the questions is more about the default (implicit) behaviour.

With current default behaviour you will need 3 entities:

1. One that imports the explicit stated wasi:http/handler
   (imports wasi:http/types)
2. One that exports the explicit stated wasi:http/handler
   (imports wasi:http/types)
3. One that exports the explicit stated wasi:http/types

By default 1. and 2. are already assigned to the worlds a and b but the 3. is missing.
This means by default you don't know where 3. comes from.

The only way to solve this is to go outside of the default behaviour and assign the 3. separately.
Either to the component targeting the world export wasi:http/handler or a third component or the host.
This requires you to know how wasi:http/handler is defined else you wouldn't even know that you need to take care of it.

What makes this even more confusing is that if you use interfaces for organization likes the types.wit files in wasi with a central interface for shared types it behaves differently than if you would just inline your types in your interface directly.
This always forces the developer to actually know the details how an interface is defined because it could look identical from the outside but it will behave differently.

[Game4Freak]

If the default behaviour would be:

• Imports dependencies are resolved as additional imports.
• Exports dependencies are resolved as additional exports.

We would always know where everything comes from and there are no implicit imports be default.
This means if my component exports wasi:http/handler it must also export wasi:http/types.
If I dont want to implement wasi:http/types I could explicitly add wasi:http/types as import and link it to the export.
Then I also have wasi:http/types no specified where it comes from but this is explicitly defined and not the default.
In addition I know that my component will provide wasi:http/types to whoever it's going to be plugged in, so for that other component it's already defined where wasi:http/types comes from.

[lukewagner]

By default 1. and 2. are already assigned to the worlds a and b but the 3. is missing.
This means by default you don't know where 3. comes from.
[...]
Either to the component targeting the world export wasi:http/handler or a third component or the host.

This is indeed the desired behavior for WASI HTTP (and the original motivation for splitting types out from handler): we usually want the host to implement wasi:http/types and so by default there are 3 entities (component a, component b, and the host). Or, if we're virtualizing WASI HTTP, "the host" gets replaced with "the virtual HTTP impl component".

[Game4Freak]

Yeah and it makes sense for the wasi packages but long term there will be more than just wasi.
While wasi can heavily rely on the host by default, custom interfaces will also use the same wit language as wasi does but they cant rely on the host that easily.
It's probably extremely uncommon that a host will implement something from a custom interface.
So the custom interfaces will be kind of stuck in this unresolved dependency issue where they dont know who will provide it (in the default behaviour).

[lukewagner]

Alright, so if we set aside WASI HTTP and consider the general case, I think the remaining root problem here might be that #402 was never merged and implemented (I think it just fell off the radar). In particular, with #402, exporting an interface that uses some other interface that isn't already imported or exported would turn into a validation error (instead of today succeeding and injecting an import, which can be surprising behavior, given the ambiguity of the choice), and so you'd need to explicitly add an import or export to your world to say what you want.

[Game4Freak]

Yeah I agree, that should take care of it (somehow overlooked it while searching the issues and pulls when creating this one).
By forcing expliclit import or export on dependencies of an exported interface, you will always know where the used dependencies come from, instead of falling into the current trap where it's pretty much undefined by default.

[lukewagner]

Great, thanks for the consideration and digging into this issue! I commented on and rebased #402, so hopefully we can get to that before long.