The `extends` field should support referencing OADs by their `$self`

[handrews]

See OAI/Arazzo-Specification#451 for an analogous requirement to support their Source Description Object's url field behaving the same way. It involves language about parsing potential target OAD documents supplied to the implementation for their $self field before deciding that no OAD document matches the URI.

IIRC, Overlay documents are never a target of any sort of references, so they don't really need their own $self at this time (although it might be nice for all three specs to support it, which would future-proof Overlays if they ever do need to be referenced — the above PR adds it for Arazzo).

[handrews]

I've filed #316 for a $self within the Overlay specification, and move the comment about that that was here to that new issue. This way the two parts can be addressed separately if that is desired.

[baywet]

Collecting notes from today's meeting (neither Lorna or Henry were here to provide additional context). It'd be helpful to include additional information about why this would be needed, and how that would work for implementers.

[baywet]

On the how, this provides significant context https://spec.openapis.org/oas/v3.2.0.html#parsing-documents

[baywet]

Some quick thoughts: this adds significant requirements to the implementers of the Overlay specification. Now they need to parse OpenAPI 3.2.0 document (as in, understand the document better than just a bunch of JSON nodes), to fish out the self. And that schema might evolve over time, which they'd need to keep track of.

Maybe a good candidate for 1.3 ?

[baywet]

This also impacts the "manifest" discussion that's been floating around in the ecosystem. Essentially, if we had a manifest that'd describe "in order to get the result you need in your context, take that OpenAPI description, apply overlay A, B, C, and use arazzo Alpha", we'd probably need resource locators instead of identifiers to "be able to find the documents". And once you have the documents, which already contain the identifiers, why duplicate the identifiers in the manifest?

That last question came from the use we (my immediate team) makes of overlay, where we essentially never use extends.
We might want to deprecate extends all together is usage is low in version 2.0

[handrews]

@baywet

Now they need to parse OpenAPI 3.2.0 document (as in, understand the document better than just a bunch of JSON nodes), to fish out the self.

If they're referencing OAS 3.2 documents, then they need to do this anyway- that is a requirement of OAS 3.2, and Overlays can't negate the requirement. It's already the requirement within OAS 3.2 (and now Arazzo 1.1) so it's not like this is a problem tools will be seeing for the first time.

You do have to bootstrap a bit, which is why $self, like openapi (and arguably jsonSchemaDialect), is in the root object, so you don't have to understand the whole document. Just that there's a root object and you have to check a few fields. JSON Schema works the same way with $id and $schema. All of these are fields that tell you how to parse the rest of the document, so you have to be able to look at them prior to doing the full parsing.

And that schema might evolve over time, which they'd need to keep track of.

The $self field can't be moved in a minor release as that would break compatibility, so "the schema might evolve over time" is not a real concern. Except for a major release, and that's always an "all bets are off" compatibility break, so d

It'd be helpful to include additional information about why this would be needed, and how that would work for implementers.

I'm puzzled- IIRC you were there for the entire discussion for OAS, and there's also the issues and PRs for Arazzo. It's the same feature everywhere, which is the whole point: to make parsing and referencing work consistently across our specification ecosystem instead of having special rules for each one.

Nothing here is new. We debated it fully over the past several years before putting it in 3.2. What is the concern?

[handrews]

I suppose I really should have phrased this as "the extends field should acknowledge that parsing OAS 3.2 requires respecting $self", as no specification can negate another specification's requirements.

[handrews]

This also impacts the "manifest" discussion that's been floating around in the ecosystem. Essentially, if we had a manifest that'd describe "in order to get the result you need in your context, take that OpenAPI description, apply overlay A, B, C, and use arazzo Alpha", we'd probably need resource locators instead of identifiers to "be able to find the documents". And once you have the documents, which already contain the identifiers, why duplicate the identifiers in the manifest?

First, keep in mind that in really large orgs, you may have a lot of OpenAPI-related documents and not all of them are needed all the time for every use case. Also, presumably the manifest conveys additional information like the order of overlays to be applied.

Now let's take two scenarios that you would need to get approved with whoever does your security audits. You tell your auditor that you have a tool that controls how you interact with APIs (and these APIs could potentially do important/dangerous things, like change medical records for example, as I once ran these scenarios past an audit expert who worked for a major medical insurance org).

1. Location version: When processing instructions, this tool may encounter an arbitrary network or filesystem address, fetch whatever is there, assume it is correct, and process the instructions contained therein.
2. Identity version: When processing instructions, this tool may encounter an arbitrary identifier. It checks pre-configured, approved-by-the-auditor locations for documents that self-identify with the given identifier.

The first version will not get approved in any sort of higher-security environment (again, I have actually verified this with a security auditor- the look on her face at option 1 was hilarious). In some environments, you won't even have the network access necessary.

The second one can get approved (although depending on how high-security we're talking about, you might need additional mechanisms to trust that the contents of the documents match their identity claims, but that's mostly orthogonal to identity vs location).

Another possible analogy here is running a local PyPI or NPM server. In a higher-security environment you don't necessarily want to pick up everything on a public server. You want packages to be screened and approved by your open source governance process, and audited for security risks. So when you pip install or npm install, you are giving it an identifier, and depending on how the tool is configured, it might fetch the package from different locations.

[baywet]

I'm puzzled- IIRC you were there for the entire discussion for OAS, and there's also the issues and PRs for Arazzo

Yes, and in the context of an OpenAPI document, or set of documents, it makes perfect sense to me to have $self. To be clear, I'm not pushing against that aspect, or suggesting that the Overlay specification should somehow "remove" it. (because it can't)

It's the same feature everywhere

I disagree here. $self makes sense in the self contained context of OpenAPI descriptions. $self also makes sense in the context of Arazzo since Arazzo build upon concepts from the OpenAPI descriptions that require reference resolution.

Overlay however, does not come close to any of those concepts, the OpenAPI description is "just a set of JSON nodes to mutate". In fact, that's exactly what I'm doing in clio. With the current version of the specification, there's no need to "understand the OpenAPI description" to apply the transformations. Only the location (through extend or tooling context) is required (and the order if you have multiple overlay documents).

#296 introduces a notion of references, but those are restricted to be relative only at this stage, which I believe means we wouldn't need a $self field in this iteration?

While writing the above, the multiple package feeds example actually started resonating with me. $self is not just about resolving $refs. It's also about being able to identify a "document" regardless of its location, especially if the location changes, without requiring both sources/having to perform a diff. I think that's the distinction I (and others on the call) needed to tie things together.

If that context is correct, I think it'd be helpful to callout separate but related aspects:

• Addition of $self in overlay to uniquely identify an overlay document, regardless of its location.
• Revamping extends so it supports providing both a location (optional) and a $id (or a better name) to identify the target document.

Does that make any sense? Again, I'm not pushing back, simply trying to ensure I'm following along here.

[handrews]

$self is not just about resolving $refs. It's also about being able to identify a "document" regardless of its location, especially if the location changes, without requiring both sources/having to perform a diff.

Yes, this is what "separation of location and identity" is all about. You can move a thing around and it's still the same thing, but not all places where you might put that thing are equally accessible to every use case.

Addition of $self in overlay to uniquely identify an overlay document, regardless of its location.

This is issue #316.

Revamping extends so it supports providing both a location (optional) and a $id (or a better name) to identify the target document.

This is this issue, although I would question putting a location in, even as an option, because now you have several things, and the second one is confusing:

1. The internally-decalred identity ($self), that stays the same (whether it is an absolute identity or a partial, relative-to-location identity) regardless of location
2. An internally claimed location (your optional location field)
3. [rarely used] an externally-recorded location (this happens with the Content-Location part header in multipart/related documents, for example, essentially saying "I got this part from here")
4. The actual location from which the document was retrieved, a.k.a. the retrieval URI

Numbers 1, 3, and 4 are well-established and correspond to sections 5.1.1, 5.1.2, and 5.2.3 of RFC3986. Number 2, which you are proposing to add, disrupts RFC3986's model of how to manage identity and location and introduces ambiguity (if it disagrees with the retrieval-URI) and complexity to an otherwise well-defined process.

[handrews]

Wait, it's my turn to cross something out, as I got the topics a bit confused.

I would still avoid putting a location in extends. Let extends work like all other references. You don't want to undermine the separation of identity and location, because there are many other ways to handle that. It's a level of indirection, so you don't want to try to be both indirect and direct at the same time. You want to allow differnces between location and identity to be managed externally.

Note that there is also nothing inherently opping you from using the extends URI as a locator. And again, we want referencing (of which this is a subset) to work the same everywhere. As much as possible. The three different $refs of OAS are an accidental historical bug, not a feature :-/

[baywet]

The three different $refs of OAS are an accidental historical bug, not a feature

yup... if we can clean this up for 4.0, it'd be awesome. @MaggieKimani1, @darrelmiller and myself lost a lot of hair in "solving for refs" in Microsoft.OpenAPI (dotnet lib)...

I would still avoid putting a location in extends. Let extends work like all other references

But isn't it a "special" kind of reference though? In the sense that it can't accept any fragments? I think my confusion as to "this is not a regular reference" came from on my end: extends is always external by nature, and can't "index into" a document.

[handrews]

@baywet the "not allowing fragments" thing isn't actually special- it's just a result of the fact that OpenAPI Descriptoin documents are what RFC3986 calls "primary resources", and are by definition identified by an absolute-URI (without a fragment).

So it's not that the extends URI could have had a fragment but is forbidden from doing so. It's that OAD document URIs can't have fragments by definition, and the value of extends is an OAD document URI. The fragment rule is a consequence of how URIs and documents work, not a decision made by the Overlay specification. Does that make sense?

[baywet]

Yep, that helps a lot solidifying my understanding of the different layers here. Thanks!

With all that context in mind, I think we've already addressed the "overlay documents need a self field to identify them" part of the discussion. As a side question: why self and not id in this context? Consistency?

For the extends field, using the self of the target oad effectively means the location will need to be provided by the "tooling context" (arguments for a CLI, etc)
I believe the implicit result from having a mismatch between the extends value and the self of the target document is that the tool should error out. Did I get that part right?

[handrews]

Yep, that helps a lot solidifying my understanding of the different layers here. Thanks!

Great! Yeah there are rationales and existing standards driving all of this, but most folks aren't deeply familiar with them or with the use cases. Working in an enterprise company where we sold network devices into "air-gapped" networks (no physical connectivity to the wider internet) really expanded my notion of URI-resolution use cases! When the laws of physics prevent you from fetching a URL, you need flexibility!

As a side question: why self and not id in this context? Consistency?

Um... reasons? People on the TDC calls just liked $self better. I think it helps a bit because JSON Schema's $id is more complex with the whole embedded resource concept, and $self is intentionally much simpler. If $id were restricted to the root object of a JSON Schema document (without the embedded-in-a-document-subschema resource concept) then the behaviors would be identical. But it's easier to explain $self on its own. But I think I made up that justification after the fact.

For the extends field, using the self of the target oad effectively means the location will need to be provided by the "tooling context" (arguments for a CLI, etc)

Correct. You can see this in a lot of JSON Schema implementations where you provide a plug-in that determines how to locate a given URI. It might do a "pass-through" and treat it as a URL, or it might look in a document store database, or map it to a filesystem, or whatever. The referring reference has no idea, and that's intentional.

I believe the implicit result from having a mismatch between the extends value and the self of the target document is that the tool should error out. Did I get that part right?

That's more of a policy decision. I would consider it an anti-pattern to reference by location when a $self identifier is present, but if your tool allows the "wouldn't pass a security audit" automatic URL fetch, perhaps because it is strictly internal, there's no reason it can't work. But a tool might want to offer the option to error out, or at least log a warning.

The trick with these things is to always balance convenience (for casual, low-security environments) with rigor (for high-security environments, or environments with challenging networking configurations like intermittent line-of-sight relays). So we want to make it possible to enforce a high-security configuration, but not require it of everyone. Sometimes you're just throwing together a prototype API for an internal demo and it really doesn't matter.