Generate Go modules with the in-repo codegen and add an opt-in build-only runtime - #35
Draft
eunomie wants to merge 12 commits into
Draft
Generate Go modules with the in-repo codegen and add an opt-in build-only runtime#35eunomie wants to merge 12 commits into
eunomie wants to merge 12 commits into
Conversation
Module codegen is the one asymmetry PR dagger#15 left: clients are generated by this repository's engine-free helper, modules still go through the engine's native goSDK.Codegen. This doc scopes bringing module codegen in-repo for dagger-module.toml modules (two engine-free passes around the engine's public schema().merge(), reachable from Dang since v1.0.0-0) and adds a small, opt-in, build-only runtime module at runtime/, mirroring python-sdk and java-sdk. Legacy dagger.json modules and the engine's builtin routing are untouched. Signed-off-by: Yves Brissaud <yves@dagger.io>
The shared generator was extracted before engine beta.10 started returning optional object fields as (*T, error), nil on null. A module generated by the engine at beta.10 or later already carries those signatures, so module codegen in this repository must emit the same shape or a regenerate would rewrite the user's bindings. This is the engine's change (dagger/dagger 41472c623c) applied to the extracted files: SupportsNullableObjects gates on the schema version, the field/interface signature builders and the object and dag templates return the nullable form, and GeneratedState grows the RemovePaths/PostCommands/NeedRegenerate fields plus Apply that module generation relies on. Behaviour change for generateClient: optional-object fields now return (*T, error) on engines at or above v1.0.0-beta.10, matching the engine's own client generator. The helper test check moves to golang:1.26-alpine, the image the codegen builder moves to next. Signed-off-by: Yves Brissaud <yves@dagger.io>
The client extraction (PR dagger#15) narrowed the shared generator to what a standalone client needs. Module generation renders against the parsed module package, so this restores the plumbing that carried it: the loaded package, fileset and pass index on the template funcs behind a GoTemplateFuncsForModule constructor, the IsPartial template hook for the bootstrap pass, and a moduleGenCtx threaded through generateCode so one render path serves both modes — the client passes nil and is unchanged. Per-module binding files land under internal/dagger in module mode, where the root package is main. PackageInfo learns whether dagger.io/dagger is replaced and the module config gains the library version to pin. Signed-off-by: Yves Brissaud <yves@dagger.io>
Engine-free: the module's own source is loaded with go/packages, which needs a Go toolchain but no engine. The trace span goes, as in the client port. One check the engine's loader does not make: packages.Load runs `go list -e` and returns a package even when an import cannot be resolved, leaving the affected types `invalid` and the emitter free to drop those functions without a word. Under the engine every import resolves because credentials are forwarded; here they are not, so a parse error or an import whose module cannot be found is fatal. The root package only records the latter as a TypeError, so NeedImports is requested (without NeedDeps) to read the ListError on the imported stub. Other TypeErrors stay tolerated: stripping function bodies makes "imported and not used" routine. Signed-off-by: Yves Brissaud <yves@dagger.io>
Pure go/types and jennifer AST rendering, engine-free: the module_* type definition templates, the invoke dispatcher (modules.go), the introspection emitter that turns the module's own declarations into introspection JSON, the visit/optional helpers, the module .tmpl files (the dispatcher body of dagger.gen.go and the internal/dagger bindings package) and their tests, from dagger/dagger@v1.0.0-beta.11 with imports rewritten. The isModuleCode/isStandaloneClient/moduleRelPath helpers already live in client.go since the client extraction; the client-only Dependencies helpers are not ported because a standalone client here serves a single bound module. Adds github.com/dave/jennifer and github.com/mitchellh/mapstructure, the versions the engine uses. Signed-off-by: Yves Brissaud <yves@dagger.io>
GenerateModuleTypes bootstraps a module to a loadable state (go.mod, base bindings from the dependency schema, go get + go mod tidy post-commands, another pass while anything was scaffolded) and emits the module's own types as introspection JSON. GenerateModule takes a schema that already carries those types and renders the bindings, the invoke dispatcher, .gitattributes and .gitignore, and reports per-dependency bindings that went stale. The one engine call the engine's generator makes — merging the module's types into the schema — is left to the caller: the SDK's Dang layer runs the engine's schema().merge() between the two passes, so the binary stays engine-free. Where the engine embeds the dagger.io/dagger go.mod to seed minimum versions and copy its replace directives, this resolves the pinned library's go.mod with `go list -m -json` from outside any module (only .info and .mod are fetched) and runs the same seeding. A fresh go.mod gets the toolchain's major.minor as its go directive; `go get dagger.io/dagger@<pin>` raises it to the library's own, keeping the builder image's patch level out of users' go.mod. Self types are emitted only for schema views from v0.12.0, where older views alias schema types into the main package. .gitattributes carries the engine's three linguist-generated entries and .gitignore its surviving /.env entry — appended with the same substring de-duplication — unless dagger-module.toml sets [codegen] automaticGitignore to false, decoded with github.com/pelletier/go-toml as the engine does. Adds github.com/pelletier/go-toml. Signed-off-by: Yves Brissaud <yves@dagger.io>
The helper grows two subcommands around the caller's merge: module-types bootstraps the module and writes its own types as introspection JSON; generate-module renders the bindings from the merged schema, then writes the stale dependency bindings it removed and the go.work in effect (both relative to --output) for the caller to carry back to the workspace. Both loop while a pass reports NeedRegenerate, applying the overlay and running the post-commands in the module source directory between passes; a post-command failure is fatal, as under the engine. The flag-only client invocation stays the default so the existing clientDirectory call keeps working; generate-client is its explicit name. --module-parent-path feeds the source-map links on generated types, as the engine computes it. --module-root-path names the directory holding dagger-module.toml, so the [codegen] automaticGitignore setting is read from the module's own config rather than from whichever ancestor happens to carry one. Signed-off-by: Yves Brissaud <yves@dagger.io>
A fresh client's go.mod took its go directive from runtime.Version(), the codegen builder's full patch version. That builder moves to Go 1.26 for module generation, so a client generated today would pin users to 1.26.7 — the image's patch level, not anything the client needs. Write the toolchain's language version instead, as module generation already does: the dagger.io/dagger requirement raises the directive to whatever the library itself needs. An existing client go.mod keeps its own directive untouched, as before. Signed-off-by: Yves Brissaud <yves@dagger.io>
Mod.generate splits on config format, as python-sdk's does: a dagger-module.toml module is generated here by this SDK's own codegen; a dagger.json module keeps going through the engine's generatedContextDirectory, which is the engine's builtin Go SDK regenerating it at load anyway. The in-repo path runs the codegen helper twice around one in-engine merge. Pass A mounts the module's whole context at /src (so a parent go.work, a `replace ../lib` and a nested `source` resolve exactly as under the engine), bootstraps the module and emits its own types; the engine's schema().merge() folds them into the dependency schema — the one implementation every SDK shares, reachable from Dang since v1.0.0-0 — and pass B renders the bindings from the merged schema. The generated module root is layered onto the workspace with withDirectory, never replaced, so a file the workspace view does not carry is never reported as removed; stale dependency bindings are dropped by name, and a go.work the module was enrolled in is carried back when the caller's cwd can reach it, with an actionable error otherwise. Mod.generate is Mod.generated(ws).changes(ws): generated returns the workspace with the module's output staged onto it, and GoSdk.generate folds every managed module through it before taking one changeset at the end. Modules share files — two modules under one go.work each run `go work use .` on it — so generating them all against a common baseline and merging the changesets would keep only one of the two enrolments. An engine-free codegen has no host credentials. A module with private Go dependencies keeps the engine's codegen with a .dagger-go-sdk-engine-codegen marker, found at or above the module root like the skip marker. codegenBuilder moves to golang:1.26-alpine (the pinned dagger.io/dagger needs go >= 1.26.1) with git for `go mod tidy`'s VCS fallback, and to the module level so client and module generation share it; goSDKLibVersion is the engine's beta.11 pin. Four dagger-module.toml fixtures and e2e checks cover the generated shape and self bindings, a nested source, stale-binding removal, regenerating an already generated module as a no-op, go.work enrolment from the root, from inside the module and for two modules sharing one go.work, the marker, and a plain-toolchain build of the output. Signed-off-by: Yves Brissaud <yves@dagger.io>
The in-process checks prove the generated files; this proves they load and run. The sdk-sdk harness (now an e2e dependency, pinned as python-sdk's e2e pins it) installs this SDK into a scratch workspace with a released CLI and `dagger call`s the freshly generated toml fixture on the engine's native Go runtime — the committed-files path every `[runtime] source = "go"` module takes. The fixture is generated in-process and the staged workspace handed to the harness whole; Changeset.after holds only the touched subtree. Signed-off-by: Yves Brissaud <yves@dagger.io>
A second Dang module, go-sdk-runtime, the shape of java-sdk's: codegen is a no-op that returns the module source unchanged, keeping the SDK code-generator contract satisfied, and moduleRuntime builds the module's committed, already-generated source into its entrypoint binary — what the engine's native Go runtime does for a dagger-module.toml module and nothing more. introspectionJson is declared optional and never read: its optionality is what the engine reads (RuntimeTrustsCommittedFiles) to skip codegen at module load. A module missing its generated files is refused with the same actionable message the engine gives, before go build can fail on an undefined dag. A module opts in with `[runtime] source = "github.com/dagger/go-sdk/runtime"` or a local path. targetRuntime stays "go": the engine's builtin routing is untouched, and a ref recorded at init would resolve from the default branch rather than from this working tree. The runtime is registered as a workspace module so `load` validates it in CI, and its build container gets git: `go build` falls back to VCS when the module proxy cannot serve a dependency, the same reason codegenBuilder has it. The fixture points its runtime at runtime/ by relative path and is driven through a released CLI by the sdk-sdk harness, once to run it and once, with dagger.gen.go removed, to prove the runtime refuses to regenerate. Signed-off-by: Yves Brissaud <yves@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings Go module codegen into this repository and adds a small, opt-in, build-only runtime — the same shape python-sdk and java-sdk already went through. The engine's builtin
gorouting andtargetRuntimeare untouched: a module that does nothing special behaves exactly as before.Design:
design/go-module-codegen-and-runtime.md(problem, verified engine constraints, approach, alternatives, testing, risks, patch plan).What changes
Module codegen (dagger-module.toml modules).
Mod.generatenow splits on config format, as python-sdk's does: adagger-module.tomlmodule is generated by this SDK's own engine-free codegen helper; adagger.jsonmodule keeps going through the engine'sgeneratedContextDirectory(the engine regenerates it at load anyway). The helper runs twice around one in-engine merge — pass A bootstraps the module and emits its own types as introspection JSON, the Dang layer merges them into the dependency schema through the engine's publicschema(json:).merge(moduleTypes:, moduleName:)(reachable since v1.0.0-0, the single implementation every SDK shares), pass B renders the bindings from the merged schema. The container works on the module's whole context at/srcin the module's source subpath, exactly like the engine's codegen, so a parentgo.work, areplace ../liband a nestedsourceresolve the same way. Output matches the engine's:dagger.gen.go,internal/dagger/*.gen.go(including the module's own bindings),go.mod/go.sumpinned to the engine'sdagger.io/daggercommit,.gitattributes,.gitignore; stale per-dependency bindings are removed; ago.workthe module was enrolled in is carried back when the caller's cwd can reach it. Regeneration of an already-generated module is a no-op (checked).Escape hatch. An engine-free codegen has no host credentials, so a module with private Go dependencies keeps the engine's codegen with a
.dagger-go-sdk-engine-codegenmarker, found at or above the module root like the skip marker.Opt-in runtime.
runtime/is a second Dang module,go-sdk-runtime, the shape of java-sdk's: a no-opcodegenand amoduleRuntimethat builds the committed, already-generated source into the entrypoint binary — what the engine's native runtime does for a toml module, and nothing more. A module opts in with[runtime] source = "github.com/dagger/go-sdk/runtime"(or a local path). It refuses a module missing its generated files with the same actionable message the engine gives.Helper.
helpers/codegengains the module template set from dagger/dagger@v1.0.0-beta.11 (byte-identical modulo import paths), ago/packagesloader that fails on unresolvable modules but tolerates the type errors body-stripping produces,module-types/generate-modulemodes, and the engine's beta.11 nullable-object returns.Behaviour changes to be aware of
generateClient: optional-object fields now return(*T, error)(nil on null) on engines ≥ v1.0.0-beta.10, matching the engine's own client generator; the extracted generator predated that change. Required so module bindings do not flip signatures between an engine-generated and an in-repo-generated run.go.modfiles get the toolchain's language version (go 1.26) rather than the builder image's patch level; existing clientgo.modfiles are untouched.codegenBuildermoves togolang:1.26-alpine(the pinneddagger.io/daggerneeds go ≥ 1.26.1) and installsgitforgo mod tidy's VCS fallback.GoSdk.generatestages modules sequentially so edits to shared files (a parentgo.work) compose instead of conflicting.Not in this PR
The engine's
core/sdk/loader.gocase sdkGorouting,toolchains/engine-dev/build/sdk.go, and flippingtargetRuntimetoruntime/— that cutover is a separate workstream. Credential forwarding into the engine-free codegen and the runtime (GOPRIVATE, gitconfig, SSH agent) is out of scope; see the marker above.Testing
helpers/codegen: unit + golden tests, including the ported module template tests, loader error-kind gating,syncModReplaceAndTidyseeding/replace copy, VCS file merging,automaticGitignore.e-2-e:*): generated shape and self bindings for a toml module, nestedsource = "src", stale-binding removal, engine-codegen marker, go.work enrolment from the root / from inside the module / for two modules at once, idempotent regeneration, a plain-toolchaingo buildof the output, and — through the sdk-sdk harness with a released CLI —dagger callon the engine's native runtime and onruntime/, plus the runtime refusing a module with missing generated files.sdk-sdk:*black-box checks now exercise the in-repo codegen on scaffolded toml modules end to end (module:loads,chain:*,monorepo:*).All check groups (
e-2-e,sdk-sdk,dang-sdk) were run locally against engine v1.0.0-beta.11 before pushing.