Add models frontmatter field for custom model pricing#38276
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in 7e7a8bd / e8dc1fd.
Usage: models:
providers:
anthropic:
models:
my-custom-claude:
cost:
input: "3e-06"
output: "1.5e-05" |
model-costs frontmatter field for custom model pricingmodels frontmatter field for custom model pricing
|
…smoke-copilot Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in the latest commits:
|
There was a problem hiding this comment.
Pull request overview
Adds support for declaring custom model pricing in workflow frontmatter (models: { providers: ... }) and merging that overlay into the built-in models.json during the activation job so downstream effective-token cost accounting uses the merged catalog.
Changes:
- Propagate a frontmatter
modelsoverlay through compilation (WorkflowData.ModelCosts) and emit it asGH_AW_INFO_MODEL_COSTS. - Merge the overlay with the built-in
models.jsonduring activation and publish the merged file at/tmp/gh-aw/models.json, passed to AWF viaGH_AW_MODELS_JSON_PATH. - Update schema and golden/lock fixtures to include
/tmp/gh-aw/models.jsonin artifacts and reference the newGH_AW_MODELS_JSON_PATH.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/workflow_builder.go | Plumbs frontmatter-derived model pricing overlay into WorkflowData.ModelCosts. |
| pkg/workflow/testdata/wasm_golden/fixtures/smoke-copilot.md | Adds an example models.providers overlay to a fixture workflow. |
| pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden | Updates golden output to upload /tmp/gh-aw/models.json and export GH_AW_MODELS_JSON_PATH to the merged location. |
| pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.golden | Same golden updates for playwright-cli mode. |
| pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.golden | Same golden updates for basic copilot fixture. |
| pkg/workflow/testdata/TestWasmGolden_AllEngines/pi.golden | Same golden updates for pi engine fixture. |
| pkg/workflow/testdata/TestWasmGolden_AllEngines/gemini.golden | Same golden updates for gemini engine fixture. |
| pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.golden | Same golden updates for copilot engine fixture. |
| pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.golden | Same golden updates for codex engine fixture. |
| pkg/workflow/testdata/TestWasmGolden_AllEngines/claude.golden | Same golden updates for claude engine fixture. |
| pkg/workflow/model_aliases_test.go | Adjusts tests to validate models.providers populates ModelCosts and that it’s optional. |
| pkg/workflow/model_aliases_import_test.go | Simplifies import test expectations after removing frontmatter model-alias usage. |
| pkg/workflow/frontmatter_types.go | Replaces legacy frontmatter Models alias map with ModelCosts pricing overlay on the models key. |
| pkg/workflow/compiler_yaml.go | Emits GH_AW_INFO_MODEL_COSTS env var when ModelCosts is present. |
| pkg/workflow/compiler_types.go | Adds WorkflowData.ModelCosts to carry pricing overlay through compilation. |
| pkg/workflow/compiler_orchestrator_workflow.go | Stops passing frontmatter model aliases into alias-map validation. |
| pkg/workflow/compiler_activation_job_builder.go | Ensures /tmp/gh-aw/models.json is included in the activation artifact upload. |
| pkg/workflow/awf_helpers.go | Points GH_AW_MODELS_JSON_PATH at /tmp/gh-aw/models.json (merged output). |
| pkg/workflow/awf_config_test.go | Updates expectations for GH_AW_MODELS_JSON_PATH to reference the merged models catalog. |
| pkg/parser/schemas/main_workflow_schema.json | Defines the models frontmatter schema as a providers-keyed pricing overlay. |
| actions/setup/js/merge_frontmatter_models.cjs | Introduces runtime merge/write of built-in models.json with frontmatter overlay to /tmp/gh-aw/models.json. |
| actions/setup/js/generate_aw_info.cjs | Calls writeMergedModelsJSON() during activation job generation. |
| .github/workflows/workflow-health-manager.lock.yml | Updates lockfile to upload merged models.json and export GH_AW_MODELS_JSON_PATH=/tmp/gh-aw/models.json. |
| .github/workflows/test-workflow.lock.yml | Same lockfile updates. |
| .github/workflows/smoke-opencode.lock.yml | Same lockfile updates. |
| .github/workflows/smoke-copilot.md | Adds an example models.providers overlay to a real workflow. |
| .github/workflows/smoke-ci.lock.yml | Same lockfile updates. |
| .github/workflows/slide-deck-maintainer.lock.yml | Same lockfile updates. |
| .github/workflows/security-review.lock.yml | Same lockfile updates. |
| .github/workflows/release.lock.yml | Same lockfile updates. |
| .github/workflows/refiner.lock.yml | Same lockfile updates. |
| .github/workflows/pr-triage-agent.lock.yml | Same lockfile updates. |
| .github/workflows/poem-bot.lock.yml | Same lockfile updates. |
| .github/workflows/plan.lock.yml | Same lockfile updates. |
| .github/workflows/pdf-summary.lock.yml | Same lockfile updates. |
| .github/workflows/objective-impact-report.lock.yml | Same lockfile updates. |
| .github/workflows/jsweep.lock.yml | Same lockfile updates. |
| .github/workflows/hippo-embed.lock.yml | Same lockfile updates. |
| .github/workflows/gpclean.lock.yml | Same lockfile updates. |
| .github/workflows/firewall.lock.yml | Same lockfile updates. |
| .github/workflows/example-permissions-warning.lock.yml | Same lockfile updates. |
| .github/workflows/daily-regulatory.lock.yml | Same lockfile updates. |
| .github/workflows/daily-max-ai-credits-test.lock.yml | Same lockfile updates. |
| .github/workflows/daily-malicious-code-scan.lock.yml | Same lockfile updates. |
| .github/workflows/daily-credit-limit-test.lock.yml | Same lockfile updates. |
| .github/workflows/daily-compiler-quality.lock.yml | Same lockfile updates. |
| .github/workflows/craft.lock.yml | Same lockfile updates. |
| .github/workflows/code-simplifier.lock.yml | Same lockfile updates. |
| .github/workflows/ci-coach.lock.yml | Same lockfile updates. |
| .github/workflows/brave.lock.yml | Same lockfile updates. |
| .github/workflows/bot-detection.lock.yml | Same lockfile updates. |
| .github/workflows/archie.lock.yml | Same lockfile updates. |
| .github/workflows/agentic-token-optimizer.lock.yml | Same lockfile updates. |
| .github/workflows/agent-performance-analyzer.lock.yml | Same lockfile updates. |
| .github/workflows/ace-editor.lock.yml | Same lockfile updates. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 127/270 changed files
- Comments generated: 3
| if toolsResult.parsedFrontmatter != nil && len(toolsResult.parsedFrontmatter.ModelCosts) > 0 { | ||
| workflowData.ModelCosts = toolsResult.parsedFrontmatter.ModelCosts | ||
| } else if rawModels, ok := result.Frontmatter["models"]; ok { |
| "models": { | ||
| "description": "DEPRECATED: Named model alias definitions with ordered fallback lists, resolved recursively by AWF. Each key is an alias name (use empty string \"\" for the default policy). Each value is an ordered list of vendor/modelid glob patterns or other alias names to try in sequence. Entries defined here are merged on top of the builtin aliases; the main workflow file always wins over imported aliases. Builtin aliases include: sonnet, sonnet-6x, haiku, opus, gpt-5, gpt-5-mini, gpt-5-codex, gemini-flash, gemini-pro, small, mini, large, auto, any, agent, copilot, claude, codex, gemini.", | ||
| "deprecated": true, | ||
| "x-deprecation-message": "Frontmatter 'models' is deprecated and no longer needed. Remove this field from workflow frontmatter.", | ||
| "description": "Custom model pricing data in the same structure as models.json. Merged with the built-in models.json at runtime; frontmatter entries override matching models and fill gaps for unknown models. Useful for custom or private models, or to adjust pricing for effective-token cost accounting.", | ||
| "type": "object", | ||
| "additionalProperties": { | ||
| "type": "array", | ||
| "items": { | ||
| "type": "string", | ||
| "description": "A vendor/modelid glob pattern (e.g. 'copilot/*sonnet*') or an alias name (e.g. 'sonnet') to resolve recursively." | ||
| }, | ||
| "description": "Ordered list of model patterns or alias names to try in sequence." | ||
| }, | ||
| "examples": [ | ||
| { | ||
| "sonnet": ["mygateway/*sonnet-v3*"], | ||
| "": ["sonnet", "gpt-5-codex"] | ||
| "required": ["providers"], | ||
| "properties": { |
| } catch { | ||
| core.warning(`Failed to parse GH_AW_INFO_MODEL_COSTS: ${modelCostsEnv}`); | ||
| } |
|
@copilot add tests for import shared workflows with models and check merging rules |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Implemented in 4bc3390. Added tests for shared workflow imports with
Also wired imported |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot review all comments and address unresolved review feedback. Please summarize any remaining blockers and the next fix to land.
|
|
✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready. |
|
📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing... |
|
🚀 Smoke Antigravity MISSION COMPLETE! Antigravity has spoken. ✨ |
|
🚀 Smoke Pi MISSION COMPLETE! Pi delivered. 🥧 |
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
📰 BREAKING: Smoke Copilot - AOAI (apikey) is now investigating this pull request. Sources say the story is developing... |
|
|
|
Commit pushed:
|
Agent Container Tool Check
Result: 12/12 tools available ✅ — PASS
|
|
Smoke test Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment. Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
💥 Smoke Test: Claude — Run 27281919161Core #1-12: ✅ all passed Overall: PASS ✨ Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
Add Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
There was a problem hiding this comment.
💥 Automated smoke test review - all systems nominal!
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
💥 [THE END] — Illustrated by Smoke Claude · 102.5 AIC · ⌖ 9.05 AIC
| path: | | ||
| /tmp/gh-aw/aw_info.json | ||
| /tmp/gh-aw/model_multipliers.json | ||
| /tmp/gh-aw/models.json |
There was a problem hiding this comment.
🔍 Smoke test: verified this artifact path addition for models.json looks consistent with the new models frontmatter feature.
| GH_AW_MODEL_MULTIPLIERS_PATH="/tmp/gh-aw/model_multipliers.json" node "${RUNNER_TEMP}/gh-aw/actions/merge_awf_model_multipliers.cjs" | ||
| cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json | ||
| export GH_AW_MODELS_JSON_PATH="${RUNNER_TEMP}/gh-aw/actions/models.json" | ||
| export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" |
There was a problem hiding this comment.
🔍 Smoke test: this firewall config region looks structurally consistent — nice.
Workflows currently have no way to declare pricing for custom or private models, nor to override built-in cost entries. This adds a
modelsfrontmatter field (using the same structure asmodels.json) that merges intomodels.jsonat runtime before AWF receives it.Approach
Mirrors the existing
model_multipliers.jsonmerge pattern exactly:modelsoverlay to JSON → emits asGH_AW_INFO_MODEL_COSTSin the activation jobgenerate_aw_info.cjsmerges it over the built-inmodels.json→ writes/tmp/gh-aw/models.json/tmp/gh-aw/models.json; agent job receives it viaGH_AW_MODELS_JSON_PATHFor shared workflow imports, imported
modelsoverlays are merged together first, then the main workflowmodelsoverlay is applied last.Merge semantics: overlay wins at model granularity (whole model entry replaced); unmentioned providers/models preserved from base.
The
modelsfield exclusively accepts the models.json providers structure. The previously deprecated alias-map format is no longer supported.Changes
frontmatter_types.go,frontmatter_parsing.go,compiler_types.go,workflow_builder.go,compiler_yaml.go) — parse and emit the new field;ModelCosts map[string]anynow hasjson:"models,omitempty"and is populated directly byjson.Unmarshal; the deprecatedModels map[string][]stringfield is removedawf_helpers.go—GH_AW_MODELS_JSON_PATHnow points to/tmp/gh-aw/models.json(the merged file) instead of the setup-copied originalcompiler_activation_job_builder.go— include/tmp/gh-aw/models.jsonin activation artifact uploadmerge_frontmatter_models.cjs(new) —mergeModelCosts()+writeMergedModelsJSON(); called fromgenerate_aw_info.cjsmain_workflow_schema.json—modelsproperty requires a top-levelproviderskey (no legacy alias-array alternative)smoke-copilot.md— added samplemy-custom-claudepricing entry to demonstrate and smoke-test the featurepkg/parser/import_field_extractor.go,pkg/parser/import_processor.go,pkg/workflow/workflow_builder.go) — collectmodelsprovider overlays from imported shared workflows and merge them with main-workflow overlay precedencepkg/workflow/model_costs_import_test.go) — add coverage for shared workflow imports withmodels, including merge precedence and preservation rulesUsage
Changeset
modelsfrontmatter field for defining custom model pricing overlays and overriding built-in model costs.✨ PR Review Safe Output Test - Run 27281919161
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.comSee Network Configuration for more information.