Improve CLI workflows for coding agents

[joesobo]

Goal

Make the headless CodeGraphy CLI plus Agent Skill a useful, measurable coding-agent interface: structured JSON, bounded evidence, safe workspace control, truthful relationships, and lower model-token use on real implementation work.

Model tokens per correct task are primary. Correctness is a gate. Calls, tool output, elapsed time, fallback navigation, and indexing cost are diagnostics.

Final selected changes

Agent exploration

• search <pattern> merges live source locations, cached AST Symbols, and indexed Nodes. Sparse natural phrases use deterministic all-term File ranking.
• map <task> builds a bounded personalized File map from independent live terms, selected declarations, source-area diversity, personalized PageRank, and cached typed Relationships.
• query <node> inspects one exact File or Symbol with bounded declarations and incoming/outgoing Relationships.
• Exact Search, Target Query, Path, and targeted Relationship selectors use complete cached types unless explicitly projected; saved Graph View Scope continues shaping broad inventories.
• dependencies, dependents, and path preserve typed direction and bounded JSON.
• Public Batch was removed after agent testing showed that adaptive investigation usually needs to react between results.

Structural graph correctness

• JavaScript-family reexports are explicit reexport Relationships.
• Renamed exports are Alias Symbols.
• Calls through named, aliased, and export-star barrels resolve to implementation Symbols after both full and separate-process incremental Indexing.
• Exact File paths win before redundant Symbol-expanded path variants.

Workspace and automation contracts

• Structured settings, settings get/set/unset safely mutate validated workspace settings and refuse to overwrite malformed persisted JSON.
• Indexing reports indexedFiles, totalFound, limitReached, and the exact maxFiles recovery command.
• plugins list returns structured registry/workspace state.
• Scope mutations return compact partial results.
• Successful non-verbose Indexing keeps stderr clean.
• Plugin parsing honors --; conceptual Query help and JSON/error contracts are documented.

Generalized Agent Skill

The shipped skill explains the Relationship Graph, lifecycle, command semantics, identity, freshness, shaping, pagination, JSON output, and limitations. It does not prescribe a command sequence, call budget, search wording, stopping rule, or conditions under which CodeGraphy is permitted. Agents choose their own strategy.

Benchmark protocol

• Model: openai-codex/gpt-5.6-sol, low thinking.
• Three fresh runs per condition; mean and median both retained.
• Raw arm: no CodeGraphy executable, Graph Cache, skill, repository Markdown, source history, prior session, or shared agent home.
• Treatment: candidate CLI, exact candidate skill, and a freshly prepared per-run Graph Cache. Indexing cost is reported separately.
• Same neutral prompt, historical source, frozen dependencies, tools, and hidden validation.
• Correctness requires untouched external behavior tests, package typecheck, and unchanged manifests.
• A CLI candidate must actually be selected by treatment agents before differences are attributed to it.

The benchmark panel contains a harder Plugin-runtime disposal task and a viable relationship task with independent extension-host and Webview Plugin rejection paths. Untouched parents are red, historical solutions are green, and hidden graders assert behavior rather than historical wording. A direct previous-CLI-versus-candidate A/B follows the required raw comparison when general CLI adoption would otherwise confound attribution.

Experiment scoreboard

Percentages compare the treatment with its fresh paired raw/control condition. “Correct tokens” means mean cumulative model tokens over correct runs only.

Experiment	Correctness	Correct tokens	Calls	Output	Elapsed	Adoption	Decision
Selected Search/Query interface, hard task v10	equal	43.5% fewer	18.8% fewer	35.3% fewer	12.7% faster	4 CG calls	Keep, small-sample evidence
Connected depth-two Search neighborhoods	equal	29.2% more	34.8% more	12.8% more	2.8% faster	4 CG calls	Revert
Explicit reexport graph, selected final round	1 point lower	34.7% more	14.8% fewer	4.0% fewer	3.1% slower	4 CG calls	Keep for structural/incremental correctness, not speed claim
Prescriptive compact skill, earlier pooled clean tasks	equal	43.2% fewer	mixed	lower	mixed	1–2 CG calls	Superseded: workflow steering rejected
Frozen lexical feature + neutral generalized skill	3/3 vs 3/3	9.2% more	30.3% more	13.8% more	7.2% slower	2 CG calls in 1/3	Reject task as graph benchmark
Structural feature + neutral generalized skill	2/3 vs 2/3	42.1% fewer	4.3% fewer	6.2% more	20.1% faster	only failed run used CG	Retain as product baseline; no CLI attribution
Coverage-balanced triage <text> on lexical task	3/3 vs 3/3	21.9% more	41.0% more	13.7% more	6.1% slower	0 CG calls	Revert: no adoption
Evidence-first generalized skill	3/3 vs 2/3	232.2% more	56.1% more	27.0% more	55.6% slower	0 CG calls	Revert
Combined Triage + evidence-first skill	2/3 vs 3/3	12.4% more	31.6% more	21.6% fewer	12.1% slower	1 CG call/run	Revert: adopted but worse successful trajectories
Incoming impact <node> radius	2/3 vs 3/3	161.5% more	60.3% more	28.2% more	65.7% slower	0 Impact; 3 Search calls in 1/3	Revert: useful isolated result, no candidate adoption
Compact neutral skill subtraction	1/3 vs 3/3	40.4% more	25.3% more	21.0% fewer	8.6% faster	4 CG calls in both failures	Revert: adoption without completeness
Existing CLI on relationship fixture	3/3 vs 3/3	14.5% fewer	31.6% more	45.8% fewer	17.5% faster	CG in 3/3	Establish viable relationship benchmark
Personalized Map required raw comparison	3/3 vs 3/3	26.9% fewer	39.7% more	33.1% fewer	10.9% slower	Map in 1/3	Promising, insufficient attribution alone
Personalized Map direct CLI A/B	3/3 vs 3/3	34.7% fewer	34.4% fewer	38.1% fewer	30.5% faster	Map in 2/3	Keep
Personalized Map harder-task confirmation	2/3 vs 2/3	64.0% fewer	8.5% more	38.0% fewer	16.6% faster	Search only	Confirms existing CLI, not Map
Compact Map required raw comparison	3/3 vs 3/3	32.0% fewer	34.0% more	42.5% fewer	21.3% faster	Map in 2/3	Promising, insufficient attribution alone
Compact Map direct A/B	3/3 vs 3/3	73.2% more	10.0% more	84.5% more	45.9% slower	Map in 3/3 both arms	Revert: omitted context caused fallback Search
Fuzzy Symbol fallback	3/3 vs 2/3	1.7% fewer	53.4% more	37.2% more	115.0% slower	0 fuzzy activations	Revert: no candidate adoption

Other measured rejections recorded in ADRs:

• lazy CLI imports;
• BM25 phrase fallback;
• Target Query source embedding;
• generic connected neighborhoods;
• minimal and Symbol-preference skills;
• phrase-ranked per-term source evidence;
• default Leiden community reranking.

Seeded weighted Leiden itself was fast (45.2 ms, modularity 0.864 on 2,729 Files) but improved zero of six localization ranks and degraded two key targets, so no dependency or ranking path was added.

Durable decisions

• ADR 0003: CLI + Agent Skill are the sole agent interface; no MCP runtime.
• ADR 0007: Search and Target Query replace Batch.
• ADR 0008: deterministic phrase ranking, safe settings, indexing recovery, and retrieval rejections.
• ADR 0009: complete targeted graph types and explicit reexport semantics.
• ADR 0010: token-first clean-install evaluation history.
• ADR 0011: agents choose their own CodeGraphy strategy.
• ADR 0012: structural benchmark and tool-adoption gate.
• ADR 0013: personalized Task Map and relationship-task direct A/B.

Validation

Final source validation before the latest documentation-only commit:

• Core: 334 files / 1,443 tests.
• Plugin API: 8/8 tests.
• Release contracts: 43/43.
• Core typecheck/lint and matching Task Map role-module tests.
• Forced uncached workspace build/lint: 29/29 tasks.
• Built CLI Task Map, bound rejection, projection, and target-coverage smoke.
• Forced uncached sequential workspace test: 16/16 tasks.
• Acceptance-spec guard and diff checks.

CI run 30144140076 passed every job on the selected head, including Playwright and native/VSIX targets. Local mutation and Playwright were intentionally excluded from this experiment as requested.

Remaining queue disposition

The current queue is closed rather than padded with duplicate or weakened paths:

• one-hop neighbors duplicate exact query plus dependencies/dependents;
• watcher-assisted queries require a resident daemon, cache locking, and query mutation outside the accepted one-shot CLI architecture;
• default hub/community priors already failed localization screens, and standalone hub inventory has no coding-task adoption evidence;
• conflict/co-change requires a VCS-history contract that clean fixtures intentionally omit;
• generic two-hop inference is invalid across differently typed call/import/inherit/reference semantics.

ADR 0013 records the concrete blockers. These can be reopened only for a new architectural requirement or a viable historical task that naturally exercises the missing evidence.

Artifacts

• /tmp/codegraphy-structural-benchmark/
• /tmp/codegraphy-feature-benchmark/
• /tmp/codegraphy-feature-benchmark-v2/
• /tmp/codegraphy-token-research/
• /tmp/codegraphy-agent-benchmark/
• /tmp/codegraphy-relationship-benchmark/
• /tmp/codegraphy-relationship-benchmark-v2/

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
codegraphy	Ready	Preview, Comment	Jul 25, 2026 4:30am

[changeset-bot]

🦋 Changeset detected

Latest commit: 3b21ba4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@codegraphy-dev/core	Major
@codegraphy-dev/plugin-api	Minor
@codegraphy-dev/extension	Patch
@codegraphy-dev/tldraw	Patch
@codegraphy-dev/extension-plugin-api	Patch
@codegraphy-dev/plugin-godot	Patch
@codegraphy-dev/plugin-markdown	Patch
@codegraphy-dev/plugin-svelte	Patch
@codegraphy-dev/plugin-typescript	Patch
@codegraphy-dev/plugin-unity	Patch
@codegraphy-dev/plugin-vue	Patch
@codegraphy-dev/plugin-particles	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR