llama-cpp-profiler

llama-cpp-profiler is a Linux Rust CLI for empirically finding safe and fast llama-server configurations for local GGUF models. It measures runtime fit and speed—context, throughput, TTFT, memory headroom, and failures—not intelligence, coding quality, or model alignment.

It's a tool i use to optimize how my codex agent onboard a new local model that I want to tune for my machine's specific hardware to run with the highest tok/sec possible and using as much available VRAM as possible (with a 800MB margin).

It is designed primarily for consumer Linux systems with a single GPU. It is not a multi-GPU tuning or placement tool.

The core tool is about llama.cpp server behavior. Client harnesses such as opencode are optional export adapters because they only call the OpenAI-compatible endpoint; they do not choose or load the GGUF.

Quickstart

Install from this repository:

cargo install --path .

Discover local models and inspect one. inspect accepts either a GGUF file or a directory containing GGUFs; when a directory is passed, the largest model GGUF is selected:

llama-cpp-profiler scan ~/Models
llama-cpp-profiler inspect ~/Models/<model-or-gguf> --json

Run a bounded tuning pass and print the best observed configuration plus the exact next serve command:

llama-cpp-profiler tune ~/Models/<model-or-gguf> --preset quick --max-runs 2
llama-cpp-profiler serve ~/Models/<model-or-gguf> --profile interactive-fast --print

Or use the high-level command that tunes and prints the selected server command in one step:

llama-cpp-profiler recommend ~/Models/<model-or-gguf> --preset quick
llama-cpp-profiler recommend ~/Models/<model-or-gguf> --preset quick --agent
llama-cpp-profiler recommend ~/Models/<model-or-gguf> --preset quick --goal generation --agent

Usage

Scan a model store:

llama-cpp-profiler scan ~/Models

On a TTY, scan opens a compact searchable picker. In non-interactive shells, or with --no-tui, it prints a table. Files with mmproj or draft in the name are ignored.

Inspect GGUF metadata:

llama-cpp-profiler inspect ~/Models/<model-or-gguf> --json

This reads only the GGUF header and key/value metadata, then reports architecture, quant, native context, MoE expert counts, chat-template presence, and prior runs.

Tune safely:

llama-cpp-profiler tune ~/Models/<model-or-gguf> --preset standard

Choose what the primary recommendation optimizes for. generation favors output throughput, prompt favors prompt ingest, and balanced is the default:

llama-cpp-profiler recommend ~/Models/<model-or-gguf> --preset quick --goal prompt

All goals first prefer the highest tested KV-cache precision that clears the VRAM/swap safety gates: Q8/Q8, then Q8/Q4, then Q4/Q4. Throughput selects among configurations only after that precision choice. The default free-VRAM floor is 800 MiB and can be overridden with --min-vram-free-mib.

tune starts one llama-server at a time on localhost ports beginning at 18180, runs sanity/output/ingest probes, writes raw artifacts, and stops the server after each candidate. It can promote safer or more aggressive already-planned candidates from observed results, but stays within the selected preset or --max-runs primary-search budget. After that search, the best observed placement with meaningful CPU work triggers up to five additional topology-aware thread refinement runs. Fully GPU-resident selections skip this stage. Tuning does not run a 250k-token prompt. standard and thorough then validate the best observed candidate with one realistic combined request: roughly 25% of context (bounded to 16k–64k input tokens, with output space reserved) and up to 1024 output tokens. This final validation is outside the primary-search and thread-refinement budgets.

Use --confirm-best when repeatability matters. It reruns up to three promising candidates and uses their median throughput within each KV-precision tier before final-stage validation. The selected realistic-validation result retains the repeated-measurement count and confirmed confidence, reducing noise from clock changes, cache warming, and background desktop activity.

The default thinking probe mode keeps the reasoning-oriented baseline: generated server commands and chat probes use --reasoning on, --reasoning-budget 4096, --chat-template-kwargs '{"enable_thinking":true}', --temp 0.6, --top-p 0.95, --top-k 20, --min-p 0.0, --presence-penalty 0.0, and --repeat-penalty 1.0.

By default, tuning starts at the model's native maximum context when GGUF metadata reports it. Use --ctx to cap or explicitly select a lower context. Lower-context fallback candidates are kept in the plan and can be promoted after OOM, timeout, crash, or too-tight runs. Plain tune uses quick and never runs a near-full context probe automatically.

Quick mode skips final-stage realistic validation unless --validate-best is set:

llama-cpp-profiler tune ~/Models/<model-or-gguf> --preset quick --validate-best

The final stage scales its timeout from the selected candidate's short-probe throughput and tries candidates in precision-first order—Q8/Q8, Q8/Q4, then Q4/Q4—with balanced throughput breaking same-precision ties. It advances after a crash, timeout, safety violation, or severe retained-throughput collapse. Early EOS is recorded as incomplete but still passes. Reports show actual prompt/output token counts and retained prompt and generation throughput.

Use --probe-mode generic to omit reasoning-specific server arguments and request fields. Thinking mode remains the default.

For an explicit near-full prompt-ingest check during tuning, opt in with --near-full-ingest. The target is about 94% of the requested context, so a 266k context run targets roughly 250k estimated prompt tokens. Override it with --near-full-target-tokens.

For MoE models where you already know the useful expert-placement boundary, put specific --n-cpu-moe values at the front of the plan:

llama-cpp-profiler tune ~/Models/<model-or-gguf> --preset quick --n-cpu-moe-values 32,31,30 --max-runs 3

Explicit MoE candidates use the common high-throughput shape -ctk q8_0 -ctv q8_0 -b 16384 -ub 4096 and are still subject to the normal safety limits.

Preview the hardware-aware candidate plan without starting servers. --plan prints JSON; --json is accepted only with --plan for compatibility with agent workflows.

llama-cpp-profiler tune ~/Models/<model-or-gguf> --preset quick --plan

Inspect the current profiler, hardware, and server environment:

llama-cpp-profiler doctor --json

Run explicit near-full context:

llama-cpp-profiler fullctx ~/Models/<model-or-gguf> --profile interactive-fast

The default prompt target is 80% of the active profile's server context. Use --target-tokens to request a different target; it is capped below the active context. This is intentionally separate from tune because long prompts can take many minutes and put real pressure on VRAM, RAM, and swap.

Read ranked recommendations:

llama-cpp-profiler report ~/Models
llama-cpp-profiler report ~/Models --agent
llama-cpp-profiler report ~/Models --include-stale

The human report prints comparison tables, native versus validated context, rejected runs, telemetry safety status, candidate coverage, and the next suggested test. --agent emits stable JSON with an explicit agent schema version, model identity, environment validity, telemetry status, selected goal, exact command, metrics, validation, risk, confidence, coverage, failures, stale runs, and next action.

Export managed notes or client labels:

llama-cpp-profiler export ~/Models --markdown --dry-run
llama-cpp-profiler export ~/Models --markdown --opencode ~/.config/opencode/opencode.json --write

Markdown export updates only the managed block between <!-- llama-cpp-profiler:start --> and <!-- llama-cpp-profiler:end --> in RUNNING_NOTES.md. Opencode export is an optional adapter: it only adds labels for a client that points at the running llama-server endpoint and does not affect profiler scoring. Its context limit comes from the selected profile's validated server context. If no export target is selected, export prints the generated Markdown block.

Documentation

Commands

llama-cpp-profiler scan PATH [--no-tui]
llama-cpp-profiler inspect PATH [--json]
llama-cpp-profiler tune PATH [--ctx TOKENS] [--preset quick|standard|thorough] [--goal generation|prompt|balanced] [--probe-mode thinking|generic] [--max-runs N] [--min-vram-free-mib MIB] [--max-swap-delta-mib MIB] [--port-start PORT] [--gpu-index INDEX] [--n-cpu-moe-values VALUES] [--near-full-ingest] [--near-full-target-tokens TOKENS] [--validate-best] [--confirm-best] [--plan] [--json]
llama-cpp-profiler recommend PATH [--ctx TOKENS] [--preset quick|standard|thorough] [--goal generation|prompt|balanced] [--probe-mode thinking|generic] [--max-runs N] [--profile ID] [--port PORT] [--min-vram-free-mib MIB] [--max-swap-delta-mib MIB] [--port-start PORT] [--gpu-index INDEX] [--n-cpu-moe-values VALUES] [--near-full-ingest] [--near-full-target-tokens TOKENS] [--validate-best] [--confirm-best] [--agent]
llama-cpp-profiler fullctx PATH [--profile ID] [--target-tokens TOKENS] [--ctx TOKENS] [--probe-mode thinking|generic] [--min-vram-free-mib MIB] [--max-swap-delta-mib MIB] [--port-start PORT] [--gpu-index INDEX]
llama-cpp-profiler report PATH [--agent] [--include-stale]
llama-cpp-profiler serve PATH [--profile ID] [--port PORT] [--print] [--allow-stale]
llama-cpp-profiler doctor [--json]
llama-cpp-profiler export PATH [--markdown] [--opencode PATH] [--dry-run] [--write]

Defaults

• Tune port range starts at 18180; serve defaults to 18080.
• If GGUF metadata provides native context, that is the default requested context; --ctx caps it or supplies the fallback when native context is absent.
• Candidate plans try that native/explicit context first. Lower-context fallbacks are available for adaptive promotion after failed or too-tight runs.
• Safety defaults are --min-vram-free-mib 800 and --max-swap-delta-mib 1024.
• quick runs at most 6 candidates; standard runs at most 16; thorough runs at most 48.
• These limits apply to the primary placement search. A CPU-participating selected candidate can add up to five thread-refinement runs: llama.cpp defaults, half/all physical cores, physical/logical splits, and all logical cores. Duplicate pairs are removed on smaller systems, and an explicit pair replaces the default only at a measured balanced-throughput improvement of at least 3%.
• standard and thorough add final-stage realistic validation after thread refinement. quick adds it only with --validate-best. Failed candidates fall back in KV-precision order, with balanced score breaking same-precision ties; a failed validation disqualifies its short-probe baseline from recommendations.
• Plain tune and recommend default to quick; standard and thorough are explicit deeper modes.
• --goal balanced is the default. It places the matching balanced, interactive-fast, or prompt-replay profile first in tune output; recommend uses that primary profile unless --profile is explicit.
• --confirm-best reruns up to three safe, promising candidates, uses median measurements to select final-stage validation candidates, and preserves the repeated-measurement count in the eventual recommendation.
• Confidence is provisional for smoke evidence, benchmarked after standard/realistic validation, confirmed after repeated measurements, and full-context-validated after fullctx.
• thinking is the default probe mode; generic omits reasoning-specific arguments and request fields.
• quick runs are labeled smoke; normal standard and thorough probes are labeled standard-ingest; final-stage runs are labeled realistic; fullctx runs are labeled fullctx.
• --n-cpu-moe-values is a comma-separated MoE-only override that prepends explicit partial-MoE candidates, for example 32,31,30.
• fullctx targets 80% of the active profile's server context by default. Explicit --target-tokens values are capped below that context. tune and recommend can run one optional near-full ingest probe with --near-full-ingest.
• Stale or legacy runs are excluded from best-observed-profile selection by default.
• Export is dry-run unless --write is present.
• LLAMA_SERVER=/path/to/llama-server overrides the executable used for tune, fullctx, serve, and doctor.

Artifacts

Each model gets an isolated state directory (flat GGUFs in one directory cannot overwrite one another):

.llama-cpp-profiler/
  models/<model-path-hash>/
    manifest.json
    recommendations.json
    runs/<run-id>/
      command.sh
      server.log
      telemetry.jsonl
      request.json
      response.json
      result.json
    reports/latest.md

request.json and response.json each contain a probes array with all probe requests/responses for the run. result.json stores the exact command, GGUF metadata, probe summaries, parsed llama.cpp timing lines, client-observed TTFT, telemetry peaks/minimums, outcome, environment snapshot, validation level, and paths to raw artifacts. Final-stage results also store their baseline run, target and actual token counts, retained prompt/output throughput ratios, and whether generation ended before the 1024-token ceiling.

Profiles

Recommendations are the best observed configurations derived from current-environment passed runs and safety limits:

• interactive-fast: highest-precision safe configuration, with generation throughput breaking same-precision ties.
• interactive-safe: the same precision-first selection under the configured free-VRAM floor.
• prompt-replay: highest-precision safe configuration, with prompt throughput breaking same-precision ties.
• balanced: highest-precision safe configuration, with balanced throughput breaking same-precision ties.

Requirements

• Linux
• llama-server in PATH, or LLAMA_SERVER=/path/to/llama-server
• NVIDIA telemetry through nvidia-smi is optional. Without it, VRAM safety is unknown, and a profile cannot qualify as interactive-safe.
• Optional: a client such as opencode that talks to http://127.0.0.1:18080/v1

License

MIT. See LICENSE-MIT.

This beta supports Linux and llama-server, with consumer single-GPU setups as its primary target. Multi-GPU hardware may be detected, but the profiler does not model or optimize placement across GPUs; non-NVIDIA telemetry and backend-specific sweeps are limited or deferred. fullctx and the standard/thorough presets can be expensive and may pressure VRAM, RAM, and swap.

See docs/schema.md and docs/profiling-strategy.md for data contracts and tuning logic.