Research brief · 2026-08-30

Putting a coding model on the RTX 3090

What is actually worth running on a dedicated 24 GB Ampere card in late August 2026, measured rather than advertised, and how it fits this specific workstation.

GPU 0 · RTX 3090 · sm_86 · 24,576 MiBdriver 595.84 · CUDA 13.2host RAM 45 GBfree disk 126 GB

Recommendation

Start with patched llama.cpp behind llama-swap, serving Qwen3.8-27B and Tiel-Coder side by side. Add the vLLM lane later only if you want roughly 2× decode and can live with a research-grade fork.

Qwen3.8-27B is the strongest open coder that fits: 16/25 on SWE-bench-Live vs 12/25 for the next local model, Terminal-Bench 2.1 at 73. It is dense, so it decodes at 57–80 tok/s on llama.cpp and 104–133 on vLLM.
Tiel-Coder-35B-A3B (a requant of Ornith-1.5, 3 B active) is the fast one: 149 tok/s decode and 2,790 tok/s prefill on the same card, 8.6 min per SWE attempt vs 50 for Qwen. Weaker on the hardest tasks, better in multi-turn repair.
Both fit at 131 K context with vision on one 3090 — but only with the desktop off the card. Both monitors are currently cabled to the 3090, and the VS Code instance GNOME restored at login is rendering on it (session restore bypassed yesterday's pin).
The llama.cpp lane needs a patched build: stock llama.cpp decodes Qwen3.8 at ~34 tok/s once context is deep; the 0x7067 patch set holds 52–63 tok/s at 50–80 K. The patches are not upstream.

Your machine, checked

Everything below was read off the box this session. Three items change what you should do next.

Item	Found	Bearing on the build
Monitors	DP-1 and HDMI-A-2 both connected to the 3090 (01:00.0)	move cables The compositor must scan out through the 3090 while a display hangs off it. Move both to the 5060.
VS Code GPU process	Main process launched at login as `/usr/share/code/code ~/Desktop/repo_gdx_dispatch` with no render-node flag → GPU process on renderD129 (3090), 88 MiB	pin bypassed The desktop-file and `~/bin/code` pins are correct (by-path 02:00.0), but GNOME session restore relaunched VS Code straight from `/usr/share/code/code`, so it fell back to Mutter's primary — the 3090. Fixing the primary GPU fixes this for free.
gnome-shell	323 MiB on the 3090	udev rule Mutter picks the 3090 as primary. One udev rule flips it to the 5060 (see Freeing the card).
PCIe link	Gen3 ×8 (board splits ×16 with the 5060)	fine Only affects model load time, not decode. Batch-1 decode is VRAM-bandwidth bound.
Host RAM	45 GB total, 31 GB available	ok vLLM's 15.9 GiB checkpoint load and llama.cpp's 8 GB host prompt cache both fit. syv-ai's one RAM failure report was a 16 GB box.
Driver / toolkit	595.84, CUDA 13.2, nvidia-container-toolkit 1.20, CDI configured	ok syv-ai image needs ≥ 580. No `nvcc` on the host; both lanes are Docker builds that bring their own.
Power limit	350 W default (range 100–400)	set 250 W Every syv-ai number was taken at 250 W. A 4090 at 450 W is only +1.9% faster — the extra watts buy nothing at batch 1.
Disk	126 GB free; 29 GB reclaimable Docker build cache	ok Lane A ≈ 50 GB, Lane B adds ≈ 30 GB.
Port 11430	nothing listening	free `plan_deep/dispatchers.py` still expects an OpenAI-compatible server here.

Model candidates

Only two open models released since you tore the old stack down are both (a) better than Gemma-4-31B at code and (b) runnable on Ampere in 24 GB. The rest of what is trending on Hugging Face is either too big (GLM-5.3-Flash 321 B, Qwen3.8-Flash-Next 180 B, DeepSeek-V4) or ships in NVFP4/FP8, which the 3090 cannot execute natively.

Model	Shape	SWE-bench-Live 25	Terminal-Bench 2.1	SWE-bench Verified	Vision	Fit on 3090
Qwen3.8-27B Qwen · 2026-08-05 · Apache-2.0	27.8 B dense, hybrid Gated-DeltaNet (16 of 64 layers attention)	16 / 25 50 min/attempt	73.0	— SWE-Pro 61.7	yes	UD-Q4_K_XL 17.6 GB + MTP + mmproj → 131 K ctx
Tiel-Coder-35B-A3B = Ornith-1.5-35B-A3B requant · MIT	36 B MoE, 3 B active, 256 experts, hybrid GDN (10 of 40 layers attention)	12 / 25 8.6 min/attempt	67.8 Ornith card	79 Ornith card	yes	UD-Q4_K_S 20.9 GB → 131–184 K ctx with q8_0 KV
KAT-Coder-V2.5-Dev Kwaipilot · 2026-07-23	35 B MoE, 3 B active (Qwen3.6-35B-A3B base), text only	—	41.0	69.4	no	IQ4_XS 18.8 GB
Gemma-4-31B Google · March, no update since June	31 B dense	—	32–42	52–61	yes	QAT q4_0 ~18 GB

SWE-bench-Live column is from peculiar-ragdoll's Tiel card (same harness, all four local builds). Terminal-Bench and SWE-Verified are each vendor's own numbers; compare within a column loosely. Gemma-4 numbers are the range reported by KAT and Ornith in their comparison tables. Gemini 3.7 Flash (Aug 13) is API-only — nothing from Google runs locally that is newer than Gemma 4.

Why both, not one

They are opposites on the axis that matters for an agent loop. Qwen3.8 thinks longer and solves harder tickets; Tiel answers 2–5× faster and never replies with prose instead of code (1 of 164 vs 15 of 164 in the 0x7067 HumanEval run). llama-swap hot-swaps them on one card in ~10 s, so the choice becomes per-request.

Lane A · patched llama.cpp + llama-swap

Reference: 0x7067/qwen38-27b-rtx3090-llamacpp. A Dockerfile pins llama.cpp at 4df29be4f (2026-08-16), applies eight Ampere-specific CUDA patches, and ships llama-swap. Independently, lyth.de ran twelve configurations on a 3090 and landed on the same shape.

Measured decode, Qwen3.8-27B UD-Q4_K_XL, 131 K context resident

Build	Speculation	short prompt	~7 K prose	50–80 K agentic	Source
stock, no MTP	none	40.3	38.8 @16 K	34.0 @64 K	hardware-corner
stock b10615	built-in MTP, n=4	54.1	—	34.5	lyth.de
patched b10454	external Q4_0 drafter, n=3	57.3	—	58.6 @81 K	lyth.de (adopted)
patched, pin 4df29be4f	d48k drafter, n=5	75.7	68.2	62.9 @53 K	0x7067
stock	DFlash2 Q8 draft	59.9	—	—	z-lab discussion (10 K-token coding outputs)

The patches that matter are 0003 (a GQA-batched flash-attention kernel for quantized KV on cc < 8.9 — upstream had zero coverage for this exact head shape) and 0004 (an env cap on the MMVQ→MMQ crossover). Both were submitted upstream; neither is merged, and the repo measured llama.cpp master as −5.6% short / −1.5% long versus their pin.

VRAM budget, 131 K context (0x7067 prod profile)

weights 16,750MTP drafter 760mmproj 890KV + buffers 4,500desktop 550

022,892 MiB measured at 131 K + 550 desktop = 23,44024,576 MiB

1.1 GB of headroom with the desktop on the card, 1.7 GB without. Tiel-Coder UD-Q4_K_S at 131 K with K=V=q8_0 and vision measured 22,694 MiB; at 184 K, 23,404 MiB.

What you get

Native /v1/messages (Anthropic) and /v1/chat/completions on one port; llama-swap hot-swaps Qwen ↔ Tiel.
Host-RAM prompt cache: repeat a 54 K prompt in 0.26 s, append a turn in 1.4 s, vs 56 s cold. This is what makes agent loops feel fast, more than decode.
Vision (paste a screenshot of a broken page).
Cold load ~10 s. No compile step. Nothing to JIT at first request.
Prefill ~1,300 tok/s.

What you pay

Decode is 57–80 tok/s for Qwen3.8 — about half of the vLLM lane.
You are running a pinned commit with vendored patches. Upgrading llama.cpp means re-applying or waiting for the upstream PR.
One slot at 131 K. --parallel 2 pushed peak to 23,942 MiB, over the ceiling.
--ctx-checkpoints must stay at 32 on a hybrid model or the prompt cache silently re-prefills from zero.

Lane B · vLLM W4A16 (syv-ai)

Reference: syv-ai/qwen38-27b-rtx3090 (894 stars, pushed 2026-08-27). vLLM 0.27.1 with ~a dozen patches, an AutoRound W4A16 body requantized on first boot, GPTQ-int4 lm_head and MTP module, FP8 KV via FlashInfer, optional DFlash2 block drafter. Prebuilt 9.5 GB image on ghcr.

Profile	Context	C1 decode	at 60 K	at 100 K	4 concurrent	Stability
`CTX=fast`, MTP-4, bf16 KV	64 K	118–121	—	—	~270	stable
`CTX=fast SPEC=dflash2`	64 K	132–133	—	—	~279	one-user mode
`CTX=long`, MTP-3, FP8 KV (0x7067 deployment)	140–150 K	104	95.7	70.4	358	Xid 31 every ~40 h
same, MTP-4	150 K	faster	—	—	—	IMA crash; no fix through 0.28.0

All at 250 W. Prefill ~1,210 tok/s single-user at 1 K, ~1,000 at 100 K, so a 100 K cold prompt is ~100 s to first token; prefix cache brings a second turn on a 25 K document to 0.56 s. Quality cost of the whole quant stack: IFBench 78.3 vs 79.5 BF16, perplexity +0.6%, GSM8K 96.5%.

What you get

Roughly 2× the decode of Lane A at the same context; DFlash2 reaches 260–380 tok/s when the answer quotes its prompt (edits, refactors).
Continuous batching: four agents at once at 358 tok/s aggregate.
/v1/messages exists (vllm/entrypoints/anthropic/), plus qwen3_coder tool parser and qwen3 reasoning parser.

What you pay

Cold start 4–5 min (108 s warm). Every restart is a coffee break.
Pinned to vLLM 0.27.1 + patches; the maintainer says re-apply after upgrades. The k=4 FlashInfer crash is still open on 0.28.0.
Recurring Xid 31 MMU fault under CTX=long at ~28 K context (issue #34, same address twice, auto-recovers on restart policy).
Runs at GPU_UTIL 0.93–0.95 → 22.9–23.3 GB claimed at boot. The desktop must be off the card or the free-memory gate fails.
One GPU, one owner: llama-swap must be unloaded before this starts. Only Qwen3.8 — vLLM rejects hybrid-GDN GGUFs, so Tiel cannot live here.

Ampere-specific traps

Things that are true on this card and wrong on a 4090/5090, or that cost someone a day in the reference repos.

NVFP4 and FP8 weights are not for you. Qwen3.8-27B-NVFP4 (2 M downloads) needs Blackwell FP4 tensor cores; Qwen3.8-27B-FP8 (4.6 M) has no native FP8 GEMM on sm_86. Use GGUF Q4/Q5 or W4A16/AWQ-INT4 (Marlin). FP8 KV cache is fine — it is a storage format, handled in FlashInfer.
Mixed K/V cache types kill flash attention in llama.cpp. fattn.cu returns no kernel when K and V types differ. This is exactly why Tiel's shipped K=q8_0 V=q4_0 config prefilled at 188 tok/s and the symmetric q8_0/q8_0 config prefills at 2,792. llama-bench still prints fa = 1.
For Qwen3.8 on llama.cpp, q4_0 KV beats q8_0 and f16 on Ampere — q8_0 rides the same vector path with 1.9× the bytes; f16 is 8.4 GB at 131 K and does not fit. Tiel is a different head shape and measured fine at q8_0. Measure per model.
IQ4_XS is not a free lunch. 8% faster and 2.4 GB smaller, but KL divergence to Q6_K is 1.4× that of Q4_K_L — across two independent imatrix builds, so it is structural. UD-Q4_K_XL is the quality pick at 3% speed cost.
Sub-4-bit "GDN-aware" quants (Ridge 3.7 bpw) were 8.5× worse KLD and slower. Fewer bytes stops predicting faster below 4-bit K-quants.
FlashInfer JIT-compiles its FP8-KV kernel at first request. The server health-checks green, then every request fails if nvcc is missing. The syv-ai image includes it; a hand-rolled venv must too.
The 250 W limit is a host setting (nvidia-smi -pl), not persistent across reboots, and containers cannot set it.
Marlin repack CUDA fault that wedges the card until reboot — reported on sm_80 (CMP 170HX) with a w8a16 target; not reproduced on sm_86 with the shipped W4A16. Stay on the shipped target.

Wiring it into what you already run

OpenAI-compatible callers (works today)

~/.claude/tools/plan_deep/dispatchers.py — gemma helper at localhost:11430/v1, default model gemma-4-26b-a4b. Serve on 11430 and either alias that name in llama-swap or change the default.
gdx_dispatch tools/pre_commit_fast_reject.py — fails open today; lights up on the same port.
/redteam skill — was Qwen2.5-Coder on vLLM; a Qwen3.8 endpoint is a drop-in.
Any OpenAI-SDK tool: Aider, Continue, OpenCode, n8n.

Claude Code (technically works, officially unsupported)

Claude Code speaks Anthropic Messages to ANTHROPIC_BASE_URL with ANTHROPIC_AUTH_TOKEN. llama-server and vLLM both implement /v1/messages (+ count_tokens on llama.cpp).
Anthropic's docs: it "doesn't support routing Claude Code to non-Claude models through any gateway." Expect adaptive-thinking and beta-header fields to be ignored or rejected; CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 quiets most of it.
Model discovery only lists ids containing "claude" — set ANTHROPIC_MODEL explicitly instead.
Realistic role: a local subagent for reviews, test generation, and bulk edits — not the main driver. Ornith publishes Terminal-Bench "(Claude Code)" harness numbers, so these models are trained for that loop.

Freeing the card

Three changes take the desktop's 411 MiB (and, more importantly, its contention) off the 3090. Do them before either lane; Lane B's memory gate will fail otherwise.

Move both monitor cables to the 5060

DP-1 and HDMI-A-2 are on the 3090 today. A display that stays on the 3090 forces Mutter to keep a scanout path there even after the primary changes.

hands-on · 2 min

Tell Mutter the 5060 is primary

# /etc/udev/rules.d/61-mutter-preferred-primary-gpu.rules
SUBSYSTEM=="drm", ENV{DEVTYPE}=="drm_minor", ENV{DEVNAME}=="/dev/dri/card[0-9]", \
  SUBSYSTEMS=="pci", ATTRS{vendor}=="0x10de", ATTRS{device}=="0x2d05", \
  TAG+="mutter-device-preferred-primary"

0x2d05 is the GB206 / RTX 5060 from lspci -nn. Then log out and back in (or reboot).

config · 5 min + relogin

Relaunch VS Code through the pinned launcher

The running instance was restored by GNOME at login without the --render-node-override flag, so it is on the 3090 right now. Once the udev rule makes the 5060 primary, an unflagged launch lands on the 5060 anyway; until then, quit and relaunch via the dock entry or ~/bin/code. Check with nvidia-smi --query-compute-apps=pid,used_memory --format=csv — nothing should be on GPU 0.

2 min

Pin 250 W

sudo nvidia-smi -i 0 -pl 250      # not persistent; wrap in a systemd oneshot

Batch-1 decode is bandwidth-bound; the reference numbers were all taken at 250 W and a 450 W 4090 only gained 1.9%. Cooler, quieter, same speed.

config · 5 min

Build plan

Lane A image

Clone 0x7067, docker build (CUDA 12.8 devel base, arch 86, patches applied, llama-swap v230 bundled). Builds llama-server, llama-bench, llama-perplexity, test-backend-ops.

~25 min build · 8 GB image

Weights

unsloth/Qwen3.8-27B-GGUF: UD-Q4_K_XL (17.6 GB) + MTP/mtp-Q4_0 (1.4 GB) + mmproj-BF16 (0.9 GB). peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP: MTP-UD-Q4_K_S (21.8 GB, MTP head embedded) + its mmproj.

~42 GB · 10–20 min on your link

llama-swap on :11430

Two backends, one resident at a time, TTL unload. Qwen: 131 K single slot, -ctk q4_0 -ctv q4_0, --spec-type draft-mtp -md mtp-Q4_0 --spec-draft-n-max 3, --jinja, --reasoning-format separate, keep --ctx-checkpoints 32. Tiel: 131 K, -ctk q8_0 -ctv q8_0 (symmetric!), embedded MTP. Alias gemma-4-26b-a4b → Tiel so plan_deep works unchanged.

~1 h incl. smoke tests

Verify

llama-bench at depth 0 / 16 K / 64 K for both; watch nvidia-smi peak stays under 23.5 GB; hit /v1/messages and /v1/chat/completions with a tool call; time a second-turn prompt-cache hit.

~30 min

Optional · Lane B

Only after A is boring. docker compose --profile single up from syv-ai with SPEC=dflash2 PREFIX_CACHE=1, stop llama-swap first. Compare on your real prompts, not the benchmark ones.

~1 h + 20 GB requant + 9.5 GB image

Before starting

You wrote in June that you were "moving off local LLM inference entirely." The models got materially better in August, but the reason may not have changed. If this is a two-hour experiment, do Lane A only and leave the vLLM lane alone.

Sources

Qwen/Qwen3.8-27B — model card, benchmark tables, architecture
unsloth/Qwen3.8-27B-GGUF — quant sizes, MTP and mmproj files
ornith-ai/Ornith-1.5-35B-A3B — Ornith 1.5 card and config
peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF — SWE-bench-Live comparison of local builds
Kwaipilot/KAT-Coder-V2.5-Dev — KAT benchmark table incl. Gemma-4-31B
0x7067/qwen38-27b-rtx3090-llamacpp — patches, PERFORMANCE.md, quant-selection.md, vllm-companion.md, tiel-vs-qwen
syv-ai/qwen38-27b-rtx3090 — README, docs/docker.md, start_qwen.sh, issues #27 #34 #39
Mateleo/qwen3.8-3090 — first-run llama.cpp setup and defaults
lyth.de — twelve configurations on a 3090
hardware-corner — Qwen3.8 27B VRAM by context
z-lab DFlash2 discussion #9 — Q4 vs MTP vs DFlash2 on a 3090
unsloth discussion #68 — 150 K context command lines
llama.cpp server README — Anthropic endpoints, --spec-type draft-mtp
vllm/entrypoints/anthropic — Messages API router
Claude Code gateway protocol and connect — env vars, support stance
mutter/doc/multi-gpu.md — preferred-primary udev rule
Gemini API changelog — no Gemma releases June–August