Test report · generated 2026-08-30 09:24

Three vLLM models on one RTX 3090, measured

Qwen3.8-27B, Ornith-1.5-35B-A3B (Tiel's weights) and DiffusionGemma-26B-A4B, each booted through llama-swap on this workstation, timed with the same harness, and judged by whether the code they returned actually runs.

RTX 3090 · 24,576 MiB · 250 W · desktop moved to the 5060llama-swap v251 · 127.0.0.1:11440vLLM 0.28.0 (Ornith, DiffusionGemma) · syv-ai 0.27.1 fork (Qwen)

Scoreboard

Qwen3.8-27B

PASS
syv-ai vLLM 0.27.1 fork · W4A16 fast variant · DFlash2 (7 drafts) · 64 K ctx · bf16 KV
157–198tok/s decode
1,180tok/s prefill
22,897 MiBVRAM serving
external code tests 14/24 over 3 sample(s), thinking off

Ornith-1.5-35B-A3B

PASS
stock vLLM 0.28.0 · AutoRound int4 + MTP k=2 · 32 K ctx · language-model-only
204.6tok/s decode
6,775tok/s prefill
22,229 MiBVRAM serving
external code tests 18/32 over 4 sample(s), thinking off

DiffusionGemma-26B-A4B

PASS with workarounds
stock vLLM 0.28.0 · AWQ-INT4 symmetric g32 · block diffusion, canvas 256 · TRITON_ATTN
423.3tok/s decode
2,003tok/s prefill
23,535 MiBVRAM serving
external code tests 16/24 over 3 sample(s), thinking off
Decode = server-reported completion tokens ÷ streaming time after first token, on a 100-token coding prompt. Prefill = prompt tokens ÷ time-to-first-token on a 6.1 K-token prompt. Code tests: the model's own pytest cases, and an 8-case external set it never saw, both executed.

Verdict

What to run, and how

Qwen3.8-27B is the coder. With thinking off and a 4 K budget, or thinking on with reasoning_effort=low, it was the only model to pass every case — and it did so on the first sample. It is the slowest to prefill (1,180 tok/s) and the slowest to boot, but the prefix cache makes multi-turn work fine.

Ornith-1.5-35B is the fast lane: 200 tok/s, 6,800 tok/s prefill, 0.3 s cached turns. On this quote-heavy prompt its code was inconsistent (4/8, 6/8, one that does not compile). Good for review passes, bulk edits, and anything where speed matters more than the last 20% of correctness.

DiffusionGemma-26B is an experiment that works: 300–420 tok/s of real code at 6/8, but it needs the Triton attention backend, rejects sampling parameters, and gets little from the prefix cache. Keep it for drafts and for watching where diffusion LMs go.

Small round: none of the 4–9 B models beat the big three on this prompt. Gemma-4-12B (official QAT int4) is the one worth keeping — 6/8 twice, compact, tools and thinking behave. The bf16 9 B models are bandwidth-bound at 44 tok/s (slower than the 27 B W4A16 with a drafter) and Ornith-9B, despite its agent-harness benchmarks, scored 0/8 single-shot. The 4 B class loops and mis-quotes; quantizing it makes it 2.3× faster, not more correct.

Two rules for all of them on this card: never leave Qwen-family thinking at default effort in an agent loop (both Qwen and Ornith spent 8 K tokens deliberating over a 30-line parser and never answered), and give code replies at least 4 K tokens — three "failures" in the first pass were my own truncation.

Speed

modeldecode, short (tok/s)decode at 6 K ctxprefill (tok/s)cached 2nd turn TTFTtool callAnthropic API
Qwen3.8-27B157–198159.31,1800.77 sOK (qwen3_coder parser)OK
Ornith-1.5-35B-A3B204.6138.06,7750.31 sOKOK
DiffusionGemma-26B-A4B423.3382.42,0031.87 sOK (gemma4 parser)OK
Reading the numbers

Ornith and Qwen are autoregressive with speculative decoding, so tok/s moves with draft acceptance and with content (code accepts better than prose). DiffusionGemma denoises a 256-token canvas at a time; its 300–420 tok/s is real, but a 12-token answer still costs a full canvas, and its prefix cache does little because the canvas is re-encoded each block. Thinking-on runs report tok/s over reasoning tokens.

Quality of returned work

Same prompt to every model: write parse_door_size(s) -> (width_in, height_in) for forms like 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84", plus eight pytest cases. The reply is extracted and executed twice — against the model's own tests, then against an external set. A tool-call check (correct function and argument) and an Anthropic-format request round it out.

Qwen3.8-27B PASS

thinking off
sample 1external 8/8own 8/8all external cases pass159 lines
sample 2external 6/8own 5/8fails: 120", 192"134 lines
sample 3does not run ^ E SyntaxError: unexpected character after line continuation character !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
returned code, first 42 lines
```python
import re
from typing import Optional

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parse garage-door size strings into (width_in, height_in) in inches.
    
    Supported formats include:
    - "16x7" (feet)
    - "16' x 7'" (feet)
    - "16 ft by 7 ft" (feet)
    - '192" x 84"' (inches)
    - Mixed forms like "10'6\" x 7'0\"" (feet and inches)
    
    Returns:
        Tuple of (width_in_inches, height_in_inches)
    """
    s = s.strip().lower()
    
    # Try to parse using a general regex that captures numbers and units
    # We'll look for patterns like: number(unit) [separator] number(unit)
    
    # First, try to detect if the primary unit is inches (no feet, just inches)
    # or feet (default)
    
    # Let's parse tokens: find all number+unit pairs
    # Units: ft, ', foot, feet, in, ", inch, inches
    # Separators: x, by, *
    
    # Remove common separators and normalize
    # Let's use regex to find two dimension components
    
    # Pattern for a single dimension: 
    # Can be: number(ft/in) or number' number" (mixed) or just number (assume ft unless specified as inch)
    
    # Let's split on 'x', 'by', '*' etc.
    # Replace separators with a single delimiter
    s_norm = re.sub(r'\s*(x|by|\*)\s*', '|', s)
    
    parts = s_norm.split('|')
    
… (118 more lines)
thinking on · default effort
sample 1does not run ^^^ E SyntaxError: invalid syntax !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
returned code, first 42 lines
We need answer user's request: "Write a Python function `parse_door_size(s: str) -> tuple[int, int]` that parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"" into (width_in, height_in) in inches. Handle feet, inches, and mixed forms. Include 8 pytest cases. Reply with code only."

Need provide code only. Need think details. Function parse string into inches. Need support feet, inches, mixed forms. Examples: "16x7" likely feet? Garage-door sizes: "16x7" means 16 ft x 7 ft? Usually width and height in feet. "16' x 7'" feet. "16 ft by 7 ft". "192\" x 84\"" inches. Handle mixed forms e.g., "16' 6\" x 7' 2\""? Mixed forms could be "16' 6 x 7' 2"? Need parse both width and height as either integer/inches or feet or feet+inches, with units optional. Could implement robust regex.

Need include 8 pytest cases. Reply code only, likely full code with function and tests. Must be valid Python. Need decide behavior for ambiguous no unit: "16x7" likely feet if both numbers < some threshold? Garage door sizes "16x7" are feet. "192x84" likely inches. We need handle feet, inches, and mixed forms. Without units maybe infer: if value >= 24? inches? But 16x7 likely feet => 192,84. If no unit and numbers small (<12 maybe inches? But height 7 could be feet). How infer? Could treat unitless as feet if values seem like feet (<=12? but width 16 >12). Garage doors: width 7-24 ft, height 6-14 ft. Inches can be 84-192. So no units could be feet if both values < 15? 16 >15? Hmm 16x7: 16 ft, 7 ft. Inches 192x84. Need infer no-unit: maybe if any number >= 24, inches; if both <= 12, feet? But 16 is >12 but likely feet. Could use threshold: if both numbers < 30? feet? But 24 ft doors? 24 inches? Mixed? Need robust for typical garage door.

Spec: "parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"" into (width_in, height_in) in inches. Handle feet, inches, and mixed forms." No explicit ambiguous no unit inference. Could require units for inches; unitless default to feet? But "192 x 84" perhaps should be inches? They only show "192\" x 84\"" with quotes, not unitless. If unitless default feet, "192x84" -> 2304,1008, not inches. But might be desired if no units: numbers large => inches. We can infer.

Need produce only code. Need include pytest cases. Could define parse_door_size(s): normalize, replace commas? Use regex to find two measurements separated by x, X, by, space. Each measurement can be e.g.:
- "16"
- "16'"
- "16 ft"
- "16 feet"
- "16'"
- "16'6" (16 feet 6 inches) maybe no inch mark? 
- "16' 6\""
- "16 ft 6 in"
- "16 ft. 6 in."
- "16'6\""
- "16' 6''" (double prime for inches)
- "16"
- "192\""
- "192 in"
- "192 inches"
- "192"

Need return tuple[int,int]. Probably if missing units infer. Mixed forms: one part feet, another inches? e.g. "16 ft x 84 in". Or combined feet+inches for each dimension.

Need design regex. Simpler: split string on separators: x, X, by, ' x ', etc. But if measurement includes space like "16 ft 6 in" contains spaces. Need split into two tokens at separator. Use regex split: `re.split(r'\s*(?:x|by|×)\s*', normalized)`? But "16' x 7'" ok. "16 ft by 7 ft" ok. If string has no x but just space? "16' 7'"? Could handle if not split. But separator may be `x` or `by` or `×` or comma? We'll support x/X/×/by.

Normalize: lower? Need preserve quotes. Replace unicode quotes? Could map: `s = s.replace('×', ' x ').replace('by', ' x ')`? But unit "by" only separator. If measurement includes "feet" maybe no. Better split with regex that captures separator including `by`. Need be careful "16 ft by 7 ft": split at ` by ` => ["16 ft", "7 ft"].

For each measurement, parse to inches.

Possible approach: parse each dimension with regex:
- If contains feet marker and inch marker: pattern `(?P<ft>\d+(?:\.\d+)?)\s*(?:'|ft(?:s?)?\.?)\s*(?P<in>\d+(?:\.\d+)?)\s*(?:"|in(?:ches?)?\.?)`? Need handle `16' 6"`, `16 ft 6 in`, `16'6"`, `16 ft 6 in.`, maybe `16'6`.
- If contains feet marker only: `(?P<ft>\d+(?:\.\d+)?)\s*(?:'|ft(?:s?)?\.?)`
- If contains inches marker only: `(?P<in>\d+(?:\.\d+)?)\s*(?:"|in(?:ches?)?\.?)`
- If unitless: infer.

But mixed forms could be "16 ft x 84 in". parse each dimension individually.

… (247 more lines)
thinking on · reasoning_effort=low
sample 1external 8/8own 8/8all external cases pass76 lines
sample 2external 8/8own 8/8all external cases pass87 lines
returned code, first 42 lines


```python
import re


def parse_door_size(s: str) -> tuple[int, int]:
    """Parse garage-door size strings like '16x7', '16' x 7'', '16 ft by 7 ft',
    '192" x 84"' into (width_inches, height_inches)."""
    s = s.strip().lower()

    # Normalize separators: "by" -> "x", strip extra spaces around x
    s = re.sub(r"\bby\b", "x", s)
    s = re.sub(r"\s+x\s+", "x", s)

    parts = s.split("x")
    if len(parts) != 2:
        raise ValueError(f"Cannot parse door size: {s!r}")

    def _parse_part(part: str) -> int:
        part = part.strip()

        is_inches = (
            '"' in part
            or " in" in part
            or " inch" in part
            or " inches" in part
        )

        m = re.search(r"[\d]+(?:\.\d+)?", part)
        if not m:
            raise ValueError(f"No number found in: {part!r}")

        num = float(m.group())

        if is_inches:
            return int(num)
        # bare numbers and feet default to feet
        return int(num * 12)

    return (_parse_part(parts[0]), _parse_part(parts[1]))

… (37 more lines)
  • Best code of the three when given room: thinking off with a 4 K budget scored 8/8 on the first sample; thinking on with reasoning_effort=low scored 8/8 own and 8/8 external in 2,243 tokens.
  • Default thinking effort never answered: 8,192 tokens of reasoning, no final code — same failure as Ornith. Set chat_template_kwargs.reasoning_effort="low" for agent loops.
  • One of three no-think samples does not compile (\'|' quoting inside a raw regex) — the quote-heavy prompt trips both Qwen-family models about one time in three.
  • Prefill 1,180 tok/s is the price of a dense 27 B; Ornith prefills 5.7× faster. Prefix cache (0.77 s) makes multi-turn fine.
  • Prep (download + requant) ran ~35 min on CPU; first boot compiles for ~4 min and caches to the qwen-cache volume.

Ornith-1.5-35B-A3B PASS

thinking off
sample 1external 4/8own 6/8fails: 16', 16x7, 8'x7', 9x882 lines
sample 2external 6/8own 6/8fails: 16x7, 9x869 lines
sample 3does not run ^ E SyntaxError: unterminated string literal (detected at line 13) !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 4external 8/8own 6/8all external cases pass80 lines
returned code, first 42 lines
```python
import re
import pytest


def parse_door_size(s: str) -> tuple[int, int]:
    """Parse a garage-door size into a (width_in, height_in) tuple (in inches).

    Accepts forms such as:
        "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\""
    Feet (' / ft) and inches (\" / in) are recognized and mixed forms are
    allowed, e.g. "4' 6\" x 8'".
    """
    tokens = re.findall(r"[+-]?\d+(?:\.\d+)?\s*(?:'|"'"'"'"|ft|in|inches|inche)?", s)
    tokens = [t for t in tokens if t.strip()]
    if len(tokens) < 2:
        raise ValueError(f"Could not parse dimensions from {s!r}")

    def to_inches(token: str) -> float:
        token = token.strip().lower()
        if token.endswith(("ft", "feet")):
            return float(re.sub(r"(ft|feet)?", "", token)) * 12.0
        if token.endswith(("in", "inches", "inche")):
            return float(re.sub(r"(in|inches|inche)?", "", token))
        # No unit: default to inches if it's a bare number, but allow bare feet
        # only when a trailing ' or the whole value already looked like feet.
        return float(token)

    w = to_inches(tokens[0])
    h = to_inches(tokens[1])
    return round(w), round(h)


@pytest.mark.parametrize(
    "s, expected",
    [
        ("16x7", (16, 7)),
        ("16' x 7'", (16, 7)),
        ("16 ft by 7 ft", (16, 7)),
        ('192" x 84"', (192, 84)),
        ('4\' 6" x 8\'', (54, 96)),
        ("4 ft 6 in by 8 ft", (54, 96)),
… (7 more lines)
thinking on · default effort
sample 1no final answerhit the token budget while still reasoning (265 lines of reasoning saved)
returned code, first 42 lines
The user wants a Python function `parse_door_size(s: str) -> tuple[int, int]` that parses garage-door sizes in various formats into (width_in, height_in) in inches.

Formats to handle:
- "16x7"
- "16' x 7'" (feet with apostrophe)
- "16 ft by 7 ft"
- "192" x 84"" (inches with double-quote)

Need to handle feet, inches, and mixed forms.

Let me write a robust parser using regex.

The idea:
- Extract two numbers
- Determine their units (feet or inches)
- Convert everything to inches

Let me think about the units. Feet can be denoted by `'` (apostrophe) or `ft`. Inches can be denoted by `"` (double quote) or `in`.

If no unit is specified (like "16x7"), they're already in inches (standard garage door notation where 16x7 means 16 feet x 7 feet? Wait, no).

Actually, garage door sizes like "16x7" typically mean 16 feet wide x 7 feet tall. But the question says "into (width_in, height_in) in inches." Hmm.

Wait, let me reconsider. "16x7" - is that feet or inches? In garage door context, "16x7" means 16 feet wide by 7 feet tall. But the function returns inches.

Hmm, but then "16' x 7'" would be 16 feet x 7 feet = 192" x 84". And "192" x 84"" would be 192 inches x 84 inches.

So these should be consistent! "16' x 7'" = 192 x 84, and "192" x 84"" = 192 x 84. Yes!

So "16x7" — if interpreted as feet — would be 192 x 84. That makes sense with the garage door convention.

Wait, but that's ambiguous. Let me think again. The examples:
- "16x7" 
- "16' x 7'" 
- "16 ft by 7 ft" 
- "192" x 84""

If "16' x 7'" = 16 feet x 7 feet = 192 x 84 inches.
And "192" x 84"" = 192 inches x 84 inches.

So both give (192, 84). That means "16x7" is also feet, giving (192, 84).

… (499 more lines)
  • Thinking mode never finished the coding task: 4,096 then 8,192 tokens of reasoning (25 KB) with no final answer. Use thinking off for code, or budget ≥16 K tokens.
  • Code quality on the quote-heavy parser prompt was inconsistent across 3 no-think samples: 4/8, 6/8, and one reply that does not compile (unterminated string inside a raw regex). Bare-feet forms (16x7) were the common miss.
  • Marlin linear + Marlin WNA16 MoE kernels (Ampere path).
  • Thinking on by default; reasoning lands in a separate field. Budget max_tokens for it or pass enable_thinking=false.
  • Asymmetric int4 checkpoints (cyankiwi, ulkaa) are rejected by vLLM's MoE kernels; MIRALABS' symmetric one is 25.6 GB — too big. biMEMO's was the only single-card fit.

DiffusionGemma-26B-A4B PASS with workarounds

thinking off
sample 1external 6/8own 6/8fails: 16x7, 9x854 lines
sample 2external 6/8own 6/8fails: 16x7, 9x857 lines
sample 3external 4/8own 5/8fails: 16, 16x7, 8'x7', 9x854 lines
returned code, first 42 lines
```
import re
import pytest

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parses garage door strings into (width_inches, height_inches).
    Handles formats: 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84"
    """
    # Normalize string: remove whitespace and lowercase
    s = s.lower().strip()
    
    # Split by common delimiters: 'x', 'by', or just whitespace
    parts = re.split(r'x|by', s)
    if len(parts) != 2:
        raise ValueError(f"Could not parse door dimension string: {s}")

    def to_inches(part: str) -> int:
        part = part.strip()
        # Extract numbers
        match = re.search(r'(\d+)', part)
        if not match:
            raise ValueError(f"Could not find dimension in: {part}")
        
        value = int(match.group(1))
        
        # Check for units
        if "ft" in part or "'" in part:
            return value * 12
        elif '"' in part or "in" in part:
            return value
        else:
            # Default to inches if no unit provided
            return value

    width = to_inches(parts[0])
    height = to_inches(parts[1])
    return (width, height)

@pytest.mark.parametrize("inp, expected", [
    ("16x7", (192, 84)),
    ("16' x 7'", (192, 84)),
… (13 more lines)
thinking on · default effort
sample 1external 5/8own 6/8fails: 16x7, 8'x7', 9x850 lines
returned code, first 42 lines
python
import re
import pytest

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parses garage door strings into (width_inches, height_inches).
    Handles formats: 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84"
    """
    # Normalize the string
    s = s.lower().strip()
    
    # Split by common delimiters: 'x', 'by', or whitespace
    parts = re.split(r'\s*[xbby]\s+', s)
    if len(parts) != 2:
        raise ValueError(f"Could not parse string: {s}")

    def to_inches(dim: str) -> int:
        dim = dim.strip()
        # Extract numeric part
        match = re.search(r'(\d+)', dim)
        if not match:
            raise ValueError(f"Could not find dimension in: {dim}")
        
        value = int(match.group(1))
        
        # Check for units
        if "'" in dim or "ft" in dim:
            return value * 12
        elif '"' in dim or "in" in dim:
            return value
        # Default fallback to inches if no unit provided
        return value

    return to_inches(parts[0]), to_inches(parts[1])

@pytest.mark.parametrize("input,expected", [
    ("16x7", (192, 84)),
    ("16' x 7'", (192, 84)),
    ("16 ft by 7 ft", (192, 84)),
    ("192\" x 84\"", (192, 84)),
    ("8'x8'", (96, 96)),
… (10 more lines)
  • Default FlashInfer backend crashes at CUDA-graph capture: plan() gets a per-request causal tensor where it wants a bool. FLASH_ATTN refuses head_dim 256. TRITON_ATTN works.
  • temperature / seed / min_p are rejected (HTTP 400) for diffusion models in 0.28.0.
  • "Reply with exactly: ok" returns an immediate EOS (1 token) in both thinking modes; ordinary prompts are fine.
  • Prefix cache barely helps (1.87 s second turn) — canvas re-encoding dominates.

Fit on the card

modelcheckpointweight loadresidentKV budgetVRAM servingcold start
Qwen3.8-27Bsyv-ai requant of Qwen/Qwen3.8-27B (AutoRound W4A16 body + GPTQ-int4 lm_head/MTP) + DFlash2 W4A16 drafter6.7 s (+0.3 s drafter)15.02 GiBpinned 5.58 GB → 68,605 tok (1.05× at 64 K)22,897 MiB~4.5 min first boot (compile), ~1.5 min warm
Ornith-1.5-35B-A3BbiMEMO/Ornith-1.5-35B-A3B-int4-AutoRound-MTP (20.9 GB; the same weights Tiel-Coder is quantized from)28.9 s + 3.7 s MTP18.77 GiB2.09 GiB → 70,870 tok (2.16× at 32 K)22,229 MiB~3 min
DiffusionGemma-26B-A4Bcyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4 (17.2 GB)17–21 s15.58 GiB5.04 GiB → 102,853 tok (3.14× at 32 K)23,535 MiB~2.5 min
"Resident" is vLLM's own "Model loading took … GiB". VRAM serving is nvidia-smi during the bench. Cold start is container start → first token, warm compile cache.

The stack

One endpoint, three containers

  • llama-swap on 127.0.0.1:11440 speaks OpenAI and Anthropic; model ids qwen3.8-27b, ornith-1.5-35b, diffusiongemma-26b (aliases qwen, tiel, dgemma).
  • Each model is a docker run pinned to the 3090 by UUID; one resident at a time; docker stop on swap.
  • gemma-4-26b-a4b is aliased to Qwen so ai-pitlane's reviewer/local_ai and plan_deep work unchanged.

ai-pitlane playground

  • Project llm-stack-bakeoff: probe → bench Qwen → bench Ornith → bench DiffusionGemma → summary, all python_script nodes, $0 cap, manual launch. Writes ai-queue/llm-stack/bakeoff.md.
  • Design doc /design/llm-stack-3090 carries the architecture diagram.
  • venv rebuilt; 1,662 tests pass.

Small-model round

Same harness, same prompt, same scoring — models in the 4–12 B class on the 3090, in bf16 where they fit and with an official or well-formed int4 where they don't, plus a 4-bit copy of the 4 B to measure what quantization costs.

Ornith-1.5-9B

runs, but 0/8 on code
stock vLLM 0.28.0 · bf16 (19.3 GB) · 64 K ctx · 3090
44.4tok/s decode
3,670tok/s prefill
VRAM serving
external code tests 0/32 over 4 sample(s), thinking off

Qwen3.8-9B-Distill

runs; 4/8 best
stock vLLM 0.28.0 · bf16 (19.3 GB) · 64 K ctx · 3090
44.8tok/s decode
3,665tok/s prefill
VRAM serving
external code tests 8/32 over 4 sample(s), thinking off

Qwen3.5-4B

runs, but unusable for code
stock vLLM 0.28.0 · bf16 (9.3 GB) · 128 K ctx · 3090
76.4tok/s decode
6,369tok/s prefill
VRAM serving
external code tests 4/32 over 4 sample(s), thinking off

Gemma-4-12B-it

best of the small round · 6/8
stock vLLM 0.28.0 · official QAT W4A16 (10.3 GB) · 64 K ctx · 3090
78.7tok/s decode
2,280tok/s prefill
21,003 MiBVRAM serving
external code tests 6/32 over 4 sample(s), thinking off

Qwen3.5-4B AWQ-INT4

runs, but unusable for code
stock vLLM 0.28.0 · AWQ-INT4 (4.0 GB) · 128 K ctx · 3090 — 4-bit vs the bf16 build
172–175tok/s decode
6,790tok/s prefill
VRAM serving
external code tests 0/32 over 4 sample(s), thinking off
modeldecode, short (tok/s)decode at 6 K ctxprefill (tok/s)cached 2nd turn TTFTtool callAnthropic API
Ornith-1.5-9B44.443.83,6700.15 sOKFAIL (thinking ate the budget)
Qwen3.8-9B-Distill44.844.23,6650.15 sOKFAIL (thinking ate the budget)
Qwen3.5-4B76.475.06,3690.09 sOKFAIL (400-token budget consumed by thinking)
Gemma-4-12B-it78.771.02,2800.09 sOK (gemma4 parser)OK
Qwen3.5-4B AWQ-INT4172–175166.36,7900.09 sOKFAIL (thinking ate the budget)
modelcheckpointweight loadresidentKV budgetVRAM servingcold start
Ornith-1.5-9Bornith-ai/Ornith-1.5-9B bf16 — Qwen3.5-9B base + coding/agentic RL; card: Terminal-Bench 46.2, SWE-bench Verified 70.6~15 s≈18 GiB (bf16)not captured~2 min
Qwen3.8-9B-Distillempero-ai/Qwen3.8-9B-Distill — full-parameter distillation of Qwen3.8-2.4T into the Qwen3.5-9B architecture (~70 K teacher traces)~15 s≈18 GiB (bf16)not captured~2 min
Qwen3.5-4BQwen/Qwen3.5-4B bf16 — the official small baseline~10 s≈9 GiB (bf16)not captured (container swapped out before the log was read)~2 min
Gemma-4-12B-itgoogle/gemma-4-12B-it-qat-w4a16-ct — Google's own quantization-aware int4, compressed-tensors5.0 s8.28 GiB288,717 tokens (4.4× at 64 K)21,003 MiB~2 min
Qwen3.5-4B AWQ-INT4cyankiwi/Qwen3.5-4B-AWQ-4bit — same model as above at 4-bit, to measure what quantization costs~5 s≈3.5 GiBnot captured~2 min

Ornith-1.5-9B runs, but 0/8 on code

thinking off
sample 1external 0/8own 0/8fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x8146 lines
sample 2does not runport parse_door_size E ModuleNotFoundError: No module named 'parse_door_size' !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 3external 0/8own 0/10fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x882 lines
sample 4external 0/8own 0/8fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x8146 lines
returned code, first 42 lines
```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    s = s.strip()
    # Format: "16 ft by 7 ft" or "16 feet and 8 inches"
    ft_parts = re.findall(r'(\d+)\s*feet?', s, re.IGNORECASE)
    if ft_parts:
        ft = int(ft_parts[0])
        st = ft * 12
        inch_parts = re.findall(r'(\d+)\s*inch(?:es)?', s, re.IGNORECASE)
        if inch_parts:
            st += int(inch_parts[0])
        return (ft * 12, st)

    # Format: '16\' x 7\'', "16\" x 84\"", 16x7, "16' 8\" x 7'"
    m = re.search(r"(\d{1,3})('?\"?| x | by )(\d{1,3})('?\"?)", s)
    if m:
        w = int(m.group(1))
        h = int(m.group(2))
        inches = (w if m.group(1) == w else 0)
        inches2 = (h if m.group(1) == h else 0)
        width_in = inches + inches2
        return (width_in, height_in)
```

I broke my thinking here since I didn't finish writing the code. Let me rebuild this properly:

```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    s = s.strip()

    # Format: "16 ft by 7 ft", "16 feet and 8 inches", "16' x 7'", "16\" x 84\"", "16x7", "16' 8\" x 7'"
    ft_parts = re.findall(r'(\d+)\s*feet?', s, re.IGNORECASE)
    if ft_parts:
        ft = int(ft_parts[0])
        st = ft * 12
        inch_parts = re.findall(r'(\d+)\s*inch(?:es)?', s, re.IGNORECASE)
        if inch_parts:
            st += int(inch_parts[0])
… (189 more lines)
thinking on · reasoning_effort=low
sample 1external 0/8own 0/0fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x821 lines
returned code, first 42 lines
The user wants a Python function that parses garage-door sizes into (width_in, height_in). Let me think about the requirements:

1. Parse strings like:
   - "16x7" → 16 inches x 7 inches
   - "16' x 7'" → 16 ft x 7 ft = 192x84
   - "16 ft by 7 ft" → 16 ft by 7 ft = 192x84
   - "192\" x 84\"" → 192 inches x 84 inches
   - Handle feet, inches, and mixed forms

2. Return a tuple of (width_in, height_in) in inches.

3. Include 8 pytest cases.

Let me think about the parsing logic:

- Numbers can be in feet or inches.
- Feet indicated by `'` or `ft` or `feet` or `feet "` etc.
- Inches indicated by `"` or `in` or `inch` or `inches`.

So I need to extract numbers and their units, then convert feet to inches (multiply by 12), and handle the width/height order.

Let me write a robust regex-based parser.

Approach:
1. Find all number-unit pairs.
2. For each, determine if it's feet or inches.
3. Convert to inches.
4. First number is width, second is height.

Let me think about what counts as feet vs inches:
- `'` (apostrophe) → feet
- `"`, `\"` → inches
- `ft`/`feet`/`foot` → feet
- `in`/`inch`/`inches` → inches

But careful: `x` unit (like "cm in x ft")? Probably not needed. But "16 ft by 7 ft" uses "by". Let me handle general.

Let me write the parser:

```python
import re

… (509 more lines)
  • 44 tok/s: 19.3 GB of bf16 weights and no drafter, so decode is pure bandwidth — slower than the 27 B W4A16 + DFlash2 build and the 35 B MoE. An int4 build would roughly triple it, but every published int4 of this model carries bf16 vision/embedding tensors and is 8.6–18 GB.
  • With thinking off it still reasons in the open: sample 1 was 11 code fragments separated by "Let me write the actual clean version now…", cut off at 4,096 tokens. reasoning_effort=low ran to 8,192 tokens with no answer.
  • Its cleanest sample (878 tokens, 10 own tests) raises IndexError: no such group on every input — a regex group-index bug. 0/8 on all four samples.
  • The card's SWE-bench 70.6 is earned inside an agent harness with tools and iteration; single-shot code generation on a quote-heavy spec is a different task, and here it did not deliver.

Qwen3.8-9B-Distill runs; 4/8 best

thinking off
sample 1external 4/8own 0/0fails: 120", 16', 192", 8'x7'101 lines
sample 2does not runE ^ E SyntaxError: unterminated string literal (detected at line 11) !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 3no final answerhit the token budget while still reasoning (154 lines of reasoning saved)
sample 4external 4/8own 0/0fails: 120", 16', 192", 8'x7'101 lines
returned code, first 42 lines
```python
import re
from typing import Tuple


def parse_door_size(s: str) -> Tuple[int, int]:
    """
    Parse garage-door size strings into a tuple of (width_in, height_in) in inches.

    Supported formats:
        - "16x7"
        - "16' x 7'"
        - "16 ft by 7 ft"
        - "192\" x 84\""
        - "16 ft by 7 in"
        - "16' x 7\""
        - "2'4\" x 9'6\""
        - "36 in x 84 in"

    Args:
        s: Input string representing a door size.

    Returns:
        A tuple (width_in, height_in) where both values are integers in inches.

    Raises:
        ValueError: If the input string cannot be parsed.
    """
    s = s.strip()

    # Pattern for: A_unit x/unit Feet/ft by Feet/ft/inches Inch/in or similar
    # Capture two components separated by 'x' or 'by' with unit labels:
    match = re.match(
        r"^\s*"
        r"(\d+(?:'\d+)?\s*(?:ft?|inches?)?|\"\d+?(?:/?\"?\s*(?:ft?|inches?)?)?)\s*"
        r"(?:x|X|x?|by?)"
        r"\s*"
        r"(\d+(?:'\d+)?\s*(?:ft?|inches?)?|\"\d+?(?:/?\"?\s*(?:ft?|inches?)?)?)\s*$",
        s,
        re.IGNORECASE,
    )
    if not match:
… (60 more lines)
thinking on · reasoning_effort=low
sample 1no final answerhit the token budget while still reasoning (377 lines of reasoning saved)
returned code, first 42 lines
Thinking Process:

1.  **Analyze the Request:**
    *   Function name: `parse_door_size`
    *   Input: `s: str`
    *   Output: `tuple[int, int]` representing (width_in, height_in) in inches.
    *   Input formats to handle: "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"".
    *   Logic: Convert feet and inches to total inches, parse dimensions.
    *   Constraints: Must include 8 `pytest` test cases.
    *   Output format: Code only.

2.  **Determine Parsing Logic:**
    *   Need to strip units (ft, ft in, ", ").
    *   Need to split dimensions (x, by).
    *   Need to handle mixed units (e.g., "5'6"" or "5 ft 6 in" or "16' 7\"" or "16 ft 8 in"). The problem description gives examples like "16' x 7'" and "192\" x 84\"", implying the whole dimension might be in inches or feet. It also says "Handle feet, inches, and mixed forms".
    *   Let's look closer at the examples:
        *   "16x7" -> 16, 7
        *   "16' x 7'" -> 16*12 + 0, 7*12 + 0 (assuming no sub-inches) OR 16 inches width, 7 inches height. Wait, usually door sizes are like 16' x 8'. So 16 feet is 192 inches.
        *   "16 ft by 7 ft" -> 16*12, 7*12.
        *   "192\" x 84\"" -> 192, 84.
    *   "Mixed forms" implies inputs like "5'6" or "5 ft 6 in".
    *   Regex approach is best here.
    *   Split string by separators ('x', ' by ', ' ' ) to get two parts.
    *   For each part, parse numbers and units.
    *   Units: 'ft', "'", '"', 'in', 'inch' (maybe).
    *   If 'ft' unit present: multiply value by 12.
    *   If '"' unit present: treat as inches.
    *   If 'in' unit present: treat as inches.
    *   If just number: treat as inches (common default or assume feet? In "192\" x 84\"", it's inches. In "16x7", could be feet or inches? Usually stripped units are ambiguous. However, for the function signature `parse_door_size`, returning a specific int means making a decision. Let's assume for bare numbers, if the other has 'ft' maybe all are feet? No, "16x7" in door context likely means feet if it's a garage door. But often "16x7" just means inches in casing context. Let's look at "192\" x 84"". "16 x 7" (without quotes) -> if it's feet, (192, 84). If inches, (16, 7).
    *   Actually, a robust parser for "Width" "Height" usually looks for the trailing unit.
    *   Let's refine the parsing logic per component:
        1.  Replace slashes, quotes with delimiters.
        2.  Split by 'x', ' by ', ' ' (careful with spaces).
        3.  Strip unit suffixes from each numeric part.
        4.  If unit is 'ft' -> *12.
        5.  If unit is 'in' or '"' -> result as is.
        6.  If no unit -> assume inches? Or feet? The prompt examples "16x7" -> likely 16 inches, 7 inches? Or 16 feet, 7 feet? Garage door standard 16' x 7' is possible. But "16x7" usually implies inches in DIY unless 'x' implies feet (like U.S. drywall is usually feet width).
        7.  Actually, "Handle feet, inches, and mixed forms" suggests if I write "16' x 7'", 16 is feet. If I write "16" x 84"", 16 is inches.
        8.  So the rule is:
            *   Nominal Number: > 1 -> Feet (Garage doors are usually >100 inches). But "16x7" might be small shed door.
            *   Better approach: If `ft` found, convert. If `"` found, it's inches. If bare number, assume inches (since "192\" x 84\"" are inches). Wait, if input is "16x7", output (16, 7).
            *   Wait, let's look at "16 ft by 7 ft".
… (335 more lines)
  • Same 44 tok/s bandwidth ceiling as Ornith-9B (identical architecture, bf16, no drafter).
  • The best-behaved small model: three compact no-think replies (0.9–1.3 K tokens) with no restart loops. But only one compiles — 4/8, getting the bare-feet forms right and the inch-quoted ones wrong — and it wrote no tests of its own; the other two fail on quote escaping (unterminated string; invalid syntax).
  • reasoning_effort=low is not honored (8,192 tokens, no answer) — the distill did not inherit Qwen3.8's effort control.
  • Usable for review/summarization at 9 B; not reliable enough for generation on this kind of spec.

Qwen3.5-4B runs, but unusable for code

thinking off
sample 1external 2/8own 0/1fails: 10, 16, 16', 16x7, 8'x7', 9x895 lines
sample 2no final answerhit the token budget while still reasoning (361 lines of reasoning saved)
sample 3does not run ^ E SyntaxError: unexpected character after line continuation character !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 4external 2/8own 0/1fails: 10, 16, 16', 16x7, 8'x7', 9x895 lines
returned code, first 42 lines
```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"" 
    into (width_in, height_in) in inches.
    
    Handles:
    - Numeric only (assumed inches)
    - Numeric + feet (ft, ', space-separated scalar)
    - Numeric + inches (numeric or "). In mixed cases, last unit implies height unit.
    - Space, ' x ', or " by " as separators.
    """
    s = s.strip()
    
    # Split by common separators: x, ' x ', " by ", spaces/generic separators
    # Try to split by ' x ' or " x " or " by " first
    split_by_x = re.split(r'[\s]*\s*x\s*[\s"']*|[\s]*\s*by\s*[\s"]*', s, flags=re.IGNORECASE)
    if len(split_by_x) != 2:
        # If no split by 'x' or 'by', assume single string; split on spaces between numbers if needed
        # For simplicity, assume single or exactly space-separated parts; try to split by space
        # If numbers contain ', or quotes, strip them.
        parts = [p.strip() for p in s.split()]
    else:
        parts = split_by_x
    
    def parse_number許 unit_str:
        unit_str = unit_str.replace('"', '').strip()
        # Handle ' ft ', ' ft', ' feet ', ' feet', 'ft ', ' ft'
        unit = unit_str.lower()
        
        if unit == 'ft' or unit == 'feet':
            if unit_str.isdigit():
                return int(unit_str), 'ft'
        # If there's a decimal point, we consider inches if the digit is a single-char diff.
        # But typical usage is whole number.
        
        # Now treat as given number with inches or ft.
        # We support both: "16" (16 inches), "16'" or "16 ft", "16 ft 6" (mixed).
        # We'll convert to a float then to integer at end, handling the last part as inches if possible
        # If the unit is absent, assume inches.
… (359 more lines)
thinking on · reasoning_effort=low
sample 1does not runE import re E IndentationError: unexpected indent !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
returned code, first 42 lines
Thinking Process:

1.  **Analyze the Request:**
    *   Function name: `parse_door_size`
    *   Input: string `s` (example: "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"")
    *   Output: tuple of two integers `(width_in, height_in)`
    *   Requirements: Handle feet (`'`, `ft`, `''`?), inches (`"`, `inch`, no suffix), mixed forms.
    *   Constraint: Handle the logic correctly to extract numbers and units.
    *   Constraint: Include 8 pytest test cases.
    *   Constraint: Reply with code only.

2.  **Determining Conversion Logic:**
    *   1 foot = 12 inches.
    *   Input strings might contain:
        *   Pure inches (e.g., "192\" x 84\"") -> Note: The example uses backticks in the prompt's example `"` but usually, it's just a character. I need to handle `"` (which is often escaped in prompt strings) literally.
        *   Pure feet (e.g., "16' x 7'", "16' x 7'")
        *   Feet abbreviation (e.g., "16 ft by 7 ft")
        *   Mixed (e.g., "16 ft x 8'")
    *   Separators: `x`, `x`, `hyphen`, `space`, `by`.
    *   Numbers: Need to be floats or integers for intermediate calculation, then converted to int for result.

3.  **Reference Handling:**
    *   The prompt example shows `192\"`. This looks like Python string representation. I should assume the input string will contain the literal characters `'` or `ft` or `"` or `x` or ` by ``.
    *   Wait, the prompt example `192\" x 84\"` suggests the actual input string might contain `"` (double quote) but escaped in the documentation. Or it might contain the character `'`.
    *   Commonly used units:
        *   `'` (apostrophe) = foot
        *   `ft` = foot
        *   `"` = inch
        *   No unit = assume inches or feet? Usually, sizes like `16x7` imply just numbers, but context implies units. Let's assume if no unit is specified but numbers are given, they might be ambiguous. However, for robustness, usually, explicit units are preferred. Let's excuse assume no explicit unit is treated as inches (since doors are often expressed as inches, but garage doors are often 7ft high (~84in) or 7ft high (84in). 192 inches is ~16ft wide. 84 inches is ~7ft high. So `192x84` are definitely inches).
    *   Logic refinement:
        1.  Extract all numbers from the string.
        2.  Determine units (`'`, `ft`, `"`, empty).
        3.  Since the input format is `N1 unit1 by unit2` or `N1 unit1 x N2 unit2`, regex is best.
        4.  Regex: `r'(\d+(?:\.\d+)?)\s*(?:'|\s*ft\s*)?\s*(?:x|\s*by\s*)?\s*(\d+(?:\.\d+)?)\s*(?:'|\s*ft\s*)?\s*(?:'?|\"|\s*\#)??'?` ... This is getting complicated.
        5.  Simpler approach: Replace units with a number.
            *   `ft` -> 12, `'` -> 12.
            *   `"` -> 1, empty -> 1 (assume inches if no unit found? No, wait).
            *   If I see `192"`, it's 192 inches.
            *   If I see `16`, is it 16 inches or 16 feet? In garage doors, width might be 16 (meaning 16ft or 6'4"? 16 is usually a shorthand for 16ft). Height 7 usually means 7ft (84in).
            *   However, the prompt asks to handle mixed forms.
            *   Let's look at standard regex patterns for mixed units:
                *   `^\s*(\d+(?:\.\d+)?)\s*([\'\"]?\salt)?` -> This is tricky.
… (403 more lines)
  • Never finishes. All three no-think samples hit the 4,096-token cap and the reasoning_effort=low run hit 8,192: the model writes a complete solution, announces "Final clean output below:", and starts again. Its own test expectations were wrong too (16x7 → (16, 7)).
  • Scored on the first block that compiles: best sample 2/8 external (only the inch-quoted forms); the other three have no compiling block at all.
  • Decode is only 76 tok/s despite 4 B parameters: no speculative decoding and bf16 weights, so it is pure bandwidth — the 27 B W4A16 + DFlash2 build is 2× faster.
  • Fine for classification/extraction chores; not a coder on this prompt.

Gemma-4-12B-it best of the small round · 6/8

thinking off
sample 1external 0/8own 0/0fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x842 lines
sample 2does not run ^ E SyntaxError: unterminated string literal (detected at line 26) !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 3external 6/8own 1/1fails: 16x7, 9x845 lines
sample 4external 0/8own 0/0fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x842 lines
returned code, first 42 lines
```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    # Regex to find numbers and their associated units (ft, ', ", feet)
    # Matches: 16, 16', 16", 16 ft, 16 feet
    pattern = r"(\d+)\s*(?:'|\"|ft|feet|inch|inches)?"
    matches = re.findall(pattern, s.lower())
    
    def convert_to_inches(match_str: str) -> int:
        num_part = match_str[0]
        unit_part = match_str[1]
        val = int(num_part)
        
        if any(u in unit_part for u in ("'", "ft", "feet")):
            return val * 12
        return val

    # Extract all numeric parts converted to inches
    sizes = []
    for m in matches:
        sizes.append(convert_to_inches(m))
    
    if len(sizes) >= 2:
        return (sizes[0], sizes[1])
    return (0, 0)

if __name__ == "__main__":
    import pytest

    @pytest.mark.parametrize("input_str, expected", [
        ("16x7", (16, 7)),
        ("16' x 7'", (192, 84)),
        ("16 ft by 7 ft", (192, 84)),
        ("192\" x 84\"", (192, 84)),
        ("8x9", (8, 9)),
        ("10ft x 8ft", (120, 96)),
        ("12' x 12'", (144, 144)),
        ("24 inch x 36 inch", (24, 36)),
    ])
    def test_parse_door_size(input_str, expected):
        assert parse_door_size(input_str) == expected
… (1 more lines)
thinking on · reasoning_effort=low
sample 1external 6/8own 0/0fails: 16x7, 9x849 lines
returned code, first 42 lines
```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parses garage door sizes into a tuple of (width_in, height_in).
    Supports formats like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"".
    """
    # Normalize the string: remove quotes and whitespace, convert to lowercase
    clean_s = s.replace('"', '').lower().replace("'", "").replace(" ", "")
    
    # Regex to find numbers and the separator (x, by, or just a space/gap)
    # We look for parts that represent dimensions
    parts = re.findall(r'(\d+)', clean_s)
    if len(parts) < 2:
        raise ValueError("Could not find two dimensions in input string.")
    
    raw_w = int(parts[0])
    raw_h = int(parts[1])
    
    # Determine if units are feet or inches based on keywords
    is_feet = "ft" in clean_s or "'" in s
    is_inches = '"' in s or "inch" in clean_s
    
    # Logic to decide multiplier:
    # If "ft" or "'" is present, treat as feet unless "inch" is also present.
    # If no units are provided, we assume inches (standard for door codes like 16x7).
    # However, if "ft" is found, multiply by 12.
    
    multiplier = 1
    if is_feet and not is_inches:
        multiplier = 12
        
    return (raw_w * multiplier, raw_h * multiplier)

if __name__ == "__main__":
    import pytest

    def test_parse_door_size():
        assert parse_door_size("16x7") == (16, 7)
        assert parse_door_size("16' x 7'") == (192, 84)
        assert parse_door_size("16 ft by 7 ft") == (192, 84)
… (8 more lines)
  • The small model to keep. Compact replies (480–880 tokens), thinking mode that actually returns (645 tokens at effort=low), tool calls and the Anthropic route both working, and 6/8 on two of four samples.
  • Fails are the same two bare-feet forms (16x7, 9x8) as its DiffusionGemma sibling — a family habit of reading unitless numbers as inches. One sample scored 0/8 and one did not compile.
  • Wrote one pytest case instead of the eight asked for — it under-delivers on the test half of the prompt.
  • Marlin int4 at 8.3 GiB resident leaves a 289 K-token KV pool: this is the model to give a whole repository to.

Qwen3.5-4B AWQ-INT4 runs, but unusable for code

thinking off
sample 1does not run ^ E SyntaxError: unterminated string literal (detected at line 32) !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
sample 2no final answerhit the token budget while still reasoning (288 lines of reasoning saved)
sample 3external 0/8own 0/0fails: 10, 120", 16, 16', 16x7, 192", 8'x7', 9x8159 lines
sample 4does not run ^ E SyntaxError: unterminated string literal (detected at line 32) !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
returned code, first 42 lines
```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    """
    Parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\""
    into (width_in, height_in) in inches.
    
    Supported formats:
    - Plain numbers: "10x10"
    - With units: "10' 9\"", "10'09\"", "10 ft 9 in", "10' 9'", "10 ft by 9 ft"
    Logic:
      - If both parts contain 'ft' or inches without a 'prime', treat as converted fully to inches.
      - If primes (') are present, treat the number before the prime as feet.
      - If a mix: assume feet as inches * 12 if no inch part.
    """
    s = s.strip()
    
    s = s.lower()
    
    # Handle single number (no x or by)
    s = re.sub(r'\s*x\s+', ' ', s)
    s = re.sub(r'\s+by\s+', ' ', s)
    
    s = s.strip()
    
    # Determine if we have prime feet or just numbers
    has_prime = "'" in s
    has_decimals_or_inches = '"' in s
    
    # Normalize spaces
    s = s.replace("ft", " ").replace("in", " ").replace("ft ", " ").replace("in ", " ")
    s = s.replace('ft "', "ft ').replace(' in "', "in ").replace('ft-', ' ft ').replace('in-', ' in ')
    
    parts = s.split('by')
    if len(parts) > 1:
        parts = [p.strip() for p in parts]
    else:
        parts = [s, '']
    
    parts = [p.strip() for p in parts]
    
… (91 more lines)
thinking on · reasoning_effort=low
sample 1does not runE SyntaxError: closing parenthesis ')' does not match opening parenthesis '[' !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
returned code, first 42 lines


```python
import re

def parse_door_size(s: str) -> tuple[int, int]:
    s = s.strip()
    numbers = []
    units = [''] * len(numbers)
    
    i = 0
    while i < len(s):
        match = re.match(r'(\d+(?:\.\d+)?)', s[i:], 0)
        # Find all numbers in sequence
        if match:
            num = float(match.group(1))
            numbers.append(num)
            # Determine unit
            current = s[i:]
            unit_match = re.match(r'(ft|feet|\'|\")', current[l().group(1)), end()
            unit_match.group
        i += match group.findlen ends with
    else:
        if numbers[0]:
            numbers = [int(round(number)) for number in numbers]
        else:
            numbers = [0, 0]
    
    # Return as tuple
    return (numbers[0], numbers[1])


# 8 pytest cases

def test_parse_door_size_basic():
    assert parse_door_size("16x7") == (192, 84)

def test_parse_door_size_feet():
    assert parse_door_size("16' x 7'") == (192, 84)

def test_parse_door_size_ft_by():
    assert parse_door_size("16 ft by 7 ft") == (192, 84)
… (13 more lines)
  • 2.3× faster than the bf16 build (175 vs 76 tok/s): with no speculative decoding, decode is pure weight bandwidth and int4 Marlin reads a quarter of the bytes.
  • Same restart habit as bf16 — two of three no-think samples looped to the 4,096 cap; reasoning_effort=low did finish (2,749 tokens).
  • The two replies that finished have real syntax errors (an unterminated string literal; a ) closing a [). 0/8 on every sample.
  • Conclusion for the 4 B class: quantize it for speed if you use it at all, but this prompt is beyond it.

Controlled re-test: prompt + sampling

Same task, three changes: the unit convention stated in the prompt (bare numbers are feet), a system prompt demanding exactly one fenced block with no revisions and raw single-quoted regexes, and temperature 0.2, top_p 0.95, presence_penalty 1.0 instead of each checkpoint's temperature 1.0 default. DiffusionGemma keeps its default sampling (vLLM 0.28 rejects the parameters for diffusion models). Each model runs in its best baseline mode, three samples.

modelmodebaseline · external passes (per sample)won't compiletuned · external passes (per sample)won't compile
Qwen3.8-27Bthinking · effort=low16/16 8 · 80/224/24 8 · 8 · 80/3
Ornith-1.5-35B-A3Bthinking off18/32 4 · 6 · err · 81/48/24 err · 4 · 41/3
DiffusionGemma-26B-A4Bthinking off16/24 6 · 6 · 40/38/24 0 · 0 · 80/3
Ornith-1.5-9Bthinking off0/32 0 · err · 0 · 01/47/24 5 · 0 · 20/3
Qwen3.8-9B-Distillthinking off8/32 4 · err · err · 42/412/24 2 · 6 · 40/3
Qwen3.5-4Bthinking off4/32 2 · err · err · 22/414/24 0 · 8 · 60/3
Gemma-4-12B-itthinking off6/32 0 · err · 6 · 01/422/24 8 · 8 · 60/3
Qwen3.5-4B AWQ-INT4thinking off0/32 err · err · 0 · err3/46/24 2 · 0 · 40/3

Playground run (ai-pitlane)

The llm-stack-bakeoff graph launched from the dashboard as a transient systemd unit: probe the endpoint, bench each model in turn (one GPU, so the chain is sequential, with a llama-swap swap between models), then write ai-queue/llm-stack/bakeoff.md.

runnodeswalloutcome
graph-20260830T062950…p461826probe ok → bench_qwen ok → bench_ornith ok → bench_dgemma failed → summary failed200 sfailed at bench_dgemma — llama-swap was still running the pre-Triton DiffusionGemma command (config edited, process not restarted)
graph-20260830T063545…p498911probe ok → bench_qwen ok → bench_ornith ok → bench_dgemma ok → summary ok273 scomplete — all three models benched through one endpoint with two swaps; artifact ai-queue/llm-stack/bakeoff.md written
graph-20260830T134547…p3051319 (small)probe ok → bench_ornith9b ok → bench_qwen9b ok → bench_qwen4b ok → bench_gemma12b ok → bench_qwen4b_awq ok → summary ok536 scomplete — five small models benched sequentially through :11440 with four swaps; artifact ai-queue/llm-stack/small-bakeoff.md
  • Per-node wall incl. swap: Ornith-9B 175 s, Qwen3.8-9B-Distill 104 s, Qwen3.5-4B 120 s, Gemma-12B 74 s, Qwen3.5-4B-AWQ 62 s. Speeds reproduced within 1 tok/s of the manual runs.
  • The run's replies were scored as a fourth sample each: Ornith-9B 0/8, Qwen3.8-9B-Distill 4/8, Qwen3.5-4B 2/8, Gemma-4-12B 0/8 (its four samples are 0 · err · 6 · 6 — high variance), Qwen3.5-4B-AWQ does not compile.
  • Graph llm-stack-small-bakeoff, 7 python_script nodes, $0.00.

Process notes

1. Establishing ground truth on the machine before touching models

  • Read the GPU state with nvidia-smi, lspci -nn, and sysfs DRM connectors, not from memory notes. This caught two things the notes had wrong: both monitors were cabled to the 3090, and the VS Code instance GNOME restored at login had bypassed the render-node pin.
  • Fixed at the source: cables moved to the 5060, a mutter-device-preferred-primary udev rule for the 5060 (PCI id 10de:2d05), a full logout. Verified by a new gnome-shell PID and the 3090 dropping from 383 MiB to 31 MiB of stubs. A lock/unlock did not re-pick the GPU — only a real session restart does.
  • Set the 3090 to 250 W via a systemd oneshot, because every reference benchmark for this card was taken at 250 W and batch-1 decode is bandwidth-bound (a 450 W 4090 measured only +1.9%).

2. Model selection — primary sources over blog posts

  • Pulled the Hugging Face API (creation dates, download counts, safetensors sizes, config.json architectures and quantization_config) instead of trusting "best local LLM 2026" articles, several of which still recommended 2024 models.
  • Read each model card's own benchmark table, then cross-checked against a third party's comparison that ran all candidates on one harness (peculiar-ragdoll's SWE-bench-Live 25-problem set). Vendor numbers were kept but labeled.
  • Ampere filter applied up front: NVFP4 (Blackwell only) and FP8 weights (no native sm_86 FP8 GEMM) were excluded regardless of download counts.

3. Serving-stack research

  • Two reference deployments for Qwen3.8-27B on exactly this card were read in full: syv-ai/qwen38-27b-rtx3090 (vLLM 0.27.1 fork) and 0x7067/qwen38-27b-rtx3090-llamacpp (patched llama.cpp), including their issue trackers (the FlashInfer+MTP k=4 crash, the recurring Xid 31, the sm80 Marlin repack fault, the 16 GB RAM failure).
  • For Ornith/Tiel and DiffusionGemma there was no single-3090 vLLM report, so the checkpoints were vetted by reading quantization_config directly: asymmetric int4 MoE (cyankiwi/ulkaa Ornith) is rejected by vLLM's Marlin WNA16 MoE kernels; MIRALABS' symmetric requant is 25.6 GB (too big for one card); biMEMO's AutoRound int4 at 20.9 GB with an intact BF16 MTP head was the only fit. For DiffusionGemma, cyankiwi's AWQ-INT4 was confirmed symmetric (group 32) from its config before download.
  • vLLM support was verified in the source tree at the v0.28.0 tag: DiffusionGemmaForBlockDiffusion and Qwen3_5MoeMTP in the model registry, --diffusion-config / --language-model-only / --speculative-config in arg_utils.py, and the diffusion sampler defaults read from generation_config.json.

4. Swapping three models on one card

  • vLLM itself is one-model-per-process; its Sleep Mode frees VRAM but does not switch models. llama-swap v251 fronts arbitrary commands (its README lists vLLM and Docker explicitly), so one docker run per model with cmdStop: docker stop gives a single OpenAI + Anthropic endpoint on :11440 with one model resident at a time. Aliases keep old callers (gemma-4-26b-a4b) working without code changes.
  • Each container is pinned to the 3090 by GPU UUID so nothing can land on the 5060.

5. Test harness

  • bench.py (stdlib only): streamed chat to measure TTFT and decode tok/s from server-reported completion_tokens; a fixed coding prompt (garage-door size parser + 8 pytest cases); a ~6 K-token prefill test followed by a second turn to measure prefix-cache TTFT; an OpenAI tool call checked for the right function and argument; an Anthropic /v1/messages call.
  • quality.py: extracts the code block from the saved reply, runs the model's own pytest cases, then an external 8-case set the model never saw. "Quality" here means the code executes and passes, not that it reads well.
  • Thinking is on by default for Ornith and Qwen; each is measured both ways because reasoning tokens inflate tok/s and eat the max_tokens budget (the first Anthropic test "failed" only because 32 tokens all went to the thinking block).

6. What went wrong and what was done about it

  • Ornith download stalled on a .gitignore.lock because ~/.cache/huggingface is root-owned from an old container run; the --local-dir download recovered on its own, but the xet cache fell back to a slow path. Not fixed (needs sudo); noted.
  • DiffusionGemma on vLLM 0.28.0 crashed at CUDA-graph capture: flashinfer prefill_wrapper.plan() argument #14 "Expected bool but got Tensor" — a FlashInfer 0.6.16 API mismatch on the bidirectional-attention path. No matching upstream issue existed. Worked through alternative attention backends and eager mode (see results).
  • Container images pulled at the same time as 38 GB of weights; the Qwen lane (image → 20 GB download → CPU requant → compile) is the long pole and was started first for that reason.

7. ai-pitlane as the playground

  • venv built, 1,662 tests green. New project llm-stack-bakeoff (5 python_script nodes, $0 hard cap, manual trigger) chains probe → bench Qwen → bench Ornith → bench DiffusionGemma → summary, sequential because one GPU. Design doc plans/design/llm-stack-3090.md carries the architecture diagram, per the repo's rule that diagrams never go on the canvas.

8. The playground run, and one more lesson

  • Launched llm-stack-bakeoff through the dashboard (POST /api/graph/launch, a transient systemd unit). probe → Qwen → Ornith ran clean (Ornith's warm swap took 93 s, not 3 min); DiffusionGemma "exited prematurely" through llama-swap even though the same flags booted directly.
  • Cause: llama-swap reads its YAML at startup and I had edited the DiffusionGemma entry (Triton backend, Gemma4 tool parser) without restarting it, so the run used the old FlashInfer command and hit the known crash. Restarted llama-swap, re-verified, re-launched. Rule: edit config → restart llama-swap (and do not pkill -f a pattern that matches your own shell).

9. Small-model round (afternoon)

  • Candidates chosen from the HF API the same way (created since May, ≤ 12 B, Ampere-compatible weights), then vetted by size: every published int4 of a Qwen3.5-9B-family model carries bf16 vision/embedding tensors and is 8.6–18 GB, so the 9 B models were run in bf16 instead. Doug ruled the RTX 5060 out for inference, so the planned 5060 lane was dropped before it ran.
  • Download throughput was per-connection capped (~5 MB/s from the HF CDN); five parallel streams gave 30 MB/s. The fixed ~/.cache/huggingface ownership (chowned) removed the xet permission errors but was not the bottleneck.
  • Scoring rule added: when a reply's concatenated code does not compile (a model that restarts itself, or truncation), score the first block that compiles and defines the function. This rescued exactly one sample (Qwen3.5-4B, 2/8) and left every genuine syntax error as a failure.
  • Benches were queued as a single serial chain gated on each download's completion, so the GPU never idled while the next weights were arriving. A missing exit= stamp in the five-stream downloader stalled the chain once; a watcher that stamps logs when the download process exits fixed it.
  • Finding that generalizes: at batch 1 on a 3090, a bf16 9–10 B model with no drafter decodes at ~44 tok/s — slower than a 27 B W4A16 model with speculative decoding. Weight bytes per token, not parameter count, set the speed.

10. Controlled re-test — prompt + sampling

  • Found that every model had been benchmarked at temperature 1.0 (each checkpoint's generation_config default; the harness sent nothing). Re-ran all eight with three changes: the unit convention stated in the prompt (bare numbers are feet), a system prompt demanding one fenced block with no revisions and raw single-quoted regexes, and temperature 0.2 / top_p 0.95 / presence_penalty 1.0. DiffusionGemma kept default sampling (vLLM 0.28 rejects those parameters for diffusion models). Each model in its best baseline mode, three samples, ~30 min through llama-swap.
  • Effect: compile failures fell from 11 of 33 baseline samples to 2 of 24; restart loops disappeared everywhere (replies 420–1,100 tokens instead of hitting the cap). Correctness moved most where the unit convention was the miss (Gemma-12B 0·err·6·6 → 8·8·6; Qwen3.5-4B never-compiles → 0·8·6). Qwen3.8-27B went from 8/8 on most samples to 8/8 on all three with all own tests passing. Ornith-35B did not improve (err·4·4) and still wrote a double-quoted raw regex containing " despite the instruction.
  • Interpretation: about half of the "quality" gap in the baseline was mine — an ambiguous spec and chat-default sampling. The remaining gap (Ornith-9B, the 4 B class) is the model.

Files

  • ~/llm-stack/config/llama-swap.yaml — the three model definitions with every flag and the reasons for them
  • ~/llm-stack/bin/bench.py, bin/quality.py, bin/build_report.py — harness and this page's generator
  • ~/llm-stack/RESULTS.md, PROCESS.md, logs/ (boot logs, bench outputs, saved replies)
  • ~/Desktop/ai-pitlane/ai-queue/operations/projects/llm-stack-bakeoff.graph.json
  • ~/Desktop/local-ai-research-2026-08-30/ — the parallel 11-agent research pass (synthesis in 12-SYNTHESIS.md)
  • Research brief that preceded this: 3090 Local Coding Stack