By the numbers
| October 14 | October 17 | |
|---|---|---|
| Scored runs | 192 | 275 (+83: 41 cloud, 42 local) |
| Benchmarks | 6 | 8 (two new, both C++) |
| Model families | 18 | 21 (Claude Sonnet 4.5, Nemotron 3.5 Super and Bolt) |
| Setups that ran every benchmark | 28 of 30 (all six) | 27 of 33 (all eight): 15 local, 12 cloud |
Every combined score on the dashboard now averages all eight benchmarks. Another 14 runs are not on the dashboard: a 12-run experiment with flash attention turned off on the two larger Gemma models (slower overall, so their original runs stay), one run spoiled by a shared chat, and one run stuck behind a Visual Studio window (both below).
Two new benchmarks, both in C++
7 · Review, C++
A small MoSearch change (8 files) that makes the Settings dialog’s OK button either save or say it could not. The change is small, but its main issue only shows up if the reviewer reads dialog code the change does not touch. It is scored out of 3 against our real review, and it now separates models more than any other benchmark:
| Setup | Score | What it found |
|---|---|---|
| GPT-6.1 Sol (low and high), Grok 4.6 and 4.7 (xhigh) | 3 / 3 | The main issue, the second issue, and bonus findings |
| Claude Sonnet 5.5 high, Claude Haiku 5.5 xhigh | 1.5 / 3 | The second issue and a bonus finding |
| Qwen3.8-27B UD-Q6_K, low thinking (best local) | 1.5 / 3 | The second issue and a bonus finding |
| GPT-6 Luna, GPT-5.6 Terra, GLM-5.3-Flash | 0 to 1 / 3 | At most the second issue or a bonus |
| Most other local models, and Claude Sonnet 4.5 | 0 / 3 | Approved the change; several repeated a wrong assumption from the change’s own plan |
Swift and two other Qwen3.8-27B setups also found the second issue, with lower scores.
8 · Code, C++ bug
Symptom only: progress updates that should be throttled happen for every folder and file. The model has to find the faulty comparison (it was always true, so the throttle never throttled) in three places and fix it.
Most models found and fixed it, so the difference is in the details. Windows’ millisecond counter is 32 bits and wraps around about every 49.7 days. A fix written as now > last + interval can stop updating right after the wrap; now - last >= interval stays correct. Eleven setups scored 6 of 6, including two local ones (Nemotron 3.5 Super and Qwen3.8-27B at Q4). GLM-5.3-Flash did it in under a minute.
Three new models
Claude Sonnet 4.5 (thinking): a yardstick from a year ago
We added an older frontier model to see where local models stand against the cloud of about a year ago. Sonnet 4.5 scored 45% on all eight benchmarks, the same as Qwen3.6-35B-A3B at Q6_K running on our desktop (45%), and close on both halves: 79% against 81% on code, 10% against 8% on reviews. Today’s Claude Sonnet 5.5 high scores 86%, and most of that gain is in reviewing: 72% against 10%. Artificial Analysis agrees: it rates Sonnet 4.5 (reasoning) 21 on its Intelligence Index, near Qwen3.6’s 18. Sonnet 4.5 ran in Cursor, which offers thinking on or off with no effort level.
Nemotron 3.5 Super VL 120B-A12B: Qwen3.6’s quality at twice the time
NVIDIA’s new mixture-of-experts model, at Unsloth’s UD-Q4_K_XL quant (81 GB). It fits in our 96 GB of graphics memory with a 256K-token context (88 GB) and in real runs reads about 310 tokens a second and writes about 27. We hoped it would be a faster Qwen3.8-27B. It scored 46% on all eight benchmarks: about Qwen3.6-35B-A3B (45% in 45 minutes) at twice the time (96 minutes), and well below Qwen3.8-27B (63–65%, about 300 minutes). Code was strong (85%, with perfect scores on benchmarks 5 and 8); reviews were weak, as they are for every local model. Its vision add-on did not load in Lemonade’s llama.cpp, so we ran it text only.
Nemotron 3.5 Bolt 74B-A9B: Super’s score in a quarter of the memory
Bolt is a smaller relative of Super (74B parameters, 9B active) that NVIDIA compressed to about 2 bits per weight so it fits in 24 GB. It needs a newer llama.cpp (build b11702) than Lemonade includes, so we pointed Lemonade at that build for Bolt alone. It scored 46% on all eight benchmarks, about the same as Super and Qwen3.6, using 24 GB of graphics memory instead of Super’s 88, in about 125 minutes. Its first try at benchmark 6 went wrong for a reason outside the model’s reasoning: it tried to build by starting Visual Studio with an option that does not exist, Visual Studio opened a help window that waited for someone to click Close, and nobody did at 3 AM. The model sat idle for 104 minutes until our 2-hour limit stopped it. We now block Visual Studio in benchmark runs (builds go through MSBuild) and re-ran that benchmark: 20 minutes, and a better fix. On another benchmark Bolt reported every test passing while two of its own new tests failed.
A side result: a newer llama.cpp reads faster
While we had that newer llama.cpp, we timed it against Lemonade’s own build in the same session. Prompt reading was 37% faster for Qwen3.6-35B-A3B and 7–8% faster for Qwen3.8-27B; writing speed did not change. We will move to it when Lemonade includes it, and re-time every model then.
Where local models stand
| Setup | All 8 | Reviews (1–3, 7) | Code (4–6, 8) | Time, all 8 |
|---|---|---|---|---|
| Claude Sonnet 5.5 high (cloud) | 86% | 72% | 100% | 21 min |
| GPT-6.1 Sol low (cloud) | 78% | 61% | 95% | 15 min |
| Qwen3.8-27B UD-Q6_K, medium (local) | 64% | 37% | 91% | 290 min |
| GLM-5.3-Flash high (cloud) | 60% | 23% | 96% | 27 min |
| Nemotron 3.5 Super (local) | 46% | 8% | 85% | 96 min |
| Qwen3.6-35B-A3B UD-Q6_K (local) | 45% | 8% | 81% | 45 min |
| Claude Sonnet 4.5 thinking (cloud) | 45% | 10% | 79% | 21 min |
Shaded rows run locally on our desktop. On code, the best local models are close to the cloud: Qwen3.8-27B scores 90–95% on the four coding benchmarks against 95–100% for the best cloud setups, but takes 15 to 20 times as long. On reviews the gap is wide: the best local setups find about a third of what our real reviews found, the best cloud setups 61–72%. All percentages here are our benchmarks only, without the Artificial Analysis blend the dashboard chart can add.
Good value in the cloud
- GPT-6.1 Sol low: 78% on all eight benchmarks in 15 minutes, and one of the few setups that found benchmark 7’s main issue.
- Claude Haiku 5.5 xhigh: 81%, a small model close to the top.
- GLM-5.3-Flash high: a low-cost model that scored 96% on code but 23% on reviews. Good for clear coding tasks, not for checking work.
How we test, tightened
- One new chat per run. Once, by accident, we switched models inside a chat that had just fixed a bug, and the new model went looking for its predecessor’s exact fix. We threw that run out, and we now check every session log for a single model.
- Session logs checked. Before scoring a cloud run we read the tool’s own log: one model, memory off, no outside tools, nothing read from outside the benchmark.
- Local runs overnight. After a fresh restart, one model at a time, with a 2-hour limit per run, and every benchmark reset before the next run.
- Flash attention stays on. Turning it off looked faster on a single prompt, but in real agent runs the two larger Gemma models wrote 30–60% slower, so those runs are not on the dashboard.
Next: looking under the hood
A small monitor records what the PC is doing about once a second during every local run: GPU, CPU and disk use, memory, and whether the model is reading, writing, or waiting while the agent runs a tool. The first numbers already show differences worth a closer look: while the model was working, the GPU was busy nearly all of the time for some models and under half of it for Gemma 4 31B on one benchmark. We plan to turn this into a per-model view of how each one uses the hardware, and to write it up separately.
Also coming: Claude Sonnet 4 as a second, older yardstick; Qwen3.8 Flash-Next-Coder; and a vision benchmark for image keywords.