Mo-Search Local-first desktop search

News · 2026-10-14

AI models on our real code: local vs cloud.

We build MoSearch, MoContext and MoVault with AI help every day, so we test the models on the same work: reviewing real changes, planning a feature, and making C# and C++ changes in our own codebase. Here is how today's cloud models compare with models running free on a desktop PC.

The test machine: one mini PC, 128 GB, no separate graphics card

Minisforum MS-S1 Recipe

Every local result on this page ran on a Minisforum MS-S1 MAX: an AMD Ryzen AI Max+ 395 (“Strix Halo”) with Radeon 8060S graphics and 128 GB of LPDDR5X-8000 shared by the CPU and GPU. In AMD’s Adrenalin app we give the GPU 96 GB of it, enough for a 27B model at Q6_K with a 256K-token context, or two mid-size models loaded at once. The models run in AMD’s Lemonade server (llama.cpp on Vulkan) under Windows 11, driven by the OpenCode coding agent. Nothing leaves the machine.

How we tune before we benchmark. Each setting change is timed on the same 6,000-token code prompt with a 256-token reply, three runs, comparing runs two and three (the first is a warm-up), then confirmed on a 20,000-token prompt. Writing speed is capped by memory bandwidth (about 256 GB/s), so tuning mostly moves prompt reading. One setting paid off: the micro-batch size, which helped the dense 27B model at 128 (+7%) and the 35B mixture-of-experts model at 1024 (+20%), opposite directions, so measure each model. Flash attention, full GPU offload and the Windows power mode were already at their best; the ROCm backend did not see the GPU, so we stay on Vulkan. Only then does each model run the six benchmarks below, with fixed answer keys.

Composite
Single bench
AA weight
Show
Local · free, runs all dayCloud · paid per use

All six benches

Local AI Speed on our benches

Average tokens per second over each model's bench runs, from llama.cpp's own counters. Reading is taking in the prompt and files; writing is generating, thinking included. These are long agent sessions, so they run slower than short-prompt tests. Sorted by combined speed.

Fun (and a little scary) things the models did

  • The one-line test file. Gemma 4 26B “fixed” a feature by rewriting a 1,000-line test file onto a single line. The build failed.
  • “All tests pass.” Claude Haiku 4.5 wrote a test that failed, then reported that every test passed.
  • The fix that does nothing. Nemotron 3.5 Lightning was asked to make a second program follow the dark/light switch. It never sent that program a message, then said the change works immediately.
  • While I’m in here… Gemma 4 31B, working on the theme fix, also renamed an unrelated license message.
  • Confident and wrong. Swift-Qwen3.8 cited “documented .NET behavior” for its only finding. We ran it: .NET doesn’t do that.
  • “Ready to commit.” On our hardest review, GPT-6 Luna at medium, high and max effort all called a change with a blocking bug ready; max took 8 minutes to say so. Grok 4.6 found the blocker; the newer Grok 4.7 argued it was fine.
  • Thinking less took longer. Qwen3.8 at low thinking wrote one and a half to two times more text than at medium, so it was no faster.
  • A model that remembered the answers. Codex kept memories across our benchmark runs, so later GPT runs started out knowing what earlier runs had found. One “discovery” vanished when we re-ran it with memories off. Every affected run has been redone.
  • The blue screen. With nobody at the PC, Windows turned the screen off in the middle of a long local run; with the GPU at full load, the graphics driver missed its power deadline and Windows crashed. Long runs now keep the screen on.
  • And an honest one. GPT-5.6 Terra at xhigh effort could not run its tests because Windows blocked a freshly built file, and said exactly that instead of claiming success.

The six benches

Every bench is a frozen snapshot of our own repository at a real moment in development, with history removed so a model cannot look up the answer. Each answer key comes from the real review or the real fix that shipped.

  • 1 · Review, hard. Coming in v26.1 A 13-file change adding MoContext's Folder / Whole index access ceiling (+880 lines). The key is the one blocker and three issues our real review found; all were fixed before the change was committed.
  • 2 · Review, medium. MoContext's daily automatic backup (7 files, +1,360 lines). Three issues from the real review, plus two bonus issues that AI reviewers found later.
  • 3 · Plan. Write the worker plan for that auto-backup from the orchestrator's prompt, scored out of 19 against the real plan review.
  • 4 · Code, apply a review. Apply three review fixes in C#, build, and pass the 166-test suite.
  • 5 · Code, small feature. A real item we are fixing: an unreachable backup folder should back off for an hour, not ten minutes. Change the code, add a test that would catch a regression, pass the suite.
  • 6 · Code, C++/MFC. Symptom only: "a running AutoIndex keeps its old colors when you switch MoSearch between dark and light". The model has to find the cross-process cause in MoSearch's C++ and fix it; scored against the fix we shipped.

How it is scored, and what to keep in mind

  • Quality is points earned against each bench's fixed answer key. Wrong claims cost points; a failing build or test caps a coding score at 50%. Composites average a setup's benches; time is their sum.
  • Who scored it. Claude (Anthropic's Opus 5.5) scored every run, including runs by Claude models, against those fixed keys. The keys came first, from our real reviews.
  • Real findings. Some review findings are real issues in our code. We fix them outside the benchmark, so the table words them plainly rather than in detail.
  • Small samples. One or two runs per setup. Reviews vary a lot from run to run, so close scores are within noise.
  • Times. Local times are measured. Cloud times are wall-clock readings taken while running each model in the editor.
  • AA weight blends in the Artificial Analysis Intelligence Index (v4.3.2, read from their public model pages 2026-10-10 to 12), scaled so the best tested setup is 100%. Estimated entries (quantized local models, nearest effort setting) count at half weight and show hollow. Our six-bench ranking and the AA index agree closely (rank correlation 0.85), so the knob mostly moves the setups where they disagree.
  • Local setup. One desktop PC: AMD Ryzen AI Max+ 395 (Strix Halo) with 128 GB, 96 GB of it given to the GPU, running Lemonade (llama.cpp, Vulkan) and the OpenCode agent. Cloud models ran in VS Code and Cursor.

Setups