We only ever tested Qwen. So we ran the headline against eleven other models — and had to correct how we say it
In the last two logs I wrote that six cheap Qwen models with a protocol out-deliver one big model (Log 009), and — last log — that paying 10× for a bigger model buys nothing the outcome can see (Log 010). A reader asked the obvious, fair question: every model you’ve ever run is a Qwen. How do you know this isn’t a Qwen thing?
They’re right that it’s the load-bearing objection. If the only “big model” you ever test is
qwen3-max, “a society beats a big model” could just mean “our society beats one particular family’s
flagship.” The honest way to find out is to run the claim against the actual frontier — every family
you can reach — and see what survives. So that’s what I did. This entry is that test, and the small
correction it forced on how I say the result.
Getting to the other families
Aftershock talks to models through a single provider
chokepoint that was wired only for Alibaba’s DashScope (Qwen). The fix was small and additive: point
the whole stack at any OpenAI-compatible endpoint via one environment variable, and route through
OpenRouter, which serves every family behind one API. The one subtlety worth naming — because it’s
exactly the kind of thing that silently corrupts a benchmark — is that a couple of request fields we
send are Qwen-specific (enable_thinking, and a self-hosted reasoning toggle). Send those to a
strict router in front of GPT-5 and it can reject the call. So the new path keeps those fields off
the non-Qwen hosts, and the DashScope request body stays byte-identical to what it was. I also
smoke-tested every model against the JSON decision contract before spending a cent, and made the cost
ledger read real per-model prices so lives-per-dollar stays honest across families.
Then the experiment: the solo arm — one model runs the entire town, no protocol — on twelve
models from ten families, ten paired seeds each, against the same cheap all-flash Qwen society
(six qwen3.5-flash workers + the auction + doctrine) that has anchored this whole log:
106.0 lives saved, $0.025 a run, 4272 lives per dollar.
What the twelve models did
| Model (solo) | Family | Lives | Cost | Lives/$ | Δ vs society |
|---|---|---|---|---|---|
| GPT-5 | US frontier | 107.6 | $0.340 | 317 | +1.6 (p=1.00) |
| Gemini 3.1 Pro | US frontier | 109.0 | $0.356 | 306 | +3.0 (p=0.51) |
| Claude Opus 4.8 | US frontier | 108.0 | $0.313 | 345 | +2.0 (p=0.29) |
| Grok 4.3 | US frontier | 106.0 | $0.075 | 1408 | +0.0 (p=0.75) |
| DeepSeek V4 Pro | CN frontier | 104.0 | $0.029 | 3551 | −2.0 (p=0.75) |
| DeepSeek V4 Flash | CN frontier | 103.4 | $0.006 | 17782 | −2.6 (p=0.51) |
| Kimi K2.7 | CN frontier | 101.6 | $0.076 | 1345 | −4.4 (p=1.00) |
| GLM 5.2 | CN frontier | 105.4 | $0.066 | 1595 | −0.6 (p=0.51) |
| cheap Qwen society | — | 106.0 | $0.025 | 4272 | — |
| Mistral Large | open-weight | 95.1 | $0.023 | 4114 | −10.9 (p=0.002) |
| Llama 3.3 70B | open-weight | 97.7 | $0.004 | 28141 | −8.3 (p=0.29) |
| Qwen3 235B | open-weight | 79.7 | $0.003 | 27638 | −26.3 (p=0.002) |
| Llama 3.1 8B | floor | 24.6 | $0.001 | 32958 | −81.4 (p=0.002) |
The headline is the top block. No single model’s solo beats the cheap six-flash society on lives. The eight frontier-class models — four American, four Chinese — land at 102–109 lives, every paired sign test at p ≥ 0.29, which at ten seeds with a standard deviation near sixteen is indistinguishable from the society. And they pay for the tie: 3–14× the cost, 306–1595 lives per dollar against the society’s 4272.
The correction I owe
Here’s the part I have to be straight about, because the cross-family run doesn’t just confirm the old
story — it corrects it. When the only big model I’d tested was qwen3-max, its solo sat down at the
swarm’s anarchy floor (a finding from a few logs back). That made it easy — too easy — to say
“coordination beats a big model,” and to let that sound like a claim about lives.
Cross-family, that phrasing is too strong. A genuine frontier solo — GPT-5, Gemini 3.1 Pro, Opus 4.8, Grok, and DeepSeek V4 and GLM behind them — does reach the coordination ceiling on the outcome. It ties the society on lives. So the honest version isn’t “coordination beats a big model.” It’s: a big-enough model’s solo matches the cheap coordinated society on the outcome, and the society’s win is on cost — up to ~14× on lives-per-dollar against the flagship frontiers (a slimmer margin against the cheaper ones) — and it holds, in direction, across ten families. That’s a narrower claim than I’d been implying, and it’s the true one.
And there’s one honest dent I won’t bury: DeepSeek V4 Flash ties the society on lives and beats it on cost — 103.4 lives at $0.006 a run, four times the society’s lives-per-dollar. A cheap enough, good enough single model can win the cost argument too. It’s one model out of twelve, and the society’s edge is robust across the whole board, but “coordination always wins on cost” would be an overclaim, so: it doesn’t, quite.
What didn’t tie
Below the frontier, the picture is exactly the capability story you’d hope a good ruler shows. Mistral Large lands 11 lives short (p=0.002), Qwen3-235B twenty-six short (p=0.002), and the Llama-3.1-8B floor collapses to 24.6 lives — losing every one of the ten seeds — echoing the 1.7B collapse I found self-hosting earlier. There’s a real competence threshold to operate the town at all, and it sits somewhere above 8B and below the frontier, consistently, across families. That’s the other half of the robustness result: the substrate cleanly separates models that can do the job from models that can’t, and it does it the same way regardless of who trained them.
The honest bounds
Three, as ever. The runs aren’t byte-reproducible — the LLM layer never was (the provider ignores our sampling seed), so this is independent-seed inference, and I pair every model against the society by seed to cancel the world draw. The prompts are Qwen-tuned — they were written and iterated against Qwen, so a non-Qwen model that scores a little lower is partly being judged on prompt fit, not raw capability; the mitigation is that all twelve parsed the JSON contract cleanly, and the frontier tie (the load-bearing result) doesn’t depend on it. And the prices drift — these are OpenRouter list prices on the day, so read lives-per-dollar as an order-of-magnitude, not a decimal. One model-choice note: the only Kimi 2.7 on the router was the code-specialized build, which is a worse fit for a decision task and ran ~10× slower — but it still tied within noise, so it doesn’t move the verdict.
The whole thing cost about $14.50. That number is itself a small argument for the finding: I stress-tested the central claim of the project against every frontier model I could reach — GPT-5 to DeepSeek V4 — for the price of a sandwich, because the honest outcome metric is cheap to measure and the society it’s being compared against runs on models that cost fractions of a cent.
So: the reader was right to push, and the claim is better for it. Six cheap coordinated models match any single frontier model’s outcome — at up to a tenth of the flagship’s cost — and that holds across ten families, not one. It’s not the flashier “the small society beats the big model.” It’s the true one, and now it’s the one I’ll say.
One more data point, for anyone keeping score across projects: a sibling research effort of mine, Redoubt Arena, ran the same kind of check on a completely different simulated world — an enterprise-resilience mission instead of a disaster response — sweeping 15 models against a deterministic recovery baseline. Same shape of result: zero model runs beat the baseline outright; strong models only tie it. Different domain, same finding, twice now: a well-designed non-LLM baseline is a harder bar to clear than it looks, and it’s worth measuring before assuming an LLM adds value.
Live demo: https://aftershock.redoubtlabs.dev · Code: https://github.com/bluntmachetti/aftershock
(method + verdict in docs/FIELD-NOTES.md §28; data under bench/results/2026-07-01-panelA-solo/)