swarm.arnao.ai — a wind tunnel for harnesses

One line: a console where I point a swarm of cheap models (and a strategic Fable planner) at a real goal, set a hard budget, and watch the orchestration think — with an external verifier deciding what's true and a same-cost single-model baseline running next to it so the whole thing stays honest.

The reframe that makes it matter: this is not a dashboard I built to watch my agents. It's a wind tunnel where harness designs compete — same task, same dollar cap, same eval — and the output is a cost/quality frontier, not a light show. The animation is the replay layer over real evidence. Nobody remembers a dashboard; people cite a leaderboard.

Why this, why now

I keep building harnesses by hand — a swarm here, a self-improving loop there, a router that picks the cheap model until it needs the expensive one. Every one is a one-off script and a gut feeling about whether it was worth it. There's no instrument. No way to prove that ten cheap agents beat one big call at the same price, or that a verification loop actually earned its extra tokens.

swarm.arnao.ai is that instrument. You give it a goal in plain language, pick a mode, set the caps, and hit launch. You watch it run in a mission-control view — the swarm graph, the rolling tail logs, the dollars ticking up by model — and at the end you get a verified result, a full provenance trail, and a point on a cost/quality curve you can compare against every other way you could have solved it.

It's a true experiments tool: max cost, max agents, max time, max loops are a contract the run cannot break. Every run is sealed, seeded, and gets a permalink you can replay or re-run. You can fork a run, change one dial, and A/B the harness.

The thesis (this is the LinkedIn spine)

I'm deliberately not leading with "the harness matters." That fight is over and I didn't win it — Berkeley's compound-AI-systems essay, DSPy, flow engineering, and Karpathy's LLM-as-CPU riff all said it first and louder. Repackaging consensus is the opposite of naming a discipline.

What nobody has built is the metrology. Everyone agrees orchestration matters and then argues about it in anecdotes and cherry-picked blog benchmarks — no controlled conditions, no baselines, no error bars, no reproducibility. Harness engineering has no instruments. swarm.arnao.ai is the wind tunnel.

And July 2026 handed the wind tunnel its first exhibit. ARC-AGI-3 launched in March with every frontier model below 1%; by July the best official public-set score was still in the low teens (the comparison point [schema] itself cites). Then Impossible Research's [schema] changed nothing about the model — the harness makes the model write each game's mechanism as an executable program, backtest it against observed reality, and plan inside it — and posted ~99% on the public set with Opus 4.8 + Fable 5 (announcement). Same weights, roughly 7× the score — the loudest harness-is-the-lever claim of the season. Note the word: claim, not receipt. The HN thread did the wind tunnel's job by hand and came back empty: no held-out score published, spend unconfirmed (commenters estimated ~$25k — hearsay, labeled as such), no same-cost baseline, no error bars. If the number holds, harness engineering just had its defining demo; if it collapses, the field had no instrument that could have told us either way. That's the gap this tool exists to close — the argument made by someone else, at full volume.

That's the ownable claim, and it reframes everything else. The beliefs the field proclaims become hypotheses the instrument tests, not proclamations I add to the pile:

Compose an economy of models, don't pick one — cheap labor for the many moves, an expensive consultant (Fable 5) for the few pivotal ones. The tool measures when that actually beats one big call.
The unit of design is the topology, not the prompt — prompts are variables; topologies are the experiment.
Reliability comes from loops, not from IQ — generation is cheap; verification is the moat. The tool proves it by killing unverified winners on screen.
Cost is a first-class dial, and the Pareto frontier is the truth — "Can it do X?" is the wrong question; "what does X cost, and where's the frontier?" is the right one.

And the meta-move only a tool can make: treat harnesses like software — versioned, diffed, benchmarked, forked, with error bars. "I built the instrument, and here's what it measured" survives a skeptic. "Here are my five beliefs about orchestration" is LinkedIn wallpaper.

Lines I can actually post:

"Harness engineering has no instruments. So I built the wind tunnel."
"Capability without cost is marketing. The Pareto frontier is the receipt." (the best line — own it)
"Generation got cheap. Verification is the moat."
"Nobody in agent-land shows error bars. That's the whole problem."
"Stop asking which model. Start designing the economy of models."

(Prepared answer for the obvious knife — Anthropic's finding that token budget explains ~80% of multi-agent variance: "Exactly. That's why every result here is same-cost. The instrument exists to isolate the other 20%.")

How it works

See the target-state architecture for the full picture. The flow:

Control plane (guardrails). Before anything runs, the experiment contract is set: MAX COST $ · MAX AGENTS · MAX TIME · MAX LOOPS. This is a live budget enforced everywhere — breach means a graceful stop, never a silent overrun. A dry-run cost estimate shows before you spend a cent.

Planner — Fable 5, strategic and budget-capped. One premium call decomposes the goal into independent worker tasks and picks the mode's strategy. Fable is a thin sliver of the cost; the cheap models do the volume. This is the mixture-of-models thesis made concrete.

Cheap-capable worker swarm. Haiku / Gemini-Flash / Sonnet, routed cheapest-first. A worker that hits a low-confidence gate escalates up to Fable — and that escalation is a visible beat on screen, not a hidden branch.

SWARM — N candidates explored in parallel, each with its own context.
LOOP — evaluator → optimizer, repeated, capped at 2–3 iterations with stop-on-converge. Returns diminish sharply — reported figures put iteration 1 at ~60% of errors caught, iteration 2 ~25%, with over-optimization degrading output past pass 4–5 (Future AGI) — so the cap and the convergence check are load-bearing, not decoration.
COMBO — a loop of swarms.

External verifier — the gate. This is the load-bearing wall. Nothing is "done" until something the swarm cannot fake says so: an objective scorer, a unit-test suite, a benchmark, a solver's optimum, or an adversarial verify pass that kills weak winners by majority-refute. Without an external signal, a self-grading loop is theater — LLMs largely can't self-correct on their own opinion (Huang et al., DeepMind). Only verified candidates advance.

Synthesis + the same-cost baseline. The winner is assembled from verified parts (best candidate plus grafted ideas from runners-up), with a full provenance trail: plan → tried → killed → won. And running alongside every mission is the honesty check — one Fable call at the swarm's exact dollar spend. It answers the one question a skeptic always asks: did the swarm actually beat one big model at equal cost? (Anthropic's own finding: token budget alone explains ~80% of multi-agent performance variance. If you can't beat the same-cost baseline, you just spent more money.)

The mission-control console

One screen answers three questions at a glance: who is doing what, why, and what it costs.

Live swarm graph — nodes are agents, edges are handoffs; state is visible (active / verifying / killed / done). The Fable planner is the purple hub; cheap workers are the amber constellation.
Rolling tail logs — a mission-control event stream (handoffs, escalations, verifier verdicts), auto-scrolling, not a raw dump.
Cost-by-model strip — dollars ticking up per model tier, cheap-dominant with Fable a thin sliver. The thesis, live.
Meters vs. caps — cost / agents / time / loops filling toward their ceilings.
Convergence curve (loop mode) — score vs. iteration climbing to the threshold, then "✓ converged, stopped early." The "it's actually learning" money shot, honest because it's a real external metric.
Goal input + mode toggle + RUN / PAUSE / STOP, plus a global kill switch.

The controls and dashboard live on the mission-control view — this slots into the existing Command console as a new tab.

The demos (ranked by how well they survive a skeptic)

The test each must pass: an external oracle (not my own judgment), can't be pre-baked, visually legible, one clear aha. "Audience picks the input" is a variance grenade with cheap models, so the honest version is a curated-and-disclosed pool: ~15 pre-vetted tasks, audience picks which one, and I say out loud that I vetted them for demo-time feasibility (not for outcome). Curation disclosed is credibility; curation discovered later is rigging.

A. The same-cost race — swarm vs. one Fable call, split-screen. (Lead with this.) "AI fixes a bug" is table stakes in 2026 and gets polite nods. The race is what gets screenshotted. Left panel: one Fable 5 call with budget $X. Right panel: the swarm with the identical $X. Same real open issue, same repo test-suite as oracle (nobody in the room wrote it), both cost meters ticking against the same ceiling. The money moment: the single model burns its budget and ships a patch the tests reject, while the swarm's verifier visibly kills two bad candidates and passes a third. That's the thesis as theater — and it's honest theater because the oracle is external. The bug-fix is the substrate; the same-cost head-to-head is the demo. One structural honesty rule, because a live race is n=1 and sometimes the single model wins (this document says so itself): the demo leads with the precomputed n≥5 distribution, variance bands on screen, then runs one live instance as flavor — and the live result gets appended to the distribution in front of the room, whatever it is. The instrument doesn't care who wins. That line, delivered while the dot lands wherever it lands, is the demo.

B. Beat a benchmark the models have never seen. (Most rig-proof.) Questions/data published after the models' training cutoff. Can't be memorized; ground truth checkable live. Aha: orchestration + tools beats a bigger tool-less model.

C. Red-team vs. blue-team self-play. (Most legible.) One swarm writes a small program or claim; another tries to break it. Score = did blue survive. A closed adversarial game with a hard win condition — you can't rig both sides. Aha: rigor emerges from cheap models pointed at each other.

The trading demo — a two-act kill (this is how Byron asked for it, done honestly). A swarm searching thousands of strategies over historical data is literally an overfitting machine; a winning equity curve is the most screenshotted object on the internet and the fastest way to look like every backtest charlatan on FinTwit. So I don't sell the win — I let the room fall for it, then execute it:

Act 1 — "What the swarm believes." Run the search for real. The swarm surfaces its best in-sample candidate and renders the seductive result exactly as you'd want: green equity curve, Sharpe 1.62, CAGR 18.4% vs. buy-and-hold. Let the room feel the pull. This is the trading demo — a real swarm really finding a real in-sample winner.
Act 2 — "What the verifier proves." The same external-verifier gate that kills bad patches in Demo A now runs on the strategy, live: walk-forward out-of-sample split, Deflated Sharpe, Probability-of-Backtest-Overfitting, a shuffle test. The out-of-sample curve flatlines on screen. The verdict stamp lands in the identical visual language as a killed swarm candidate: KILLED — OVERFIT (in-sample Sharpe 1.62 → OOS 0.1, PBO 0.87). The curve the audience just coveted dies in front of them.

Hard rule, enforced in result.html: the winning curve never appears in any frame or screenshot without the verdict stamp in the same frame. The strongest possible ending is that nothing survives: "the swarm generated 4,000 strategies and the harness refused to lie to me about a single one" beats any Sharpe ratio, and it makes the trading demo the emotional proof of the verification thesis rather than a liability. Never touches a real brokerage. (Safer sibling, same optimization flavor without the FinTwit smell: solve a public routing problem against an exact OR-Tools solver — solver = ground-truth optimum, routes redraw on a map, no profit claims.)

The experiment library — greenfield in front, solved runs behind it

The console ships with two kinds of experiments, and the split is the on-ramp design:

Solved experiments — the gallery. A shelf of finished, sealed runs anyone can open without spending a cent: the same-cost race with its verifier kills, the seating chart converging, the trading strategy dying its two-act death. Every gallery piece is a replay (deterministic, from cache, free) with a fork button: change one dial — worker count, loop cap, model tier, one constraint — and re-run it as your own experiment. Tweak-one-dial forks are how a visitor becomes an experimenter: the parent run is the control group, the fork is the hypothesis, and the console renders the delta. Nobody starts from a blank page.

Greenfield — bring a new problem. Type a goal the console has never seen. The planner drafts the experiment contract (caps, mode, worker plan) and proposes an oracle — but the planner proposes, it never approves. The system being measured must never pick its own measuring stick; that conflict of interest is the first thing a serious critic looks for, so the design closes it structurally: oracles come from a typed library (unit-test suite, exact solver, held-out data, executable checker — free-form checkers flagged as such), the proposed oracle is shown to you and approved as part of the sealed contract before a cent is spent, and a model outside the swarm runs an adversarial pass on the oracle itself (can it be gamed? does the checker actually check?). The greenfield rule is absolute: no approved oracle, no measured run. If the best available check is an LLM opinion, the run still executes — but its result card is watermarked exploratory structurally, in the pixels, with the same anti-crop treatment as the trading verdict stamp. Never measured. That gate is what keeps greenfield from collapsing into a chatbot with extra steps.

The canonical gallery walkthrough: seat 120 wedding guests. Complex enough to be real — it's an NP-hard constraint problem — and relatable enough that nobody in any audience needs it explained: couples together, feuding relatives apart, table capacities respected, the friends-of-nobody table minimized. The oracle is arithmetic, not opinion: a constraint checker counts violations, and OR-Tools CP-SAT computes the exact optimum for the instance, so "how close did the swarm get" is a number, not a vibe. One run demos all three primitives on screen: SWARM (five cheap models propose seatings in parallel; worst candidates killed on violation count), LOOP (the violations-per-iteration curve stepping down to "✓ converged"), and the same-cost baseline (one Fable call at identical spend — usually beaten, and when it isn't, the chart says so, because that's what an instrument is for). Typical run cost: under $0.60. And the obvious knife gets defused on screen, in one sentence, before any OR person says it out loud: CP-SAT solves this instance exactly, in milliseconds, for pennies — tasks with known optima are how the instrument is calibrated; then the calibrated instrument points at tasks no solver can touch. Seating is a calibration target, not a use case, and the result card says so. Second family, same shape: delivery routing against the solver's optimum, drawn on a map. To be explicit about a decision this document previously left open: the launch lead is Demo A, the same-cost race. Seating is the follow-along; the ARC-AGI-3 arena is the Phase-3 article. One lead, chosen.

Who can run it — and the $2.00 rule

Watching is public. Spending requires a name. Every sealed run, replay, and gallery piece is open to anyone with the link — replays serve from cache and cost nothing, and that includes the precomputed n≥5 galleries with their variance bands: the instrument's flagship output is free to everyone, always. Anything that spends real tokens — a fork, a re-run, a greenfield mission — requires Google sign-in first. One click, no passwords. What the sign-in is for is stated plainly on the login screen:

Abuse prevention and spend attribution. Every run bills to an identity, so a viral day degrades gracefully into "join the queue" instead of a four-figure surprise, and misuse is revocable per account.
A per-account allowance (default $3/day) for metered runs — and bring-your-own-API-key removes the ceiling entirely, which is how a real experimenter runs a real two-arm n≥5 A/B (~$6+) without asking anyone's permission. The free tier watches and forks; the BYO tier researches.
Aggregate usage research, disclosed. Which task families get run, fork-vs-greenfield ratio, where runs get abandoned — analyzed in aggregate to improve the instrument, published only as aggregate stats. Role ("builder / student / other") is optional and self-declared, never inferred from your account. Run ownership on public permalinks is opt-in — the default is an anonymous "verified account" badge, because publishing someone's identity is their decision, not mine.

The $2.00 rule. Before any run spends a cent, the dry-run estimator prices it. Estimate over $2.00 → the console warns and steers: it proposes the nearest in-bounds variant — fewer workers, cheaper tier, smaller n, tighter loop cap — with its predicted cost and what you give up (fewer reps = wider error bars; the tradeoff is rendered, not hidden). Accepting the steer is one click; going over $2.00 anyway is deliberate, visible, and still wall-capped by the contract (MAX COST remains the hard stop). The estimator's estimate-to-actual gap is tracked and shown on every result card — an estimator that lies is worse than no estimator. Defaults everywhere are tuned so the canonical demos land between $0.30 and $1.50.

What makes it a true experiments tool (not a dashboard)

Statistical repetition — the feature that makes it an instrument and not a lava lamp with a ledger. Swarms are stochastic; a single race is n=1 and tells you nothing — run it again and the result can flip. So every A/B and every Pareto point is n≥5 runs with variance bands, and the verdict is "B beats A, p<0.05 at this budget" or "no significant difference." Nobody in agent-land ships error bars. Error bars are the whole difference between an instrument and a screensaver — and, quietly, the best marketing.
Sealed, replayable runs. Every run captures seed, model IDs+versions, prompts, tools, caps, and the full event trace with cached completions. /run/ab39f2 replays deterministically (byte-identical from cache) or re-runs (fresh, stochastic). That distinction is the honesty guarantee — and I state it precisely, because "reproducible" is a claim you get caught on.
Same-cost baseline, attached automatically to every run. One Fable call at the swarm's exact spend. Not a feature you opt into — the default comparison on every mission.
Harness A/B. Race "one strong agent" vs. "10 cheap + 1 judge" vs. "debate-and-adjudicate" on the same task, budget, and eval — each n≥5, with significance. The thesis, empirical instead of rhetorical.
Harness diff — the numbers, not the story. The quantitative diff is real and novel and measured straight from the traces: "B spent 3× more on verification; A's candidate diversity collapsed at step 4." I ship those numbers. The causal narrative ("B's judge caught a hallucination") is an LLM opinion over traces — I render it, clearly labeled as generated, never as measured fact. Most tools stop at the leaderboard; the quantitative diff is the part nobody's seen.
Cost/quality Pareto frontier — scoped per task-family. Points in (cost, quality) space compose into a frontier only within a task family (bug-fix, research-summary, optimization); "quality 0.91" doesn't mean the same thing across families, so the frontier is always labeled with its family. Drag a budget slider and the recommended topology changes under your finger. The signature visual — kept honest.
Fork + a library of plays. Named topologies (map-reduce, debate-and-judge, generate-critique-revise, tournament). Fork any run, change one variable, see your delta vs. the parent. Git for harnesses.

Roadmap — one real thing before four fake ones

The trap this kind of project dies in: a gorgeous front end over a faked backend. The visible 10% is animation; the invisible 90% is the orchestration engine, deterministic replay, cost accounting, and eval harness. So:

Phase 0 — Run 0: point the wind tunnel at my own citations. This proposal leans on the field's most-cited folklore — Anthropic's "token budget explains ~80% of variance", the loop-decay curve. Before launch, replicate them with the instrument, pre-registered: hypothesis and analysis plan published (and hashed) before the runs execute. The launch article writes itself — "the field argues in anecdotes, so the first thing my instrument did was test the field's three most-cited claims; here's what survived" — and nobody can accuse me of cherry-picking claims the field picked for me.
Phase 1 — one real vertical slice. Demo A (repo bug-fix) end-to-end, genuinely working: real swarm over cheap models on my fleet, real test-suite verifier, real cost ledger, sealed + replayable + forkable. If one demo is truly real, the tool is real.
Phase 2 — the frontier. Add the same-cost baseline, harness A/B, and the Pareto view. Now it's an instrument, not a toy.
Phase 3 — open the wind tunnel (the endgame). Model leaderboards are everywhere; a **leaderboard of harnesses — fixed tasks, fixed dollar budgets, external oracles, anyone submits a topology, every entry sealed and replayable — does not exist. When "state of the art on HarnessBench-BugFix at $2.00**" becomes a phrase other builders use, the discipline has a venue and I own it. The person who named a discipline isn't the one with the best dashboard; it's the one whose arena other people compete in and cite. Harness diff, the play library, and self-designing harness all live here. (Self-designing harness stays aspirational, not a demo — a real search doesn't produce legible generational improvement in five minutes.)

The named first arena event: the ARC-AGI-3 harness frontier. The arena half-exists already — an external oracle nobody can fake (the game engine), published human baselines, and a public field: [schema] at ~99% (spend unconfirmed), Executable World Models at 58% RHAE on a $200/mo subscription, Duck Harness an order of magnitude cheaper again on open weights. But "everyone at $2.00 a game" would be a category error — capping a $25k-class harness at $2 measures the cap, not the harness. The honest event is the budget frontier: each runnable harness at several budget points ($0.50 / $2 / $10 a game, n≥5, variance bands), plotted as cost-vs-score curves. Entrants must reproduce on my hardware before their name goes on the board — Duck is open source and qualifies today, EWM's method is published, and [schema] gets an open invitation rather than a proxy reimplementation I'd then be accountable for. (Their metric is also a design lesson: RHAE squares the human-efficiency ratio, so doubling actions quarters the score — punishing exploration. Proof that metric shape steers harness design, and why this leaderboard's headline number is quality-per-dollar with the formula published.)

The mockups in this proposal (index.html, experiment.html, result.html, loop.html) are the look and feel — simulated data, real design. They show what Phase 1–2 renders to; they are not the working engine.

Self-critique — Fable adversarial pass

I ran this whole proposal through Fable 5 as a ruthless critic (demoability / utility / harness-LinkedIn). It changed the spine. What it caught, and what I did:

"The harness is the product" is not mine to claim — it's prior art (compound-AI-systems, DSPy, Karpathy's CPU line). Leading with it makes me a repackager. Fix: inverted the thesis to the metrology — "harness engineering has no instruments; here's the wind tunnel" — which is ownable. Cut the CPU line.
Every run is n=1 — a single stochastic race proves nothing. Fix: made statistical repetition (n≥5, variance bands, significance verdicts) the must-have feature, not a nice-to-have.
The bug-fix demo is table stakes and lands as a polite nod. Fix: rebuilt the lead demo as the split-screen same-cost race, which is the thesis as honest theater.
The winning-backtest mockup is the single most reputation-dangerous artifact — a screenshot of "Sharpe 1.62 beats buy-and-hold" makes me indistinguishable from a FinTwit charlatan, and winning curves always escape. Fix: rebuilt result.html as a two-act kill; the winning curve never renders without its verdict stamp in the same frame.
Fabricated-precision numbers (the loop 60%/25% stats) read as measured findings. Fix: sourced them inline, or they'd unravel trust in the real citations next to them.
Harness diff as a causal story is grading with the tech I'm selling. Fix: ship the quantitative diff as fact, label the narrative as generated.
The Pareto frontier is chart junk across task families. Fix: scoped it per family.

The one bold idea it surfaced is now the endgame below.

Second Fable pass (2026-07-16, on the greenfield / library / access additions). Same treatment, new material. What it caught, and what changed:

I called schema's 99% a "receipt" while proving in the same paragraph that it's unadjudicated. Fix: it's a claim — the exhibit is that nobody can check it, not that it's true. My credibility now attaches to having flagged it, not to the number.
Tweet as primary source, HN hearsay stated as fact — inside a metrology manifesto. Fix: the score cites the project page; the $25k is labeled an unconfirmed commenter estimate.
The bake-off was a category error — capping a $25k-class harness at $2/game measures the cap, not the harness. Fix: reframed as a budget-frontier event (several price points, cost-vs-score curves), with a reproducibility gate before any entrant is named.
The greenfield oracle was circular — the planner picked its own measuring stick. Fix: typed oracle library, user-approved sealed contract before spend, adversarial oracle check from outside the swarm, anti-crop exploratory watermark.
The auth copy told users they're content for my personal brand. Fix: consent-forward rewrite — abuse prevention, allowance, disclosed aggregate research; role self-declared; permalink ownership opt-in.
The $3/day cap couldn't afford the tool's own flagship feature (a two-arm n≥5 A/B costs ~$6+). Fix: precomputed n≥5 galleries free for everyone; bring-your-own-API-key removes the ceiling.
Three candidate lead demos is a decision, not a lineup. Fix: decided — the same-cost race leads (restructured distribution-first: "the instrument doesn't care who wins"), seating is the gallery follow-along with its CP-SAT knife defused as calibration, ARC is the Phase-3 article.

Its bold idea — Run 0, pre-registered replication of the field's most-cited claims as the launch content — is now Phase 0 of the roadmap.

The honest risks (so I'm not caught by them)

"Looks fake" is the default read of any animated dashboard. Antidote: audience picks the input, raw logs + verifier verdicts are visible, and the permalink lets them re-run it after I leave the room. Click a glowing node → see the actual tokens, or it's a screensaver.
Dead air kills live demos. Real swarms over cheap models are slow. Decide per demo what's live vs. replayed, label the replays (labeling increases trust), pre-warm, keep the scrubber moving.
Cost runaway, especially with self-improving loops. Caps aren't enough: hard global kill switch, per-run and per-day ceilings, spawn-depth limits, dry-run estimate before spend, sandboxed tools. One loop that bills $400 overnight is a story I don't want.
Unfalsifiable metrics. If quality = an LLM judge, critics say I'm grading with the tech I'm selling. Use external oracles wherever possible; when it is an LLM judge, say so and show judge-vs-human agreement.
Model/pricing drift. "Cheap-but-capable" and its price move monthly. I make that a feature — the frontier recomputes as prices change — and acknowledge the conclusions expire.
"So what does it produce?" Every run drops a durable artifact someone wants — a patch, a report, a strategy spec, a reusable play — or it's a very expensive lava lamp.

Mockups

index.html — the live mission-control swarm console
experiment.html — the experiment setup (goal, mode, model routing, guardrails)
result.html — a finished run with provenance, cost breakdown, and the two-act overfitting kill
loop.html — a self-improving loop converging live
architecture.svg — target-state architecture
present.html — the self-running narrated demo, in my voice (script)