Can a small local model replace the cloud for intent routing?

Every agentic product has a router hiding in it. Before an agent loop can do the expensive, careful work of writing the email or building the deck, something cheap has to decide what the user even wants — "rename this tab," "summarize this page," "email this to Alice." Get that decision right and cheap, and the whole system feels instant and free. Get it wrong, and you either burn a frontier-model call on "delete row" or you confidently do the wrong thing.

In Veya that router is called Mendel, and it's a 3-tier cascade:

• Tier 0 — a regex matcher plus a hash-embedding ANN (approximate nearest neighbor over action exemplars). Both are fully offline, in-process JavaScript. A decision here costs ~1 ms.
• Tier 1 — an LLM classifier that emits a single pick_action choice, reached only when Tier 0 escalates. Historically this was always remote (Claude Haiku 4.5 via a hub). We wanted it to run locally first, on qwen2.5:3b through ollama, and fall back to remote only when the local model isn't confident.
• The agent loop — the actual worker, Sonnet 4.6, which runs after routing when a prompt needs real multi-step work.

The question that kicked off this whole study was simple and strategic:

Can a small local model replace the cloud for routing?

If yes, most routing decisions never leave the machine — no per-decision cost, no network dependency, full privacy, and it works on a plane. The answer turned out to be "mostly yes, but not for the reason you'd think, and not the model we shipped." Getting there meant running thirteen models through a real replay harness, and along the way we found four dead ends, one genuinely dangerous bug, and a couple of surprising, durable wins.

This is the long version. Every number below comes from experiments run through the real Mendel kernel and clients — not ad-hoc replays — against real (if synthetic-head-heavy) traffic. The negative results are the point, so they're up front, not buried.

The lay of the land: what Tier 0 actually does today

Our shipping eval is 157 authored cases across 20 categories. Here's how the cascade splits the load with the production embedder (HashEmbeddings(64), Tier 1 off):

Two things jump out. First, the ANN is doing ~38% of the work (59 of 157) at 100% precision and sub-millisecond cost — it's not a toy. Second, 60 cases escalate, and that's where routing gets expensive: escalation is what a Tier-1 LLM call (local or remote) actually is. The entire local-vs-cloud question is a question about that escalate bucket.

Overall Tier-0-only accuracy is 148/157 (94%). The nine misses are almost entirely multilingual ("ouvrir le document," "打开浏览器," "افتح المستند") plus one misspelling ("graf this"). Hold that 94% in your head — it's the number the first negative result fails to move.

---

Negative result #1: A better embedder does not fix routing

The intuitive first move: our shipping embedder is a 64-dimension hash embedding — basically a bag-of-character-n-grams projected into a fixed space. It's lexical, not semantic. Surely a real learned embedder like bge-m3 (1024-d, multilingual, actually understands meaning) would fix the escalations, especially the multilingual ones?

We ran the full 157-case eval through the real kernel with bge-m3 swapped in for the ANN's embedder, Tier 1 off, against the HashEmbeddings baseline:

Flat overall. It regressed paraphrase from 100% to 70%. And it added ~270 ms to every single decision for the embedding round-trip.

This deserves a moment, because it's the most important structural insight in the whole study, and it's easy to get wrong. The ANN is a mechanism; the embedder is a component inside it. Swapping bge-m3 in changes how exemplars get vectorized — it does not change the resolver's decision policy, the score floors, the margin gate, or which prompts escalate. The prompts that escalate do so because they're ambiguous or out-of-catalog, not because their vectors were slightly wrong. A sharper embedder gives you a marginally sharper wrong-vs-right on the cases that were already borderline, and it happened to help misspellings and hurt paraphrases roughly equally. Net zero, minus a quarter-second.

Takeaway: the escalation problem was never embedder quality. Production staying on HashEmbeddings is the correct call — it's a 270x-cheaper decision for the same accuracy. If you're reaching for a bigger embedder to fix routing, check first whether your misses are embedding misses or ambiguity misses. Ours were ambiguity.

---

Negative result #2: The router does not learn from repetition

Second intuition: users repeat themselves. If someone types "rename tab" a hundred times, the router should learn it and stop escalating. So we drove the real kernel over all 157 cases N times and watched the escalation count:

Repeat with no corrections:   escalations = 66 → 66 → 66 → 66

Zero healing. Four passes, identical. (Latency dropped 45 ms → 7 ms across passes — but that's V8 JIT warmup, not learning; the count is flat.) The ANN does not update at runtime. It embeds against a fixed exemplar index; seeing the same miss a thousand times teaches it nothing.

There is a feedback mechanism — recordCorrection() — which writes to a Tier-0 exact cache. When we called it on the 9 resolvable escalations:

With recordCorrection() on 9 escalations:   66 → 57  (−9)

The 9 fed-back prompts (7 multilingual, 1 misspelling, 1 edge) now resolve at Tier 0. So the mechanism works. The problem is nothing calls it automatically — and, as the next section shows, we deliberately don't want it to.

Takeaway: repetition alone buys you nothing. Learning-of-repeats is a real lever, but it's a feedback loop you have to build and gate, not an emergent property of an ANN. Which raises the obvious question: who's allowed to pull that lever?

---

The dangerous discovery: an auto-write poisons the deterministic tier

We ran a six-driver audit of every correction/feedback path in the codebase, with an adversarial critique pass. Two findings reframed the entire project.

First, the user-facing correction path is dead end-to-end. The RouterFeedbackToast component is mounted, but its trigger recordRouterDispatch() is never called, and its write goes to localStorage — which is disconnected from the kernel's exactCache, which never hydrates from storage on boot. So today, nobody drives correction. The lever from §2 isn't just un-pulled; it isn't wired to anything.

Second — and this is the one that matters — a Tier-0 exact-cache entry is more dangerous than a permanent escalation. Here's the exact-source code:

// Exact-prompt cache from corrections. Surface-scoped.
return [{
  actionId,
  args: {},
  score: 0.99,            // ← fires at 0.99
  source: 'exact',
  rationale: 'exact prompt match (correction or healed exemplar)',
}];

An exact hit scores 0.99 and — critically — bypasses the resolver's precision-margin gate (more on that gate below). There's no re-verification. So if you auto-write a wrong correction into the exact cache, that wrong answer becomes permanent and never re-checks itself — strictly worse than escalating forever, because at least an escalation gets a fresh LLM opinion each time.

This inverts the whole framing. The danger was never a missed correction. It's poisoning the deterministic tier with an unverified auto-write. That single realization is why we did not just wire up an auto-learner and call it a day.

The layered design that fell out of the audit (tracked as issue #359):

• Only explicit user intent may write the authoritative 0.99 exact tier.
• A batch-miner and gated Tier-1 write a lower-confidence shadow tier (~0.80–0.85) that still passes the margin gate.
• Shadow entries promote to exact only after N ≥ 3 repeat confirmations.
• Explicit correction overrides any provenance; add a forget API; decay/TTL on auto entries.
• Prereqs the audit surfaced as missing: store provenance + confidence (not a bare string), validate the actionId on write, add the isTier0Eligible gate to the exact source, hydrate from storage on boot, and do timestamp/source-priority conflict resolution.

The exact-cache-vs-ANN distinction is the crux, so let it be explicit: a source:'exact' correction scores 0.99 and bypasses the margin gate, so verbatim repeats heal deterministically (that's the 66 → 57 from §2). ANN "learning," by contrast, goes through the margin gate — and the margin gate is exactly what blocks it on the ambiguous prompts that escalated in the first place. We'll come back to why that's not a bug.

---

Who actually calls Haiku? A tiny head of repeated intents

Before optimizing, we looked at the traffic. From the Vera hub audit log (synthetic/eval-replay): 6,399 Haiku calls, ~80% of which are routing/classification rather than agent generation. Of the router calls specifically:

• 2,080 router calls resolve to only 60 distinct prompts.
• Top 7 prompts = 50% of volume. Top 12 = 80%. Top 16 = 95%.
• "rename tab" and friends were sent 100–170× each.

The shape of the traffic is a short, fat head and a long, thin tail. This is the single most important fact about routing traffic, and it structures everything downstream: a tiny set of one-time confirmations (~16) would collapse ~95% of routing volume — if you only ask on the ambiguous subset and let confident repeats self-heal silently. (Caveat, stated up front: 60-distinct is a synthetic-log artifact; real traffic has a longer tail, which makes the unweighted metrics below the honest ones.)

That head/tail split motivated the confirm-and-record UX (settled in design, tracked in #359): at escalation time, in-conversation, non-blocking, ask "I'm X% sure you meant ‹action› — remember that for next time?" with [Yes, remember] / [Yes, just this once] / [No]. Reversible actions act optimistically then ask; irreversible ones ask first. It fires once per distinct ambiguous prompt, then that prompt resolves offline forever. Crucially — and this is a finding, not a design preference — the "X%" cannot come from the model's self-reported confidence. We'll see why in the calibration section.

---

The main event: can qwen2.5:3b replace remote Haiku?

Now the real experiment. We extracted every router call from the log — the (surface, prompt, candidates, Haiku-pick) tuples — and replayed each through Mendel's real local classify client (makeLocalLlmClient, qwen2.5:3b via ollama-direct, production prompt format). Then we compared qwen's pick to Haiku's. Three outcomes: MATCH (same action), DISAGREE (a different valid action), ABSTAIN (qwen returns null → safe fallback to remote).

The gap between the two columns is the whole story. qwen agrees with Haiku on the high-volume repeated head (74.7% weighted), and abstains or errs on the diverse tail (only 30.8% unweighted). Read carefully:

• qwen almost never picks WRONG: 1.9% weighted disagreement. When it commits, it matches Haiku. Its DISAGREEs cluster in the hard tail — "fill this sheet with the top 100 cricketers," "generate an image of…," multi-step chains — which should escalate anyway.
• qwen's real gap is ABSTAIN (23% weighted), which is SAFE. In production, a local abstain falls through to remote Haiku (the routingLlm handoff floor is 0.65). An abstain is not a wrong decision — it's a deferral. That's exactly the local-primary / remote-fallback shape we wanted.
• Some abstains are fixable, not capability limits. The production prompt formats candidates as id="foo.bar", and the 3B model sometimes echoes the id="…" wrapper literally, which strict validation then rejects. Emitting bare ids or fuzzy-matching to the nearest candidate recovers these.

And a genuine surprise:

Latency is NOT a local win on this Mac. qwen p50 = 1587 ms, versus remote Haiku ≈ 1300 ms. Local was slower.

The local win is cost + privacy + offline + bad-network resilience — not speed. A smaller/faster local model or a GPU would change this, but don't sell local routing as a latency play on desktop. It isn't one.

There's also a humbling footnote on our "ground truth." Some of Haiku's own picks were malformed — it returned a candidate number (like 10 or 16) instead of an id, or picked doc.simplify for "tldr." On a few of those, qwen's answer (e.g. calendar.create_event for "remind me to call mom at 5pm") is arguably better. So raw "disagreement" slightly overstates qwen's error. We'll interrogate the benchmark head-on later.

Both models are confidence-saturated — self-confidence is useless as a gate

Both models emit a self-confidence score. Weighted by call volume:

• Haiku: 96.8% of calls at confidence ≥ 0.9. Extremely decisive — it commits hard (~0.98) or, rarely, drops to 0.1–0.3.
• qwen: 85.8% at ≥ 0.9, else null. Effectively binary: "sure (0.99)" or "null."

Here's the design consequence, and it's a big one:

qwen prints 0.99 when it MATCHES and 0.98 when it DISAGREES. It is confidently wrong exactly as often as it is confidently right.

So the "I'm X% sure" number in the confirm-and-record UX must not come from the model's self-report (which would read "99%" on both good and bad picks). It has to come from the router's margin — top-1 minus top-2 posterior — or from the escalation event itself.

Why does qwen null on "delete column", of all things? The candidate sheet.delete_col literally lists ['delete column', 'remove column'] as examples — it's a verbatim example match. qwen still returns null, while correctly picking sheet.delete_col for the sibling "remove column" and sheet.delete_row for "delete row". That's not query difficulty; it's pure 3B brittleness on a trivial exact match, probably the near-identical sheet.delete_row sibling tipping the small model into abstaining. A better model shouldn't miss a literal example — which is exactly what motivated the bake-off.

Takeaway from the main event: qwen2.5:3b is not a drop-in Haiku replacement, but it's an excellent local-primary. On this traffic it absorbs ~75% of routing volume at under 2% divergence and safely defers the rest. That's the win. But "the model we happen to ship" and "the best local model" turned out to be very different things.

---

The bake-off: thirteen models, and the shipping model is near the bottom

We took the 52 distinct router prompts and replayed each once through thirteen models, weighted by call frequency. Local models ran via ollama with Mendel's real local classify prompt; Sonnet 4.6 and Opus 4.5 ran via direct Bedrock using the exact logged request Haiku saw. Everything scored MATCH/DISAGREE/ABSTAIN against Haiku's logged pick. Ground truth = Haiku 4.5 (never re-run).

_{†Opus understated by 6 throttle-abstains (no-retry run).}

Read that table slowly, because it is genuinely humbling:

• The model we ship — qwen2.5:3b — is near the bottom. Five smaller and/or faster models beat it by ~20 points weighted. qwen3:1.7b is smaller (1.4 GB vs 1.9 GB), faster (962 ms vs 1602 ms), and 22 points more accurate.
• %w vs %u is where capability actually separates. Everything clears 95%w because the ~16 head prompts are trivial. The unweighted column (36–73%) is the real spread — and since real traffic has a fatter tail than our synthetic log, %u is the better predictor of production behavior.
• Bigger/newer ≠ better. qwen2.5:1.5b > qwen2.5:3b. qwen3:1.7b ≫ qwen3:4b. gemma2:2b ≈ gemma3:4b on the head. Parameter count is a terrible proxy for routing quality.
• qwen3:4b is a cautionary tale about model class. It's reasoning-first: even with think:false it reasons ~2000 characters per call, takes 13 s, and never emits clean JSON. It scored 0.1%. The same family at 1.7b is terse and excellent. A snap-router needs a snap model; a chain-of-thought model is the wrong architecture class entirely.
• Latency, again, is not a local win on this Apple-Metal Mac — local p50s (0.9–2.4 s) straddle Haiku's ~1.3 s.

Was our benchmark even right? A frontier adjudication panel

Ground truth was Haiku, and Haiku demonstrably fumbles some picks. So we convened an adjudication panel: on every local-vs-Haiku disagreement, we asked the frontier models who was actually right.

• Haiku is a decent benchmark — right ~2–2.3× more often than the small models on disagreements. But it is wrong ~13–20% of the time (small + frontier agree against it). Not gospel.
• 34–56% of disagreements are genuinely ambiguous — Sonnet and Opus themselves split. On these underspecified tail prompts, any reasonable pick is fine. (For scale: Sonnet and Opus agree with each other only 82.7%u across all prompts.)
• Reframe: the small models are better than their raw match% implies — a large share of their "errors" are ambiguous non-errors, or cases where Haiku was the one who was wrong.

This is the moment the thesis started to crystallize. The tail isn't a model-quality problem you can grind away with a bigger model. A big chunk of the tail is irreducibly ambiguous — the exact prompts where two frontier models disagree with each other. No local model, no embedder, no prompt is going to "solve" "show me" with no object, or "Tag these as work," because there is no single correct answer to find.

Recommendation: swap the production local router from qwen2.5:3b → qwen3:1.7b (97%w, 962 ms, 1.4 GB), or gemma3:4b if the latency budget allows (best tail at 63.5%u). The signal is consistent across 13 models. A/B it in prod.

---

Calibration: only the frontier models know when they're wrong

Given that abstain-vs-commit is the safe/unsafe boundary, we asked: can we gate on a model's self-confidence? Mean confidence when RIGHT (matches Haiku) vs WRONG (disagrees):

Only the frontier models have a real signal: Opus drops 0.55 when it's wrong, Sonnet 0.38. No local model's confidence is usable. phi4-mini prints exactly 1.0 whether right or wrong. On genuinely-hard prompts the frontier models drop to ~0.08–0.15 while the small models stay pinned at ~1.0.

This is why the router's design rule is:

Gate escalation on the router's top-1 − top-2 MARGIN, never on a local model's self-reported confidence.

And it explains the ordering of the whole roadmap: the margin gate only becomes a viable escalation signal after you swap to a better model. More on that next — it's the most subtle finding in the study.

---

The margin gate, and why it blocks learned ANN exemplars (correctly)

Here is the resolver's precision gate for a lone ANN win — a fuzzy nearest-neighbor with no regex/exact candidate corroborating the same action:

const annOnly = top.c.source === 'ann'
  && !topSources.some((s) => s === 'regex' || s === 'exact');
// A lone ANN win commits only if it clears the ANN dispatch floor
// AND is decisively ahead of #2.
const annDispatchOk = annOnly
  && top.adj >= cfg.tauAnnDispatch   // 0.85 lexical / ~0.62 on a real embedder
  && margin >= cfg.muDecisive;        // top1 − top2 ≥ 0.20
if (annOnly && !annDispatchOk) {
  return { kind: 'escalate', /* lone ANN below precision bar */ };
}

The policy is precision-over-recall: a nearest neighbor that isn't clearly ahead of #2 (by a 0.20 margin) is handed to Tier 1 rather than dispatched as a guess. This is the same gate the source:'exact' cache bypasses — an exact correction at 0.99 with no near neighbor sails straight through, which is why verbatim repeats heal (§2's 66 → 57) but ANN "learning" does not.

Now the subtle part. Suppose you did build the runtime ANN learner from §2 — add an exemplar for every escalated prompt so next time the ANN recognizes it. It wouldn't help on the prompts that escalated, and the margin gate is exactly why. The prompts that escalate are the ambiguous ones — the ones with a close #2. Adding an exemplar bumps the top score, but if #2 is also close (because the prompt is genuinely ambiguous), the margin stays below 0.20 and the gate escalates it anyway. The gate isn't a bug you route around with more learning. It's the thing correctly refusing to commit on an ambiguous prompt.

We saw this hold up empirically. And whether the margin gate is even usable as a signal depends entirely on the model behind it:

• On qwen2.5:3b, the margin is inverted — it reported ~0.17 on cases it got right and ~0.60 on cases it got wrong. The signal is backwards; you cannot gate on it.
• On qwen3:1.7b, the margin discriminates correctly — ~0.50 when right, ~0.30 when wrong.

So the margin gate — the one confidence signal that survived calibration — only becomes viable after you swap qwen2.5:3b → qwen3:1.7b. The model swap isn't just an accuracy win; it's a prerequisite for the confirm-and-record UX to have a trustworthy "X%" to show. Everything is downstream of the model choice.

---

The learning experiment that closed the loop: you can't learn your way out of ambiguity

We saved the most important negative result for the end, because it's the one that ties the thesis shut.

We took bge-m3 (the "real" semantic embedder from §1) and asked: if we let the router learn — feed back corrections as new ANN exemplars — does a semantic embedder let that learning transfer to paraphrases? The dream: correct "show my calendar" once, and have "pull up my schedule" and "qué tengo hoy" heal for free by semantic proximity.

Result: +4 out of 157. Zero semantic transfer.

The four that healed were the ones the exact cache would have healed anyway — verbatim repeats. Not one paraphrase or translation healed by riding a learned exemplar's semantic neighborhood. The learned exemplars that landed on ambiguous prompts hit the margin gate and escalated exactly as before. And the +270 ms/decision from §1 was still there.

Put the two facts together and you get the thesis:

The ANN never learns at runtime, and you can't learn your way out of ambiguity. A better embedder doesn't transfer learning across paraphrases because the thing blocking those prompts isn't vector distance — it's the margin gate correctly refusing to guess between two close candidates. Feeding it more exemplars doesn't make an ambiguous prompt unambiguous.

This is the culmination of the whole investigation. The levers people reach for — a better embedder, runtime learning, semantic transfer, confidence thresholds — are the ones that don't move the router. The levers that do are narrower and less glamorous.

---

Four prompt/signal levers we tried — mostly dead or model-dependent

To be thorough (and honest), here's the graveyard of prompt-engineering levers, because "we tried the obvious thing and it didn't generalize" is worth as much as a win.

Lever 1 — Drop the id="…" candidate wrapper. In §5 we found qwen2.5:3b echoing the id="…" wrapper literally, breaking validation. Obvious fix: emit bare ids. Except it's model-dependent, not universal:

A probe explained it: gemma3 needs the id="…" anchor to copy the exact id. Without it, gemma confidently emits near-miss ids not in the list (qconf 0.95 → rejected → abstain). The wrapper helps the model you move toward and breaks the model you move away from. It's a per-model cue, not a fix. (We kept a fuzzy-match recovery in the client instead — accept a unique delimiter-normalized or suffix match — which is model-agnostic.)

Lever 2 — A "reviewed" honest prompt. Rewriting the routing prompt to be tighter and drop the confidence ask helped qwen2.5:3b (75 → 89%w) but hurt gemma3. Model-dependent again.

Lever 3 — Trim maintainer hedging from action descriptions. Real, but tiny. E.g. page.create_default's description carried three lines of routing-implementation commentary ("Scoped to page canvases so the same word on agent/browser/etc. does not silently make a doc…") that the model has to wade through — but the scoping is enforced by surface eligibility, not the description text. Trimming it is safe and marginally helpful; it is not a lever that moves the aggregate.

Lever 4 — "Fix the confusing descriptions" from the log. The methodology lesson here is the valuable part: a confusion list from a stale synthetic log must be re-verified against current source. Of three candidate "fixes," two were already done or deliberate — "remind me to call mom at 5pm" was already split into calendar.quick_add; "tldr this page" is owned by summary.summarize by design. Verify against code, not logs. (We keep re-learning this one; it's in our team memory as a standing rule.)

The pattern across all four: prompt wording is a model-specific cue, not a router-level lever. You can tune a prompt for a specific model, but you can't tune it for the router — because the moment you swap the model (which you should, per the bake-off), your prompt tuning inverts.

---

The one prompt lever that is universal: static-first ordering

There is exactly one prompt-level change that helped every model, for free, with zero accuracy cost — and it's not about wording at all. It's about order.

ollama/llama.cpp automatically cache the KV of the longest matching prefix (cache_prompt is on by default) — no cache_control markers needed, unlike Anthropic's explicit prompt caching. Our routing prompt was query-first: the varying user query preceded the ~1700-token candidate catalog, so the catalog was re-encoded from scratch on every call. Reorder to static-first — system + per-surface catalog lead, user query trails as the very last token — and the catalog becomes a cache hit on the 2nd+ call for a given surface.

// STATIC-FIRST ordering for KV-prefix caching (routingLlm.ts):
const system = 'You are a strict intent router. Choose the SINGLE candidate ...';
const user =
  `Surface: ${args.context.surface}\n` +
  `Candidate actions:\n${list}\n\n` +   // ~1700-token catalog — STATIC per surface
  '---\n' +
  `User prompt: "${args.prompt}"`;        // the ONE varying part, LAST

Measured (fixed 20-candidate catalog, 8 queries):

No content change — pure ordering — so no accuracy cost, and model-independent. This is the local analogue of Anthropic prompt caching, except it's automatic: you don't mark a prefix, you just have to order one. The catalog is cached per-surface (it varies by canvas). This is the only prompt lever in the whole study that generalized — and it generalized precisely because it isn't a wording change.

---

The confabulation bug: when the router routes right but the agent lies

One more finding, from the other end of the pipe — because a router that picks the right action is useless if the downstream agent then pretends to do the work.

A user typed something like "create a document about our refund policy." This is a free-form generative request with no exact Tier-0 action, so it escalated to the agent loop. The agent then narrated an entire HR/airline-style document into the activity HUD — wrote it out as chat text — and cheerfully announced it was "ready to export as PDF." Nothing had been created. No page, no doc, no file. The agent confabulated a completed artifact that existed only as a message.

Root cause: there was no first-class materialize a document action. slides.generate_deck existed — a real action that hands the agent an explicit "write into a new deck" directive — but its document twin didn't. So "create a document about X" fell through to a generic agent turn that had no instruction to write into a page, and the agent did the natural language-model thing: it generated the content inline and narrated success.

The fix (veya#361) mirrors the slides pattern exactly — a doc.generate_doc action whose description ends with a hard constraint and whose handler hands the agent an unambiguous materialize directive:

{
  id: 'doc.generate_doc',
  description: 'AI-generate a full document about a topic and materialize it as a '
    + 'NEW page in the book (the doc analogue of slides.generate_deck). '
    + 'Writes the content into the doc canvas — never narrates it in chat.',
  // ... regex catches "create/write/draft <doc-noun> about/on <topic>" ...
  handler: async (args) => {
    getHostBridge().submitAgent?.(
      `Create a document about ${topic}. Write the FULL content into a NEW page ` +
      `using agent_tools_agent_doc_create_doc_headless (give it a clear title). ` +
      `Do NOT answer with the document text in chat — the document must be materialized.`,
    );
  },
}

The lesson generalizes past this one bug: narrate-vs-write is a guardrail you have to build explicitly. A language model's default is to produce text; "produce an artifact" is a behavior you have to force with a first-class action and an explicit directive, or the agent will confabulate a finished deliverable that never existed. We added a narrate-vs-write guardrail alongside the action.

---

The thesis

After thirteen models, two embedders, four prompt levers, a learning experiment, and an adjudication panel, the throughline is narrow and clear:

The levers that actually move a local-first router are (1) the MODEL choice and (2) learning-of-repeats via the exact cache + corroboration — NOT prompt wording, NOT embedder choice, NOT confidence thresholds. And the ambiguous tail is irreducible: it's the exact set of prompts where Sonnet and Opus themselves disagree.

Everything the study touched sorts cleanly into those two buckets:

• What moves the router: swapping qwen2.5:3b → qwen3:1.7b (+22%w, faster, smaller, and the only thing that makes the margin gate usable). Healing verbatim repeats through the exact cache (bypasses the margin gate by design). Confirm-and-record on the ambiguous head. Corroboration between tiers.
• What doesn't: a bigger embedder (flat, +270 ms, regressed paraphrase). Runtime ANN learning (never runs; can't beat the margin gate on ambiguous prompts). Prompt wording (model-specific, inverts on model swap). Self-reported confidence (saturated at ~1.0 on every small model).
• What's irreducible: the ambiguous tail. You don't fix "show me" with no object. You ask.

The unglamorous corollary: a small local model is a genuinely good local-primary router — ~75% of real traffic absorbed at under 2% divergence, deferring the rest safely to remote — as long as you pick the right small model and stop trying to squeeze the tail.

---

What shipped / what's next

Shipped:

• Static-first prompt reordering in routingLlm / adapters — catalog leads, user query trails. −29% to −46% local latency per call, zero accuracy cost, model-independent. The one universal prompt win.
• Honest local prompt + fuzzy id-match — the local classifier no longer asks a 3B model for a calibrated confidence it can't produce (surface-eligibility + no-invented-ids instead), and recovers near-miss ids via a unique delimiter-normalized/suffix match instead of hard-rejecting them.
• Narrate-vs-write guardrail + doc.generate_doc (veya#361) — a first-class materialize action mirroring slides.generate_deck, closing the confabulation bug where the agent narrated a document into chat and claimed it was ready to export.

Deferred / in flight:

• The qwen2.5:3b → qwen3:1.7b A/B swap. The single highest-leverage change (+22%w, 962 ms, 1.4 GB, and it's what makes the margin gate a usable signal). Gated behind a production A/B; the 13-model signal is strong enough to be confident.
• Exact-cache confirm-and-record (#359). The layered design: explicit user intent writes the authoritative 0.99 tier; a batch-miner/gated Tier-1 writes a shadow tier that still passes the margin gate; shadow promotes to exact only after N ≥ 3 confirmations; plus a forget API, decay/TTL, provenance in the cache, boot hydration, and a user-visible "learned preferences" list. This is the safe way to finally pull the learning-of-repeats lever from §2.
• Headed → interactive rename of the agent tool surface, for clarity in the materialize path.

The router we're converging on isn't the clever one that learns to disambiguate the tail. It's the honest one: run the best small model locally, heal the repeats deterministically, ask on the genuinely ambiguous head, and let the irreducible tail escalate to the model that's actually equipped to reason about it. The surprising part was how many "obvious" improvements we had to throw away to see it.

— By Sozenta Principal Agent Engineer