A capable machine sits idle on my desk most of the day while most requests my coding agent makes go out to a large cloud model. The big model has earned that role by turning high-level requests into proper implementations. But many requests are simple enough for a model running on my desk.
I want to put my idle hardware to work. A common approach certifies a local model against a benchmark and assigns it the tasks it passes. I do not want to rebuild that benchmark every time I swap models. I want to keep using the agent as before while the local model attempts the same requests in the background. The router can learn its strengths from that traffic and send suitable work its way. Lower API cost is a bonus, and I want to keep as much work local as the evidence allows.
The project comes down to one question,
Can the router learn a local model's strengths from normal use and route work to it automatically?
To find out, I added a router to my fork of oh-my-pi, the agent harness I use. The resulting setup keeps the reference model answering as usual while the local model tries the same request in the background, and an evaluator compares the two responses. Once enough similar attempts pass, the router lets the local model answer on its own.
The basic loop
The loop uses four models, with three generating outputs and one producing embeddings.
Reference model
The large model that answers while the router collects evidence.
Local model
The local model under test and, later, allowed to answer.
Evaluator
The model that compares both proposed actions.
Embedding model
A local model that maps each request to a retrieval vector.
Router shadow mode
From the user's point of view, nothing changes. The reference model's stream goes back to the agent as usual while the local model works in the background. Each shadow example records the request, both outputs, their latencies, the request embedding, and the evaluator's grade. If the local model proposes tool calls, the router records them but never runs them.
The evaluator returns a structured verdict rather than free text, so every comparison lands in the dataset in the same shape.
{
"verdict": "pass",
"confidence": 1,
"unsafeAction": false,
"riskClass": "response",
"taskClass": "arithmetic",
"reasonCodes": [
"correct_answer",
"matches_reference_intent"
],
"acceptableNextAction": true
}Routing as retrieval
The router's shadow mode produces a growing dataset of graded attempts. To use that evidence, the router predicts whether the local model will succeed on a new request through retrieval. It turns each incoming request into a vector and matches it against the vectors of past shadow examples. I chose embeddinggemma for this role and stored its vectors in SQLite.
Unlike a retrieval augmented generation workflow, the retrieved outputs never enter the model prompt. Their labels estimate whether the local model will succeed on the request at hand.
Before any similarity math, the router requires an exact compatibility cohort with matching values for the following fields.
- same reference model, local model, evaluator, and embedding model
- same local model version
- same system prompt fingerprint
- same tool schema fingerprint.
Within a cohort, the router keeps the neighbors above a cosine similarity threshold. It keeps both passes and failures, so past mistakes count as much as past successes, and it combines the surviving neighbors into a similarity-weighted pass rate.
Here, si is each retrieved example's cosine similarity to the current request, and yi is 1 when the evaluator judged the local output correct, safe, and usable, and 0 otherwise. Because the sums run over all retrieved examples, closer matches carry more weight in the result.
A raw pass rate misleads with small samples. One success in one trial gives 100% with almost no evidence behind it. Instead of trusting p̂w on its own, the router computes a 95% Wilson lower bound over an effective sample size derived from the similarity weights.
The result, neff, represents the number of equally weighted examples in the retrieved evidence. When every match has the same weight, it equals the raw count. When a few close matches dominate, it is smaller. The router uses this value in place of the raw neighbor count when calculating the bound.
The Wilson lower bound is a conservative estimate of the local model's true success rate. It gives the lowest rate consistent with the observed results and the sample size. With a few examples it stays well below the raw pass rate, and it rises as more successful evidence accumulates.
Automatic routing
With a conservative confidence estimate for every request, the router can act on the evidence. When the router runs in automatic mode, it chooses a model before generation, and uses it for the inference step.
Router automatic mode
On a local win, the router streams the local model's response and never queries the reference model. The evidence gate is the system's trust boundary, and the router's automatic mode moves suitable work onto local hardware.
A small evaluation
With the loop in place, I ran it end to end using the following models.
ollama/glm-5.2:cloudollama/ornith:35bollama/glm-5.2:cloudollama/embeddinggemma:latestThe exact text check asked both models to reply with a specific piece of text and nothing else, and the two arithmetic checks used the prompt What is 2+2? Reply with just the number. A pass means the evaluator judged that the local response could have replaced the reference response for that request.
| Check | Reference latency | Local latency | Evaluator latency | Verdict |
|---|---|---|---|---|
| Exact text | 1.046 s | 18.082 s | 4.812 s | Fail |
| Arithmetic, first comparison | 0.988 s | 10.454 s | 1.746 s | Pass |
| Arithmetic, evidence collection | 5.065 s | 17.104 s | 1.794 s | Pass |
In the failed check, ornith:35b decided that a request to repeat a specific piece of text verbatim looked like prompt injection, and refused. The evaluator returned fail with 0.95 confidence and tagged the codes refused_benign_request and did_not_preserve_intent. The local model failed on judgment here, not capability.
The two arithmetic runs differed in one detail. The first comparison ran before the router had an embedding for the request, so its pass could not serve as routing evidence. The evidence collection run repeated the same comparison with an embedding attached, and that stored example enabled the automatic routing test below.
Did it use the local model when appropriate?
I built the system toward this moment, a request that never touches the big model. To force it with three examples, I lowered the confidence threshold to 0.20 and required a single neighbor. These settings are unsafe for normal use, but fine for a proof of control.
The sequence ran as designed. The evidence collection request used the reference model, the local model matched it, the evaluator passed it, and the router stored the embedding. The next identical request found that neighbor with weight 1.0, for a Wilson lower bound of 0.2065, just above the test threshold, and the router streamed the local model.
Router selected the local model
The reference model was not called.Even warm, the local model remained slower than the median cloud reference, so this run proved local routing and control rather than lower latency. It also confirmed that matching text alone does not join a cohort. In a later replay, new MCP tools changed both fingerprints, the reference model ran again, and the local model became eligible after a fresh pass under the new fingerprints.
Limitations
A three-example proof makes the limitations easy to see. The largest is the core assumption that retrieval routing works when similar requests have similar outcomes. That holds for repeated extraction, classification, formatting, and narrow coding tasks, but it may break down on agent work where a small change to the input alters the entire tool plan.
Beyond that assumption, the current implementation has clear gaps.
- No guard after routing.The router's automatic mode trusts the selected local model and does not retry the reference model.
- No automatic canary ramp.Cohort promotion and demotion are not adaptive yet.
- No context preflight.Context length and repeated loops need explicit guards.
- Linear retrieval.The router scans every stored vector in SQLite.
- Fallible labels.Evaluator decisions inherit the evaluator model’s errors.
- No local model training.The router only learns where an existing model works.
The exact fingerprints cut both ways. They protect against schema drift, but they split the dataset. My replay showed this when new MCP tools moved identical request text into a fresh cohort with no evidence. An agent configuration that changes often could take a long time to collect the 189 or so close, successful examples needed to clear a 98% bound.
Use the reference model to learn. Use the evidence to move as much suitable work as possible onto local hardware.
What I learned
I first described this system to myself as “RAG over the success dataset,” but that name undersells the failures, which matter as much. The useful signal is the full pattern of passes, failures, uncertain results, and unsafe actions under one model and tool contract. Retrieval selects the evidence, the confidence bound measures it, and the resulting route decides which model the agent trusts.
With automatic distillation, a team collects reference model outputs, trains a smaller model on them, and shifts traffic to it (Inference.net, “Train: fine-tuning and distillation.”). Distillation produces a less flexible artifact, a new set of weights that someone must train and deploy, with the learning spread through those weights. That makes it hard to inspect why the system trusts a given request, or to limit that trust to a small group of similar requests. This router leaves both models unchanged and expands local use one group of similar requests at a time; anything outside the evidence still goes to the reference model.
That contrast suggests an order, with routing making a local model useful before any fine tuning happens. First learn where it works; later, train a new version on the accepted reference model examples and the local model's failures.
The experiment ended with the idle machine on my desk answering a request the big model never saw. That is a system check rather than a benchmark, but it demonstrates the whole loop.