The Model Router
Two trained classifiers behind an LLM agent that decides which one to call.
- Timeline
- 2 classifiers · 1 routing agent
- Role
- Solo build
- Status
- Built and running
- Primary stack
- Python · scikit-learn · LangGraph · Streamlit
Headline metrics
- What it tackles
- Language models get handed decisions that small trained classifiers make better, faster and cheaper, and a fluent answer never tells you whether anything was actually measured.
- What it delivers
- Two trained classifiers, risk scoring at 92.4% accuracy and spam detection at 98.4%, behind a LangGraph agent that picks the right tool, calls it, words the result, refuses when neither model applies, and logs the tokens and cost of every turn.
Why I built it
I kept seeing teams reach for a language model to make decisions that a small trained classifier makes better, faster and cheaper. I wanted to show the split cleanly: let the model route and explain, and let the trained models decide.
The problem in full
A language model asked whether a message is spam will answer. It will answer fluently, it will sometimes be right, and nothing in the reply will tell you which time this was. That is the failure this build is arranged against: a confident answer with nothing measured behind it, indistinguishable from a real one.
Training a classifier is a solved exercise. Two of them are two solved exercises. What is not solved by default is the join: an agent that reads an arbitrary message, works out which trained model it belongs to, actually calls it, and puts the determination back into words a person can act on. Routing a risk question to a spam classifier returns a confident and completely wrong answer, which is worse than returning nothing, so tool selection has to be right, not plausible.
It is a deliberate argument about architecture. The model that talks is allowed to route and to explain. It is never allowed to judge.
System design
I trained two classifiers in scikit-learn, each measured on a held-out test split. A random forest scores risk on a structured record at 92.4% accuracy and 0.977 ROC AUC. A multinomial naive Bayes model over TF-IDF features classifies free text as spam or not spam on a public SMS corpus at 98.4% accuracy and 0.990 ROC AUC. The two figures are different tasks on different data, so the system never stages them as a contest. Each stands on its own axis.
Each model is wrapped as a callable tool with a strict contract. risk_score takes the record as given and returns a trained prediction, not a generated guess. spam_check does the same for text. The one-sentence description on each tool is the real interface, because the agent selects a tool by reading that sentence. Those two sentences say what the tool is for and when to reach for it, and nothing about what a good answer looks like, because that is not the agent's decision to make.
A LangGraph agent sits in front, with LangChain binding the tools and a small model doing the reading. One turn runs five steps: message in, with no schema and no routing hint. The agent resolves which tool applies, deciding which and never what. The bound classifier is invoked with the input as given. The trained model returns its determination, the only step in the turn a trained model performs. Then the agent puts the result into plain language and names the model that produced it.
There is a fourth path, and it matters most. When a message matches neither tool, the agent calls nothing and says plainly that it has no trained model for the question. Ask it which customer will churn next quarter and it declines, because answering anyway is the exact failure the system exists to prevent. The whole thing ships as a chat interface, so someone who has never opened a notebook can use it, and every call logs its prompt tokens, completion tokens, cost, and the tool invoked.
The calls that made it work
The agent routes, the trained models decide
A random forest beats a general model at risk scoring, and a naive Bayes classifier runs spam detection in milliseconds at 98.4% accuracy. The language model contributes what it is genuinely good at, reading an arbitrary message and choosing the right instrument, and the determination always comes from a model trained to make it.
Refuse when neither model applies
The most important branch in the router is the one that ends nowhere. A question neither classifier was trained on gets no tool call and a plain statement saying so, because a fluent unmeasured answer is indistinguishable from a real one, and that ambiguity is the thing being engineered out.
Treat the tool description as the interface
The agent picks a tool by reading the sentence that describes it, which makes those sentences the highest-consequence text in the system. Each one states what the tool is for and when to use it, and deliberately nothing about what a good answer looks like.
Log tokens and cost on every call
An agent that decides for itself how many tool calls to make is an agent that can spend unpredictably. Every turn records prompt tokens, completion tokens, cost, and the tool invoked, which is the difference between a system you can demonstrate and a system you can run where a bill arrives.
The results in full
The system routes a raw message to the right trained model, returns a measured decision with a plain-language explanation naming the model that made it, and accounts for every token it spent doing so. Risk scoring holds 92.4% accuracy at 0.977 ROC AUC and spam detection holds 98.4% at 0.990, each on its own held-out test set.
The same shape carries directly into my commercial work: a language model deciding which deterministic step to run, then invoking it, is the pattern behind the qualification and enrichment systems I build for outbound. The domain changes and the arrangement does not, and drawing the line where the model that talks may route and explain but never judge is most of the design work in any agent that touches a real process.
Continue reading
The Podcast Pipeline
Finds founders interviewed on small podcasts, reads the episode, and writes about what they said.
The Enrichment Engine
A recruiting engine rebuilt to decide before it pays for data, then pointed at two more verticals.
The Listing Auditor
A pipeline that audits a brand's live Amazon listing and writes the email about what it found.