Stop sending every task to your biggest model.

Your hardest task is setting the price for everything

A team finds a model that handles its most difficult documents, plugs it into the product, and sends every request there. Complex analysis, simple extraction, rewriting a sentence: all the same route. It is an understandable starting point. It can be an expensive permanent arrangement.

The alternative is not to replace the strongest model with a cheaper one everywhere. It is to decide which work needs which capability, and to check that the decision does not quietly damage the product.

That second part matters. A lower invoice is not a successful optimization if customers spend more time correcting answers. Routing is a quality-control problem with a cost benefit, not a procurement trick.

Start with a bounded workload whose results you can assess. Extracting a few fields from a known document type is a more tractable first experiment than deciding which model should answer every question a company receives.

Two architectures, two different bets

Routing before generation chooses a model from the request and available context. It avoids generating multiple answers, but the router must predict which model will be sufficient before seeing the result. Simple rules can be enough for a narrow workload; a learned router is another option.

RouteLLM studies the latter approach, training routers with preference data to select between stronger and weaker models. Its evaluations show that routing can improve the cost-quality tradeoff under the tested conditions. They do not establish a savings rate for your documents, tools, or deployment. ^[1]

A cascade starts with one model, evaluates its output, and escalates selected cases. It gains evidence from the first answer but pays for that attempt even when it is discarded. FrugalGPT explores learned cascades and demonstrates their potential on its evaluated tasks. Its 2023 prices and model choices are historical, not a current shopping list. ^[2]

Choose based on what you can observe. If task type reliably distinguishes easy work, route early. If difficulty becomes visible only after checking an answer, a cascade may be worth the extra step. If neither gives a trustworthy signal, a single stronger workflow may remain the better option.

The gate is the product

Consider extracting a supplier name, invoice number, currency, and total from a document. A compact model produces structured fields. A gate checks whether required fields exist, whether the currency is supported, and whether the extracted values have identifiable locations in the source.

These checks do different jobs. Valid JSON establishes structure. Arithmetic checks can catch certain inconsistencies. A source location helps a reviewer inspect a claim. None proves that every field is semantically correct: a model can copy the wrong valid-looking number from a different part of the page.

Build the gate around known failure modes. A quoted total from an unrelated attachment should fail. Two conflicting totals should trigger escalation. An unsupported document type should stop or enter a separate workflow. Do not force every case through a gate built for the easiest examples.

Asking the model “are you confident?” can supply a signal to evaluate, but it should not become the sole acceptance rule. Fluent certainty is exactly what many incorrect answers already have. An additional model judge also needs evaluation; agreement between models is not independent evidence when they share the same blind spot.

The key error is a false acceptance: a wrong result that the gate approves. An unnecessary escalation costs money and time. A false acceptance can quietly corrupt the work. Set the tolerance according to the task and distinguish severe failures from harmless formatting issues.

A cheap answer can still make an expensive system

Here is an illustrative batch of 1,000 tasks. These are invented per-task costs for arithmetic, not vendor prices or measured performance. Assume a direct stronger-model workflow costs $0.02 per task, or $20 for the batch.

In a cascade, each first attempt costs $0.002. Every first output receives a check costing $0.001. Twenty percent of tasks escalate, with an additional $0.02 charged for each escalated attempt.

That is a 65% reduction in these specified service charges. It excludes orchestration, storage, final review, further checking of escalated answers, and the consequences of mistakes. If those costs differ between the two workflows, add the difference before deciding which is cheaper.

At an assumed review cost of $60 an hour, thirteen additional minutes of human work across the batch would consume the entire $13 service saving. That is not an argument against routing. It is a reminder that tiny API prices can make people-time the dominant cost.

Under the same assumptions, the cascade service cost per task is $0.003 plus $0.02 multiplied by the escalation rate. It is lower than $0.02 only while escalation stays below 85%. This is a service-cost threshold, not a quality or profitability threshold. Changing document lengths, checks, caching, or retry behavior changes it.

Average latency can hide a bad experience

A cascade makes escalated users wait for the first answer, the gate, and then the next attempt. Even if most requests finish quickly, the slower group can receive a worse experience than if it had gone directly to the stronger model.

Measure end-to-end latency for the complete workflow. Look at the distribution and the escalated subset, including queueing, retries, and review. A pleasing average can conceal the exact cases your most demanding customers care about.

Use a bounded escalation policy. One failed attempt does not justify an endless chain of increasingly expensive models. After the defined limit, return an explicit unresolved result or route to an authorized reviewer. Some tasks lack the evidence needed for any model to answer them.

For interactive use, a router that identifies likely difficult requests early may be worth more than a small additional service saving. For overnight processing, sequential latency may matter less. The right architecture depends on the product's deadline, not a universal preference for small models.

Evaluate the accepted answers, not just the failures

Build a representative, permitted evaluation set before tuning the routing rule. Separate development examples from a held-out test set. Include unusual layouts, incomplete inputs, conflicting evidence, and the languages the product genuinely supports. Repeatedly tuning against the supposed test set turns it into another development set.

Compare complete policies: one direct workflow against the proposed routed or cascaded workflow. Record accepted accuracy, severe false acceptances, escalation rate, unresolved cases, review effort, cost, and latency. Where outcomes are stochastic, repeated runs or an appropriate uncertainty analysis can help distinguish a real change from noise.

During a limited rollout, audit a sample of accepted first-pass results as well as escalated results. Reviewing only failures hides the mistakes your gate failed to notice. Keep that audit sample representative; oversample sensitive cases separately and label them accordingly.

Model changes, prompt edits, new customer documents, and altered tool access can all change the gate's behavior. Version the policy and keep a rollback path. Before any routing decision, enforce data-handling requirements so a cheaper provider never receives information it is not authorized to process.

Start with two routes and one task you understand. Expand only when you can explain why the cheaper path is sufficient and show how the system detects when it is not. The winning number is still cost per accepted result, with an acceptance standard worth trusting.