AI Model Routing in Production: The Architecture Pattern Your Development Team Probably Skipped

Model routing is the practice of directing different AI tasks to different model tiers based on what each task actually requires. A classification task goes to the cheapest model. A summarisation task goes to a mid-tier model. Complex reasoning — if it exists in the workflow — goes to the most capable model. Each task hits the cheapest endpoint that meets its quality threshold.

This pattern reduces API costs by 80–95% in most production AI systems. It is the single most impactful architectural decision for AI cost efficiency. And most development teams skip it.

Why is model routing not standard practice?

Because it requires more engineering work upfront for a benefit the client doesn't see until production volume arrives.

A development team building without routing picks one model, writes one set of prompts, and ships. A team building with routing has to decompose every workflow into discrete tasks, evaluate each task across multiple model tiers, engineer task-specific prompts for each tier, build the routing logic itself, and instrument monitoring to track quality and cost per route. That's 3–5x more engineering work for a result that looks identical to the client during the demo.

The cost difference only becomes visible at scale. At 100 API calls per day, nobody notices. At 10,000, it's $300/day versus $16/day. At 50,000, the difference is existential — $1,500/day versus $80/day.

Vendors building AI systems on fixed-price contracts have no incentive to do this extra work. They're paid to deliver a working product, not to optimise its operating cost. The client pays the API bill, not the vendor.

How does model routing actually work?

A routing layer sits between your application logic and the AI provider's API. When a task needs to be processed, the router evaluates what kind of task it is and directs it to the appropriate model tier. There are three common routing strategies:

Static routing assigns a fixed model tier to each task type at build time. Extraction tasks always go to Haiku. Summarisation always goes to Sonnet. This is the simplest approach and handles 90% of use cases. The mapping is determined during the architecture phase by testing each task across model tiers and selecting the cheapest one that meets the quality threshold.

Complexity-based routing analyses the input before selecting a model. Simple inputs (short, structured, standard format) go to the cheap tier. Complex inputs (long, unstructured, edge-case-heavy) go to a more capable tier. This works well for tasks where input complexity varies significantly — most inputs are easy, but occasional ones are genuinely hard.

Fallback routing starts with the cheapest model and escalates. Send the task to Haiku first. If the output fails a quality check (confidence score, format validation, consistency check), automatically retry on Sonnet. If that fails, escalate to Opus. Most calls resolve on the first attempt. The system only pays for expensive models when cheaper ones demonstrably can't handle the specific input.

In practice, we use a combination. Static routing for the bulk of well-understood tasks. Fallback routing for tasks where edge case frequency is unpredictable.

What does the routing architecture look like in code?

The routing layer isn't a separate service. It's an architectural pattern built into the AI layer of the application. At its simplest, it's a configuration map:

Each task in the system has a defined skill — a discrete, testable function with its own prompt, model assignment, and quality threshold. The orchestrator calls skills in sequence, each routing to the appropriate model. The task definition includes the model tier, the prompt template, the expected output schema, and the quality validation function.

This is what we mean by agent skill design. Each skill is a self-contained unit that can be tested independently, swapped to a different model tier without touching the rest of the system, and monitored for cost and quality in isolation.

What's the catch?

Two things.

First, each model tier has different prompt requirements. A prompt that works on Opus won't necessarily work on Haiku. Opus is forgiving — it infers intent from vague instructions. Haiku is literal — it needs explicit structure, output format definitions, and edge case handling built into the prompt. Writing effective prompts for cheaper models is a different skill from writing prompts for capable ones.

Second, you need quality validation at the task level. When you route a task to a cheaper model, you need a way to verify the output meets your threshold. For structured tasks (extraction, classification), this is straightforward — schema validation, field completeness checks, confidence scoring. For generative tasks (summarisation, content creation), quality validation is harder and sometimes requires a second model call to evaluate the first one's output.

Both of these are solvable problems. They're just not problems that a team doing basic API integration thinks about, because they don't surface until production volume makes the cost problem visible.

Can you retrofit routing into an existing system?

It depends on how the system was built.

If tasks are already separated into distinct function calls with clean interfaces, adding routing is a configuration exercise. Define the model tier per task, write task-specific prompts for each tier, add validation, deploy. Timeline: 2–4 weeks.

If the system uses monolithic prompts — one API call performing multiple tasks in a single pass — routing requires decomposing the pipeline first. Break the monolithic call into discrete tasks, test each task independently across model tiers, build the routing layer, and rebuild the orchestration. Timeline: 6–10 weeks.

In both cases, the product doesn't change. The user sees the same interface, provides the same inputs, gets the same outputs. What changes is the engineering underneath — and the monthly API cost.

What does this look like in a real production system?

We built an operations AI platform that scaled from 50 to 80+ agents in three months. The system analyses call recordings, scores agent performance, identifies coaching opportunities, and generates reports.

Each of those is a different task with different complexity requirements. Transcription preprocessing runs through a lightweight model. Scoring against a predefined rubric — pattern matching with structured output — runs on the cheapest tier available. Identifying nuanced coaching opportunities that require understanding conversational dynamics routes to a mid-tier model. Generating management-level performance summaries that need to be well-written routes to a capable model.

If every call had gone through one model, the cost per agent would have been high enough that scaling from 50 to 80+ agents would have been economically impossible. Model routing was the difference between a system that could scale and one that couldn't survive its own growth.

That's what a properly architected AI system looks like. Not one model doing everything. Multiple models, each handling what they're best at, orchestrated by a routing layer that was designed before the first line of code was written.

If your system doesn't have this layer, it's missing the most important piece of production AI architecture. And the team that built it either didn't know to include it or wasn't scoped to build it.