“Which model should we call?” used to sound like an API question. In a growing number of organizations, it is closer to a procurement, performance-engineering, and architecture question.
That change follows a simple development: there are now many models, offered through many providers, with materially different strengths, prices, latency profiles, deployment options, and contractual conditions. Stripe’s announced agreement to acquire OpenRouter is a useful signal. OpenRouter’s single API spans more than 400 models from more than 80 providers, with routing criteria that include task complexity, price, speed, reliability, latency, throughput, and provider-specific costs.
The emerging job is not necessarily a new title. It may sit across AI platform engineering, architecture, procurement, inference operations, or product engineering. But its responsibility is becoming recognizable: decide which model should handle which work, under what constraints, with what fallback, and based on what evidence.
Routing is a policy decision disguised as infrastructure
A naïve router asks: “Which model is cheapest?” A useful router asks a more specific question: “What is the least expensive model that meets the quality, latency, reliability, privacy, and operational requirements of this request?”
Those requirements vary by task. A customer-support classifier may need predictable structured output and low latency. A difficult coding task may justify a slower, more capable model. A high-volume summarization pipeline may favor a smaller model, especially if its quality is adequate after testing. A regulated workflow may require a particular region, retention policy, or provider agreement regardless of token price.
This is why routing belongs in architecture reviews, not only in application code. The route determines more than an invoice. It can affect data residency, outage exposure, observability, response consistency, tool-use behavior, and the amount of human review required downstream.
The four disciplines behind a serious routing function
1. Procurement: compare the whole service, not the headline token price
Model prices are easy to compare badly. Input and output tokens may have different rates. Cached input, batch processing, priority service, and long-context requests can alter the calculation. A provider’s nominal price also says little about retries, rate limits, support, minimum commitments, egress, or the engineering cost of switching.
A routing owner should maintain a model-and-provider inventory with fields such as:
- input, output, cached, and batch pricing;
- context and output limits;
- documented rate limits and observed throughput;
- latency distributions, not merely average latency;
- availability and timeout behavior;
- data-use, retention, residency, and contractual terms;
- supported capabilities, including tool calls, structured output, vision, and streaming;
- fallback and migration options.
The result is closer to a technology bill of materials than a list of model names. It should be reviewed when prices, policies, model versions, or business volumes change.
2. Performance engineering: measure the task, not the leaderboard
General benchmarks can help with orientation, but routing decisions need workload-specific tests. A model that performs well on a public coding benchmark may not be the best choice for an organization’s internal repositories, naming conventions, tool schemas, or security controls.
Build a representative evaluation set from real requests, with sensitive material removed or controlled. Label the outcomes that matter: factual correctness, valid JSON, successful tool selection, code-test success, refusal behavior, escalation need, and acceptable style. Then record cost, time to first token, total latency, timeout rate, retry rate, and completion length.
Do not collapse the result into one score too early. A weighted score can hide a serious failure mode. For example, a model with excellent average quality but frequent malformed tool calls may be unsuitable for an automated workflow. A slower model may be economically preferable if its answers reduce costly human review.
Use a champion-and-challenger process: keep a currently approved route, test alternatives against the same corpus, and promote a challenger only when it clears explicit quality and operational thresholds. Provider-reported claims should be treated as inputs to a test plan, not as proof that a model will perform similarly in your environment.
3. Architecture: make model choice replaceable
Routing becomes expensive when model-specific assumptions leak throughout an application. A resilient design separates the business task from the provider call.
Define an internal capability contract. It might specify that a “classification” operation returns a fixed schema, confidence or abstention fields, a model identifier, and a trace identifier. A “draft response” operation might specify tone constraints, citation requirements, and a maximum latency budget. Provider adapters then translate that contract into individual APIs.
Keep prompts, schemas, tool definitions, safety rules, and route policies versioned. Record which model snapshot and provider served each request. Preserve enough information to reproduce a decision without retaining sensitive user content unnecessarily.
Design fallbacks deliberately. A fallback might be another provider, a smaller model, a queued workflow, or a human review path. It should not silently change the task’s meaning. If structured output is mandatory, the fallback must support the same contract or trigger a controlled escalation.
4. Governance: decide when not to route automatically
Some requests should not be sent to the cheapest available model—or to any external model. Routing policy needs exclusion rules for confidential data, high-impact decisions, unsupported languages, unusually long context, or actions that require a human approval step.
Teams should also distinguish between a model being technically available and being approved for a particular use. Procurement and legal requirements may differ by business unit. A model can be excellent in an evaluation and still be unusable for a workflow whose data-handling terms do not fit the organization.
A practical routing table
A starting policy can be simple and explicit:
| Task class | Primary objective | Possible route | Escalation trigger |
|---|---|---|---|
| High-volume extraction | Valid schema and low unit cost | Small or mid-sized model with strict output validation | Schema failure or low confidence |
| Complex analysis | Quality and evidence handling | More capable model with a larger latency budget | Missing evidence, ambiguity, or policy flag |
| Interactive assistance | Fast perceived response | Low-latency model, possibly followed by refinement | Low confidence or user request for depth |
| Sensitive workflow | Approved data handling and auditability | Contract-approved provider or controlled deployment | Unapproved data, action, or jurisdiction |
The exact table will differ by organization. The important point is that routing rules should be readable by product, security, finance, and engineering stakeholders—not buried in a conditional statement.
What this means for people building careers
The strongest candidates for this work will combine several kinds of fluency. They will understand enough machine learning to reason about capability and degradation; enough systems engineering to manage latency, retries, rate limits, and failure modes; enough finance to model total cost; and enough procurement and governance to evaluate provider commitments and restrictions.
They will also be comfortable writing decision records. A useful record explains why a route was selected, what evidence supports it, what risks remain, and what event should trigger reevaluation. That is more valuable than memorizing the latest model names because model names and prices will keep changing.
A compact portfolio project could demonstrate the skill without requiring a large production system. Take one workload, create a redacted evaluation set, connect three model providers or local models behind a common interface, and compare quality, schema validity, latency percentiles, failure rates, and estimated monthly cost at several volumes. Add policy rules for sensitive inputs and a fallback path. Publish the test methodology and limitations.
Be precise about what the project proves. It does not prove that one model is universally best. It proves that you can turn an ambiguous model-choice problem into a measurable operating policy.
The career signal
Model routing is becoming strategically important because intelligence is no longer a single fixed dependency. It is a portfolio of services with different trade-offs and changing economics. The teams that treat that portfolio as interchangeable infrastructure may reduce costs, but they can also create hidden quality, compliance, and reliability problems. The teams that treat it as a permanent single-model commitment may miss better options.
The emerging discipline sits between those extremes: abstract enough to change providers, specific enough to preserve task quality, and evidence-driven enough to justify the choice. That is the model-routing career opportunity—not choosing an API once, but building the decision system that keeps choosing well.
Tom Whitfield is the accountable human editor of AI Career Brief, covering skills, roles, and smart moves for working in the age of AI.
