Choosing a Model for a Production Feature

The question usually arrives already narrowed. "Should we use the big model or the small one for this?" It is the right question asked at the wrong time. Until you know what the output has to look like to be acceptable, you have nothing to compare a cheaper model against.

So the order matters more than the choice. Establish the ceiling first, then walk down until the quality breaks, and stop one step above that.

Start with the most capable model you can call

Build the first version of the feature against the strongest model available to you, even if you have already decided you cannot afford it in production. You are not picking a model yet. You are answering a prior question: is this feature possible at all, and what does it look like when it works?

This does two things. It separates "the model cannot do this" from "the prompt is wrong," which are otherwise indistinguishable and will cost you a week of rewriting instructions for a job that was never going to land. And it gives you a reference output to compare against later, which is the only way "good enough" turns into a measurable claim instead of a feeling in the room.

If the strongest model cannot do the job reliably, stop and change the shape of the problem. Nothing downstream will help. Usually the task needs to be decomposed into smaller steps, or it needs information you have not given it.

Then step down and measure

Once the ceiling exists, try the next model down with the same prompt and the same inputs. Change one thing at a time. If you swap the model and rewrite the prompt in the same commit, you have learned nothing and you will not be able to reconstruct which change did what.

Measuring here does not require a research setup. For most production features it means three things:

• A fixed set of inputs: twenty to fifty real examples saved to disk, including the ugly ones. The demo case is not an input set.
• A definition of correct a person can apply in seconds: often just "would I have shipped this output without editing it," answered yes or no.
• A record of the failures, not only the rate: a model that fails softly on hard inputs is very different from one that fails confidently on easy ones, and a percentage hides that difference completely.

The interesting result is rarely "the small model is worse." It is usually "the small model is fine on most inputs and falls apart on one specific kind." That kind is often something you can detect up front, which turns the decision into routing rather than selection. Send the easy inputs to the fast model and escalate the rest. You get most of the savings and most of the quality, and you get a clean place to log how often escalation happens.

Latency budgets are not one number

Where the work sits in the user's attention decides how much time you have, and the two cases are far apart.

Interactive work is anything a person is watching happen: a suggestion in an editor, a reply in a conversation, a command explained in a terminal. The budget is measured against patience, and the property that matters most is time to first token rather than total time. Streaming changes the felt duration enormously. A response that starts in under a second and finishes in six feels attached to what you are doing. The same six seconds behind a spinner feels like the tool went away. For something living inside a live workflow, that one property can matter more than a quality difference you could measure between two candidates.

Background work is anything triggered by an event rather than a keystroke: processing an upload, summarizing a recording after it ends, classifying a queue overnight. Here latency is nearly free and quality is nearly everything. Use the larger model, use the longer prompt, let it take thirty seconds. Nobody is watching. That also means the cost of a wrong answer is higher, because nobody is watching to catch it.

Most real features contain both kinds of work, which is another reason a single product-wide model choice is usually the wrong unit of decision.

Make the model a setting, not a constant

The model identifier belongs in configuration, read at runtime, per call site. Not a constant compiled into the function that builds the request.

The reasons are boring and they all arrive eventually. Models get deprecated on someone else's schedule. A better one appears for your specific job. A customer has a requirement about which provider may see their data. You want the cheap model in development and the strong one in production. Outputs get strange after a provider-side change and you need to roll back at three in the afternoon, and that should be an environment variable rather than a deploy.

Per call site matters as much as configurable. A product with one global model setting forces the summarizer and the classifier and the chat surface to move together, and they have genuinely different requirements. Name the jobs, give each one its own setting, and let them fall back to a sensible default.

We built that shape into Path, which puts Claude Code and Codex behind one local API and lets you switch between them with the model string. Whatever you use, the property to preserve is that changing which model runs a job never requires touching the code that defines the job.

What breaks when you swap

Swapping is easy to write and easy to underestimate, because the failures are quiet.

• Prompts are coupled to models more than they look. A prompt tuned against one model usually carries small workarounds for that model's habits. On a different model those workarounds become bugs.
• Output format discipline varies. Smaller models are likelier to wrap structured output in prose, add a preamble, or drop a field. A parser written against a model that never did that will break on a model that does, and it will break in production rather than in your editor.
• Refusal and hedging behavior differs. A transformation one model performs without comment, another may decline or wrap in caveats, especially on unusual input.
• Long context is handled differently. The same large input can be used carefully by one model and effectively skimmed by another, and the output will look confident either way.

The defense is the input set from earlier, run against every candidate before it goes live, with a parser that fails loudly instead of quietly accepting something malformed.

The order that works

Establish the ceiling with the strongest model you can call. Write the prompt and the parser against it until the feature is genuinely good. Save the inputs you used along the way. Then step down one model at a time, changing nothing else, and watch for where it breaks. Ship the cheapest model that still clears your bar on your inputs, keep the stronger one configured for the inputs that need it, and write the choice somewhere a person can change without a build.

The mistake we see most often is the reverse. A team starts with a small model because of cost, never gets the feature to feel right, and concludes the idea does not work. Usually the idea was fine. The measurement never existed, so there was nothing to tell them how far from the ceiling they were standing.