Production patterns and cost

you route every model call through one service so keys, logging, caching and cost attribution live in one place instead of scattered through the codebase.

LLM gateway / proxy

you send the easy requests to the small cheap model and only the hard ones to the expensive one.

Model routing

you try the cheap model first and escalate to the strong one only when the cheap answer fails a check.

Cascade / fallback chain

the provider had an incident and your feature stayed up, because a second model was configured and had actually been tested.

Multi-provider failover

after enough consecutive failures you stop calling the provider at all for a while, instead of queueing thousands of doomed requests behind a dead dependency.

Circuit breaker

the provider went down and users got a cached answer or an honest error, because you decided that in design rather than at 3am.

Graceful degradation

nobody is waiting on this call, so it goes on a queue with bounded concurrency and the 429 storm stops.

Queueing and async processing

you render tokens as they land, show which tool is running, and handle the stream that dies halfway through.

Streaming UX patterns

the identical request comes in again and you return the stored answer, with no chance of it being a subtly different answer.

Exact-match caching

the second person asks roughly the same question in different words and you serve them the stored answer without calling the model.

Semantic caching

you allocate the window in code, so much for retrieved context and so much for the answer, instead of discovering the ceiling at 2am.

Token budgeting

you finally attribute spend per feature and find that a summariser almost nobody uses is a third of the bill.

Cost per request

the average looked fine and one user in twenty was waiting eleven seconds, because output length varies enormously.

Percentile latency (p50, p95, p99)

one row per call with model, prompt version, tokens, cost and outcome, which is your debugging record and your cost ledger in one table.

Structured logging of LLM calls

the prompt sits in version control, gets reviewed, is gated by the eval suite, and can be rolled back like any other deploy.

Prompt and configuration versioning

you run the candidate on live traffic without showing anyone its answers, and compare the logs afterwards.

Shadow deployment

you are five abstraction layers deep trying to see the prompt that was actually sent, and start wondering what the framework is buying you.

Frameworks: build vs buy