The API layer

you POST a list of messages and get one assistant message back, and every provider's version of this is roughly the same shape.

Messages API / chat completions

you resend the whole conversation on every call, because the server is not keeping it for you.

Messages array

every message carries a label, and the model weights the system one above the user one, though not absolutely.

Role (system, user, assistant)

you put the standing instructions at the top once, instead of repeating who the model is and what it must never do in every user message.

System prompt

the conversation got expensive because every new message resends the entire history.

Turn / multi-turn

it is HTTP. Each request stands alone, exactly like a REST endpoint without sessions.

Statelessness

you read the usage block in the response rather than estimating, because that is the only honest number for what the call cost.

Completion / response object

you set it too low and the answer got cut off mid-sentence, then set it too high and a runaway response ate the budget.

max_tokens

you tell it to halt the moment it emits the closing fence, and the trailing explanation never gets generated at all.

Stop sequence

you treated a truncated response as a complete one, because nothing in your code branched on the finish reason.

Stop reason / finish reason

the conversation got long enough that the call started failing, and you have to drop or summarise the old turns before you can keep going.

Context overflow

the raw scores exist before any probability does, and temperature is a knob applied to those, not to the words.

Logits

you draw the next token from the distribution instead of always taking the top one, which is why two identical calls disagree.

Sampling

you always take the top token, get the same answer on every run, and occasionally watch it repeat itself in a loop.

Greedy decoding

for anything you will parse with code, run at or near 0.

Temperature

you cut off the improbable tail instead of reshaping the whole distribution, and you move this or temperature, not both.

Top_p (nucleus sampling)

you cap sampling at the k most likely tokens, a knob your local stack exposes and most hosted APIs no longer do.

Top_k

you ask for the probabilities back and use them as a rough confidence signal, or to spot where the model was torn between two answers.

Logprobs

you passed a seed, got the same answer most of the time, and learned not to build correctness on that.

Seed and determinism

the twenty-second wait becomes an answer that visibly starts in under a second, without the model getting any faster.

Streaming

the streaming you implemented is a long-lived HTTP response emitting data lines, with no WebSocket anywhere in it.

Server-sent events (SSE)

the long prompt made the wait before the first word noticeably worse, even though generation ran at the same speed after that.

Time to first token (TTFT)

the first token came back fast and then the words trickled out, which is a different number from the one you were tracking.

Inter-token latency

the batch job wants tokens per second across the fleet and the chat box wants this one answer now, and you cannot tune for both.

Latency vs throughput

your calls start coming back 429 once traffic climbs, and what you are hitting is requests per minute or tokens per minute, not a bug in your code.

Rate limits (RPM, TPM)

you retry the failed call, wait twice as long, retry again, and add a little randomness so your whole fleet does not retry in lockstep.

Exponential backoff

nobody is waiting for these ten thousand answers, so you submit them as one file overnight and pay half.

Batch API

put static instructions and documents at the top, volatile user input at the bottom; caching works on prefixes.

Prompt caching

you compute cost per request on your real traffic shape and find the output tokens dominate the number the pricing page quoted.

Pricing model

the key lives in an environment variable and a secrets manager, never in the repo, because one leaked key and a quiet weekend is a five-figure invoice.

API key hygiene