AI Engineer

you asked for a citation and got a real-looking one that does not exist, delivered with the same confidence as the parts that were true.

Hallucination

you sent the same prompt twice and got two different answers, and nothing in your code changed between the calls.

Non-determinism

you asked about a library released last month and the model has never heard of it.

Knowledge cutoff

the conversation got long enough that the call started failing, and you have to drop or summarise the old turns before you can keep going.

Context overflow

you put the important instruction halfway down a long prompt and the model behaved as if it were not there.

Lost in the middle

your calls start coming back 429 once traffic climbs, and what you are hitting is requests per minute or tokens per minute, not a bug in your code.

Rate limits (RPM, TPM)

everything it does, the chat and the code and the reasoning, comes out of one trained habit of guessing the next token.

Large language model (LLM)

every API call you make is inference, and nothing you send in one of them changes the weights for the next.

Inference

'the model learns from our conversations' is false for API usage; nothing you send updates the weights.

Training vs inference

extremely sophisticated autocomplete, run one token at a time.

Next-token prediction

every bill, rate limit and context error you will ever see is denominated in tokens, not words or requests.

Token

the same string cost a different number of tokens on the other provider, because each model family ships its own.

Tokeniser

everything you send and everything it says has to fit one budget, and the retrieved documents are what usually blow it.

Context window

you halved the prompt and the bill barely moved, because the long generated answer was where the money actually went.

Input vs output tokens

when you say the prompt you mean the parts your application controls, which is most of what actually got sent.

Prompt

you put the standing instructions at the top once, instead of repeating who the model is and what it must never do in every user message.

System prompt

you resend the whole conversation on every call, because the server is not keeping it for you.

Messages array

every message carries a label, and the model weights the system one above the user one, though not absolutely.

Role (system, user, assistant)

it is HTTP. Each request stands alone, exactly like a REST endpoint without sessions.

Statelessness

for anything you will parse with code, run at or near 0.

Temperature

you set it too low and the answer got cut off mid-sentence, then set it too high and a runaway response ate the budget.

max_tokens

you treated a truncated response as a complete one, because nothing in your code branched on the finish reason.

Stop reason / finish reason

the twenty-second wait becomes an answer that visibly starts in under a second, without the model getting any faster.

Streaming

examples teach format and edge-case handling far more reliably than prose descriptions of the format.

Few-shot / in-context learning

you tell the model to work through it step by step, and the answer gets better on anything with arithmetic or several constraints at once.

Chain of thought

the prompt lives in version control with named placeholders, not concatenated inline at three different call sites.

Prompt template

you add answer only from the provided documents, and say so if they do not cover it, and the invented details stop.

Grounding via context

a coordinate for meaning; similar meanings land close together.

Embedding

you were about to stand up a dedicated vector product and realised the database you already operate has an extension for it.

Vector database

you look up the relevant bits of your own documents first and paste them into the prompt, so the answer comes from your data instead of the model's memory.

Retrieval-augmented generation (RAG)

you split the document into pieces before you store it, because embedding a whole 40-page PDF as one blob retrieves nothing useful.

Chunking

you cut at four hundred tokens and lost the sentence that straddled the boundary, so you make adjacent chunks overlap and try again.

Chunk size and overlap

you fetch fifty candidates cheaply and then have a slower, better model reorder them before you show the top five.

Reranking

you need JSON your code can parse rather than prose, so you constrain the shape of the response instead of writing a regex over the answer.

Structured output

you write out the types, required fields and enums once, and use the same thing for tool parameters and for the response shape.

JSON schema

the model replies with the name of one of your functions and its arguments instead of an answer, and your code runs it and hands the result back.

Function calling / tool calling

the model kept picking the wrong tool until you rewrote the description like documentation for a junior engineer.

Tool definition / schema

you stop capping it at one tool call and let the model keep calling tools until it decides it is finished.

Agentic loop

you stop scripting the steps and let the model choose which tool to call next, in a loop, until it decides it is finished.

Agent

you stop saying the answers feel better and produce a number you can put next to last week's number.

Eval

you have a second model score the first one's output against a rubric, because there is no exact answer to diff against.

LLM-as-judge

you re-run the whole scored set after a prompt tweak or a model upgrade, so you find out you broke case 12 before your users do.

Regression testing

nothing in your code changed but the outputs are worse than they were in March, because the traffic or the model underneath moved.

Drift

you retry the failed call, wait twice as long, retry again, and add a little randomness so your whole fleet does not retry in lockstep.

Exponential backoff

put static instructions and documents at the top, volatile user input at the bottom; caching works on prefixes.

Prompt caching

the second person asks roughly the same question in different words and you serve them the stored answer without calling the model.

Semantic caching

you allocate the window in code, so much for retrieved context and so much for the answer, instead of discovering the ceiling at 2am.

Token budgeting

you finally attribute spend per feature and find that a summariser almost nobody uses is a third of the bill.

Cost per request

the provider had an incident and your feature stayed up, because a second model was configured and had actually been tested.

Multi-provider failover

SQL injection's cousin, except there is no clean equivalent of parameterised queries, so you defend in depth instead.

Prompt injection

you validate and allowlist what the model returned before your code acts on it, because that output is untrusted input to everything downstream.

Output validation as a security boundary

you put deterministic checks around the model, allowlists and schema validation and refusal filters, because you cannot rely on it policing itself.

Guardrails

you redact before the text leaves your network, and you know what the provider retains and for how long.

PII handling