Lifestyle
What Is Inference? The Computation Behind Every Response After Model Training
Inference uses a trained model to process new inputs and produce outputs, with weights fixed throughout. This article breaks language model inference into prefill and decode, explains the differences between time to first token, end-to-end latency and throughput, distinguishes inference from reasoning and cloud from local execution, and shows what batching, quantization and caching each save.
About 8 min read

Contents
- 1. How Inference Differs from Training
- 2. The Two Phases of a Response: Process the Prompt, Then Generate Tokens
- 3. Example: Two Ways to Ask a Guesthouse Assistant
- 4. Two Kinds of Slowness: Time to First Token and End-to-End Latency
- 5. Reducing Costs: Batching, Quantization and Caching
- 6. Cloud vs. Local: Whose Hardware, and Who Is Queuing?
- 7. Terminology: Inference, Reasoning and Test-Time Compute
Inference uses a trained model to process new inputs and produce outputs. Each time you press send, all computation from reading the prompt to generating the final token is inference. Model parameters, such as weights, remain fixed throughout; adjusting weights is part of training. Another term often confused with inference is reasoning, which refers to a model's ability to derive conclusions through multiple steps. This site uses the Traditional Chinese term "tuilun" for inference and "tuili" for reasoning; the reason is explained at the end.
This article uses a fictional guesthouse booking assistant to explain the two phases of a response, why longer outputs usually take longer, the difference between two latency metrics, and what batching, quantization and caching each save. It mainly draws on Google's glossary, two research papers and official documentation from various providers. It lists no prices or speed figures, since these change with the model and hardware.
How Inference Differs from Training
Google's Machine Learning Glossary defines inference in traditional machine learning as applying a trained model to unlabeled examples to make predictions; for large language models, it means using a trained model to generate a response to an input prompt. Training is the process of determining model parameters, including weights and biases: the system reads examples and gradually adjusts its parameters. The dividing line is whether parameters are changed. The glossary also notes that inference has a slightly different meaning in statistics; this article uses its machine learning meaning.
During inference, weights do not change because of your particular question. If you include a few examples in the prompt, the model can follow their format, but those examples are simply inputs to this computation. Whether a provider stores conversations and later uses them for training is a matter of product terms and settings, separate from inference itself.
| Aspect | Training | Inference |
|---|---|---|
| Weights | Gradually adjusted | Fixed; used only for computation |
| Frequency | Concentrated before model completion or during version updates | Computation required for every request |
| Cost structure | Compute and data investment before model completion | Accumulates with request count and length |
The Two Phases of a Response: Process the Prompt, Then Generate Tokens
An inference request to a large language model has two phases. The first is processing the prompt (prefill): all prompt tokens are already known, so they can be processed together in parallel. This also computes probabilities for the first new token and stores intermediate results for each position—keys and values, collectively called the KV cache. The vLLM paper calls this the prompt phase; Pope et al. call it prefill.
The second phase is generating tokens one by one (decode): each step feeds only the previous new token into the model to compute the next one, until an end token or the length limit is reached. The KV cache for earlier tokens is already stored, so each step only needs to compute the new token's contribution. However, each step must wait for the preceding one, so the steps cannot run in parallel. The vLLM paper notes that this phase struggles to fully utilize GPU compute capacity, is constrained by memory reads, and accounts for most of a single request's latency.
This is why longer outputs take longer: more tokens require more repeated steps, and each step must read model weights and the KV cache. Longer inputs mainly increase the work in the single prefill phase and enlarge the KV cache that subsequent steps must read. OpenAI's latency guide likewise notes that generating tokens is almost always the step with the highest latency; unless the input is very long, it is usually not the main cause.
Read the full description
Timeline. A request first passes through queuing and networking, then enters prefill, where the entire prompt is processed in parallel in one pass. It then enters decode, generating one token per step until completion. Time to first token covers queuing, networking and prefill; generation time covers the tokens generated one by one afterward. Their sum is end-to-end latency. Longer inputs mainly increase time to first token; longer outputs mainly increase generation time. Weights remain fixed throughout, with no updates.
Example: Two Ways to Ask a Guesthouse Assistant
The following examples have not been tested. Prompt A: paste in lengthy check-in instructions and ask, "What time is checkout?" The input is long and the output is one sentence. Prompt B: provide no reference material and simply request a very long introduction to walks around the neighborhood. The input is short and the output is long. A is expected to take longer to display its first character because the instructions must be processed first, but it should finish quickly afterward. B should display its first character quickly, but most of its end-to-end latency will be spent generating many tokens one by one.
If measurements differ from these expectations, do not force this model to fit. A long time to first token may result from queuing, network round trips, or a longer prompt than expected. Slow generation may result from many concurrent requests or a longer output than expected. NVIDIA's documentation notes that TTFT typically includes queuing, prefill and network latency; identifying the bottleneck requires measuring each phase separately.
Two Kinds of Slowness: Time to First Token and End-to-End Latency
Time to first token (TTFT) runs from submitting a request to receiving the first token. End-to-end latency runs through the final token; NVIDIA's documentation expresses it as TTFT plus generation time. Anthropic's documentation notes that TTFT is especially important for streaming output: streaming lets users see the beginning sooner, improving perceived speed, but does not change the total token count.
Throughput measures how many tokens the entire system can generate per second. NVIDIA's documentation explains that as more requests are processed concurrently, total system throughput increases until resources are fully utilized, but each user's individual speed decreases. Latency and throughput therefore involve a trade-off.
Reducing Costs: Batching, Quantization and Caching
The cost of each request comes from how much hardware time it occupies; Pope et al. measure this as chip time or money per token. In their analysis of very large Transformer inference across multiple accelerator chips, they note that small batches can offer lower latency but cost more per token; for offline processing, larger batches are usually the most economical. Three common approaches to reducing costs are outlined here:
- Batching: combining multiple requests for computation. The vLLM paper notes that the KV cache is large and grows and shrinks dynamically, so wasted memory limits batch size; it proposes PagedAttention to address this.
- Quantization: representing weights with lower-precision numbers to reduce memory requirements. Pope et al.'s experiments compare bfloat16 weights with weights quantized to int8.
- Caching: reusing previously computed results, such as the KV cache for an identical prefix. OpenAI's guide recommends placing variable content toward the end of the prompt.
For details, see Quantization and Prompt Caching; prompt caching saves work only during prefill and does not shorten decode.
Cloud vs. Local: Whose Hardware, and Who Is Queuing?
Cloud inference runs on a provider's cluster, serving many people simultaneously. Requests must queue, may be batched with other people's requests, and involve network round trips, all of which contribute to your wait. The main things you can adjust are prompt and output length, whether to stream, and model size. Both Anthropic's and OpenAI's latency documentation list these options.
Local inference usually avoids network round trips and other people's requests, but you must supply the compute and memory, and your device limits model size. The llama.cpp project describes its goal as local and cloud inference across a range of hardware, offering integer quantization to reduce memory usage. Both environments have prefill and decode; the difference is who supplies the resources and how many people share them.
Terminology: Inference, Reasoning and Test-Time Compute
Traditional Chinese translations of inference and reasoning are inconsistent. Google's Traditional Chinese Machine Learning Glossary translates inference as "tuilun" and uses "tuili" to describe a model's reasoning. The National Academy for Educational Research's LeCi terminology site lists both translations across subject glossaries: statistical terminology uses "tuilun," while computer terminology uses "tuili"; inference engine is rendered as "tuili yinqing" (reasoning engine) or "tuili ji" (reasoning machine). Thus, "tuilun = inference, tuili = reasoning" is this site's convention for distinguishing the two concepts, not a dictionary rule; terms such as "inference engine" must be read in context.
Related concepts: Reasoning Models are a class of models designed for multistep problems; generating their answers still involves inference. Test-Time Compute is a strategy of investing more compute during inference. For other terms, see the AI Terminology Index.
Found this guide helpful? Share it with someone who needs it
Lifestyle
Agentic Retrieval-Augmented Generation (Agentic RAG): When to Search Again
Agentic RAG lets models make retrieval decisions, choosing sources, rewriting queries or searching further based on the question and available evidence. Using a concrete LangGraph workflow and a comparison of venue-use conditions, this article contrasts it with fixed RAG and explains stopping criteria, permissions, query logs and cost control. Learn to judge whether a task needs dynamic exploration or a predictable fixed workflow is enough.
Lifestyle
Graph Retrieval-Augmented Generation (GraphRAG): Understanding a Document Collection Through Relationships
GraphRAG uses entities and relationships in retrieval to answer questions about cross-document connections and overall themes. Drawing on Microsoft's original paper and official docs, this article explains entity extraction, community summaries, and local and global search, using local event records to show why different questions need different paths. It also shows how missing links, mistaken name merges, and distorted summaries affect answers: a pretty graph is not verified fact.
Lifestyle
Knowledge Graphs: Making Entities and Relationships Clear
Knowledge graphs organize entities and relationships to trace who took part in what, where and when through links. Using community events, this article covers entity identity, relationship direction, sources and time; compares Resource Description Framework (RDF) and property graphs; and explains why graphs are not equivalent to visualizations or reliable facts. Learn when to build one and how it works with tables, vector search and graph retrieval-augmented generation (GraphRAG).
Lifestyle
Hybrid Search: Balancing Exact Terms and Meaning
Hybrid search merges full-text, vector and other retrieval results to cover exact codes and natural-language needs. An equipment-manual query explains candidate retrieval, deduplication, rank fusion and reranking, including how Reciprocal Rank Fusion (RRF) merges results by rank. Hybrid search is not always more accurate. Learn to test quality with code, paraphrase and no-answer queries and spot biases from permissions, filters and duplicate content.
Articles that cite this one
Sources
- Google: Machine Learning Glossary (entries including inference and training)Checked:
- Google: Machine Learning Glossary (Traditional Chinese version; inference translated as tuilun)Checked:
- Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM, SOSP 2023)Checked:
- Pope et al.: Efficiently Scaling Transformer Inference (2022)Checked:
- NVIDIA: NIM LLMs Benchmarking, Metrics (definitions of TTFT, ITL and throughput)Checked:
- Anthropic: Reducing latency (official documentation)Checked:
- OpenAI: Latency optimization (official documentation)Checked:
- llama.cpp: Project overview (official implementation documentation for local inference)Checked:
- National Academy for Educational Research's LeCi: inference entry (translation comparison across subject glossaries)Checked:
- National Academy for Educational Research's LeCi: exact-match results for inference engine (glossaries use tuili yinqing or tuili ji)Checked:
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.