THE MINIMALIST INFERENCE
Inference Explained in 100 Words
Ravindra Dastikop
Understand less. Accomplish more.
Why This Book?
Inference can sound complicated.
Inputs.
Tokens.
Models.
Logits.
Probabilities.
Decoding.
Generation.
Caching.
Latency.
Outputs.
But understanding Inference does not have to begin with complexity.
This book strips Inference down to what matters.
10 ideas.
10 words each.
100 words.
Enough to begin.
How to Read This Book
Read one idea.
Stop.
Think about it.
Try it with AI.
Then move to the next.
The purpose is not to know everything about Inference.
The purpose is to understand enough to accomplish something with Inference.
01 — INFERENCE
Inference uses trained models to generate outputs from new inputs.
Training and inference are different stages.
Training develops the model's learned parameters.
Inference uses those parameters.
When you give an LLM a prompt and receive a response, the trained model is being used to process your input and generate output.
It does not need to retrain itself for every prompt.
Instead, it applies what was learned during training to the information currently available.
Think:
Training builds.
Inference uses.
This distinction is fundamental to understanding how deployed AI systems work.
02 — INPUT
Input provides information the trained model processes during inference execution.
Inference begins with input.
For a language model, that input might include:
Your prompt.
Instructions.
Conversation history.
Retrieved information.
Tool results.
Or other available context.
The model processes this information when generating its response.
Changing the input can therefore change the output even though the model's trained parameters remain unchanged.
This is why context and instructions matter so much when working with AI.
The model remains trained.
The information available for the current inference changes.
03 — TOKEN
Tokens provide discrete units models process throughout language model inference.
Language models process tokens.
Your text is first represented as a sequence of tokens.
Those tokens might correspond to words, parts of words, punctuation, or other units determined by the tokenizer.
During inference, the model processes representations of these tokens.
During autoregressive generation, newly generated tokens extend the sequence used for subsequent predictions.
What looks to us like a flowing sentence therefore emerges through token-level processing.
This connects directly with The Minimalist LLM and The Minimalist Transformer.
04 — MODEL
Models transform input representations using parameters learned previously during training.
The model contains parameters learned during training.
During inference, those parameters are used to transform input representations.
For a Transformer-based language model, information passes through multiple layers.
Attention relates information across token positions.
Other neural network components transform those representations further.
The model eventually produces scores for possible next tokens.
The crucial point is simple:
The model is applying learned parameters.
It is not normally relearning them from your individual prompt.
05 — LOGIT
Logits represent model scores for possible tokens before probability normalization.
The model must determine what token might come next.
It produces numerical scores for candidate tokens.
These scores are called logits.
A logit is not itself a normalized probability.
Instead, logits represent the model's relative scores across possible outputs before normalization.
Conceptually:
Token A → 2.1
Token B → 4.7
Token C → 1.3
These values provide the raw material from which a probability distribution can be produced.
06 — PROBABILITY
Probabilities express normalized likelihoods across possible next token candidates generated.
The model has produced logits.
Those scores can be converted into a probability distribution across candidate tokens.
Now different possible next tokens have different relative likelihoods.
Conceptually:
Token A → lower probability
Token B → higher probability
Token C → lower probability
The model therefore does not simply contain one predetermined next word.
It produces scores from which possible next tokens can be selected.
How that distribution becomes an actual output depends on decoding.
07 — DECODING
Decoding determines how model predictions become tokens in generated output.
The model has produced predictions.
Now the system needs to choose what becomes output.
That process is called decoding.
Different decoding strategies can produce different behavior.
A system may choose the highest-scoring token.
Or it may sample according to a probability distribution.
Parameters and decoding methods can therefore influence how deterministic or varied generation becomes.
The model produces predictions.
Decoding turns those predictions into selected tokens.
08 — GENERATION
Generation repeatedly predicts tokens until specified stopping conditions are reached.
One token rarely makes a useful response.
So the process repeats.
The model predicts a token.
A token is selected.
That token extends the sequence.
The model predicts again.
This is autoregressive generation.
What appears to us as a paragraph is produced incrementally through repeated next-token prediction.
Generation continues until a stopping condition is reached.
That might be an end-of-sequence signal, a configured token limit, or another stopping rule.
09 — CACHE
Caching reuses previous computations to make autoregressive generation more efficient.
Autoregressive generation repeatedly needs information from previously processed tokens.
Recomputing the same attention key and value states for every previous token at every step would be inefficient.
A KV cache stores previously computed key and value states.
These stored states can then be reused during subsequent generation steps.
As new tokens are generated, new states can be added to the cache.
This reduces redundant computation, although the cache itself consumes memory.
Caching therefore introduces an important engineering trade-off:
10 — LATENCY
Latency measures time required for inference to produce useful outputs.
Inference is not only about what the model produces.
It is also about how quickly the result arrives.
Latency measures delay.
For an interactive AI system, this can include how long users wait before receiving the first generated token and how long the complete response takes.
Latency can be affected by many factors.
Model size.
Input length.
Output length.
Hardware.
System load.
Inference optimizations.
A useful AI system therefore needs more than capable models.
It also needs practical inference performance.
The Entire Book in 100 Words
INFERENCE
Inference uses trained models to generate outputs from new inputs.
INPUT
Input provides information the trained model processes during inference execution.
TOKEN
Tokens provide discrete units models process throughout language model inference.
MODEL
Models transform input representations using parameters learned previously during training.
LOGIT
Logits represent model scores for possible tokens before probability normalization.
PROBABILITY
Probabilities express normalized likelihoods across possible next token candidates generated.
DECODING
Decoding determines how model predictions become tokens in generated output.
GENERATION
Generation repeatedly predicts tokens until specified stopping conditions are reached.
CACHE
Caching reuses previous computations to make autoregressive generation more efficient.
LATENCY
Latency measures time required for inference to produce useful outputs.
The Minimalist Inference
You do not need to understand everything about Inference before you begin.
Remember ten things:
Inference.
Input.
Token.
Model.
Logit.
Probability.
Decoding.
Generation.
Cache.
Latency.
Everything else can come later.
Provide input.
Represent it as tokens.
Use the trained model.
Produce logits.
Form probabilities.
Apply decoding.
Select tokens.
Continue generation.
Reuse cached computation.
Manage latency.
Then understand how a trained model becomes a working AI experience.
One Final Question
The next time an LLM generates a response, don't begin with:
“How did it know the answer?”
Begin with:
“How did the trained model turn my input into generated tokens?”
Then think inference.
