THE MINIMALIST POST-TRAINING
Post-Training Explained in 100 Words
Ravindra Dastikop
Understand less. Accomplish more.
Why This Book?
Post-Training can sound complicated.
Fine-tuning.
Preferences.
Rewards.
DPO.
Reinforcement learning.
Safety.
Alignment.
Evaluation.
Generalization.
Human judgment.
But understanding Post-Training does not have to begin with complexity.
This book strips Post-Training down to what matters.
10 ideas.
10 words each.
100 words.
Enough to begin.
How to Read This Book
Read one idea.
Stop.
Think about it.
Try it with AI.
Then move to the next.
The purpose is not to know everything about Post-Training.
The purpose is to understand enough to accomplish something with Post-Training.
01 — POST-TRAINING
Post-training adapts pretrained models toward useful, controllable, and desired behaviors.
Pretraining gives a model broad capabilities.
But broad capability does not automatically produce the behavior people want.
A model may need to become better at:
Following instructions.
Answering helpfully.
Respecting preferences.
Solving particular tasks.
Avoiding undesirable behavior.
Post-training is the broad stage in which additional training techniques are applied after pretraining to shape model behavior.
Different methods can be used.
PRETRAINING → POST-TRAINING → DESIRED BEHAVIOR
Pretraining builds broad capability.
Post-training shapes how that capability is expressed.
02 — SUPERVISED FINE-TUNING
Supervised fine-tuning teaches models desired responses using curated example demonstrations.
One important post-training method is supervised fine-tuning, often abbreviated SFT.
People provide examples demonstrating desired behavior.
Conceptually:
PROMPT → DESIRED RESPONSE
The model trains on these examples.
Its parameters are adjusted so responses become more consistent with the demonstrated patterns.
Examples might teach:
Instruction following.
Response formats.
Task behavior.
Conversational behavior.
Domain-specific responses.
SFT therefore gives the model examples of what good responses should look like.
EXAMPLES → SFT → ADAPTED MODEL
Demonstration becomes training.
03 — PREFERENCE
Preference data compares candidate responses to express desired model behavior.
Sometimes showing one good response is not enough.
We may instead compare alternatives.
Suppose a model generates:
Response A
and
Response B
An evaluator indicates which response is preferred.
Why?
Perhaps one is:
More accurate.
More helpful.
Safer.
Clearer.
Better aligned with instructions.
Many such comparisons create preference data.
PROMPT → RESPONSE A vs RESPONSE B → PREFERENCE
Preference data communicates relative desirability rather than merely providing one target response.
This creates another way to shape model behavior.
04 — REWARD
Reward models learn preference signals used to guide model optimization.
Human preferences can be used to train a reward model.
The reward model learns to assign scores reflecting patterns in preference data.
Imagine evaluators repeatedly preferring one response over another.
Those comparisons provide evidence about desirable behavior.
A reward model attempts to learn that signal.
PREFERENCES → REWARD MODEL → REWARD SIGNAL
The reward signal can then help guide subsequent model optimization.
This is an important component of traditional RLHF pipelines.
The reward model does not replace human values or judgment.
It approximates patterns represented in the preference information used to train it.
05 — DPO
Direct preference optimization trains models directly from paired preference data.
Reward modeling followed by reinforcement learning is not the only way to learn from preferences.
Direct Preference Optimization, or DPO, provides another approach.
DPO works with preference pairs such as:
Prompt.
Chosen response.
Rejected response.
Instead of separately training a reward model and then performing a reinforcement-learning stage, DPO directly optimizes the language model using preference data.
Conceptually:
CHOSEN RESPONSE > REJECTED RESPONSE
becomes a learning signal.
PREFERENCE PAIRS → DPO → ADAPTED MODEL
This makes DPO an important modern preference-optimization method.
06 — REINFORCEMENT LEARNING
Reinforcement learning improves model behavior using rewards from evaluated outputs.
Another post-training approach uses reinforcement learning.
The model produces outputs.
Those outputs receive rewards according to some evaluation mechanism.
Optimization then seeks behavior associated with higher reward.
Conceptually:
MODEL → OUTPUT → REWARD → LEARNING
In RLHF-style systems, human preference information can contribute to the reward signal through a learned reward model.
Other reinforcement-learning approaches may use different reward mechanisms.
The important idea is:
Instead of only imitating example responses, the model can learn from feedback about the quality of generated behavior.
07 — SAFETY
Safety training reduces harmful behaviors while preserving useful model capabilities.
A capable model should not simply become more effective.
Its behavior must also be managed appropriately.
Post-training can help models learn to avoid or appropriately handle undesirable requests and behaviors.
This might involve:
Safety examples.
Preference data.
Policies.
Adversarial examples.
Specialized training.
Evaluation.
The challenge is balance.
A model that responds dangerously is undesirable.
But a model that unnecessarily refuses harmless requests is also less useful.
The objective is therefore not simply:
REFUSE MORE
It is:
REDUCE HARM + PRESERVE USEFULNESS
Safety is part of shaping model behavior responsibly.
08 — EVALUATION
Evaluation measures whether post-training improves targeted behaviors without regressions elsewhere.
Training changes a model.
Evaluation tells us what those changes actually accomplished.
Did instruction following improve?
Did safety improve?
Did reasoning performance change?
Did helpfulness increase?
Did another capability deteriorate?
Post-training can improve one behavior while unintentionally affecting another.
Evaluation therefore needs multiple relevant measurements.
A useful comparison is:
MODEL BEFORE POST-TRAINING
versus
MODEL AFTER POST-TRAINING
Then measure the differences.
Post-training without evaluation tells us what we changed.
Evaluation tells us whether the change was actually useful.
09 — GENERALIZATION
Generalization checks whether learned behaviors transfer beyond training examples reliably.
A model should not merely reproduce its post-training examples.
It should behave appropriately on new situations.
Suppose training examples teach the model to follow instructions in hundreds of cases.
The important question is:
What happens when it receives an instruction it has never encountered?
Useful learned behavior should extend beyond specific examples.
Conceptually:
TRAINING EXAMPLES → LEARNED BEHAVIOR → NEW SITUATIONS
This is generalization.
Post-training succeeds more meaningfully when desired behavior transfers to relevant unseen inputs rather than remaining tied to memorized examples.
10 — JUDGMENT
Human judgment defines desired behavior, evaluates outcomes, and guides improvement.
Post-training ultimately raises a human question:
What behavior do we actually want?
Training methods cannot answer that question independently.
People decide:
What counts as helpful.
What counts as harmful.
Which trade-offs matter.
Which preferences deserve representation.
Which evaluations matter.
Whether performance is acceptable.
Technical methods can optimize objectives.
But objectives themselves embody choices.
VALUES → TRAINING OBJECTIVES → MODEL BEHAVIOR → HUMAN EVALUATION
Post-training therefore remains connected to human judgment.
We do not merely train models.
We decide what kinds of behavior we are trying to create.
The Entire Book in 100 Words
POST-TRAINING
Post-training adapts pretrained models toward useful, controllable, and desired behaviors.
SUPERVISED FINE-TUNING
Supervised fine-tuning teaches models desired responses using curated example demonstrations.
PREFERENCE
Preference data compares candidate responses to express desired model behavior.
REWARD
Reward models learn preference signals used to guide model optimization.
DPO
Direct preference optimization trains models directly from paired preference data.
REINFORCEMENT LEARNING
Reinforcement learning improves model behavior using rewards from evaluated outputs.
SAFETY
Safety training reduces harmful behaviors while preserving useful model capabilities.
EVALUATION
Evaluation measures whether post-training improves targeted behaviors without regressions elsewhere.
GENERALIZATION
Generalization checks whether learned behaviors transfer beyond training examples reliably.
JUDGMENT
Human judgment defines desired behavior, evaluates outcomes, and guides improvement.
The Minimalist Post-Training
You do not need to understand everything about Post-Training before you begin.
Remember ten things:
Post-Training.
Supervised Fine-Tuning.
Preference.
Reward.
DPO.
Reinforcement Learning.
Safety.
Evaluation.
Generalization.
Judgment.
Everything else can come later.
Start with a pretrained model.
Demonstrate desired behavior.
Collect preferences.
Learn reward signals where appropriate.
Optimize directly from preferences when useful.
Learn from rewards when appropriate.
Train for safer behavior.
Evaluate the changes.
Test generalization.
Keep humans responsible for defining and judging desired outcomes.
Then understand how broad model capability becomes more useful behavior.
One Final Question
The next time you hear “post-training,” don't begin with:
“Which training method should we use?”
Begin with:
“What behavior do we want the model to exhibit?”
Then think Post-Training.
