THE MINIMALIST TOKENIZATION
Tokenization Explained in 100 Words
Ravindra Dastikop
Understand less. Accomplish more.
Why This Book?
Tokenization can sound complicated.
Text.
Tokens.
Vocabulary.
Subwords.
Algorithms.
Encoding.
Special tokens.
Sequences.
Decoding.
Efficiency.
But understanding Tokenization does not have to begin with complexity.
This book strips Tokenization down to what matters.
10 ideas.
10 words each.
100 words.
Enough to begin.
How to Read This Book
Read one idea.
Stop.
Think about it.
Try it with AI.
Then move to the next.
The purpose is not to know everything about Tokenization.
The purpose is to understand enough to accomplish something with Tokenization.
01 — TOKENIZATION
Tokenization converts input into units that language models can process.
Humans see sentences, words, punctuation, and meaning.
Language models need numerical inputs.
Tokenization provides the bridge.
A tokenizer takes input and divides or encodes it into units called tokens according to its tokenization system.
Those tokens can then be mapped to numerical identifiers that the model processes.
A token is therefore not necessarily a word.
It might represent a word, part of a word, a character, bytes, punctuation, or another learned unit depending on the tokenizer.
INPUT → TOKENIZATION → TOKENS
Tokenization turns human-readable input into model-processable units.
02 — TOKEN
Tokens are discrete units models receive and generate during processing.
A token is a unit produced by a tokenizer.
Consider:
unbelievable
One tokenizer might represent it as one token.
Another might divide it into several subword tokens.
The exact result depends on the tokenizer and its vocabulary.
For language models, tokens matter because models operate over sequences of token representations rather than simply receiving ordinary written words as humans perceive them.
During generation, autoregressive language models also produce outputs token by token.
TEXT → TOKENS → MODEL
Tokens are the model's working units of language representation.
03 — VOCABULARY
Vocabulary defines the token set a tokenizer can represent directly.
A tokenizer needs a collection of known tokens.
That collection is its vocabulary.
Each vocabulary entry is associated with an identifier.
Different models can use different vocabularies.
A vocabulary might contain:
Common words.
Subwords.
Characters.
Punctuation.
Byte-derived units.
Special tokens.
The tokenizer uses its vocabulary and tokenization method to represent incoming text.
Subword methods help keep vocabularies manageable while still representing uncommon words using smaller known units.
VOCABULARY → TOKENS → IDs
04 — SUBWORD
Subwords represent frequent language pieces while decomposing less familiar words.
Whole-word tokenization creates a problem.
Language contains enormous numbers of words, names, variations, and newly created terms.
Subword tokenization offers a useful compromise between whole words and very small units.
Common sequences can remain together.
Less familiar words can be divided into smaller known pieces.
For example, a tokenizer might split a rare word into several subwords that already exist in its vocabulary.
This helps models represent unfamiliar words without requiring every possible word to have its own vocabulary entry.
WORD → SUBWORD + SUBWORD
05 — ALGORITHM
Algorithms determine how input sequences are divided into available tokens.
Different tokenizers use different algorithms.
Common subword approaches include:
BPE.
WordPiece.
Unigram.
They do not all learn or select tokens in the same way.
BPE learns useful merges.
WordPiece uses its own vocabulary-building and segmentation approach.
Unigram begins with candidate pieces and learns which pieces should remain.
SentencePiece is a tokenization framework that can implement approaches such as BPE and Unigram directly from raw text.
The key point is simple:
Token boundaries are determined by the tokenizer—not by spaces alone.
06 — ENCODING
Encoding maps tokenized input into numerical identifiers models can process.
The model does not receive token strings in the way humans read them.
Tokens are mapped to numerical IDs from the tokenizer's vocabulary.
Conceptually:
TEXT → TOKENS → TOKEN IDs
For example:
Token A → ID 125
Token B → ID 731
Token C → ID 42
The actual identifiers depend entirely on the tokenizer vocabulary.
Those IDs can then be transformed into numerical representations used by the model.
Tokenization therefore connects human-readable input with the numerical processing performed by the model.
07 — SPECIAL TOKEN
Special tokens represent structural purposes beyond ordinary input content units.
Not every token corresponds directly to ordinary content.
Models may use special tokens for structural or task-specific purposes.
Depending on the model, these might indicate:
Beginning of a sequence.
End of a sequence.
Padding.
Masking.
Separation between sequences.
Different architectures and tokenizers use different special tokens.
Special tokens therefore provide information that helps structure or control how token sequences are interpreted.
CONTENT TOKENS + SPECIAL TOKENS → MODEL INPUT
08 — SEQUENCE
Sequences arrange tokens in order for models to process context.
After tokenization, the model receives an ordered sequence of token IDs.
Order matters.
Compare:
Dog bites man.
Man bites dog.
The same words can create very different meanings when arranged differently.
Transformer models process token sequences while using positional information and attention mechanisms to model relationships within those sequences.
Tokenization therefore does more than create isolated pieces.
It creates the ordered units over which the model operates.
INPUT → TOKENS → ORDERED SEQUENCE → MODEL
These tokens also occupy space within the model's available context.
09 — DECODING
Decoding converts generated token identifiers back into human-readable output text.
Generation eventually needs to return to human-readable language.
The model produces or selects token identifiers.
The tokenizer's decoding process converts those identifiers back into token strings and reconstructs readable output according to its rules.
Conceptually:
TOKEN IDs → TOKENS → TEXT
Different tokenization systems may require different reconstruction rules.
Subword pieces may need to be joined.
Whitespace may need to be reconstructed.
Special tokens may need to be handled appropriately.
Tokenization therefore works in both directions:
ENCODE → MODEL → DECODE
10 — EFFICIENCY
Efficient tokenization balances vocabulary size, sequence length, and coverage effectively.
Tokenizer design involves trade-offs.
Use very large units and the vocabulary can become enormous.
Use very small units and sequences can become much longer.
Subword tokenization provides a practical middle ground.
Frequent patterns can use compact tokens.
Less common patterns can be represented through combinations of smaller tokens.
Byte-level approaches can provide broad coverage because arbitrary text can ultimately be represented using byte-based units.
Tokenization therefore influences:
Vocabulary size.
Sequence length.
Representation.
Computation.
Context usage.
TEXT → EFFICIENT REPRESENTATION → MODEL
The Entire Book in 100 Words
TOKENIZATION
Tokenization converts input into units that language models can process.
TOKEN
Tokens are discrete units models receive and generate during processing.
VOCABULARY
Vocabulary defines the token set a tokenizer can represent directly.
SUBWORD
Subwords represent frequent language pieces while decomposing less familiar words.
ALGORITHM
Algorithms determine how input sequences are divided into available tokens.
ENCODING
Encoding maps tokenized input into numerical identifiers models can process.
SPECIAL TOKEN
Special tokens represent structural purposes beyond ordinary input content units.
SEQUENCE
Sequences arrange tokens in order for models to process context.
DECODING
Decoding converts generated token identifiers back into human-readable output text.
EFFICIENCY
Efficient tokenization balances vocabulary size, sequence length, and coverage effectively.
The Minimalist Tokenization
You do not need to understand everything about Tokenization before you begin.
Remember ten things:
Tokenization.
Token.
Vocabulary.
Subword.
Algorithm.
Encoding.
Special Token.
Sequence.
Decoding.
Efficiency.
Everything else can come later.
Start with input.
Divide it into tokens.
Use a vocabulary.
Represent unfamiliar words with smaller units.
Apply a tokenization algorithm.
Map tokens to IDs.
Add special tokens where required.
Create an ordered sequence.
Process it with the model.
Decode generated IDs back into readable output.
Then understand how language crosses the boundary between humans and models.
One Final Question
The next time you hear “token,” don't begin with:
“How many words is that?”
Begin with:
“How does this model's tokenizer represent this input?”
Then think Tokenization.
