AI Engineering Living Glossary #001: Token Explained | Cost, Context & Performance

AI ENGINEERING | LIVING GLOSSARY | #001

TOKEN

The Small Unit That Shapes AI Cost, Context and Performance

AIYING — Accomplishing with AI

Every interaction with a large language model involves tokens. They are fundamental to how language models process information, generate responses and account for computational usage.

Understanding tokens is an essential first step towards designing reliable and efficient AI applications.

Engineering insight: AI engineering is not simply about giving a model more information. It is about supplying the right information within practical limits of cost, context and performance.

Contents

1. What Is a Token?
2. Why Tokens Matter
3. How Tokenization Works
4. Industry Applications
5. Engineering Decisions
6. Practical Example
7. Try It Yourself
8. Connected Glossary Terms
9. Living Update
10. Key Takeaway
11. References

1. What Is a Token?

A token is a unit of text represented by a language model's tokenizer. Depending on the tokenization method, it may correspond to a complete word, part of a word, punctuation or another text fragment.

A language model does not necessarily process text one word at a time. Instead, text is converted into a sequence of token identifiers.

These identifiers are then represented numerically for processing by the model.

Important: One token is not necessarily one word. Token boundaries vary between models and tokenizers.

↑ Back to contents

2. Why Tokens Matter

Tokens affect several important dimensions of AI engineering.

Dimension	Engineering significance
Context window	Determines how much tokenized information can fit within the model's context limits.
Cost	Many commercial language-model APIs charge according to input and output token usage.
Latency	Input processing and output generation contribute to response time.
Prompt design	Encourages efficient use of instructions, examples and relevant context.
Application reliability	Token limits can affect whether sufficient context and complete outputs are available.

↑ Back to contents

3. How Tokenization Works

Consider the sentence:

AI engineering creates useful systems.

A tokenizer divides the text into units according to its vocabulary and algorithm.

A simplified conceptual representation is:

Input text
    ↓
Tokenizer
    ↓
Token identifiers
    ↓
Model processing
    ↓
Predicted output tokens
    ↓
Decoded response

The actual token boundaries and identifiers depend on the tokenizer used.

For autoregressive language generation, the model predicts successive tokens based on the preceding context.

↑ Back to contents

4. How the AI Industry Uses Tokens

A. API pricing

Many AI providers publish prices per million input and output tokens. Some also distinguish cached input tokens or other usage categories.

B. Retrieval-Augmented Generation

RAG systems retrieve relevant documents and place selected information into a model's context. Token budgets influence how much retrieved material can be included.

C. AI agents

Agents may repeatedly exchange instructions, tool results, intermediate information and responses with a model.

Without careful context management, repeated interactions can increase token consumption and cost.

D. Enterprise AI

Organizations monitor token usage to forecast operating expenses, optimize applications and identify inefficient workflows.

↑ Back to contents

5. What Must an AI Engineer Decide?

A practical AI engineer considers questions such as:

• How much context does this task require?
• Which information is genuinely relevant?
• Can retrieved documents be filtered?
• Would caching reduce repeated processing?
• How much output should the model generate?
• Does a longer prompt improve task quality?
• What is the cost per successful task?

Engineering principle: Optimize for successful task completion, not merely the smallest possible token count.

Excessive compression may remove instructions or evidence that the model needs. The best solution balances cost, quality, speed and reliability.

↑ Back to contents

6. Practical Engineering Example

Suppose an AI application handles 100,000 requests per month.

Assume each request uses:

• 2,000 input tokens
• 500 output tokens

Monthly consumption would be:

Category	Monthly tokens
Input	200 million
Output	50 million
Total	250 million

For illustration, suppose input tokens cost $1 per million and output tokens cost $4 per million.

The estimated monthly model usage cost is:

Input:  200 × $1 = $200
Output:  50 × $4 = $200

Total = $400 per month

These are hypothetical rates, not current prices from any specific provider.

The example excludes caching adjustments, infrastructure, storage, retrieval, tools and other operational expenses.

If input usage falls by 25% without reducing answer quality, the illustrative monthly input cost falls from $200 to $150.

The saving is $50 per month, or $600 annually at the same workload.

↑ Back to contents

7. Try It Yourself

Mini Experiment: Prompt Efficiency

Task: Summarize a technical document using an AI model.

Step 1: Write a detailed prompt with extensive instructions.

Step 2: Rewrite it using shorter, clearer instructions while retaining the requirements.

Step 3: Run both prompts using the same model and settings.

Step 4: Record input tokens, output tokens, response time and answer quality.

Step 5: Compare the results.

The objective is not to prove that shorter prompts are always better. It is to discover the most effective prompt for the task.

↑ Back to contents

8. Connected Glossary Terms

Tokens are connected to several other AI engineering concepts:

• Context Window: The amount of tokenized context a model can accommodate.
• Inference: The process of running a model to produce predictions or outputs.
• Latency: The time required to produce a response.
• Prompt Engineering: Designing effective instructions and context.
• RAG: Retrieving external information for model-grounded generation.
• Context Engineering: Selecting, organizing and managing information supplied to the model.

These topics will be explored in subsequent Living Glossary entries.

↑ Back to contents

9. Living Update

Editorial baseline: November 2026

Token management extends beyond shortening prompts. Modern AI engineering workflows also consider:

• Prompt and context caching
• Retrieval filtering and ranking
• Context compression
• Efficient handling of conversation history
• Input/output token budgeting
• Quality-aware cost optimization

Multimodal systems may also account for images, audio and video through model-specific tokenization or usage mechanisms.

Because pricing, context limits and tokenization methods vary by provider and model, engineers should consult the current documentation for their selected systems.

Living Glossary note: This entry should be revisited as tokenization methods, pricing structures and context management practices evolve.

↑ Back to contents

10. Key Takeaway

A token is a small unit of language-model processing, but token management is a major AI engineering responsibility.

Good AI engineering is not about maximizing the number of tokens supplied to a model. It is about delivering the information necessary to accomplish a task accurately, efficiently and reliably.

Think like an engineer: What information does the model actually need to succeed?

↑ Back to contents

11. References and Further Reading

• OpenAI — Tokenizer
• OpenAI — API Documentation
• Anthropic — Developer Documentation
• Hugging Face — Tokenizers Documentation

↑ Back to contents

AI Engineering — Living Glossary

One Term. One Industry Connection. One Practical Insight. One Action.

AIYING — Accomplishing with AI

Episode #001 | TOKEN

Next: #002 — EMBEDDING

↑ Return to top