AIYING · ALL OF AI FOR ALL OF US · INFERENCE ENGINEERING · C21 — TUTORIAL #001
vLLM Prefix Caching: Why Can a Shorter Prompt Still Be Slower?
By Ravindra Dastikop
Measure reusable prefixes, not just total prompt length
Shortening a prompt reduces the amount of input. But if the change replaces an early shared prefix, the server may have to recompute more of the remaining input instead of reusing cached work. A shorter request can therefore take longer to produce its first token under some conditions.
Prefix caching reuses previously computed model state for matching initial token sequences. It is different from caching and replaying a complete answer. vLLM’s Automatic Prefix Caching documentation explains that this saves input-processing work, called prefill, rather than directly accelerating generation of new output tokens, called decode.
A practitioner report filed in the vLLM repository on 26 October 2026 describes cache reuse dropping after context compaction changed an early part of a conversation. That report motivates investigation; its reported gains are not independently reproduced here or general promises.
Example: the document assistant that rewrites its introduction
Consider a fictional service answering questions about the same manual. Its request builder places a fresh request timestamp before the manual. Another version keeps the stable instructions and manual first, then appends request-specific details and the question.
Both requests can contain similar information, but they offer different matching prefixes. The exact benefit depends on tokenization, cache block boundaries, model architecture, routing, and whether the relevant cache entries remain available.
Compare these layouts in an isolated serving environment:
A: stable instructions | request timestamp | manual | question
B: stable instructions | manual | request timestamp | question
This is an experiment design, not measured results. Keep the instruction hierarchy and required information intact. Do not rearrange prompts merely to gain cache hits if the change harms correctness.
Run a cold-versus-warm comparison
- Pin the setup: record vLLM version, model revision, tokenizer, chat template, hardware, routing, and cache configuration. Confirm prefix caching is enabled for the deployed model. Use documentation matching your installed version.
- Prepare requests: use one fixed manual and several questions. For each question, construct A and B with identical content except placement of the timestamp. Inspect the rendered token sequences to confirm where they first differ.
- Warm each condition separately: reset the prefix cache using your version’s supported mechanism, or restart an isolated test server and complete non-prefix warm-up. Send a seed request, then questions sharing that condition’s prefix. Avoid cross-condition cache contamination.
- Measure: collect client time to first streamed token, input and output token counts, cache-reuse measurements, total latency, and answer correctness. Keep output limits, sampling settings, and concurrency fixed. Check metric names and units for your version.
- Repeat and compare: run multiple cold/warm cycles, alternating condition order. Compare median and tail latency across enough requests to make those summaries meaningful; retain individual measurements.
Start at concurrency one to inspect the mechanism, then repeat under representative traffic. Low-load improvement does not establish production improvement.
If cache hits rise but latency does not improve, inspect queueing, output generation, eviction, and routing to different replicas. Hybrid architectures can impose additional cache-matching constraints. A high cache-hit figure alone is not the user-facing outcome.
Your turn: What request-specific content appears before your largest reusable prompt segment?
Sources and scope: vLLM Automatic Prefix Caching documentation and vLLM issue #63743, checked 17 November 2026. The issue is a contributor’s report, not a verified general benchmark. The example and tutorial are AIYING proposals; no serving experiment was executed for this post.

