The migration question that needs a benchmark, not an announcement
You have a vLLM deployment on GPUs. Someone points at a TPU announcement and asks whether you should move. The honest answer is that you cannot know from the announcement alone, because the thing that decides serving cost is not the model — it is the shape of your traffic and where the time goes inside the request lifecycle. This article is a methodology guide, not a results report: it will not give you TPU-vs-GPU numbers, but it will show you how to generate your own.
Google announced new vLLM and SGLang inference engines on TPU, built as native PyTorch backends powered by TorchTPU. Per the announcement, the engines were built in collaboration with the open-source project maintainers and include optimized TPU kernel implementations, production-ready features like KV cache management with TPU Sync (the post names TPU Sync but does not describe the mechanism), and disaggregated serving. The post also claims public benchmarks show state-of-the-art performance and industry-leading cost efficiency on Ironwood TPU for long-context and agentic workloads — a vendor claim to test, not a result to trust.
That is the sourced claim. Everything below about why it matters and how to evaluate it is my analysis, not the vendor's.
What a native PyTorch backend actually changes
A serving engine is three things stacked: a scheduler that batches requests, a model execution path (attention, matmuls, sampling), and a KV cache that holds per-request state between decode steps. Historically, running an engine on non-GPU hardware meant either a separate kernel set maintained outside the main project or a fork that drifted.
A native PyTorch backend changes the maintenance story more than the API story. The engine's execution path targets TorchTPU, so the kernels and cache plumbing live inside the project rather than in a vendor fork. For you, that means:
- The model you serve is still a PyTorch model. You are not porting weights or rewriting attention.
- Bug fixes and kernel improvements land in the upstream project, not in a patch you carry.
- Your debugging tools are PyTorch tools, which is a real reduction in cognitive load.
What it does not mean is that your GPU-tuned configuration transfers. Batch sizes, tensor-parallel degree, max sequence length, and cache block sizing are all hardware-shaped. Treat the backend as a new deployment target, not a flag flip.
KV cache management is the latency lever
During decode, every generated token needs the attention keys and values for all previous tokens. Recomputing them is wasteful, so engines cache them. The KV cache is typically the largest growing consumer of memory during decode, and often the largest overall at high concurrency, because it scales with the number of concurrent requests and sequence length. How the engine allocates, evicts, and shares it decides how many concurrent requests you can hold before latency collapses.
The announcement says the TPU engines ship KV cache management with TPU Sync. The post does not describe the mechanism, so I will not guess at it. What I can say is the generic rule every serving engine follows: when the cache fills, the scheduler must preempt or queue, and queued requests show up as tail latency, not average latency.
This is where most benchmark comparisons go wrong. A GPU deployment tuned to a specific concurrency and a TPU deployment with default cache settings are not comparable. If you want a fair read, fix the concurrency, fix the prompt and output length distribution, and watch p99 rather than mean.
If you have not built that discipline yet, the latency-budget framing in why your AI feature feels slow is the right starting point — it forces you to separate queue time from compute time before you blame hardware.
Disaggregated serving and your latency budget
Disaggregated serving splits prefill (processing the prompt) from decode (generating tokens) onto separate workers. The rationale is that the two phases have different compute profiles: prefill is a big parallel matmul, decode is sequential and memory-bound. Running them on the same device means one starves the other under mixed load.
The announcement lists disaggregated serving as a production feature in these engines. It does not publish a latency breakdown, so any number I gave you here would be invented. What I can do is describe the tradeoff structurally.
Approach — Strength — Cost · Co-located prefill + decode — Simpler to operate, one pool — Prefill spikes inflate decode latency · Disaggregated — Prefill and decode scale independently — Extra network hop for KV transfer, more moving parts
The extra hop is the part people forget. KV state for a long prompt is large, and moving it between workers costs bandwidth and adds a step to the critical path. For short prompts and low concurrency, disaggregation can make things worse (my analysis, not a sourced result). It pays off when you have long prompts (long-context and agentic workloads, which is exactly what the vendor benchmark targets) and enough traffic to keep both pools busy.
A senior engineer does not enable disaggregation by default. They measure prefill time and decode time separately first. If prefill is a small fraction of your p99, splitting it buys you nothing and costs you an operational surface.
A benchmark harness you control
These engines are configured through their own launch and API surfaces; the exact TPU-specific flag names are not in the announcement, so the snippet below is deliberately generic: it shows the shape of a benchmark harness you control, not a vendor invocation. The point is that the harness lives in your repo and runs against whatever endpoint you point it at.
// bench.mjs — run with: node bench.mjs
// Measures p50/p95/p99 latency (nearest-rank) against an OpenAI-compatible endpoint.
// Assumes the response includes an OpenAI-style usage.completion_tokens field.
// Note: some servers require a flag (e.g., stream_options) to return usage.
const ENDPOINT = process.env.ENDPOINT ?? "http://localhost:8000/v1/chat/completions";
const CONCURRENCY = Number(process.env.CONCURRENCY ?? 8);
const REQUESTS = Number(process.env.REQUESTS ?? 64);
const PROMPT = "Summarize the following log line in one sentence: ";
async function oneRequest(i) {
const started = performance.now();
const res = await fetch(ENDPOINT, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: process.env.MODEL ?? "default",
messages: [{ role: "user", content: PROMPT + "event=" + i }],
max_tokens: 128,
temperature: 0,
}),
signal: AbortSignal.timeout(60_000),
});
if (!res.ok) {
throw new Error(`HTTP ${res.status}: ${await res.text()}`);
}
const body = await res.json();
const tokens = body?.usage?.completion_tokens ?? 0;
return { ms: performance.now() - started, tokens };
}
async function run() {
const results = [];
let cursor = 0;
async function worker() {
while (cursor < REQUESTS) {
const i = cursor++;
try {
results.push(await oneRequest(i));
} catch (err) {
console.error(`request ${i} failed:`, err.message);
}
}
}
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
const lat = results.map((r) => r.ms).sort((a, b) => a - b);
const totalTokens = results.reduce((s, r) => s + r.tokens, 0);
console.log(`ok=${lat.length} failed=${REQUESTS - lat.length}`);
if (lat.length === 0) {
console.log("all requests failed; no percentiles to report");
return;
}
if (totalTokens === 0) {
console.warn("warning: no completion_tokens reported; throughput numbers are unreliable");
}
// Nearest-rank percentile, clamped to the array bounds.
const pct = (p) => {
const idx = Math.ceil((p / 100) * lat.length) - 1;
return lat[Math.min(lat.length - 1, Math.max(0, idx))];
};
console.log(`p50=${pct(50).toFixed(0)}ms p95=${pct(95).toFixed(0)}ms p99=${pct(99).toFixed(0)}ms`);
console.log(`tokens=${totalTokens}`);
}
run().catch((err) => {
console.error("benchmark aborted:", err);
process.exit(1);
});Run it at three concurrency levels against both backends, with the same model, same prompt set, same max_tokens. Anything else is a comparison of two different experiments. Note that the harness uses a placeholder prompt; swap in your real prompt set before trusting results.
The output is intentionally minimal. You want to see whether p99 scales smoothly with concurrency or falls off a cliff — the cliff is where the KV cache is saturating, and that is the number that decides your capacity plan.
How to structure a benchmark that does not fool you
Vendor benchmarks are useful for forming a hypothesis. They are not evidence about your workload. The announcement's benchmarks target long-context and agentic workloads on Ironwood TPU; if your traffic is short chat turns at low concurrency, that result may not transfer at all.
Concrete rules that hold regardless of hardware:
- Pin the model and tokenizer. Different quantizations or tokenizer versions make the comparison meaningless.
- Use your real prompt distribution. Synthetic fixed-length prompts hide the prefill variance that dominates real p99.
- Report p95 and p99, not mean. Mean latency is the most flattering number you can compute.
- Sweep concurrency. A single-concurrency result tells you almost nothing about a production pool.
- Measure warm and cold. First-request latency after a deploy includes compilation and cache warmup; exclude it or report it separately, but do not hide it.
- Count failures. A backend that drops requests under load will look fast if you only average the successes. The harness above logs failures for this reason.
As an illustrative heuristic, not a threshold: if p99 is several times p50 — pick your own ratio — that gap is worth investigating. It could point to scheduler contention, but it could equally be cache saturation, prefill variance, or queueing at a proxy. The p50 tells you the hardware is fine; the p99 tells you something in the pipeline is struggling — and fixing that is often cheaper than migrating.
The broader discipline of picking the right numbers before you commit to an architecture is covered in six numbers to check before you ship an AI feature, and it applies just as well to a backend migration.
Failure modes you will actually hit
Configuration transfer. Copying a GPU batch size onto a TPU pool is the most common first mistake. The memory hierarchy and parallelism strategy differ; start from the engine's defaults and tune one axis at a time.
Cache sizing under bursty traffic. Steady-state benchmarks miss the burst case where concurrency triples for thirty seconds. Size the KV cache for the burst, or accept preemption and measure how bad it gets.
Disaggregation without headroom. Splitting prefill and decode into two pools means each pool needs its own headroom. Under-provision either and you have moved the bottleneck, not removed it.
Benchmark contamination. Running your load generator on the same host as the server, or through a proxy that adds its own queue, will show you the proxy's latency. Keep the generator off the serving host.
Silent fallbacks. As a general risk to watch for (not a documented behavior of these TPU backends), if a kernel path is unavailable for your model shape, the engine may fall back to a slower path without erroring. Watch throughput per request, not just success rate.
Lock-in, honestly
The code you write is PyTorch, and vLLM and SGLang expose OpenAI-compatible HTTP servers (verify for the TPU build you deploy), so the model layer is portable. The lock-in is elsewhere:
- Capacity and pricing. TPU availability and cost structure are vendor-specific. Your capacity plan does not transfer back to GPUs without re-benchmarking.
- Operational tooling. Monitoring, autoscaling, and debugging integrations are shaped by the platform. That is the real switching cost.
- Feature parity drift. The announcement describes these as new engines. Features present on the GPU path may not be present on the TPU path yet; the post does not enumerate parity, so verify against the project docs for the specific version you deploy.
The pragmatic position: keep the serving contract (OpenAI-compatible HTTP) as the boundary, keep your benchmark harness in your repo, and you can move between backends without rewriting application code. It is the same boundary-placement instinct as prompt injection is an architecture problem — decide where the seam belongs, and the implementation behind it becomes swappable.
What to do this week
If you are curious about TPU serving, the cheap experiment is: run your existing benchmark harness against a TPU endpoint with the same model and prompt set, at three concurrency levels, and compare p95 and p99. Do not migrate anything. Just get the number.
If the p99 is better at your target concurrency and the operational cost is acceptable, then disaggregated serving is worth a second experiment. If it is not, you have saved yourself a migration and learned where your actual bottleneck is — which is usually the scheduler or the prompt distribution, not the silicon.
In my read, the announcement is a real change in the maintenance story for TPU serving, and that is worth something. It is not a reason to skip the benchmark.
Further reading
- A small team's AI eval stack
- Prompt injection is an architecture problem
