The release is real. The comparison needs a footnote.
A year ago, most inference traffic was one prompt in and one response out. Agent workloads broke that pattern. A single task now fans out into dozens of model calls that share long prefixes, arrive in bursts, and expect the system to remember what the previous step computed. Serve that traffic the old way, with replicas answering in rotation, and each replica recomputes context its neighbour already holds.
llm-d is the open-source attempt to fix the routing layer rather than the model. It sits in front of serving engines such as vLLM and SGLang, tracks which replica holds which cached prefix, and sends each new request to a replica that can reuse that work. Version 1.0, announced on 20 October at the PyTorch Conference, is less about new tricks than about operability: stable v1 APIs, priority and fairness controls so bursty agent loops do not starve interactive users, a highly available router, and rolling upgrades that move prefill and decode pools together without dropping active sessions.
Reading the 85%
The number Google leads with deserves a slower read. On an agentic coding workload running on Ironwood TPUs (TPU 7x), Google reports that prefix-aware routing plus host-memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4 times. The comparison point, stated in Google's own post, is a plain Kubernetes Service. That is the default way to spread traffic across replicas, with no knowledge of what any replica has cached.
That baseline matters, because it is close to no routing intelligence at all. The result supports the claim that this combined configuration can improve serving performance on a prefix-heavy workload. It does not isolate how much of the gain comes from routing alone, and it does not establish an advantage over a tuned vLLM or SGLang deployment, a commercial inference gateway, or the previous llm-d release. Google does not offer those comparisons in the announcement.
The production figures further down the same post are more useful, because they come from systems that were already running — but they are still company-reported. Google says its Gemini platform serving path doubled its prefix cache hit rate from 35% to 70% through this routing layer, cutting time-to-first-token by more than 35% for coding agents. Those are Google's figures for its own serving path, not an independent comparison against competing systems. Google also relays Snap's report of prefix cache hit rates of up to 75–80% in production; that figure is Snap's as relayed by Google, and it was not independently verified in the sources reviewed here. Neither example establishes that llm-d beats a tuned alternative under comparable conditions.
What 1.0 actually changes for a team serving its own models
If you already run vLLM behind a load balancer, 1.0 is the first llm-d release that asks to be treated as infrastructure rather than as an experiment. The v1 API commitment, multi-tenant flow control and zero-downtime upgrades are the features operations teams usually wait for before adopting a serving layer, and they are the substance of this release.
The project also ships Prism, an open dashboard that Google says lets users inspect the benchmark configurations behind its published numbers and compare them with runs from their own cluster. That is a practical starting point for testing whether the reported gains transfer to a different environment. It is not independent validation: the results still depend on the workload, hardware and configuration used. Before treating the headline figures as planning assumptions, reproduce the comparison against your own tuned serving stack.
Two cautions before budgeting around it. First, llm-d remains a CNCF Sandbox project, the foundation's entry-level hosting tier. Version 1.0 marks a release milestone; it should not be mistaken for evidence that the project's governance and ecosystem have reached the maturity of a more established CNCF project. Second, the strongest published results are on Google's own stack, GKE with Ironwood TPUs or managed Lustre storage. The stack is designed to run on any cloud and accelerator, and contributors include AMD, Intel and Mistral, but those results are evidence for Google's configuration: they do not establish equivalent gains on other clouds, accelerators or serving stacks, and contributor participation is not a cross-platform performance benchmark.
Sources
- [1] Google Cloud Blog, 'llm-d 1.0, an open source stack for distributed inference', 20 October 2026 — primary source: the 1.0 feature list, the 85% and 2.4x figures and their plain Kubernetes Service baseline, the Gemini and Snap production figures, the keynote demonstration and the Prism dashboardRead source
- [2] SiliconANGLE, 'IBM-backed llm-d framework adds controls for running AI models in-house', 20 October 2026Read source
- [3] Unite.AI, 'llm-d 1.0 Debuts at PyTorch Conference with Production Inference Stack', 20 October 2026Read source