Google Cloud
Oct 21, 2026

Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters

Google Cloud has released llm-d 1.0, the first stable version of its distributed inference stack. The open-source, Kubernetes-native serving layer is designed to combine vLLM or SGLang model servers into coordinated inference pools, with a focus on improving performance for self-hosted AI agents and reinforcement learning workloads. The release adds cache-aware routing, tiered KV cache offloading, and prefilling/decoding disaggregation. According to Google, on Ironwood TPU coding workloads, the median time to first token fell by more than 85%, while throughput increased by 2.4x. The llm-d project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. It is now a CNCF Sandbox project with more than 500 contributors from over 25 organizations. The update was cited by Techub, with MarkTechPost named as the source in the brief.

910
Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters