Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters
Google Cloud has released llm-d 1.0, the first stable version of its distributed inference stack. The open-source, Kubernetes-native serving layer is designed to combine vLLM or SGLang model servers into coordinated inference pools, with a focus on improving performance for self-hosted AI agents and reinforcement learning workloads. The release adds cache-aware routing, tiered KV cache offloading, and prefilling/decoding disaggregation. According to Google, on Ironwood TPU coding workloads, the median time to first token fell by more than 85%, while throughput increased by 2.4x. The llm-d project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. It is now a CNCF Sandbox project with more than 500 contributors from over 25 organizations. The update was cited by Techub, with MarkTechPost named as the source in the brief.

