Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters

Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters

N
News Editor
2026-10-21 06:30:08
Google Cloud has released llm-d 1.0, the first stable version of its distributed inference stack. The open-source, Kubernetes-native serving layer is designed to combine vLLM or SGLang model servers into coordinated inference pools, with a focus on improving performance for self-hosted AI agents and reinforcement learning workloads. The release adds cache-aware routing, tiered KV cache offloading, and prefilling/decoding disaggregation. According to Google, on Ironwood TPU coding workloads, the median time to first token fell by more than 85%, while throughput increased by 2.4x. The llm-d project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. It is now a CNCF Sandbox project with more than 500 contributors from over 25 organizations. The update was cited by Techub, with MarkTechPost named as the source in the brief.

Google Cloud has released llm-d 1.0, the first stable version of its distributed inference stack. The open-source, Kubernetes-native serving layer is built to combine vLLM or SGLang model servers into coordinated inference pools and is aimed at improving performance for self-hosted AI agents and reinforcement learning workloads.

What is new in llm-d 1.0

The 1.0 release introduces cache-aware routing, tiered KV cache offloading, and prefilling/decoding disaggregation, adding a set of features tailored for distributed inference deployments.

Performance figures shared by Google

According to Google, on Ironwood TPU coding workloads, llm-d reduced median time to first token by more than 85% and increased throughput by 2.4x.

Project background

The llm-d project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. It is currently a CNCF Sandbox project and has more than 500 contributors from over 25 organizations.

The brief was published by Techub and cited MarkTechPost as the source.

This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan.
9000

Disclaimer:

The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.

Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.

Google Cloud releases llm-d 1.0 distributed inference stack for TPU and GPU clusters | Bit.Fan