Research firm SemiAnalysis has released a report that breaks down the underlying architecture of large-model inference services. The report says that as mixture-of-experts, or MoE, models become mainstream, AI inference is no longer a single compute task. Instead, it has developed into a more complex pipeline made up of Prefill, Midfill, Decode Attention, and Decode Experts.
Resource demands vary across inference stages
According to the report, those stages place clearly different demands on compute, memory bandwidth, and networking. In most inference scenarios, memory bandwidth has more economic value than capacity. SemiAnalysis says high-bandwidth memory can improve token generation efficiency, while idle HBM only adds cost.
Single pipeline stages may need 400 GB to 500 GB of fast local memory by 2027
SemiAnalysis estimates that by 2027, an individual pipeline stage could require about 400 GB to 500 GB of local fast memory. At the same time, KV cache that has already been processed should be moved promptly to CPU DRAM and lower-cost network storage, so that scarce HBM resources are not occupied unnecessarily.
Scheduling is described as a key infrastructure layer
The report identifies the scheduling layer as a critical part of AI inference infrastructure. Prefill and Midfill have relatively predictable processing times, while decode duration can vary widely, making task backlogs and delay-feedback oscillation more likely. SemiAnalysis says stable buffering, scheduling across different time scales, and tiered KV cache management will directly shape throughput efficiency in inference clusters.
Integrated and disaggregated designs come with different trade-offs
On architecture choices, the report compares integrated and disaggregated approaches. In the integrated design, Prefill and Decode are placed on the same device, which reduces cross-node KV cache transfers but requires hardware that combines high compute density with high memory bandwidth. In the disaggregated design, dedicated nodes are assigned to different stages, allowing hardware optimization for each role, though at the cost of more data movement and higher network overhead.
SemiAnalysis says the trade-off between the two approaches will ultimately depend on whether future accelerators can combine both high compute performance and high memory bandwidth in a single general-purpose design.
Simulator projections cover Kimi K3 on Nvidia systems
The report also uses a simulator to project Kimi K3 performance on Nvidia B200, B300, and GB200 systems. The results show that GB200 has an advantage in some low-latency scenarios, while the peak single-GPU HBM residency in most frontier configurations stays below 80 GB. The report says this adds to the case for prioritizing memory bandwidth and storage orchestration.