Wafer AI said it recently deployed Kimi K3 on AMD MI355X GPUs, reporting that a model which had required 16 NVIDIA B200 GPUs across two servers could instead be run inside a single AMD server with eight MI355X cards.
In its test with 1,024 input tokens and 400 output tokens, the MI355X setup delivered 952 tokens per second in aggregate throughput and 118 tokens per second for single-user generation. Wafer said that on a per-node basis, throughput was about 3.8 times that of the 16-card B200 configuration, and that the cost-performance ratio also came in above both B200 and B300.
Memory capacity was the main constraint
Kimi K3 has 2.8 trillion parameters. The article says the model weights alone require more than 1.5 TB of memory, before accounting for the KV cache needed for a million-token context window.
An eight-card B200 server offers 192 GB of memory per GPU, or about 1.5 TB in total. According to the report, that makes it difficult to fit the model weights in full, let alone leave room for KV cache, which is why the B200 deployment had to span two servers and 16 GPUs.
B300 carries 288 GB per GPU and can hold the model within a single node. AMD’s MI355X also comes with 288 GB per card, bringing an eight-GPU server to roughly 2.3 TB, enough to keep the deployment on one machine.
That matters beyond simple server count. Once a model runs across nodes, every generated token may require data synchronization over the network. The article notes that even with a roughly 195 Gb/s RoCE v2 network, cross-node communication still slows decoding. MI355X avoided that by keeping the full model inside one node through its larger memory footprint.
Throughput and cost figures published by Wafer
Wafer’s final numbers showed peak aggregate throughput of 952 tokens per second on eight MI355X GPUs, with single-stream generation at 118 tokens per second.
For comparison, the dual-node 16-card B200 deployment reached 498 tokens per second in total throughput, which works out to about 249 tokens per second per node. On that basis, the article says MI355X’s single-node throughput was about 3.8 times the average per-node throughput of the B200 setup. For single-user generation, MI355X’s 118 tokens per second also topped B200’s 90.
B300 still led on raw speed. An eight-card B300 node posted 1,568 tokens per second of aggregate throughput and 172 tokens per second for single-stream generation, or about 1.65 times the total throughput of MI355X.
Wafer then applied hourly price assumptions of $2.5 per MI355X, $4.25 per B200, and $6 per B300. Under that pricing, MI355X delivered roughly 48 tokens per second of peak throughput per dollar, versus about 7 for B200 and about 33 for B300.
That left B300 as the faster option, while MI355X came out ahead on unit-cost efficiency. The article says that trade-off may matter more than a simple race for the performance crown for data centers running open models at scale.
ROCm needed fixes, but not a ground-up rewrite
The report says AMD’s biggest data-center GPU problem has often been software rather than hardware. A model that runs directly on CUDA may require framework changes, operator patches, or even low-level kernel rewrites on ROCm.
In Kimi K3’s case, though, AMD provided support that the article described as close to day-one readiness. Wafer said the model could mostly run directly on MI355X, with follow-up work centered on a small number of compatibility issues and performance tuning.
One issue appeared in speculative decoding. Because Kimi K3 itself did not provide the draft-model parameters needed for MTP or EAGLE, Wafer used an external block-diffusion draft model. That setup ran directly on CUDA, but in ROCm the first real request triggered a scheduler error because a function called top_k_renorm_prob was missing from the ROCm branch.
The function’s role, as described in the article, is straightforward: select the top-k values from a probability distribution, zero out the rest, and renormalize the retained probabilities. Wafer said it patched that logic with a standard PyTorch function, without writing a custom GPU kernel and without redesigning the speculative decoding system.
After the fix, speculative decoding lifted single-stream performance by about 2.2x, improved single-stream performance under medium concurrency by about 1.7x, and increased peak aggregate throughput by about 18%. The article also says the system was then able to reach peak throughput under higher concurrency.
Fixing the time-to-first-token problem
Throughput was not the only metric covered. The report also focused on TTFT, the wait from request submission to the first generated token appearing.
On that measure, MI355X initially lagged. For a cold-start prefill task of about 172,000 tokens, MI355X took about 51 seconds, while B300 took about 23 seconds. For models that support million-token contexts, prefill workloads can become extremely large, making first-token latency a visible part of the user experience.
Wafer eventually traced most of the gap to one attention kernel. Under Kimi K3’s eight-way tensor parallel setup, each GPU handled 12 attention heads. AMD AITER’s faster MLA prefill kernel, however, supports shapes in multiples of 4, 8, or 16. Since 12 heads did not fit those requirements, the system fell back to the slower generic Triton implementation.
The workaround was simple: pad the 12 attention heads to 16, call the existing fast kernel, then recover the 12 heads actually needed after the computation. The article says this did not change the model structure and did not require a new assembly kernel. It only added four zeros.
After the optimization, the AITER MLA kernel reached a stable prefill speed of about 13,000 tokens per second, versus roughly 4,000 to 7,000 tokens per second on the previous Triton fallback path. Cold prefill time was cut by around two to three times. The report says this does not change final decoding throughput, but it does reduce the wait before users see the first token.
What the test did and did not show
The article also cautions that a single test does not prove AMD has fully caught up with NVIDIA. B200 was forced into a cross-node setup because of memory limits, B300 still led on absolute performance, and ROCm’s tooling, framework support, and developer ecosystem still trail CUDA.
Even so, the piece argues that open models are moving quickly into the trillion-parameter era. When a model grows beyond what one server can comfortably hold, memory capacity stops being a spec-sheet detail and starts shaping communication cost, deployment complexity, and eventual throughput. On that reading, AMD’s strategy of packing more HBM onto each GPU is turning into a system-level advantage.
The article cites related X posts from Wafer AI and Chirag Asarpota. It also notes that the piece originated from the WeChat public account "机器之心," with the author identified as "关注LLM的".