Tsinghua team open-sources Puro-2B, saying a 2B-parameter pretraining run can be done on RTX 5090s for about $4,400

A team led by professors Chen Wenguang and Lü Kaifeng at Tsinghua University has released Puro-2B, a fully open-source pretraining pipeline built around low-cost hardware and reproducible training. The project includes training data, framework, configurations, code, and model weights, and the team says it trained a model with roughly 2 billion parameters from scratch on a consumer GPU cluster based on NVIDIA’s RTX 5090, using as many as 1.4 trillion tokens. According to the paper, a version trained with about $4,400 in pretraining compute outperformed Qwen2-1.5B on the average score across 15 tasks, while a stronger version trained for about $6,900 came close to Qwen2.5-1.5B. The report contrasts that with far higher estimated replication costs for other open projects, including about $200,000 for OLMoE-1B/7B and about $719,000 for SmolLM3-3B. The paper attributes the lower cost to a combination of design choices rather than one change alone: the use of RTX 5090 hardware, FP8 training, the MuonH optimizer, curriculum model averaging, and a structured data-selection workflow. The team also introduced a Puro Cost Scaling Law to estimate expected model capability at different budget levels before running a full training cycle.