Pretraining at the billion-parameter scale has remained out of reach for many smaller labs. Open-weight model families such as Qwen have made downstream experimentation easier, but they have not fully disclosed the complete pretraining recipe. Other projects, including OLMo and SmolLM, have gone further by releasing data and training details, yet replication is still expensive. The article cites estimates of about $200,000 to reproduce OLMoE-1B/7B and about $719,000 for SmolLM3-3B, even at this scale.
Against that backdrop, a team led by Tsinghua University professors Chen Wenguang and Lü Kaifeng introduced Puro-2B, a fully open-source pretraining pipeline that releases training data, the training framework, configuration files, and model weights.
What Puro-2B released
According to the paper, the team trained a model with roughly 2 billion parameters from scratch on a compute cluster built with consumer-grade RTX 5090 GPUs, using as many as 1.4 trillion tokens.
In a broad evaluation spanning 15 tasks, the paper says a version trained with about $4,400 in pretraining compute surpassed Qwen2-1.5B on the average metric across those tasks. Its best version, trained at a cost of about $6,900, delivered average performance close to Qwen2.5-1.5B. The report presents this as a sizable reduction in training cost relative to other open pretraining setups.
The team also published the paper, model checkpoints, training data and mixture ratios, training code, and data-processing code.
- Paper: https://arxiv.org/pdf/2608.27370
- Open-source collection: https://huggingface.co/collections/thu-pacman/puro-2b
- Models and intermediate checkpoints: https://huggingface.co/collections/thu-pacman/puro-2b
- Training data and ratios: https://huggingface.co/datasets/thu-pacman/Puro-2B
- Training code: https://github.com/thu-pacman/Puro-Megatron
- Data processing code: https://github.com/thu-pacman/Kaiyuan-Spark
A cost-performance curve for planning runs
The team also tried to answer a practical question: what level of model quality is achievable when the budget falls even lower. To do that, it fit an empirical cost-performance relation it calls the Puro Cost Scaling Law.
The method starts from the same first-stage checkpoint, then varies the training budget for the second stage under a uniform data recipe. That produces a fitted relation between average performance and total cost. In practice, the curve is meant to let researchers estimate what a given budget can buy before committing to the longest full training run.
The low-cost recipe is built from five parts
The paper breaks the system into five pieces: hardware choice, low-precision training, optimizer changes, higher efficiency in using high-quality data, and data selection. The idea is not a single shortcut. It is a full-stack attempt to lower cost across different layers of the training process.
More compute per dollar with RTX 5090
The team begins with a basic constraint: how much compute a fixed budget can actually buy. Using the GPU rental prices adopted in the paper, it found that RTX 5090 offers about 2.77x the BF16 peak throughput per dollar of NVIDIA’s H200. In other words, on a cost basis, the consumer GPU can provide more raw compute than widely used data-center cards, while also being easier to obtain.
That comes with trade-offs. RTX 5090 has only 32 GB of memory and no NVLink, so data-center training settings cannot simply be copied over.
To adapt the platform, the team adjusted drivers and system settings to enable PCIe P2P communication within each node. It says that raised one-way GPU-to-GPU bandwidth from 31.5 GB/s to 56 GB/s and cut communication latency from 14.3 microseconds to 0.4 microseconds. Across nodes, it enabled GPU Direct RDMA, which increased 24-GPU AllReduce bus bandwidth from 8.87 GB/s to 19.93 GB/s.
The training strategy was also tuned for billion-parameter workloads and the hardware profile of RTX 5090. The team used a combination of data parallelism and pipeline parallelism, avoiding tensor parallelism because of its heavier communication pattern. It also redistributed compute across pipeline stages, adjusted micro-batch sizes, and placed optimizer states according to memory usage to balance memory pressure across nodes.
The paper says these changes pushed overall MFU above 70%, turning the cost advantage of consumer hardware into a practical training-cost advantage.
FP8 training and its trade-off
Once cheaper compute was in place, the next step was to improve efficiency at the precision level. Because RTX 5090 is based on the Blackwell architecture, the team used FP8 training as part of the stack. It followed the fine-grained block quantization approach used in DeepSeek-V3, placing matrix multiplication in major linear layers on the FP8 path while keeping numerically sensitive operations and training states in BF16 or FP32.
FP8 can reduce accuracy, so the team ran controlled experiments at several model sizes in what it calls a scaling ladder. Across five model scales, the paper reports that FP8 produced validation loss about 0.0031 to 0.0039 higher than BF16. The gap was small and stable across scales.
At the same time, training throughput improved by 1.36x. After converting the loss gap into the extra compute needed to reach the same loss, the paper still estimates a net efficiency gain of about 1.34x.
From nominal learning rate to effective learning rate
The optimizer is another major lever because it affects how much each update actually changes the model. The team used MuonH, which the paper describes as a variant of Muon that normalizes the update magnitude and then projects weights back to a fixed radius after the update. That setup is intended to control both the update norm and the weight norm.
With that structure, effective learning rate, or ELR, can be set directly instead of drifting as weight scale changes during training.
The paper points to a key comparison. If standard Muon is used without the fixed-radius weight projection, but its learning rate is dynamically adjusted so that its ELR trajectory matches MuonH, the loss curves almost overlap. Based on that result, the team argues that MuonH’s advantage should not be attributed only to norm constraints. The ELR trajectory itself appears to be the more informative lens.
Another comparison found that when weight decay was not enough to constrain the weight norm, standard Muon saw ELR fall faster early in training, and loss dropped earlier as well. MuonH kept a gentler decay and performed better later. That shifted the focus from choosing an optimizer name to designing a better ELR schedule.
The team then ran ablations on the share of the decay phase under different peak learning rates and training-token budgets. The result: higher peak learning rates and longer training runs usually require longer decay periods. Based on that, Puro-2B uses a power schedule in stage one that does not depend on a preset endpoint and supports continued training, while stage two uses a long linear decay to leave room for final loss convergence.
In scaling-ladder comparisons, the paper says MuonH delivered about 1.19x training efficiency relative to a tuned Muon baseline.
Putting high-quality data later in training
Data efficiency depends not only on what data is used but also on when it is shown to the model. The paper argues that high-quality samples are limited in proportion but highly influential, and that placing them later in training improves their use. Many open datasets already provide sample-level quality scores.
The Puro team used those signals to move higher-quality samples toward the later part of training within each source, then aligned and mixed data blocks from different sources according to training progress. That allows the model to move gradually into higher-quality regions while keeping local source ratios roughly stable, avoiding abrupt changes in domain composition at the same time quality is changing.
But ordering the data is not enough by itself. The team’s earlier work found that when learning rates become too low late in training, high-quality data that appears near the end may no longer move the parameters much. To address that, it used Curriculum Model Averaging, or CMA. The method keeps the learning rate constant late in training so the model can continue absorbing higher-quality data, then averages weights across the final checkpoints to reduce the training noise introduced by the higher rate.
The ablation results in the report say that model averaging on top of a standard decaying-learning-rate trajectory does not deliver stable gains by itself. What matters is the combination of data order, late-stage learning rate, and model averaging.
Using the Puro Cost Scaling Law for estimation, the paper says that compared with random ordering under Uniform data distribution, curriculum data plus learning-rate decay is equivalent to about 1.65x better cost efficiency, while the full CMA setup reaches about 2.40x.
A structured workflow for choosing data
After solving data order, the next question is which data to keep. The team built a layered comparison workflow across data source, data slice, and capability evaluation. For large datasets with quality scores, it sampled slices from different score ranges. For other eligible sources, it used random slices.
Each candidate slice starts from the same checkpoint, continues training under the same configuration, and is then evaluated on a fixed suite covering math, code, Chinese, and general capabilities. That makes it possible to compare sources against each other and to spot quality differences inside a single source.
The proxy experiments described in the article also found that, relative to math ability, code ability weakened general capability more. On that basis, the team increased the weight of math data in the second stage and combined domain coverage with quality filtering to settle on the final data mix. The related dataset-evaluation results were released in the report.
More than a lower compute bill
Seen as a whole, Puro-2B addresses efficiency at several levels: hardware cuts the price of each unit of compute, FP8 increases peak efficiency and shortens execution time, MuonH improves the speed of loss convergence, CMA raises the effective use of training data, and proxy evaluation helps identify data that is more worth training on. Together, those pieces form the Puro low-cost training pipeline.
The article also points to a research use case. Researchers often want to know whether a pretraining improvement still matters after post-training. If only final open weights are available, they cannot control what happened during pretraining. Puro-2B is presented as a workable and affordable path for changing the pretraining method, continuing with fine-tuning, and checking the final effect.
To test that, the Puro team took CMA and Uniform pretrained weights and applied the same fine-tuning pipeline to both. In two math fine-tuning experiments, models initialized from CMA pretrained weights improved GSM8K accuracy by 1.77 percentage points and 2.02 percentage points. In Tulu-3 mixed-domain fine-tuning, the average score across 15 benchmarks rose by 1.17 percentage points and IFEval improved by 1.36 percentage points.
The article says all three comparisons showed overall gains, suggesting that the benefit of CMA can carry into post-training. In that sense, Puro-2B is framed not only as a stronger model release but also as a base pretraining workflow researchers can use to test whether pretraining changes lead to measurable final benefits.
Paper: https://arxiv.org/pdf/2608.27370
This article cites a post originally published on the WeChat public account "Xinzhiyuan" and credited to LRST.