A new self-supervised learning paper, VISReg, is being framed as a direct answer to representation collapse in JEPA-style world models, and it has already gained visible support from Yann LeCun. The Turing Award winner reposted the work multiple times and wrote, "VICReg begat SIGReg which begat VISReg," describing it as the next step in a regularization line that started with VICReg and moved through SIGReg.
The paper focuses on a long-running problem in self-supervised learning: models can appear to train successfully while mapping different inputs to the same, or only a few, embeddings. Once that happens, the system loses discriminative power. The article ties that failure mode directly to JEPA, a world-model direction LeCun has backed since 2017 through his broader push for self-supervised learning.
Why VISReg was proposed
Most mainstream approaches try to suppress collapse with heuristic training tricks such as exponential moving averages, teacher-student architectures, stop-gradient pathways, and frozen layers. The article says those choices often make training brittle, increase tuning difficulty, and weaken interpretability and scalability.
An alternative route is to constrain the representation distribution directly with regularization. LeCun’s team previously introduced VICReg, which split the objective into variance, invariance, and covariance terms. That design used covariance to control relationships across dimensions, but covariance only captures second-order statistics. Two representations can share the same mean and variance while still having very different distribution shapes.
SIGReg came later and pushed the idea further. Using the Cramér–Wold theorem and a sketching method, it aligned the full embedding distribution to a standard Gaussian. But the article says SIGReg still had two weak points. First, its gradients fade when collapse begins, so the correction signal gets weaker at the exact moment the model most needs it. Second, it does not separate scale from shape, which means the two properties interfere with each other during optimization, especially on long-tail, low-quality, and low-rank data.
VISReg is designed around those two issues.
Paper: https://arxiv.org/abs/2606.02572
Code and pretrained weights: https://github.com/HaiyuWu/visreg
Project page: https://haiyuwu.github.io/visreg/
How the method works
VISReg keeps the variance term from VICReg to control scale, but replaces covariance-based shape control with a sketching objective built on Sliced Wasserstein Distance, or SWD. It then uses stop-gradient to separate scale and shape during optimization. The article describes the full regularization target as three parts.
Scale regularization
The first term constrains the variance of each dimension and is meant to stop amplitude collapse. Its key property, according to the paper, is that the gradient approaches a constant when collapse happens. That keeps the recovery signal alive instead of letting it vanish, which is exactly where SIGReg struggled.
Shape regularization
The second term normalizes the representation first, removing the effect of scale, and then constrains shape on its own. The crucial step is a normalization that includes stop-gradient on the standard deviation. In practice, that means optimizing the shape loss does not feed back and alter scale, so the two targets no longer interfere with each other.
After normalization, VISReg aligns the geometric shape of the distribution to an isotropic Gaussian with Sliced Wasserstein Distance. The argument rests on the Cramér–Wold theorem. As stated in Lemma 3.1 of the paper, two distributions are equal if and only if all one-dimensional projections on the unit sphere are equal. In other words, if enough random one-dimensional slices of a high-dimensional representation are each aligned to a Gaussian, then the whole high-dimensional distribution is aligned as well. The article stresses that this turns full-shape matching into a series of inexpensive one-dimensional sorting operations instead of relying only on second-order statistics.
The combined objective
The third term is a centering loss that pulls the batch mean toward the origin. Those three regularizers are then combined with the prediction objective used in JEPA and LeJEPA, where embeddings from different views, global plus local, are aligned to the mean of the global views. A single hyperparameter, λ, balances prediction and regularization.
The comparison with VICReg is central to the paper’s pitch. Both methods split regularization into separate components, but VISReg swaps covariance for a sliced-Wasserstein sketching target that is intended to model the full distribution shape, while the variance term remains in place to control scale.
Light implementation and scaling characteristics
The article says the implementation is small. The core regularization logic takes about 15 lines of PyTorch code.
On compute, the regularization part of VISReg has complexity O(NDK), where N is batch size, D is feature dimension, and K is the number of slices. The paper says that scales linearly with each expansion factor. By contrast, the covariance term in VICReg is O(ND²), which grows quadratically with dimension.
The reported hardware comparison also favors VISReg. At the same batch size, the method ran faster and used less memory than SIGReg on a single NVIDIA H100 GPU.
The slice count can also be distributed across multiple GPUs. If training uses M GPUs, each device can generate K/M slices, and the article says the outcome is equivalent to producing all K slices on one card. In the reported experiment, when the slice count per GPU was too small, switching to eight GPUs with 128 slices each, 1,024 slices in total, reduced the accuracy gap versus a single-GPU 1,024-slice setup from about 2.4% to 0.22%. The implication in the paper is that larger-scale training can keep K roughly constant without adding much burden to each GPU.
Results across 15 datasets
The paper compares VISReg against seven mainstream self-supervised methods across 15 datasets: 8 in-domain benchmarks, 6 out-of-distribution datasets, and ADE20K dense prediction. The list of baselines includes MoCoV3, DINO, iBOT, I-JEPA, MAE, and data2vec. The tasks span astronomy, medical imaging, remote sensing, textures, and flowers.
In-domain linear probing
To keep the comparison fair, the experiments are divided into methods that use heuristic tricks and those that do not. In the non-heuristic group, VISReg led. On ViT-B/16, in-domain linear probing reached 75.7%, ahead of MAE at 75.1%. On ViT-L/14, VISReg reached 77.0%, above LeJEPA at 75.6%.
Against heuristic-heavy methods such as iBOT and DINO, VISReg was only slightly behind on standard datasets and outperformed all methods on the DTD texture benchmark. The article presents that as evidence that its cross-domain generalization comes from the method itself rather than training tricks.
Out-of-distribution generalization
OOD generalization is where VISReg stands out most clearly in the reported results. The evaluation covers six datasets outside the ImageNet training domain: ChestXRay, RetinaMNIST, OrganAMNIST, Galaxy10, AID, and DTD.
According to the paper, VISReg achieved the best average OOD accuracy across all methods and backbone sizes, including some models that used heuristic techniques and larger backbones. As shown in Figure 4, VISReg with ViT-B/16 reached an average OOD accuracy of 70.19%, while the ViT-L/14 version posted 70.63%. Those numbers are above MAE at 67.85%, MoCoV3 at 69.46%, DINO at 69.56%, and I-JEPA at 68.55%.
Data efficiency
The article highlights one result in particular. After pretraining VISReg with ViT-L/14 on ImageNet-22K, about 14 million images, the method reached an average of 72.94% across the six OOD datasets.
That is essentially level with DINOv2, which posted 72.93% after training on LVD-142M, a dataset of 142 million images. The paper frames this as VISReg matching DINOv2 on OOD benchmarks with roughly one-tenth of the training data. For reference, the same VISReg model pretrained only on ImageNet-1K reached 70.63%.
Transfer fine-tuning
Although VISReg trailed DINO slightly on some in-domain linear probing tasks, the ranking changed after fine-tuning. The paper says VISReg beat both DINO and supervised pretraining on all five tested datasets: CIFAR-10, CIFAR-100, Flowers, ImageNet-1K, and Galaxy10.
The original article interprets that result as a sign of more uniform representations, lower redundancy, and stronger transferability. That characterization comes from the paper and source article.
Dense prediction and generative guidance
VISReg also posted competitive numbers outside classification. On ADE20K linear semantic segmentation with ViT-B/16, it achieved 30.16 mIoU, above DINO at 29.40 and MAE at 23.60, while trailing MoCoV3 at 31.69. The paper notes that this was done without heuristic training tricks, but also says dense prediction remains behind the best-performing method and is a target for future work.
In generative guidance, the experiment used SiT-B/2 in the iREPA framework for 100,000 training steps. Generation guided by VISReg features beat DINO on three of four metrics: gFID was 40.36 versus 41.15, Precision was 51.38 versus 50.51, and Recall was 61.26 versus 60.70. IS was nearly unchanged at 33.48 versus 33.47.
Robustness on lower-quality data
The paper also reports tests on lower-quality data, including the long-tail ImageNet-LT dataset and the low-rank Galaxy10 dataset. According to the article, VISReg remained stable, avoided collapse, and learned meaningful representations in those settings, while DINO failed without careful tuning.
A new regularization route for JEPA
The article’s main conclusion is that separating representation regularization into two independent components, scale and shape, produces a more stable, more efficient, and more generalizable self-supervised method than existing approaches.
Within the results reported in the paper, VISReg delivered leading or near-leading performance across image recognition, segmentation, and generative guidance, all without relying on heuristic training tricks. It also matched DINOv2 on OOD performance with about one-tenth of the data. That gives JEPA-style world models a new regularization-based answer to representation collapse.
Reference: https://arxiv.org/abs/2606.02572
The source article credits the original Chinese piece to the WeChat public account "新智元" and author LRST.