RLT

AI
2026-09-14 08:33:35

Princeton researcher proposes RLT, a transformer design that passes internal state across tokens

Princeton PhD student Yifan Zhang has introduced Recurrent Looped Transformer, or RLT, a transformer architecture that adds a persistent internal state passed from one token to the next. In a standard transformer, the next token is generated mainly by reading prior context through attention and the KV cache. RLT keeps that mechanism but adds another path: the internal state produced after one token is processed is handed directly to the following token. The report describes it as giving the model a kind of relay baton for reasoning. One token finishes, passes its current state forward, and the next token continues from there. The number of network layers each token runs through does not change, but the state chain keeps extending as the text grows longer. The paper calls this property “infinite time depth,” meaning the computation chain has no fixed upper bound in length. The report also draws a distinction between RLT and other recurrent-style architectures such as Ouro and Astra. Those approaches mainly let the same token run through the same network multiple times, while RLT focuses on continuous state transfer between different tokens.

100
Princeton researcher proposes RLT, a transformer design that passes internal state across tokens