OpenAI's upcoming Astra model employs a recurrent depth architecture, also known as a looped transformer. Unlike traditional Transformers, which process each token through a fixed stack of layers, Astra allows the same information to repeatedly pass through the same set of layers, performing multiple rounds of computation before output.
ByteDance and Seed Team's Earlier Work
This technical route is not new. ByteDance publicly explored similar methods last year. The Seed team's Ouro model is a Looped Language Model that runs a set of transformer layers in a loop, pushing more computation inside the model. This allows smaller models to achieve results close to larger ones by increasing computational effort, without generating longer chains of thought.
Safety Concerns and Mitigation
However, the architecture introduces a security issue: some reasoning happens in hidden internal states, leaving no visible text trail for humans to inspect for rule violations through the chain of thought. To address this, OpenAI has limited the extent of recurrent depth usage in Astra, ensuring it retains a readable chain of thought, and plans to add additional CoT monitoring.