Loop language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent-state organization. This article isolates those axes under matched controls and tests whether the resulting design principles transfer to Mixture-of-Experts models.
Parcae stabilizes loop models using an exact exponential decay for its recurrent state, while its input branch uses an Euler gain. We replace that gain with exact zero-order hold (ZOH) for the existing full-matrix write. In completed matched 140M controls and a 370M paper-style follow-up, validation loss improves; downstream readout remains mixed.
Loop models are becoming active again in reasoning and language modeling, with recent examples such as HRM, TRM, recurrent-depth latent reasoning, and Parcae. This post asks a simple question: what is the actual compute cost of looping? I analyze that question in an ablation chain over major loop-model variants, with special attention to how optimizer interval, gradient path, and storage policy change the FLOPs, NFE, and memory accounting.