Language Models

Loop language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent-state organization. This article isolates those axes under matched controls and tests whether the resulting design principles transfer to Mixture-of-Experts models.

Parcae stabilizes loop models using an exact exponential decay for its recurrent state, while its input branch uses an Euler gain. We replace that gain with exact zero-order hold (ZOH) for the existing full-matrix write. In completed matched 140M controls and a 370M paper-style follow-up, validation loss improves; downstream readout remains mixed.