Towards Looped Models Done Right

Benhao Huang‡†, Chufan Shi‡*, Junlin Chen‡, Shicheng Wen‡*,

Zhengzhong Liu‡, Eric Xing‡, Xuezhe Ma‡*

‡Institute of Foundation Models, *USC, †CMU

†Correspondence to: Benhao Huang <benhaoh@andrew.cmu.edu>

Code (Release Soon) | Living Blog (updating continuously)

1
2
3
4
5
6
7
8
9

@misc{huang2026loopedmodels,
  author = {Benhao Huang and Chufan Shi and Junlin Chen and
            Shicheng Wen and Zhengzhong Liu and Eric Xing and Xuezhe Ma},
  title = {Towards Looped Models Done Right---Part I:
           Topology, Input Injection, Recurrent-State Design},
  year = {2026},
  url = {https://huskydoge.github.io/husky-blog/posts/recursive_models/towards-looped-models-done-right/},
  note = {Accessed 2026-07-31}
}

1. Introduction

Loop language models repeatedly reuse the same Transformer blocks across logical depth, trading additional computation for reduced memory usage. As modern AI hardware delivers rapidly increasing compute throughput while memory capacity and bandwidth improve much more slowly [1] AI and Memory Wall
A. Gholami, Z. Yao, S. Kim, C. Hooper, M. Mahoney, K. Keutzer, (2024)
Link , this paradigm has emerged as a promising direction for scaling language models. Recent work has investigated recurrent-depth scaling [2,3,4,5] Scaling Latent Reasoning via Looped Language Models
R. Zhu, Z. Wang, K. Hua, T. Zhang, Z. Li, H. Que, B. Wei, Z. Wen, F. Yin, H. Xing, L. Li, J. Shi, K. Ma, S. Li, T. Kergan, A. Smith, X. Qu, M. Hui, B. Wu, Q. Min, H. Huang, X. Zhou, W. Ye, J. Liu, J. Yang, Y. Shi, C. Lin, E. Zhao, T. Cai, G. Zhang, W. Huang, Y. Bengio, J. Eshraghian, (2025)
LinkScaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
LinkHow Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
K. Schwethelm, D. Rueckert, G. Kaissis, (2026)
LinkParcae: Scaling Laws for Stable Looped Language Models
H. Prairie, Z. Novack, T. Berg-Kirkpatrick, D. Fu, (2026)
Link , capability profiles [6,7,8,9] Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
H. Kohli, S. Parthasarathy, H. Sun, Y. Yao, (2026)
LinkEquilibrium Reasoners: Learning Attractors Enables Scalable Reasoning
B. Huang, Z. Geng, Z. Kolter, (2026)
LinkStability and Generalization in Looped Transformers
A. Labovich, (2026)
LinkReasoning with Latent Thoughts: On the Power of Looped Transformers
N. Saunshi, N. Dikkala, Z. Li, S. Kumar, S. Reddi, (2025)
Link , as well as the roles of stability and residual dynamics [8,10] Stability and Generalization in Looped Transformers
A. Labovich, (2026)
LinkDeepLoop: Depth Scaling for Looped Transformers
S. Li, Y. Zhang, J. Guo, Q. Gu, M. Wang, (2026)
Link , architecture topology and sparsity [11,12] MoEUT: Mixture-of-Experts Universal Transformers
R. Csordás, K. Irie, J. Schmidhuber, C. Potts, C. Manning, (2024)
DOILoop the Loopies!
Z. Gao, Y. Chen, Y. Xiao, X. Yang, R. Tao, J. Zhou, B. Dai, (2026)
Link , and memory efficiency [13] Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
V. Vendrell, A. Masdemont, N. Grillo, J. Ros-Giralt, A. Behboodi, F. Massoli, (2026)
Link . Most existing loop language models can be broadly categorized into two architectural lineages, represented by Ouro [2] Scaling Latent Reasoning via Looped Language Models
R. Zhu, Z. Wang, K. Hua, T. Zhang, Z. Li, H. Que, B. Wei, Z. Wen, F. Yin, H. Xing, L. Li, J. Shi, K. Ma, S. Li, T. Kergan, A. Smith, X. Qu, M. Hui, B. Wu, Q. Min, H. Huang, X. Zhou, W. Ye, J. Liu, J. Yang, Y. Shi, C. Lin, E. Zhao, T. Cai, G. Zhang, W. Huang, Y. Bengio, J. Eshraghian, (2025)
Link and Huginn [3] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
Link .

Despite their growing influence, the architectural differences between these two families have not been systematically isolated. Ouro recurrently applies a tied full stack initialized from token embeddings, whereas Huginn places recurrence between untied prelude and coda layers, repeatedly injects a fixed prelude representation, and uses a separately initialized recurrent state. We formalize these differences along three axes—the iteration envelope, input injection, and latent-state organization—and isolate their effects through a controlled Ouro-to-Huginn transformation. We pretrain all resulting models from scratch under matched parameter scale, logical depth, and token budgets, and evaluate them using a common protocol.

The rest of this article follows three questions.

1. First, how should loop architectures be described? We introduce a common framework that separates three choices often bundled together: where recurrence is placed, how the input is reintroduced, and how recurrent states are organized.
2. Second, which of these choices actually matter? Through controlled Ouro-to-Huginn ablations, we find that placing recurrence inside a prelude–loop–coda sandwich provides the clearest gains, especially on mathematical reasoning, context extraction, and other reasoning-intensive tasks. Input injection helps models recover prompt-provided context and specifications, but can interfere with quantitative reasoning. Random initialization offers no consistent advantage, and separate high/low states add complexity without reliable gains. When these choices are combined, the full Huginn design outperforms Ouro across all ten dense benchmarks.
3. Third, do these conclusions survive in MoE models? They do. Huginn again outperforms Ouro, distributes expert load more evenly, and uses different experts meaningfully across loop iterations. It also approaches the performance of a much larger feedforward MoE while storing substantially fewer parameters.

2. Preliminary

The architecture of Ouro and Huginn differs in their iteration envelope, input injection, and recurrent-state organization.

2.1 Notations

Let $\mathbf x\in\mathcal V^{B\times L}$ denote token indices and $\mathbf x_0=E_\theta(\mathbf x)\in\mathbb R^{B\times L\times d}$ their embeddings. A general tied iterative model can be written as

Here $P_\theta$ is the prelude map [3] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
Link , $R_\theta$ is the tied iterative body, $C_\theta$ is the coda map. All three are stacks of standard Transformer blocks; the prelude and coda are applied once, whereas the recurrent core is repeatedly applied with shared parameters. $\phi_\theta$ initializes the recurrent state, and $W_\theta$ is the per-step input write. The fixed representation $\mathbf e$ is the prelude output, $\mathbf z_t$ is the model’s recurrent hidden stream, and auxiliary randomness $\boldsymbol{\xi}$ is used only by variants with random state initialization.

2.2 Comparing Ouro and Huginn

Under this notation, an Ouro-style topology sets $P_\theta$,$W_\theta$, and $C_\theta$ to identity maps, initializes $\mathbf z_0=\mathbf x_0$, and applies the same stack for $T$ iterations.

A Huginn-style topology keeps nontrivial prelude and coda maps and writes the fixed representation $\mathbf e$ into the recurrent stream at each iteration. Note that such random initialization changes the starting value of that same stream while leaving the number of persistent states unchanged.

This decomposition yields three separable design axes:

• Iteration envelope chooses the placement and sharing pattern of $P_\theta,R_\theta,C_\theta$
• Input interface chooses whether $W_\theta$ receives a persistent input condition and what it reads, e.g. token embeddings versus prelude-encoded $\mathbf e$;
• Latent-space design chooses how $\phi_\theta$ initializes the loop state, e.g. $\mathbf z_0=\mathbf e$ versus a separate draw, as well as the structures of the states, e.g. fast and slow latents.

3. Controlled Ablations of Loop-Model Design

In this section, we specify the matched model, training, and evaluation controls, then report the stepwise ablations and the complete construction path from Ouro to Huginn. Q1 changes the iteration envelope, Q2 adds the prelude-conditioned input injection, and Q3 varies latent-state initialization and organization.

3.1 Model Configuration

We evaluate the loop model variants in a matched 730M stored / 2.9B equivalent-when-unrolled**;** Each loop model stores the parameters of 28 Transformer blocks but reuses a subset of these blocks across recurrent iterations, yielding an unrolled computation path with the same logical depth as a 112-layer feedforward Transformer.

• The Ouro-style loop ties the full 28-block stack and executes it four times, giving $(28 \times 4 = 112)$ logical block executions, abbreviated as $R_{28}^4$ .
• The Huginn-style loop instead places recurrence in the middle of a feedforward envelope: an 8-layer prelude, an 8-step loop over a 12-layer tied recurrent core, and an 8-layer coda, giving $(8 + 12 \times 8 + 8 = 112)$ logical block executions, abbreviated as $P_8 R_{12}^8 C_8$.

Two feedforward baselines anchor the comparison. The 730M baseline $D_{28}$ contains 28 Transformer blocks and matches the resident parameter count of the looped models. The 2.9B baseline $D_{112}$ contains 112 independently parameterized Transformer blocks and matches their unrolled logical depth.

Transformer block. All models use the same decoder-only PreNorm backbone, ensuring that the ablations modify the recurrence pattern rather than the local block architecture. The attention module uses 24 query heads, 6 key–value heads, a head dimension of 64, and RoPE with base $10^{6}$. At a hidden dimension of 1536, the dense feed-forward branch uses a SwiGLU MLP with intermediate width 4352. In each MoE variant, the attention, normalization, and residual structure remain unchanged; only the dense feed-forward branch is replaced by a top-2 routed mixture of experts.

3.2 Training Recipe

All models are trained on the TxT360 dataset [14] TxT360: A Top-Quality LLM Pre-training Dataset Requires the Perfect Blend
L. Tang, N. Ranjan, O. Pangarkar, X. Liang, Z. Wang, L. An, B. Rao, L. Jin, H. Wang, Z. Cheng, S. Sun, C. Mu, V. Miller, X. Ma, Y. Peng, Z. Liu, E. Xing, (2024)
Link , a high-quality dataset that deduplicates 99 Common Crawl snapshots and 14 curated data sources from diverse domains. We use sequence length 8192 and an effective global batch of 512 sequences, or 4,194,304 tokens per optimizer step. We optimize with AdamW, sweep learning rates in {4 × 10−4, 6 × 10−4, 8 × 10−4}, and report the best setting. We use 200 warmup steps followed by cosine decay and set weight decay to 0.1.

3.3 Evaluation Protocols

We evaluate the models on ten benchmarks spanning several capability groups:

• ARC-Challenge, HellaSwag, MMLU, and TriviaQA emphasize factual, conceptual, and commonsense knowledge;
• BBH-CoT and DROP require reasoning over instructions, demonstrations, or passage-level evidence;
• GSM8K and MATH500 focus on multi-step quantitative and symbolic reasoning; and
• HumanEval+ and MBPP+ test code generation under explicit functional specifications.

These categories overlap, but they provide a useful lens for understanding which capabilities benefit from each architectural change.

3.4 Controlled Architecture Ablations

Q1: Does Moving Recurrence into a Sandwich Envelope Help?

Q1 isolates loop placement at fixed logical compute. The full-stack Ouro control reuses all 28 blocks for four passes, $R_{28}^{4}$, whereas the sandwich control places a tied 12-block recurrent core between eight untied prelude blocks and eight untied coda blocks, $P_8R_{12}^{8}C_8$. We compare the two architectures at 58, 115, 230, and 460 tokens per parameter (TPP), corresponding to 42B, 84B, 168B, and 336B training tokens for the 730M-parameter models.

MATH500, BBH-CoT, and DROP favor the envelope across the training budgets, whereas the gain on GSM8K is also visible. In this architecture, recurrence is localized to the middle of the network: only $R_{12}$ is reapplied, and the coda decodes its hidden state only after the final iteration. This separation may allow the recurrent core to devote more of its capacity to iterative state refinement, while the untied prelude and coda specialize in input encoding and output decoding. This interpretation is consistent with prior theoretical and empirical evidence that repeated middle-layer computation can support multi-step reasoning [3,13] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
LinkMemory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
V. Vendrell, A. Masdemont, N. Grillo, J. Ros-Giralt, A. Behboodi, F. Massoli, (2026)
Link .

Knowledge-intensive benchmarks depend substantially on information stored in the model parameters rather than evidence contained in the evaluated instance. Code-generation tasks also require derived computation, but success additionally depends on satisfying a precise executable specification and producing syntactically and semantically valid outputs. The observed pattern therefore bounds the benefit of the sandwich envelope: the results are consistent with improved reasoning over instance-provided information, but do not indicate better retrieval of learned knowledge or more reliable compliance with strict output contracts. Although the envelope benefits several reasoning benchmarks, that advantage alone is insufficient to improve the full mixture of capabilities required for code generation.

Q2: Does Input Injection Help?

Persistent input injection gives every recurrent step a direct path back to the input, reducing the need for the evolving state to carry all prompt-relevant information across iterations and potentially improving gradient propagation. Prior work on recurrent-depth and fixed-point models uses related mechanisms to keep iterative states explicitly conditioned on the input [3,5,8] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
LinkParcae: Scaling Laws for Stable Looped Language Models
H. Prairie, Z. Novack, T. Berg-Kirkpatrick, D. Fu, (2026)
LinkStability and Generalization in Looped Transformers
A. Labovich, (2026)
Link . We evaluate matched write-versus-no-write controls within two recurrent families, thereby isolating the effect of injection within each architecture.

Let $\mathbf x_0=E_\theta(\mathbf x)$ be the raw token-embedding stream and $\mathbf e=P_\theta(\mathbf x_0)$ the contextualized representation produced by untied prelude blocks. For a generic write source $\mathbf v$, both controls use the same diagonal operator [5] Parcae: Scaling Laws for Stable Looped Language Models
H. Prairie, Z. Novack, T. Berg-Kirkpatrick, D. Fu, (2026)
Link

The middle loop writes the fixed contextualized state $\mathbf v=\mathbf e$ before each recurrent-core pass. Full-stack Ouro has no untied prelude, so its corresponding intervention writes the raw token embeddings $\mathbf v=\mathbf x_0$ before each full-stack pass. Each write-versus-no-write pair isolates injection within one family.

Repeated writes make every update directly dependent on $\mathbf v$, so the write source remains available after every preceding recurrent transformation. This direct path should be particularly useful when later computation must recover task instructions, contextual evidence, demonstrations, or specifications from the prompt.

Within the middle-loop architecture, prelude-state injection improves MMLU by 2.53 points, BBH-CoT by 6.63 points, DROP by 1.39 points, HumanEval+ by 5.49 points, and MBPP+ by 4.23 points. Raw-token injection produces gains in the same broad capability groups for full-stack Ouro, including improvements of 1.79 points on MMLU, 1.80 points on BBH-CoT, 0.91 points on DROP, 2.44 points on HumanEval+, and 4.50 points on MBPP+.

These tasks depend heavily on information provided in the evaluated instance. HumanEval+ and MBPP+ require generated programs to satisfy prompt-supplied specifications; BBH-CoT provides worked examples; and DROP provides an answer-bearing passage. The consistent gains across both recurrent families therefore support the interpretation that persistent injection helps later recurrent computation retain access to task-relevant contextual information.

The effect of injection is not uniformly positive. In the middle-loop model, prelude-state injection reduces MATH500 by 3.60 points and GSM8K by 2.51 points. The corresponding effects are less negative in full-stack Ouro: raw-token injection reduces MATH500 by 1.60 points and slightly improves GSM8K by 0.53 points.

This pattern suggests that repeatedly reintroducing a fixed contextualized representation may interfere with the iterative state transformations required for quantitative reasoning. One possible interpretation is that mathematical tasks benefit from allowing the recurrent state to progressively depart from the input representation as intermediate computations accumulate. Persistent prelude-state injection may instead continually pull the state back toward its input-conditioned representation. The present experiments establish the behavioral trade-off but do not isolate its mechanism, which may depend on the write strength, injection location, or interaction between the fixed prelude state and the evolving recurrent state.

With their respective input-conditioning mechanisms enabled, the prelude–loop–coda model outperforms full-stack Ouro on eight of ten benchmarks, trailing only on ARC-Challenge and HellaSwag. The middle-loop model also remains stronger on MATH500 and GSM8K, even though prelude-state injection lowers its performance relative to the no-write control.

These results highlight the overall strength of the middle-loop design. The sandwich envelope and contextualized input access appear complementary on context-dependent and code-generation tasks, while the envelope advantage compensates for the negative effect of injection on quantitative reasoning. Overall, the prelude–loop–coda architecture provides the stronger foundation, although its input-injection mechanism could be further refined to better preserve mathematical reasoning performance.

Q3: Do Separate Recurrent Latent States Help?

After fixing the middle-loop topology and prelude-conditioned write, we vary only the organization of the recurrent latent state.

Random initialization completes the controlled transformation from the Ouro-style design to the standard Huginn-style configuration. The shared H/L-state variant then tests an additional hierarchical organization inspired by HRM-Text [15] HRM-Text: Efficient Pretraining Beyond Scaling
G. Wang, C. Liu, C. Wang, C. Zhou, Y. Sun, Y. Wu, S. Zhen, L. Scimeca, Y. Yadkori, (2026)
Link and TRM [16] Less Is More: Recursive Reasoning with Tiny Networks
A. Jolicoeur-Martineau, (2025)
Link . Unlike HRM-Text, which uses separate modules for the H and L updates, our matched control shares a single recurrent body across both states.

We use $H=2$ high-level cycles and $L=3$ low-level updates per cycle. The resulting schedule applies the shared recurrent body eight times—six low-level updates and two high-level updates—matching the recurrent compute of the single-state control. Let $\mathbf e=P_\theta(\mathbf x_0)$ be the contextualized representation produced by the prelude; we compare three variants:

• Input-initialized single state: $\mathbf z_0=\mathbf e$.
• Random-initialized single state: $\mathbf z_0=\boldsymbol{\xi}$, where each token state follows $\boldsymbol{\xi}\sim \mathcal{N}(\mathbf 0,\mathbf I_d/d)$.
• Shared high/low states: a high-level state is initialized from $\mathbf e$, a low-level workspace is initialized from noise, and both states are updated by the same recurrent body according to an H/L schedule.

Let

denote one application of the shared recurrent body preceded by the prelude-conditioned input write.

Thus, each low-level update receives the current high-level state through additive conditioning before entering the shared recurrent body, while each high-level update analogously receives the final low-level state from the corresponding cycle.

Random initial states are widely used in iterative models [3,7] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein, (2025)
LinkEquilibrium Reasoners: Learning Attractors Enables Scalable Reasoning
B. Huang, Z. Geng, Z. Kolter, (2026)
Link , partly because they are thought to encourage path independence on algorithmic equilibrium tasks [17] Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
C. Anil, A. Pokle, K. Liang, J. Treutlein, Y. Wu, S. Bai, J. Kolter, R. Grosse, (2022)
Link . We argue, however, that random initialization should instead be viewed as a task- and objective-dependent inductive bias, rather than as a universally beneficial design choice.

Relative to direct initialization with $\mathbf z_0=\mathbf e$, random initialization improves ARC-Challenge by 3.34 points and GSM8K by 1.22 points. However, it reduces performance by more than one point on MMLU, MATH500, HumanEval$^{+}$ and MBPP$^{+}$, while the differences on the remaining four benchmarks are within one point.

To our knowledge, prior work has not isolated the effect of random initialization in a matched, large-scale, finitely unrolled language model with persistent input injection. Across our evaluations, random initialization produces improvements exceeding one point on two benchmarks but degradations exceeding one point on four, providing no evidence of a consistent cross-task advantage in this setting. Direct initialization performs better on six of the ten benchmarks and also avoids the additional computation associated with sampling a separate initial state. These results should not, however, be interpreted as evidence against random initialization in general: our evaluations do not directly measure multi-start path independence or extrapolation to recurrent depths beyond those used during training.

HRM [18] Hierarchical Reasoning Model
G. Wang, J. Li, Y. Sun, X. Chen, C. Liu, Y. Wu, M. Lu, S. Song, Y. Yadkori, (2025)
Link introduces an H/L hierarchy through a brain-inspired separation of timescales. A fast L module repeatedly refines a local state under a fixed H-level context, after which a slower H module updates that context and initiates a new phase of low-level refinement. This alternating schedule is intended to sustain effective computational depth. HRM-Text adapts the same two-cycle, three-L-update schedule to language modeling and combines it with MagicNorm, warmup credit assignment, and PrefixLM training.

Existing evidence for the value of the H/L hierarchy is mixed. An L-only model matches HRM on Sudoku [19] Hierarchical Reasoning Models: Perspectives and Misconceptions
R. Ge, Q. Liao, T. Poggio, (2025)
Link , whereas TRM reports improvements from combining two recurrent states with shared weights on Sudoku-Extreme [16] Less Is More: Recursive Reasoning with Tiny Networks
A. Jolicoeur-Martineau, (2025)
Link . At language-model scale, HRM-Text compares a separate-module HRM architecture with a shared-weight TRM-style alternative [15] HRM-Text: Efficient Pretraining Beyond Scaling
G. Wang, C. Liu, C. Wang, C. Zhou, Y. Sun, Y. Wu, S. Zhen, L. Scimeca, Y. Yadkori, (2026)
Link . However, these comparisons also vary the update rule, effective depth, parameter count, or computational budget, making it difficult to isolate the contribution of the state hierarchy itself.

Our control holds the recurrent body, number of recurrent-body applications, input injection, parameter count, and training budget fixed. It introduces only a second recurrent state and the H/L update schedule, with the same recurrent module shared across both states. Under this matched setting, the hierarchy yields gains exceeding one point on three benchmarks and losses exceeding one point on three, while substantially degrading MATH500. Thus, the additional state and scheduling complexity provides no consistent aggregate benefit when the H and L updates share a recurrent module.

This conclusion is specific to the shared-module setting. It does not rule out benefits from the separately parameterized H and L modules used by HRM-Text. A closer HRM-Text-style variant with distinct modules for the two update streams is still under evaluation and is not included in the present comparison.

Summary

The figure below shows that the sandwich envelope and persistent input injection drive the gains, while a separate random-initialized latent state provides no consistent further benefit.

The stepwise results distinguish the contributions of the three design changes. The sandwich envelope produces the clearest gains on mathematical and multi-step reasoning tasks. Persistent input injection produces its largest additional gains on BBH-CoT and code generation, while also improving DROP and MMLU.

Introducing a separately random-initialized recurrent state has mixed effects and reverses some of the preceding gains on several benchmarks. Under this construction order, the results suggest that the broad advantage of Huginn over Ouro is driven mainly by localized recurrence and repeated access to a contextualized input representation.

4. Transfer to MoE Models

4.1 Configurations for MoE Comparison

We replace the FFN in every physical layer of the two complete loop architectures with a mixture-of-experts (MoE) layer. The Ouro-style model retains full-stack recurrence and an input-initialized recurrent state. The Huginn-style model retains the sandwich envelope, prelude-conditioned input writes, and random state initialization.

Both recurrent models contain 28 routed physical layers with 25 experts, top-2 routing, and expert width 2432. Each has 8.0B resident parameters and 793.9M active parameters per physical pass. Unrolling the recurrent computation yields approximately 3.2B active parameter applications per token. A 112-layer feedforward model without cross-layer weight sharing, denoted $D_{112}^{\mathrm{MoE}}$, provides a reference matched in logical depth and active compute, with 32.0B resident parameters. All MoE models use dot-product load-balancing loss without router bias.

4.2 Huginn-Style MoE versus Ouro

The table above shows that the endpoint difference is not uniform across tasks. Within the general benchmarks, the clearest gains for Huginn are on MMLU (+2.78) and DROP (+1.95). Its leads on HellaSwag and TriviaQA are only 0.28 and 0.07 points, respectively, while Ouro is higher on ARC-Challenge by 0.86 points and on BBH-CoT by 0.20.

The separation is larger on mathematical reasoning. Huginn improves over Ouro by 3.60 points on MATH500 and 4.70 on GSM8K. The code differences are smaller: +1.22 on HumanEval+ and +0.53 on MBPP+. Overall, the transfer advantage is concentrated on mathematical reasoning, with additional gains on MMLU and DROP, while performance on the remaining benchmarks is broadly comparable.

4.3 Routing Diagnostics

We compare normalized load-balancing loss across the two recurrent models and the feedforward reference. Uniform routing has a loss of one; lower values therefore indicate more even expert utilization.

We find that the Huginn MoE exhibits a substantially lower load-balancing loss, indicating more even expert utilization. The figure below further shows that the distribution of top-2 expert activations varies across loop iterations. This variation is functionally important: forcing iterations 2–8 to reuse the expert identities selected at iteration 1, while retaining iteration-specific mixture weights, reduces accuracy on all six evaluated tasks. This intervention provides causal evidence that iteration-specific expert selection contributes to model performance.

4.4 Huginn-MoE versus Feedforward MoE

We compare Huginn with the 112-layer feedforward MoE under the same logical depth and active-compute budget. We further compare the loop-to-feedforward performance gaps in the dense and MoE settings.

The figure below compares the dense and MoE performance gaps between Huginn and their respective 112-layer feedforward references. In panel (a), the MoE gap is smaller on eight of the nine directly comparable benchmarks, reducing the mean gap from $4.96$ to $1.71$ points. Panel~(b) shows that this pattern holds throughout training: when evaluated on the same seven benchmarks, the MoE gap is smaller at all four checkpoints.

4.5 Summary

• First, the Huginn-style MoE outperforms Ouro on eight of ten benchmarks, with its largest gains on GSM8K and MATH500 and additional gains exceeding one point on MMLU, DROP, and HumanEval+. This shows that the advantage observed in the dense setting transfers to MoE models.
• Second, Huginn achieves more balanced expert utilization, and intervention experiments show that iteration-specific expert selection contributes meaningfully to performance.
• Third, MoE substantially narrows the performance gap between Huginn and its feedforward counterpart. Although the feedforward baseline leads overall, Huginn outperforms it on DROP and GSM8K, matches it on MATH500, and reduces the mean gap from 4.96 points in the dense setting to 1.71 points in the MoE setting.

Conclusions

We compare full-stack and middle-loop language models by disentangling three design choices that standard Ouro- and Huginn-style recipes vary simultaneously: the iteration envelope, input injection, and latent-state organization. Under matched parameter scale, logical depth, and training-token budget, we find that the prelude–loop–coda sandwich architecture improves performance on mathematical reasoning, context extraction, and other reasoning-intensive tasks. Input injection primarily benefits knowledge retrieval and context access, while contributing little to quantitative reasoning. Random state initialization yields mixed results, whereas the shared H/L hierarchy provides no consistent benefit.

Our MoE transfer study evaluates the two complete architectures in a modern sparse-model setting and shows that the advantages of the Huginn-style architecture persist. The Huginn MoE achieves more balanced expert utilization and approaches—and on some tasks surpasses—the performance of its feedforward MoE counterpart, despite being trained from scratch under the same data budget and matched memory and FLOP constraints.

More broadly, the architectural design space of looped models remains underexplored and calls for substantially more controlled ablation studies. Meaningful comparisons therefore require carefully designed, apples-to-apples evaluations that match parameter scale, training budget, inference cost, and evaluation protocol. Many important questions remain open, including how loop architectures interact with conditional sparsity and how architecture–system co-design can further advance their performance–efficiency Pareto frontier.