Towards Looped Models Done Right

Fri, 31 Jul 2026 00:00:00 -0700

Benhao Huang‡†, Chufan Shi‡*, Junlin Chen‡, Shicheng Wen‡*,

Zhengzhong Liu‡, Eric Xing‡, Xuezhe Ma‡*

‡Institute of Foundation Models, *USC, †CMU

Fig. 1 Parameter scaling and benchmark performance of loop and feedforward MoE models. Left: Ouro MoE and Huginn MoE share the same scale: 8.0B resident and 0.8B active parameters, corresponding to 32.0B resident-equivalent and 3.2B unrolled-active parameter applications. Unrolled counts measure parameter applications under weight reuse. Right: Huginn MoE outperforms Ouro MoE overall, and approaches or surpasses the 112-layer feedforward MoE baseline on DROP, MATH500 and GSM8K, while remaining competitive on the other benchmarks. All models use the same data and matched training and inference FLOPs, while looped models require less memory.

Mixture of Experts on Husky'Log

Towards Looped Models Done Right