Weigao Sun
  • About Me
  • Papers
  • News
  • Recent & Upcoming Talks
    • Example Talk
  • Papers
    • A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
    • Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
    • Liger: Linearizing Large Language Models to Gated Recurrent Structures
    • MoM: Linear Sequence Modeling with Mixture-of-Memories
    • LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid
    • Minimax-01: Scaling Foundation Models with Lightning Attention
    • LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
    • Scaling Laws for Linear Complexity Language Models
    • Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
    • Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
    • HGRN2: Gated Linear RNNs with State Expansion
    • Linear Attention Sequence Parallelism
    • CO2: Efficient Distributed Training with Full Communication-Computation Overlap
    • Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  • Projects
  • Projects
    • Pandas
    • PyTorch
    • scikit-learn
  • Experience
  • Teaching
    • Learn JavaScript
    • Learn Python
  • Blog

MoM: Linear Sequence Modeling with Mixture-of-Memories

Feb 19, 2025·
Jusen Du
Weigao Sun
Weigao Sun
,
Disen Lan
,
Jiaxi Hu
,
Yu Cheng
· 0 min read
PDF Cite Code
Last updated on Feb 19, 2025
Weigao Sun
Authors
Weigao Sun
Young Scientist
← Liger: Linearizing Large Language Models to Gated Recurrent Structures Mar 3, 2025
LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid Feb 11, 2025 →

© 2025 Me. This work is licensed under CC BY NC ND 4.0

Published with Hugo Blox Builder — the free, open source website builder that empowers creators.