Papers

Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, Yiran Zhong (2024). HGRN2: Gated Linear RNNs with State Expansion. In COLM 2024.
PDF Cite Code
Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, Yiran Zhong (2024). Linear Attention Sequence Parallelism. In arXiv preprint arXiv:2404.02882.
PDF Cite Code
Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, Yiran Zhong (2024). CO2: Efficient Distributed Training with Full Communication-Computation Overlap. In ICLR 2024 (Spotlight).
PDF Cite Code
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, Yiran Zhong (2024). Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models. In arXiv preprint arXiv:2401.04658.
PDF Cite Code