litlas
Log inCreate account

HGRN2: Gated Linear RNNs with State Expansion

Authors: Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, Yiran Zhong - arXiv 2024 cited by 75

Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, limiting its expressiveness. To address this issue, we introduce a simple outer product-based state expansion mechanism, which significantly enlarges the recurrent state size without introducing any additional parameters. This enhancement also provides a linear attention interpretation for HGRN2, enabling hardware-efficient training. Our extensive experiments verify the advantage of HGRN2 over HGRN consistently across different settings and competitive with other recurrent models.

Empirical Evaluation ofGated Recurrent Neural…Empirical Evaluation of Gated Recurrent Neural Networks on Sequence ModelingThe Devil in LinearTransformerThe Devil in Linear TransformerFine-Tuning Pre-trainedTransformers into…Fine-Tuning Pre-trained Transformers into Decaying Fast WeightsScaling TransNormer to175 Billion ParametersScaling TransNormer to 175 Billion ParametersRetentive Network: ASuccessor to Transforme…Retentive Network: A Successor to Transformer for Large Language ModelsLLaMA: Open andEfficient Foundation…LLaMA: Open and Efficient Foundation Language ModelsRWKV: Reinventing RNNsfor the Transformer EraRWKV: Reinventing RNNs for the Transformer EraParallelizing LinearTransformers with the…Parallelizing Linear Transformers with the Delta Rule over Sequence LengthGated Linear AttentionTransformers with…Gated Linear Attention Transformers with Hardware-Efficient TrainingSimple linear attentionlanguage models balance…Simple linear attention language models balance the recall-throughput tradeoffZoology: Measuring andImproving Recall in…Zoology: Measuring and Improving Recall in Efficient Language ModelsScaling Laws for LinearComplexity Language…Scaling Laws for Linear Complexity Language ModelsGated Linear AttentionTransformers with…Gated Linear Attention Transformers with Hardware-Efficient TrainingLinear AttentionSequence ParallelismLinear Attention Sequence ParallelismTransformers are SSMs:Generalized Models and…Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityMoM: Linear SequenceModeling with…MoM: Linear Sequence Modeling with Mixture-of-MemoriesLonghorn: State SpaceModels are Amortized…Longhorn: State Space Models are Amortized Online LearnersComba: ImprovingBilinear RNNs with…Comba: Improving Bilinear RNNs with Closed-loop ControlLiger: Linearizing LargeLanguage Models to Gate…Liger: Linearizing Large Language Models to Gated Recurrent StructuresGated Delta Networks:Improving Mamba2 with…Gated Delta Networks: Improving Mamba2 with Delta RuleLattice: Learning toEfficiently Compress th…Lattice: Learning to Efficiently Compress the MemoryLinear-MoE: LinearSequence Modeling Meets…Linear-MoE: Linear Sequence Modeling Meets Mixture-of-ExpertsKimi Linear: AnExpressive, Efficient…Kimi Linear: An Expressive, Efficient Attention ArchitectureLASP-2: RethinkingSequence Parallelism fo…LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its HybridHGRN2: Gated Linear RNNswith State ExpansionHGRN2: Gated Linear RNNs with State ExpansionEarlier referencesFocus paperCiting papersOlderNewer

Click a node to pin it, click the empty canvas to go back to this paper, or hover to preview. Open a node’s page from its title.