v1v2v3 (latest)

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

11 June 2024

Yadong Lu

Weizhu Chen

Papers citing "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"

50 / 132 papers shown

Title
HGRN2: Gated Linear RNNs with State Expansion Zhen Qin Aaron Courville Weixuan Sun Xuyang Shen Dong Li Weigao Sun Yiran Zhong LRM 85 54 0 11 Apr 2024
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models Aleksandar Botev Soham De Samuel L. Smith Anushan Fernando George-Christian Muraru ... Koray Kavukcuoglu Demis Hassabis R. Hadsell Yee Whye Teh Nando de Frietas VLM RALM 73 29 0 11 Apr 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 128 119 0 10 Apr 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 123 2 0 03 Apr 2024
Jamba: A Hybrid Transformer-Mamba Language Model Opher Lieber Barak Lenz Hofit Bata Gal Cohen Jhonathan Osin ... Nir Ratner N. Rozen Erez Shwartz Mor Zusman Y. Shoham 90 226 0 28 Mar 2024
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models Soham De Samuel L. Smith Anushan Fernando Aleksandar Botev George-Christian Muraru ... David Budden Yee Whye Teh Razvan Pascanu Nando de Freitas Çağlar Gülçehre Mamba 103 134 0 29 Feb 2024
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval Kaiyue Wen Xingyu Dang Kaifeng Lyu 99 32 0 28 Feb 2024
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Yiran Ding Li Lyna Zhang Chengruidong Zhang Yuanyuan Xu Ning Shang Jiahang Xu Fan Yang Mao Yang RALM 94 162 0 21 Feb 2024
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks Jongho Park Jaeseung Park Zheyang Xiong Nayoung Lee Jaewoong Cho Samet Oymak Kangwook Lee Dimitris Papailiopoulos 88 75 0 06 Feb 2024
Repeat After Me: Transformers are Better than State Space Models at Copying Samy Jelassi David Brandfonbrener Sham Kakade Eran Malach 159 94 0 01 Feb 2024
In-Context Language Learning: Architectures and Algorithms Ekin Akyürek Bailin Wang Yoon Kim Jacob Andreas LRM ReLM 108 56 0 23 Jan 2024
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Zirui Liu Chia-Yuan Chang Huiyuan Chen Helen Zhou 108 117 0 02 Jan 2024
Gated Linear Attention Transformers with Hardware-Efficient Training Aaron Courville Bailin Wang Songlin Yang Yikang Shen Yoon Kim 108 180 0 11 Dec 2023
Zoology: Measuring and Improving Recall in Efficient Language Models Simran Arora Sabri Eyuboglu Aman Timalsina Isys Johnson Michael Poli James Zou Atri Rudra Christopher Ré 112 79 0 08 Dec 2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu Tri Dao Mamba 148 2,767 0 01 Dec 2023
GPQA: A Graduate-Level Google-Proof Q&A Benchmark David Rein Betty Li Hou Asa Cooper Stickland Jackson Petty Richard Yuanzhe Pang Julien Dirani Julian Michael Samuel R. Bowman AI4MH ELM 115 728 0 20 Nov 2023
Hierarchically Gated Recurrent Neural Network for Sequence Modeling Zhen Qin Aaron Courville Yiran Zhong 68 80 0 08 Nov 2023
GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling Tobias Katsch AI4TS 72 31 0 03 Nov 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 101 2,237 0 10 Oct 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 130 784 0 29 Sep 2023
Effective Long-Context Scaling of Foundation Models Wenhan Xiong Jingyu Liu Igor Molybog Hejia Zhang Prajjwal Bhargava ... Dániel Baráth Sergey Edunov Mike Lewis Sinong Wang Hao Ma 84 229 0 27 Sep 2023
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen Shengju Qian Haotian Tang Xin Lai Zhijian Liu Song Han Jiaya Jia 119 167 0 21 Sep 2023
Textbooks Are All You Need II: phi-1.5 technical report Yuan-Fang Li Sébastien Bubeck Ronen Eldan Allison Del Giorno Suriya Gunasekar Yin Tat Lee ALM LRM 161 481 0 11 Sep 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 396 12,044 0 18 Jul 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 117 1,326 0 17 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 147 342 0 17 Jul 2023
Focused Transformer: Contrastive Training for Context Scaling Szymon Tworkowski Konrad Staniszewski Mikolaj Pacek Yuhuai Wu Henryk Michalewski Piotr Milo's 65 140 0 06 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 160 540 0 27 Jun 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 93 14 0 19 Jun 2023
Block-State Transformers Mahan Fathi Jonathan Pilault Orhan Firat C. Pal Pierre-Luc Bacon Ross Goroshin 72 18 0 15 Jun 2023
Landmark Attention: Random-Access Infinite Context Length for Transformers Amirkeivan Mohtashami Martin Jaggi LLMAG 130 164 0 25 May 2023
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Uri Shaham Maor Ivgi Avia Efrat Jonathan Berant Omer Levy VLM 92 140 0 23 May 2023
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai 92 694 0 22 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,748 0 15 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 326 297 0 11 Mar 2023
Hyena Hierarchy: Towards Larger Convolutional Language Models Michael Poli Stefano Massaroli Eric Q. Nguyen Daniel Y. Fu Tri Dao S. Baccus Yoshua Bengio Stefano Ermon Christopher Ré VLM 121 307 0 21 Feb 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 125 403 0 28 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 173 37 0 15 Dec 2022
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 101 185 0 21 Sep 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 107 242 0 27 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 89 324 0 23 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 254 2,279 0 27 May 2022
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way Alex Jinpeng Wang Richard Yuanzhe Pang Angelica Chen Jason Phang Samuel R. Bowman 123 46 0 23 May 2022
Memorizing Transformers Yuhuai Wu M. Rabe DeLesley S. Hutchins Christian Szegedy RALM 100 178 0 16 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 845 9,683 0 28 Jan 2022
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 217 1,821 0 31 Oct 2021
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 345 4,569 0 27 Oct 2021
Sequence Length is a Domain: Length-based Overfitting in Transformer Models Dusan Varis Ondrej Bojar 68 56 0 15 Sep 2021
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 149 1,938 0 08 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 336 775 0 27 Aug 2021