Hungry Hungry Hippos: Towards Language Modeling with State Space Models

28 December 2022

Papers citing "Hungry Hungry Hippos: Towards Language Modeling with State Space Models"

50 / 71 papers shown

Title
Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks Mohammad Mahdi Moradi Walid Ahmed Shuangyue Wen Sudhir Mudur Weiwei Zhang Yang Liu 55 0 0 26 May 2025
Selection Mechanisms for Sequence Modeling using Linear State Space Models Umberto Casti Sandro Zampieri Fabio Pasqualetti Mamba 81 0 0 23 May 2025
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook Muyi Bao Shuchang Lyu Zhaoyang Xu Huiyu Zhou Jinchang Ren Shiming Xiang Xuelong Li Guangliang Cheng Mamba 247 0 0 01 May 2025
Future-Aware Interaction Network For Motion Forecasting Shijie Li Xun Xu S. Yeo Xulei Yang Mamba 419 0 0 09 Mar 2025
Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability Carlos E. Luis A. Bottero Julia Vinogradska Felix Berkenkamp Jan Peters 212 1 0 20 Feb 2025
Associative Recurrent Memory Transformer Ivan Rodkin Yuri Kuratov Aydar Bulatov Andrey Kravchenko 108 3 0 17 Feb 2025
State-space models are accurate and efficient neural operators for dynamical systems Zheyuan Hu Nazanin Ahmadi Daryakenari Qianli Shen Kenji Kawaguchi George Karniadakis Mamba AI4CE 180 17 0 28 Jan 2025
MambaTron: Efficient Cross-Modal Point Cloud Enhancement using Aggregate Selective State Space Modeling Sai Tarun Inaganti Gennady Petrenko Mamba 111 1 0 25 Jan 2025
MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification Yapeng Li Yong Luo Lefei Zhang Zengmao Wang Bo Du Mamba 76 65 0 10 Jan 2025
VMamba: Visual State Space Model Yue Liu Yunjie Tian Yuzhong Zhao Hongtian Yu Lingxi Xie Yaowei Wang Qixiang Ye Jianbin Jiao Yunfan Liu Mamba 260 686 0 31 Dec 2024
PTQ4VM: Post-Training Quantization for Visual Mamba Younghyun Cho Changhun Lee Seonggon Kim Eunhyeok Park MQ Mamba 99 2 0 29 Dec 2024
GG-SSMs: Graph-Generating State Space Models Nikola Zubić Davide Scaramuzza Mamba 157 1 0 17 Dec 2024
EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality Sanghyeok Lee Joonmyung Choi Hyunwoo J. Kim 166 3 0 22 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 431 0 0 21 Nov 2024
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Riccardo Grazzi Julien N. Siems Jörg Franke Arber Zela Frank Hutter Massimiliano Pontil 156 23 0 19 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 83 2 0 30 Oct 2024
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 69 4 0 24 Oct 2024
Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion Chaodong Xiao Minghan Li Zhengqiang Zhang Deyu Meng Lei Zhang Mamba 119 5 0 19 Oct 2024
Lambda-Skip Connections: the architectural component that prevents Rank Collapse Federico Arangath Joseph Jerome Sieber Melanie Zeilinger Carmen Amo Alonso 171 0 0 14 Oct 2024
Parameter-Efficient Fine-Tuning of State Space Models Kevin Galim Wonjun Kang Yuchen Zeng H. Koo Kangwook Lee 94 4 0 11 Oct 2024
SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba Xiangning Zhang Qingwei Zhang Jinnan Chen Chengfeng Zhou Yaqi Wang XiaoBo Li Xiaobo Li Dahong Qian Mamba 61 0 0 18 Sep 2024
Leveraging Joint Spectral and Spatial Learning with MAMBA for Multichannel Speech Enhancement Wenze Ren Haibin Wu Yi-Cheng Lin Xuanjun Chen Rong-Yu Chao Kuo-Hsuan Hung You-Jin Li Wen-Yuan Ting Hsin-Min Wang Yu Tsao Mamba 70 0 0 16 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 109 1 0 16 Sep 2024
Audio xLSTMs: Learning Self-Supervised Audio Representations with xLSTMs Sarthak Yadav Sergios Theodoridis Zheng-Hua Tan 79 3 0 29 Aug 2024
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 100 29 0 19 Aug 2024
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun Xinhao Li Karan Dalal Jiarui Xu Arjun Vikram ... Xinlei Chen Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin 114 103 0 05 Jul 2024
MambaLRP: Explaining Selective State Space Sequence Models F. Jafari G. Montavon Klaus-Robert Müller Oliver Eberle Mamba 237 11 0 11 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 136 66 0 11 Jun 2024
Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints Dániel Rácz Mihaly Petreczky Bálint Daróczy 90 1 0 30 May 2024
MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection Haoyang He Yuhu Bai Jiangning Zhang Qingdong He Hongxu Chen Zhenye Gan Chengjie Wang Xiangtai Li Guanzhong Tian Lei Xie Mamba 116 41 0 09 Apr 2024
Understanding Emergent Abilities of Language Models from the Loss Perspective Zhengxiao Du Aohan Zeng Yuxiao Dong Jie Tang UQCV LRM 110 53 0 23 Mar 2024
MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models Zunnan Xu Yukang Lin Haonan Han Sicheng Yang Ronghui Li Yachao Zhang Xiu Li Mamba 95 25 0 14 Mar 2024
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 314 514 0 24 Sep 2022
On The Computational Complexity of Self-Attention Feyza Duman Keles Pruthuvi Maheshakya Wijewardena Chinmay Hegde 95 124 0 11 Sep 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 87 241 0 27 Jun 2022
How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections Albert Gu Isys Johnson Aman Timalsina Atri Rudra Christopher Ré Mamba 148 97 0 24 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 79 315 0 23 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 221 2,214 0 27 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 324 3,655 0 02 May 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 471 6,231 0 05 Apr 2022
Monarch: Expressive Structured Matrices for Efficient and Accurate Training Tri Dao Beidi Chen N. Sohoni Arjun D Desai Michael Poli Jessica Grogan Alexander Liu Aniruddh Rao Atri Rudra Christopher Ré 90 93 0 01 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 194 1,944 0 29 Mar 2022
Diagonal State Spaces are as Effective as Structured State Spaces Ankit Gupta Albert Gu Jonathan Berant 111 305 0 27 Mar 2022
It's Raw! Audio Generation with State-Space Models Karan Goel Albert Gu Chris Donahue Christopher Ré 55 191 0 20 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 67 65 0 15 Feb 2022
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 94 148 0 10 Dec 2021
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 205 1,773 0 31 Oct 2021
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers Albert Gu Isys Johnson Karan Goel Khaled Kamal Saab Tri Dao Atri Rudra Christopher Ré 112 595 0 26 Oct 2021
Inductive Biases and Variable Creation in Self-Attention Mechanisms Benjamin L. Edelman Surbhi Goel Sham Kakade Cyril Zhang 70 123 0 19 Oct 2021
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding Shengjie Luo Shanda Li Tianle Cai Di He Dinglan Peng Shuxin Zheng Guolin Ke Liwei Wang Tie-Yan Liu 60 50 0 23 Jun 2021