Diagonal State Spaces are as Effective as Structured State Spaces

27 March 2022

Papers citing "Diagonal State Spaces are as Effective as Structured State Spaces"

50 / 229 papers shown

Title
Activating Wider Areas in Image Super-Resolution Cheng Cheng Hang Wang Hongbin Sun 37 10 0 13 Mar 2024
Motion Mamba: Efficient and Long Sequence Motion Generation with Hierarchical and Bidirectional Selective SSM Zeyu Zhang Akide Liu Ian Reid Richard Hartley Bohan Zhuang Hao Tang Mamba 42 62 0 12 Mar 2024
Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy Jiuming Liu Ruiji Yu Yian Wang Yu Zheng Tianchen Deng Weicai Ye Hesheng Wang 39 43 0 11 Mar 2024
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling Yair Schiff Chia-Hsiang Kao Aaron Gokaslan Tri Dao Albert Gu Volodymyr Kuleshov Mamba 27 81 0 05 Mar 2024
MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection Tianxiang Chen Zi Ye Zhentao Tan Tao Gong Yue-bo Wu Qi Chu Bin Liu Nenghai Yu Jieping Ye Mamba 59 46 0 04 Mar 2024
The Hidden Attention of Mamba Models Ameen Ali Itamar Zimerman Lior Wolf Mamba 39 58 0 03 Mar 2024
Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation Haruka Ezoe Kazuhiro Sato 28 0 0 25 Feb 2024
Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning Chi-Sheng Chen Guan-Ying Chen Dong Zhou Di Jiang Daishi Chen Mamba 49 23 0 24 Feb 2024
State Space Models for Event Cameras Nikola Zubić Mathias Gehrig Davide Scaramuzza 57 37 0 23 Feb 2024
Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers Markus Hiller Krista A. Ehinger Tom Drummond 46 1 0 19 Feb 2024
PointMamba: A Simple State Space Model for Point Cloud Analysis Dingkang Liang Xin Zhou Wei Xu Xingkui Zhu Zhikang Zou Xiaoqing Ye Xinyu Wang Xiang Bai 89 91 0 16 Feb 2024
On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era Matteo Tiezzi Michele Casoni Alessandro Betti Tommaso Guidi Marco Gori S. Melacci 19 9 0 12 Feb 2024
Scalable Diffusion Models with State Space Backbone Zhengcong Fei Mingyuan Fan Changqian Yu Junshi Huang 67 34 0 08 Feb 2024
Is Mamba Capable of In-Context Learning? Riccardo Grazzi Julien N. Siems Simon Schrodi Thomas Brox Frank Hutter 32 40 0 05 Feb 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 37 28 0 05 Feb 2024
MambaByte: Token-free Selective State Space Model Junxiong Wang Tushaar Gangavarapu Jing Nathan Yan Alexander M. Rush Mamba 44 35 0 24 Jan 2024
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model Lianghui Zhu Bencheng Liao Qian Zhang Xinlong Wang Wenyu Liu Xinggang Wang Mamba 50 710 0 17 Jan 2024
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts Maciej Pióro Kamil Ciebiera Krystian Król Jan Ludziejewski Michał Krutul Jakub Krajewski Szymon Antoniak Piotr Miłoś Marek Cygan Sebastian Jaszczur MoE Mamba 20 55 0 08 Jan 2024
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng E. Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 27 76 0 17 Dec 2023
Learning Long Sequences in Spiking Neural Networks Matei Ioan Stan Oliver Rhodes 37 11 0 14 Dec 2023
Spectral State Space Models Naman Agarwal Daniel Suo Xinyi Chen Elad Hazan 27 12 0 11 Dec 2023
Gated Linear Attention Transformers with Hardware-Efficient Training Songlin Yang Bailin Wang Yikang Shen Rameswar Panda Yoon Kim 45 142 0 11 Dec 2023
Structured state-space models are deep Wiener models Fabio Bonassi Carl R. Andersson Per Mattsson Thomas B. Schon 33 3 0 11 Dec 2023
Recurrent Distance Filtering for Graph Representation Learning Yuhui Ding Antonio Orvieto Bobby He Thomas Hofmann GNN 36 6 0 03 Dec 2023
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Tianyu Ding Tianyi Chen Haidong Zhu Jiachen Jiang Yiqi Zhong Jinxin Zhou Guangzhi Wang Zhihui Zhu Ilya Zharkov Luming Liang 27 22 0 01 Dec 2023
Diffusion Models Without Attention Jing Nathan Yan Jiatao Gu Alexander M. Rush 29 61 0 30 Nov 2023
On the Long Range Abilities of Transformers Itamar Zimerman Lior Wolf 27 7 0 28 Nov 2023
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity Zhen Qin Yiran Zhong 23 6 0 15 Nov 2023
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores Daniel Y. Fu Hermann Kumbong Eric N. D. Nguyen Christopher Ré VLM 41 29 0 10 Nov 2023
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability Jishnu Ray Chowdhury Cornelia Caragea 37 5 0 08 Nov 2023
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions Stefano Massaroli Michael Poli Daniel Y. Fu Hermann Kumbong Rom N. Parnichkun ... Atri Rudra Ce Zhang Christopher Ré Stefano Ermon Yoshua Bengio 34 19 0 28 Oct 2023
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer Qingru Zhang Dhananjay Ram Cole Hawkins Sheng Zha Tuo Zhao 27 15 0 19 Oct 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 21 41 0 04 Oct 2023
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors Ido Amos Jonathan Berant Ankit Gupta 30 24 0 04 Oct 2023
Multi-Dimensional Hyena for Spatial Inductive Bias Itamar Zimerman Lior Wolf ViT 30 4 0 24 Sep 2023
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models Zican Dong Tianyi Tang Junyi Li Wayne Xin Zhao Ji-Rong Wen RALM ALM 28 33 0 23 Sep 2023
Augmenting conformers with structured state-space sequence models for online speech recognition Haozhe Shan Albert Gu Zhong Meng Weiran Wang Krzysztof Choromanski Tara N. Sainath RALM 19 4 0 15 Sep 2023
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks Ting-Han Fan Ta-Chung Chi Alexander I. Rudnicky LRM 22 5 0 14 Sep 2023
Gated recurrent neural networks discover attention Nicolas Zucchet Seijin Kobayashi Yassir Akram J. Oswald Maxime Larcher Angelika Steger João Sacramento 36 8 0 04 Sep 2023
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer Zhen Qin Dong Li Weigao Sun Weixuan Sun Xuyang Shen ... Yunshen Wei Baohong Lv Xiao Luo Yu Qiao Yiran Zhong 43 15 0 27 Jul 2023
Efficient Beam Tree Recursion Jishnu Ray Chowdhury Cornelia Caragea 32 3 0 20 Jul 2023
Facing Off World Model Backbones: RNNs, Transformers, and S4 Fei Deng Junyeong Park Sungjin Ahn 32 24 0 05 Jul 2023
Efficient Dynamics Modeling in Interactive Environments with Koopman Theory Arnab Kumar Mondal Siba Smarak Panigrahi Sai Rajeswar K. Siddiqi Siamak Ravanbakhsh 28 3 0 20 Jun 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 43 13 0 19 Jun 2023
Block-State Transformers Mahan Fathi Jonathan Pilault Orhan Firat C. Pal Pierre-Luc Bacon Ross Goroshin 36 17 0 15 Jun 2023
2-D SSM: A General Spatial Layer for Visual Transformers Ethan Baron Itamar Zimerman Lior Wolf 28 14 0 11 Jun 2023
Decision S4: Efficient Sequence-Based RL via State Spaces Layers Shmuel Bar-David Itamar Zimerman Eliya Nachmani Lior Wolf OffRL 23 27 0 08 Jun 2023
Exploring the Promise and Limits of Real-Time Recurrent Learning Kazuki Irie Anand Gopalakrishnan Jürgen Schmidhuber 29 15 0 30 May 2023
A Quantitative Review on Language Model Efficiency Research Meng Jiang Hy Dang Lingbo Tong 25 0 0 28 May 2023
Revisiting Structured Variational Autoencoders Yixiu Zhao Scott W. Linderman BDL DRL 22 8 0 25 May 2023