What Makes Convolutional Models Great on Long Sequence Modeling?

17 October 2022

Tianle Cai

Papers citing "What Makes Convolutional Models Great on Long Sequence Modeling?"

48 / 48 papers shown

Title
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 101 185 0 21 Sep 2022
How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections Albert Gu Isys Johnson Aman Timalsina Atri Rudra Christopher Ré Mamba 166 101 0 24 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 89 324 0 23 Jun 2022
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs Xiaohan Ding Xinming Zhang Yi Zhou Jungong Han Guiguang Ding Jian Sun VLM 133 551 0 13 Mar 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 76 222 0 17 Feb 2022
A ConvNet for the 2020s Zhuang Liu Hanzi Mao Chaozheng Wu Christoph Feichtenhofer Trevor Darrell Saining Xie ViT 183 5,213 0 10 Jan 2022
SCROLLS: Standardized CompaRison Over Long Language Sequences Uri Shaham Elad Segal Maor Ivgi Avia Efrat Ori Yoran ... Ankit Gupta Wenhan Xiong Mor Geva Jonathan Berant Omer Levy RALM 102 139 0 10 Jan 2022
MetaFormer Is Actually What You Need for Vision Weihao Yu Mi Luo Pan Zhou Chenyang Si Yichen Zhou Xinchao Wang Jiashi Feng Shuicheng Yan 173 914 0 22 Nov 2021
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 217 1,821 0 31 Oct 2021
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers Albert Gu Isys Johnson Karan Goel Khaled Kamal Saab Tri Dao Atri Rudra Christopher Ré 130 606 0 26 Oct 2021
FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes David W. Romero Robert-Jan Bruintjes Jakub M. Tomczak Erik J. Bekkers Mark Hoogendoorn Jan van Gemert 132 83 0 15 Oct 2021
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding Shengjie Luo Shanda Li Tianle Cai Di He Dinglan Peng Shuxin Zheng Guolin Ke Liwei Wang Tie-Yan Liu 70 50 0 23 Jun 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 160 1,021 0 31 Mar 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 463 21,566 0 25 Mar 2021
Random Feature Attention Hao Peng Nikolaos Pappas Dani Yogatama Roy Schwartz Noah A. Smith Lingpeng Kong 109 362 0 03 Mar 2021
CKConv: Continuous Kernel Convolution For Sequential Data David W. Romero Anna Kuzina Erik J. Bekkers Jakub M. Tomczak Mark Hoogendoorn 65 126 0 04 Feb 2021
RepVGG: Making VGG-style ConvNets Great Again Xiaohan Ding Xinming Zhang Ningning Ma Jungong Han Guiguang Ding Jian Sun 286 1,602 0 11 Jan 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 389 6,802 0 23 Dec 2020
Long Range Arena: A Benchmark for Efficient Transformers Yi Tay Mostafa Dehghani Samira Abnar Songlin Yang Dara Bahri Philip Pham J. Rao Liu Yang Sebastian Ruder Donald Metzler 152 727 0 08 Nov 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 670 41,430 0 22 Oct 2020
HiPPO: Recurrent Memory with Optimal Polynomial Projections Albert Gu Tri Dao Stefano Ermon Atri Rudra Christopher Ré 123 533 0 17 Aug 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 556 2,099 0 28 Jul 2020
Rethinking Positional Encoding in Language Pre-training Guolin Ke Di He Tie-Yan Liu 84 299 0 28 Jun 2020
DO-Conv: Depthwise Over-parameterized Convolutional Layer Jinming Cao Yangyan Li Mingchao Sun Ying-Cong Chen Dani Lischinski Daniel Cohen-Or Baoquan Chen Changhe Tu OOD 66 173 0 22 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 216 1,713 0 08 Jun 2020
Neural Controlled Differential Equations for Irregular Time Series Patrick Kidger James Morrill James Foster Terry Lyons AI4TS 111 481 0 18 May 2020
Synthesizer: Rethinking Self-Attention in Transformer Models Yi Tay Dara Bahri Donald Metzler Da-Cheng Juan Zhe Zhao Che Zheng 61 338 0 02 May 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 201 2,332 0 13 Jan 2020
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu Yi Zhu Xiaojun Chang Xiaodan Liang LRM 70 242 0 18 Nov 2019
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks Xiaohan Ding Yuchen Guo Guiguang Ding Jiawei Han 72 676 0 11 Aug 2019
Disentangling neural mechanisms for perceptual grouping Junkyung Kim Drew Linsley Kalpit C. Thakkar Thomas Serre OCL 87 54 0 04 Jun 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 153 18,179 0 28 May 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 129 1,915 0 23 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 257 3,745 0 09 Jan 2019
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks Shuxuan Guo J. Álvarez Mathieu Salzmann 69 80 0 26 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,175 0 11 Oct 2018
Learning long-range spatial dependencies with horizontal gated-recurrent units Drew Linsley Junkyung Kim Vijay Veerabadran Thomas Serre 48 160 0 21 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,196 0 20 Apr 2018
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition Pete Warden 97 1,626 0 09 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 786 132,363 0 12 Jun 2017
DiracNets: Training Very Deep Neural Networks Without Skip-Connections Sergey Zagoruyko N. Komodakis UQCV OOD 57 119 0 01 Jun 2017
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning Justin Johnson B. Hariharan Laurens van der Maaten Li Fei-Fei C. L. Zitnick Ross B. Girshick CoGe 313 2,387 0 20 Dec 2016
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 338 2,898 0 26 Sep 2016
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems Martín Abadi Ashish Agarwal P. Barham E. Brevdo Zhiwen Chen ... Pete Warden Martin Wattenberg Martin Wicke Yuan Yu Xiaoqiang Zheng 287 11,150 0 14 Mar 2016
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Tim Salimans Diederik P. Kingma ODL 196 1,943 0 25 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,426 0 10 Dec 2015
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems Tianqi Chen Mu Li Yutian Li Min Lin Naiyan Wang Minjie Wang Tianjun Xiao Bing Xu Chiyuan Zhang Zheng Zhang 200 2,248 0 03 Dec 2015
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 133 2,554 0 22 Jun 2015