Adaptive Computation with Elastic Input Sequence

30 January 2023

Fuzhao Xue

Valerii Likhosherstov

Yang You

Papers citing "Adaptive Computation with Elastic Input Sequence"

50 / 50 papers shown

Title
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 74 1 0 01 Feb 2024
Scaling Vision Transformers to 22 Billion Parameters Mostafa Dehghani Josip Djolonga Basil Mustafa Piotr Padlewski Jonathan Heek ... Mario Luvcić Xiaohua Zhai Daniel Keysers Jeremiah Harmsen N. Houlsby MLLM 144 585 0 10 Feb 2023
Dual PatchNorm Manoj Kumar Mostafa Dehghani N. Houlsby UQCV ViT 37 11 0 02 Feb 2023
A Generalist Neural Algorithmic Learner Borja Ibarz Vitaly Kurin George Papamakarios Kyriacos Nikiforou Mehdi Abbana Bennani ... Andreea Deac Beatrice Bevilacqua Yaroslav Ganin Charles Blundell Petar Velivcković OOD 68 53 0 22 Sep 2022
A Review of Sparse Expert Models in Deep Learning W. Fedus J. Dean Barret Zoph MoE 89 147 0 04 Sep 2022
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung W. Fedus J. Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler AI4CE 87 102 0 21 Jul 2022
Confident Adaptive Language Modeling Tal Schuster Adam Fisch Jai Gupta Mostafa Dehghani Dara Bahri Vinh Q. Tran Yi Tay Donald Metzler 101 164 0 14 Jul 2022
The CLRS Algorithmic Reasoning Benchmark Petar Velivcković Adria Puigdomenech Badia David Budden Razvan Pascanu Andrea Banino Mikhail Dashevskiy R. Hadsell Charles Blundell 179 91 0 31 May 2022
Better plain ViT baselines for ImageNet-1k Lucas Beyer Xiaohua Zhai Alexander Kolesnikov ViT VLM 50 116 0 03 May 2022
Retrieval-Enhanced Machine Learning Hamed Zamani Fernando Diaz Mostafa Dehghani Donald Metzler Michael Bendersky 41 50 0 02 May 2022
PALBERT: Teaching ALBERT to Ponder Nikita Balagansky Daniil Gavrilov MoE 36 6 0 07 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 146 1,915 0 29 Mar 2022
End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking Arpit Bansal Avi Schwarzschild Eitan Borgnia Z. Emam Furong Huang Micah Goldblum Tom Goldstein LRM 34 24 0 11 Feb 2022
AdaViT: Adaptive Tokens for Efficient Vision Transformer Hongxu Yin Arash Vahdat J. Álvarez Arun Mallya Jan Kautz Pavlo Molchanov ViT 73 327 0 14 Dec 2021
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition Lingchen Meng Hengduo Li Bor-Chun Chen Shiyi Lan Zuxuan Wu Yu-Gang Jiang Ser-Nam Lim ViT 54 227 0 30 Nov 2021
Adaptive Token Sampling For Efficient Vision Transformers Mohsen Fayyaz Soroush Abbasi Koohpayegani F. Jafari Sunando Sengupta Hamid Reza Vaezi Joze Eric Sommerlade Hamed Pirsiavash Juergen Gall ViT 58 149 0 30 Nov 2021
PolyViT: Co-training Vision Transformers on Images, Videos and Audio Valerii Likhosherstov Anurag Arnab K. Choromanski Mario Lucic Yi Tay Adrian Weller Mostafa Dehghani ViT 49 74 0 25 Nov 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 81 101 0 25 Oct 2021
SCENIC: A JAX Library for Computer Vision Research and Beyond Mostafa Dehghani A. Gritsenko Anurag Arnab Matthias Minderer Yi Tay 65 68 0 18 Oct 2021
ViDT: An Efficient and Effective Fully Transformer-based Object Detector Hwanjun Song Deqing Sun Sanghyuk Chun Varun Jampani Dongyoon Han Byeongho Heo Wonjae Kim Ming-Hsuan Yang 115 76 0 08 Oct 2021
Exploring the Limits of Large Scale Pre-training Samira Abnar Mostafa Dehghani Behnam Neyshabur Hanie Sedghi AI4CE 84 116 0 05 Oct 2021
The Benchmark Lottery Mostafa Dehghani Yi Tay A. Gritsenko Zhe Zhao N. Houlsby Fernando Diaz Donald Metzler Oriol Vinyals 74 90 0 14 Jul 2021
PonderNet: Learning to Ponder Andrea Banino Jan Balaguer Charles Blundell PINN AIMat 111 80 0 12 Jul 2021
Scaling Vision with Sparse Mixture of Experts C. Riquelme J. Puigcerver Basil Mustafa Maxim Neumann Rodolphe Jenatton André Susano Pinto Daniel Keysers N. Houlsby MoE 67 587 0 10 Jun 2021
Scaling Vision Transformers Xiaohua Zhai Alexander Kolesnikov N. Houlsby Lucas Beyer ViT 116 1,074 0 08 Jun 2021
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks Avi Schwarzschild Eitan Borgnia Arjun Gupta Furong Huang U. Vishkin Micah Goldblum Tom Goldstein 48 74 0 08 Jun 2021
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition Yulin Wang Rui Huang S. Song Zeyi Huang Gao Huang ViT 64 189 0 31 May 2021
Consistent Accelerated Inference via Confident Adaptive Transformers Tal Schuster Adam Fisch Tommi Jaakkola Regina Barzilay AI4TS 217 70 0 18 Apr 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 480 3,952 0 18 Apr 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 68 2,136 0 11 Jan 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 324 6,657 0 23 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 460 40,217 0 22 Oct 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 82 1,142 0 30 Jun 2020
Memory Transformer Andrey Kravchenko Yuri Kuratov Anton Peganov Grigory V. Sapunov RALM 45 70 0 20 Jun 2020
Transferring Inductive Biases through Knowledge Distillation Samira Abnar Mostafa Dehghani Willem H. Zuidema 60 58 0 31 May 2020
The Right Tool for the Job: Matching Model and Instance Complexities Roy Schwartz Gabriel Stanovsky Swabha Swayamdipta Jesse Dodge Noah A. Smith 77 168 0 16 Apr 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 483 4,662 0 23 Jan 2020
Big Transfer (BiT): General Visual Representation Learning Alexander Kolesnikov Lucas Beyer Xiaohua Zhai J. Puigcerver Jessica Yung Sylvain Gelly N. Houlsby MQ 232 1,196 0 24 Dec 2019
Graph Transformer Networks Seongjun Yun Minbyul Jeong Raehyun Kim Jaewoo Kang Hyunwoo J. Kim 125 958 0 06 Nov 2019
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 195 3,458 0 30 Sep 2019
Theoretical Limitations of Self-Attention in Neural Sequence Models Michael Hahn 49 267 0 16 Jun 2019
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 165 3,490 0 19 Aug 2018
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 72 752 0 10 Jul 2018
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 258 9,687 0 25 Oct 2017
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Chen Sun Abhinav Shrivastava Saurabh Singh Abhinav Gupta VLM 131 2,386 0 10 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 519 129,831 0 12 Jun 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 307 10,412 0 21 Jul 2016
Adaptive Computation Time for Recurrent Neural Networks Alex Graves 63 544 0 29 Mar 2016
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 577 27,231 0 02 Dec 2015
Neural Turing Machines Alex Graves Greg Wayne Ivo Danihelka 88 2,318 0 20 Oct 2014