Hierarchical Transformers Are More Efficient Language Models

Hierarchical Transformers Are More Efficient Language Models

26 October 2021

Szymon Tworkowski

Michał Tyrolski

Christian Szegedy

Henryk Michalewski

Papers citing "Hierarchical Transformers Are More Efficient Language Models"

18 / 18 papers shown

Title
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models Julie Kallini Shikhar Murty Christopher D. Manning Christopher Potts Róbert Csordás 40 2 0 28 Oct 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 49 1 0 17 Jul 2024
netFound: Foundation Model for Network Security Satyandra Guthula Navya Battula Roman Beltiukov Wenbo Guo Arpit Gupta Inder Monga 25 14 0 25 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 34 15 0 28 Sep 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 22 41 0 12 Jul 2023
Hierarchical Attention Encoder Decoder Asier Mujika BDL 22 3 0 01 Jun 2023
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki E. Ponti 22 42 0 17 Nov 2022
Language models are good pathologists: using attention-based sequence reduction and text-pretrained transformers for efficient WSI classification Juan Pisula Katarzyna Bozek VLM MedIm 36 3 0 14 Nov 2022
Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation Nadine Behrmann S. Golestaneh Zico Kolter Juergen Gall M. Noroozi 22 72 0 01 Sep 2022
G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model Pan Xie Qipeng Zhang Zexian Li Hao Tang Yao Du Xiaohui Hu DiffM 44 12 0 19 Aug 2022
Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting Amin Shabani A. Abdi Li Meng Tristan Sylvain AI4TS 27 61 0 08 Jun 2022
HiP: Hierarchical Perceiver João Carreira Skanda Koppula Daniel Zoran Adrià Recasens Catalin Ionescu ... M. Botvinick Oriol Vinyals Karen Simonyan Andrew Zisserman Andrew Jaegle VLM 31 14 0 22 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 43 65 0 15 Feb 2022
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren H. Dai Zihang Dai Mengjiao Yang J. Leskovec Dale Schuurmans Bo Dai 87 77 0 12 Jul 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 255 4,796 0 24 Feb 2021
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
Pixel Recurrent Neural Networks Aaron van den Oord Nal Kalchbrenner Koray Kavukcuoglu SSeg GAN 266 2,550 0 25 Jan 2016