v1v2v3v4v5 (latest)

RoFormer: Enhanced Transformer with Rotary Position Embedding

20 April 2021

Papers citing "RoFormer: Enhanced Transformer with Rotary Position Embedding"

50 / 250 papers shown

Title
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 274 571 0 07 Mar 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 274 22 0 28 Feb 2024
SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution Chengcheng Wang Zhiwei Hao Yehui Tang Jianyuan Guo Yujie Yang Kai Han Yunhe Wang 90 7 0 27 Feb 2024
World Model on Million-Length Video And Language With Blockwise RingAttention Hao Liu Wilson Yan Matei A. Zaharia Pieter Abbeel VGen 108 82 0 13 Feb 2024
Large Language Models: A Survey Shervin Minaee Tomas Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 208 417 0 09 Feb 2024
CroissantLLM: A Truly Bilingual French-English Language Model Manuel Faysse Patrick Fernandes Nuno M. Guerreiro António Loison Duarte M. Alves ... François Yvon André F.T. Martins Gautier Viaud C´eline Hudelot Pierre Colombo 114 37 0 01 Feb 2024
Latte: Latent Diffusion Transformer for Video Generation Xin Ma Yaohui Wang Gengyun Jia Xinyuan Chen Ziqiang Liu Yuan-Fang Li Cunjian Chen Yu Qiao DiffM VGen 255 277 0 05 Jan 2024
FLTrojan: Privacy Leakage Attacks against Federated Language Models Through Selective Weight Tampering Md Rafi Ur Rashid Vishnu Asutosh Dasu Kang Gu Najrin Sultana Shagufta Mehnaz AAML FedML 138 12 0 24 Oct 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Guosheng Dong Zhiying Wu ELM LRM 209 748 0 19 Sep 2023
FLM-101B: An Open LLM and How to Train It with $100K Budget$ Xiang Li Yiqun Yao Xin Jiang Xuezhi Fang Xuying Meng ... Li Du Bowen Qin Zheng Zhang Aixin Sun Yequan Wang 130 22 0 07 Sep 2023
Physics of Language Models: Part 1, Learning Hierarchical Language Structures Zeyuan Allen-Zhu Yuanzhi Li 112 21 0 23 May 2023
Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering Zhou Yu Xuecheng Ouyang Zhenwei Shao Mei Wang Jun Yu MLLM 168 11 0 03 Mar 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 336 775 0 27 Aug 2021
Relative Positional Encoding for Transformers with Linear Complexity Antoine Liutkus Ondřej Cífka Shih-Lun Wu Umut Simsekli Yi-Hsuan Yang Gaël Richard 65 48 0 18 May 2021
Translational Equivariance in Kernelizable Attention Max Horn Kumar Shridhar Elrich Groenewald Philipp F. M. Baumann 42 7 0 15 Feb 2021
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 186 1,602 0 30 Sep 2020
Improve Transformer Models with Better Relative Position Embeddings Zhiheng Huang Davis Liang Peng Xu Bing Xiang ViT 62 132 0 28 Sep 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 203 1,786 0 29 Jun 2020
Rethinking Positional Encoding in Language Pre-training Guolin Ke Di He Tie-Yan Liu 84 299 0 28 Jun 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 165 2,754 0 05 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 880 42,379 0 28 May 2020
Learning to Encode Position for Transformer with Continuous Dynamical Model Xuanqing Liu Hsiang-Fu Yu Inderjit Dhillon Cho-Jui Hsieh 70 111 0 13 Mar 2020
How Much Position Information Do Convolutional Neural Networks Encode? Md. Amirul Islam Sen Jia Neil D. B. Bruce SSL 249 348 0 22 Jan 2020
Encoding word order in complex embeddings Benyou Wang Donghao Zhao Christina Lioma Qiuchi Li Peng Zhang J. Simonsen 61 113 0 27 Dec 2019
Are Transformers universal approximators of sequence-to-sequence functions? Chulhee Yun Srinadh Bhojanapalli A. S. Rawat Sashank J. Reddi Sanjiv Kumar 121 359 0 20 Dec 2019
CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain Chaojun Xiao Haoxiang Zhong Zhipeng Guo Cunchao Tu Zhiyuan Liu ... Tianyang Zhang Xianpei Han Zhen Hu Heng Wang Jianfeng Xu AILaw 63 63 0 20 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 485 20,342 0 23 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 373 6,469 0 26 Sep 2019
NEZHA: Neural Contextualized Representation for Chinese Language Understanding Junqiu Wei Xiaozhe Ren Xiaoguang Li Wenyong Huang Yi-Lun Liao Yasheng Wang Jianghao Lin Xin Jiang Xiao Chen Qun Liu 57 116 0 31 Aug 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 236 8,451 0 19 Jun 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 182 2,532 0 19 May 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 116 3,156 0 01 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 260 3,745 0 09 Jan 2019
Efficient Attention: Attention with Linear Complexities Zhuoran Shen Mingyuan Zhang Haiyu Zhao Shuai Yi Hongsheng Li 95 534 0 04 Dec 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,229 0 11 Oct 2018
Music Transformer Cheng-Zhi Anna Huang Ashish Vaswani Jakob Uszkoreit Noam M. Shazeer Ian Simon Curtis Hawthorne Andrew M. Dai Matthew D. Hoffman Monica Dinculescu Douglas Eck 207 486 0 12 Sep 2018
Neural Ordinary Differential Equations T. Chen Yulia Rubanova J. Bettencourt David Duvenaud AI4CE 443 5,157 0 19 Jun 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,200 0 20 Apr 2018
Training Tips for the Transformer Model Martin Popel Ondrej Bojar 72 311 0 01 Apr 2018
Self-Attention with Relative Position Representations Peter Shaw Jakob Uszkoreit Ashish Vaswani 182 2,299 0 06 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 789 132,454 0 12 Jun 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 231 2,686 0 09 May 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 171 3,289 0 08 May 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 524 4,497 0 18 Apr 2017
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions Marcin Junczys-Dowmunt Tomasz Dwojak Hieu T. Hoang 78 199 0 04 Oct 2016
The LAMBADA dataset: Word prediction requiring a broad discourse context Denis Paperno Germán Kruszewski Angeliki Lazaridou Q. N. Pham Raffaella Bernardi Sandro Pezzelle Marco Baroni Gemma Boleda Raquel Fernández 142 727 0 20 Jun 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 316 8,174 0 16 Jun 2016
A Decomposable Attention Model for Natural Language Inference Ankur P. Parikh Oscar Täckström Dipanjan Das Jakob Uszkoreit 349 1,375 0 06 Jun 2016
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 232 7,760 0 31 Aug 2015
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 136 2,555 0 22 Jun 2015