v1v2v3v4v5 (latest)

RoFormer: Enhanced Transformer with Rotary Position Embedding

20 April 2021

Papers citing "RoFormer: Enhanced Transformer with Rotary Position Embedding"

50 / 250 papers shown

Title
Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit Oleg Filatov Jan Ebert Jiangtao Wang Stefan Kesselheim 100 4 0 10 Jan 2025
Multi-subject Open-set Personalization in Video Generation Tsai-Shien Chen Aliaksandr Siarohin Willi Menapace Yuwei Fang Kwot Sin Lee Ivan Skorokhodov Kfir Aberman Jun-Yan Zhu Ming-Hsuan Yang Sergey Tulyakov DiffM VGen 169 13 0 10 Jan 2025
Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers Markus J. Buehler AI4CE 125 3 0 04 Jan 2025
Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques Lijie Tao Han Zhang Haizhao Jing Yu Liu Kelu Yao Guoting Wei Xizhe Xue 109 0 0 03 Jan 2025
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao Song 131 14 0 03 Jan 2025
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving Zihao Ye Lequn Chen Ruihang Lai Wuwei Lin Yineng Zhang ... Tianqi Chen Baris Kasikci Vinod Grover Arvind Krishnamurthy Luis Ceze 121 35 0 02 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 148 10 0 31 Dec 2024
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT Xiaotao Hu Wei Yin Mingkai Jia Junyuan Deng Xiaoyang Guo Qian Zhang Xiaoxiao Long Ping Tan VGen 138 14 0 31 Dec 2024
Parallelized Autoregressive Visual Generation Yanjie Wang Shuhuai Ren Zhijie Lin Yujin Han Haoyuan Guo Zhenheng Yang Difan Zou Jiashi Feng Xihui Liu VGen 170 16 0 19 Dec 2024
Next Patch Prediction for Autoregressive Visual Generation Yatian Pang Peng Jin Shuo Yang Bin Lin Bin Zhu ... Liuhan Chen Francis E. H. Tay Ser-Nam Lim Harry Yang Li Yuan 249 10 0 19 Dec 2024
MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis Ho Kei Cheng Masato Ishii Akio Hayakawa Takashi Shibuya Alex Schwing Yuki Mitsufuji VGen 268 17 0 19 Dec 2024
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor Chenyu Yang Shuai Wang Hangting Chen Jianwei Yu Wei Tan Rongzhi Gu Yongjun Xu Yizhi Zhou Haina Zhu Haoyang Li KELM 402 2 0 18 Dec 2024
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models Elvis Nunez Luca Zancato Benjamin Bowman Aditya Golatkar Wei Xia Stefano Soatto 189 4 0 17 Dec 2024
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration Wenhao Sun Rong-Cheng Tu Jingyi Liao Zhao Jin Dacheng Tao VGen 206 1 0 16 Dec 2024
SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer Hong Chen Zihan Wang Xianrui Li Xingwu Sun Fangyi Chen Jiang Liu Jiadong Wang Bhiksha Raj Zicheng Liu Emad Barsoum VLM 238 10 0 14 Dec 2024
jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images Andreas Koukounas Georgios Mastrapas Bo Wang Mohammad Kalim Akram Sedigheh Eslami Michael Gunther Isabelle Mohr Saba Sturua Scott Martens Nan Wang VLM 317 10 0 11 Dec 2024
SVGFusion: Scalable Text-to-SVG Generation via Vector Space Diffusion Ximing Xing Juncheng Hu Jing Zhang Dong Xu Qian Yu 182 4 0 11 Dec 2024
Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs Michael Wornow Suhana Bedi Miguel Angel Fuentes Hernandez E. Steinberg Jason Alan Fries Christopher Ré Sanmi Koyejo N. Shah 192 6 0 09 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 165 4 0 02 Dec 2024
IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models Khaled Abud Sergey Lavrushkin Alexey Kirillov D. Vatolin 179 0 0 02 Dec 2024
ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain Ali Shiraee Kasmaee Mohammad Khodadad Mohammad Arshi Saloot Nick Sherck Stephen Dokas H. Mahyar Soheila Samiee ELM 595 2 0 30 Nov 2024
Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer Shitong Shao Zikai Zhou Tian Ye Lichen Bai Zhiqiang Xu Zeke Xie DiffM 101 0 0 16 Nov 2024
More Expressive Attention with Negative Weights Ang Lv Ruobing Xie Shuaipeng Li Jiayi Liao Xingwu Sun Zhanhui Kang Di Wang Rui Yan 89 1 0 11 Nov 2024
Number Cookbook: Number Understanding of Language Models and How to Improve It Haotong Yang Yi Hu Shijia Kang Zhouchen Lin Muhan Zhang LRM 90 8 0 06 Nov 2024
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Zehua Wang Hui Xiong Hui Xiong LLMAG 148 7 0 05 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 129 13 0 31 Oct 2024
Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models Arash Marioriyad Parham Rezaei M. Baghshah M. Rohban CoGe 447 0 0 30 Oct 2024
Understanding Synthetic Context Extension via Retrieval Heads Xinyu Zhao Fangcong Yin Greg Durrett 116 2 0 29 Oct 2024
GenUP: Generative User Profilers as In-Context Learners for Next POI Recommender Systems Wilson Wongso Hao Xue Flora D. Salim 42 2 0 28 Oct 2024
Beyond Autoregression: Fast LLMs via Self-Distillation Through Time Justin Deschenaux Çağlar Gülçehre 116 5 0 28 Oct 2024
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation Zongyi Li Shujie Hu Shujie Liu Long Zhou Jeongsoo Choi Lingwei Meng Xun Guo Jiajian Li H. Ling Furu Wei VGen DiffM 145 7 0 27 Oct 2024
Scaling up Masked Diffusion Models on Text Shen Nie Fengqi Zhu Chao Du Tianyu Pang Qian Liu Guangtao Zeng Min Lin Chongxuan Li AI4CE 173 30 0 24 Oct 2024
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition Artem Basharin Andrei Chertkov Ivan Oseledets 126 1 0 23 Oct 2024
Science Out of Its Ivory Tower: Improving Accessibility with Reinforcement Learning Haining Wang Jason Clark Hannah McKelvey Leila Sterman Zheng Gao Zuoyu Tian Sandra Kübler Xiaozhong Liu 98 1 0 22 Oct 2024
Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence İlker Işık R. G. Cinbis Ebru Aydin Gol 78 0 0 22 Oct 2024
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation Hanbo Cheng Limin Lin Chenyu Liu Pengcheng Xia Pengfei Hu Jiefeng Ma Jun Du Jia Pan DiffM VGen 433 0 0 17 Oct 2024
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs Yizhao Gao Zhichen Zeng Dayou Du Shijie Cao Hayden Kwok-Hay So ... Junjie Lai Mao Yang Ting Cao Fan Yang M. Yang 124 27 0 17 Oct 2024
Artificial Kuramoto Oscillatory Neurons Takeru Miyato Sindy Löwe Andreas Geiger Max Welling AI4CE 180 10 0 17 Oct 2024
MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection Bokai Lin Zihao Zeng Zipeng Xiao Siqi Kou Tianqi Hou Xiaofeng Gao Hao Zhang Zhijie Deng 66 6 0 16 Oct 2024
MIND: Math Informed syNthetic Dialogues for Pretraining LLMs Syeda Nahida Akter Shrimai Prabhumoye John Kamalu S. Satheesh Eric Nyberg M. Patwary Mohammad Shoeybi Bryan Catanzaro LRM SyDa ReLM 154 2 0 15 Oct 2024
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 244 0 0 15 Oct 2024
Liger Kernel: Efficient Triton Kernels for LLM Training Pin-Lun Hsu Yun Dai Vignesh Kothapalli Qingquan Song Shao Tang Siyu Zhu Steven Shimizu Shivam Sahni Haowen Ning Yanning Chen 103 45 0 14 Oct 2024
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free Ziyue Li Dinesh Manocha MoE 131 9 0 14 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 84 7 0 14 Oct 2024
TULIP: Token-length Upgraded CLIP Ivona Najdenkoska Mohammad Mahdi Derakhshani Yuki M. Asano Nanne van Noord Marcel Worring Cees G. M. Snoek VLM 101 4 0 13 Oct 2024
DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation Jiatao Gu Yuyang Wang Yizhe Zhang Qihang Zhang Dinghuai Zhang Navdeep Jaitly Josh Susskind Shuangfei Zhai DiffM 97 17 0 10 Oct 2024
MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion Onkar Susladkar Jishu Sen Gupta Chirag Sehgal Sparsh Mittal Rekha Singhal DiffM VGen 80 0 0 10 Oct 2024
Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis Jinbin Bai Tian-Chun Ye Wei Chow Enxin Song Qing-Guo Chen Hefei Ling Zhen Dong Lei Zhu 131 19 0 10 Oct 2024
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching Yushen Chen Zhikang Niu Ziyang Ma Keqi Deng Chunhui Wang Jian Zhao Kai Yu Xie Chen 121 90 0 09 Oct 2024
Round and Round We Go! What makes Rotary Positional Encodings useful? Federico Barbero Alex Vitvitskyi Christos Perivolaropoulos Razvan Pascanu Petar Velickovic 121 29 0 08 Oct 2024