GQA: Training Generalized Multi-Query Transformer Models from Multi-Head
Checkpoints

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

22 May 2023

Joshua Ainslie

James Lee-Thorp

Michiel de Jong

Yury Zemlyanskiy

Federico Lebrón

Sumit Sanghai

Papers citing "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"

12 / 112 papers shown

Title
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference Siyu Ren Kenny Q. Zhu 26 27 0 09 Feb 2024
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding Zack Ankner Rishab Parthasarathy Aniruddha Nrusimha Christopher Rinard Jonathan Ragan-Kelley William Brandon 32 25 0 07 Feb 2024
TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese N. Corrêa Sophia Falk Shiza Fatimah Aniket Sen N. D. Oliveira 30 9 0 30 Jan 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 52 252 0 19 Jan 2024
CBQ: Cross-Block Quantization for Large Language Models Xin Ding Xiaoyu Liu Zhijun Tu Yun-feng Zhang Wei Li ... Hanting Chen Yehui Tang Zhiwei Xiong Baoqun Yin Yunhe Wang MQ 38 13 0 13 Dec 2023
TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models Zheng Chu Jingchang Chen Qianglong Chen Weijiang Yu Haotian Wang Ming Liu Bing Qin LRM ELM 35 9 0 29 Nov 2023
GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values Farnoosh Javadi Walid Ahmed Habib Hajimolahoseini Foozhan Ataiefard Mohammad Hassanpour Saina Asani Austin Wen Omar Mohamed Awad Kangling Liu Yang Liu VLM 42 7 0 06 Nov 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 28 3 0 07 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 126 11,099 0 18 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 78 304 0 17 Jul 2023
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng Zhang Yuxiao Dong Jie Tang BDL LRM 253 1,073 0 05 Oct 2022
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,984 0 20 Apr 2018