FactorLLM: Factorizing Knowledge via Mixture of Experts for Large
Language Models

FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models

15 August 2024

Rongyu Zhang

Yunpeng Zhang

Huanrui Yang

Kurt Keutzer

Shanghang Zhang

Papers citing "FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models"

4 / 4 papers shown

Title
DistiLLM: Towards Streamlined Distillation for Large Language Models Jongwoo Ko Sungnyun Kim Tianyi Chen SeYoung Yun 71 26 0 06 Feb 2024
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models Iman Mirzadeh Keivan Alizadeh-Vahid Sachin Mehta C. C. D. Mundo Oncel Tuzel Golnoosh Samei Mohammad Rastegari Mehrdad Farajtabar 126 60 0 06 Oct 2023
Mixture of Attention Heads: Selecting Attention Heads Per Token Xiaofeng Zhang Songlin Yang Zeyu Huang Jie Zhou Wenge Rong Zhang Xiong MoE 99 42 0 11 Oct 2022
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020