Weight subcloning: direct initialization of transformers using larger
pretrained ones

Weight subcloning: direct initialization of transformers using larger pretrained ones

14 December 2023

Mohammad Samragh

Mehrdad Farajtabar

Raviteja Vemulapalli

Mohammad Rastegari

Papers citing "Weight subcloning: direct initialization of transformers using larger pretrained ones"

10 / 10 papers shown

Title
One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models Rongguang Ye Ming Tang 12 0 0 18 May 2025
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 38 1 0 13 Oct 2024
A Review of Pseudo-Labeling for Computer Vision Patrick Kage Jay C. Rothenberger Pavlos Andreadis Dimitrios I. Diochnos VLM 50 3 0 13 Aug 2024
A deeper look at depth pruning of LLMs Shoaib Ahmed Siddiqui Xin Dong Greg Heinrich Thomas Breuel Jan Kautz David M. Krueger Pavlo Molchanov 40 7 0 23 Jul 2024
BlockPruner: Fine-grained Pruning for Large Language Models Longguang Zhong Fanqi Wan Ruijun Chen Xiaojun Quan Liangzhi Li 33 7 0 15 Jun 2024
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 40 8 0 13 Apr 2024
Efficiently Distilling LLMs for Edge Applications Achintya Kundu Fabian Lim Aaron Chew L. Wynter Penny Chong Rhui Dih Lee 50 6 0 01 Apr 2024
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 157 44 0 16 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 282 2,007 0 31 Dec 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 191 1,032 0 06 Mar 2020