Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

8 April 2024

Papers citing "Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws"

14 / 14 papers shown

Title
Data Mixing Can Induce Phase Transitions in Knowledge Acquisition Xinran Gu Kaifeng Lyu Jiazheng Li Jingzhao Zhang 39 0 0 23 May 2025
Enhancing LLMs via High-Knowledge Data Selection Feiyu Duan Xuemiao Zhang Sirui Wang Haoran Que Yuqi Liu Wenge Rong Xunliang Cai 129 0 0 20 May 2025
Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models Zhanke Zhou Zhaocheng Zhu Xuan Li Mikhail Galkin Xiao Feng Sanmi Koyejo Jian Tang Bo Han LRM 91 1 0 28 Mar 2025
Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis Jiaqi Zhao Ming Wang Miao Zhang Yuzhang Shang Xuebo Liu Yaowei Wang Min Zhang Liqiang Nie MQ 125 1 0 18 Feb 2025
How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines Ayan Sengupta Ayan Sengupta Tanmoy Chakraborty 98 0 0 17 Feb 2025
Typhoon T1: An Open Thai Reasoning Model Pittawat Taveekitworachai Potsawee Manakul Kasima Tharnpipitchai Kunat Pipatanakul OffRL LRM 147 0 0 13 Feb 2025
Do we really have to filter out random noise in pre-training data for language models? Jinghan Ru Yuxin Xie Xianwei Zhuang Yuguo Yin Zhihui Guo Zhiming Liu Qianli Ren Yuexian Zou 146 4 0 10 Feb 2025
Choose Your Model Size: Any Compression by a Single Gradient Descent Martin Genzel Patrick Putzky Pengfei Zhao Siyang Song Mattes Mollenhauer Robert Seidel Stefan Dietzel Thomas Wollmann 66 0 0 03 Feb 2025
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training Tianzhe Chu Yuexiang Zhai Jihan Yang Shengbang Tong Saining Xie Dale Schuurmans Quoc V. Le Sergey Levine Yi-An Ma OffRL 109 85 0 28 Jan 2025
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 75 9 0 11 Oct 2024
Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition Jiyeon Kim Hyunji Lee Hyowon Cho Joel Jang Hyeonbin Hwang Seungpil Won Youbin Ahn Dohaeng Lee Minjoon Seo KELM 316 4 0 02 Oct 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 105 1 0 15 Jul 2024
Knowledge Circuits in Pretrained Transformers Yunzhi Yao Ningyu Zhang Zekun Xi Meng Wang Ziwen Xu Shumin Deng Huajun Chen KELM 88 23 0 28 May 2024
Physics of Language Models: Part 1, Learning Hierarchical Language Structures Zeyuan Allen-Zhu Yuanzhi Li 80 18 0 23 May 2023