Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

20 December 2022

Zhifang Sui

Papers citing "Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers"

50 / 103 papers shown

Title
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning Jingcheng Niu Subhabrata Dutta A. Elshabrawy Harish Tayyar Madabushi Iryna Gurevych 12 0 0 16 May 2025
Big Data and the Computational Social Science of Entrepreneurship and Innovation Ningzi Li Shiyang Lai James Evans AILaw 24 0 0 13 May 2025
An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding Xiuwei Shang Zhenkan Fu Shaoyin Cheng Guoqiang Chen Gangyang Li Li Hu W. Zhang N. Yu 62 0 0 30 Apr 2025
Scaling sparse feature circuit finding for in-context learning Dmitrii Kharlapenko S. Kamath S Fazl Barez Arthur Conmy Neel Nanda 26 0 0 18 Apr 2025
Mimic In-Context Learning for Multimodal Tasks Yuchu Jiang Jiale Fu Chenduo Hao Xinting Hu Yingzhe Peng Xin Geng Xu Yang 27 0 0 11 Apr 2025
M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning in Large Vision-Language Models Yanshu Li Hongyang He Yi Cao Qisen Cheng Xiang Fu Ruixiang Tang VLM 40 0 0 06 Apr 2025
Implicit In-Context Learning: Evidence from Artificial Language Experiments Xiaomeng Ma Qihui Xu ReLM 51 0 0 31 Mar 2025
Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study Xingxuan Zhang Haoran Wang Jiansheng Li Yuan Xue Shikai Guan Renzhe Xu Hao Zou Han Yu Peng Cui 50 0 0 19 Mar 2025
Cognitive Activation and Chaotic Dynamics in Large Language Models: A Quasi-Lyapunov Analysis of Reasoning Mechanisms Xiaojian Li Yongkang Leng Ruiqing Ding Hangjie Mo Shanlin Yang LRM 47 0 0 15 Mar 2025
Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models Shaotian Yan Chen Shen Wenxiao Wang Liang Xie Junjie Liu Jieping Ye ReLM LRM 52 0 0 14 Mar 2025
TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator Deepak Vungarala Mohammed E. Elbtity Sumiya Syed Sakila Alam Kartik Pandit Arnob Ghosh Ramtin Zand Shaahin Angizi 34 1 0 07 Mar 2025
Representation Engineering for Large-Language Models: Survey and Research Challenges Lukasz Bartoszcze Sarthak Munshi Bryan Sukidi Jennifer Yen Zejia Yang David Williams-King Linh Le Kosi Asuzu Carsten Maple 102 0 0 24 Feb 2025
Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalization Zixuan Gong Xiaolin Hu Huayi Tang Yong Liu 33 0 0 24 Feb 2025
Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving Xin Xu Yan Xu Tianhao Chen Yuchen Yan Chengwu Liu ... Y. Wang Yichun Yin Y. Wang Lifeng Shang Q. Liu LRM 75 2 0 17 Feb 2025
Solving Empirical Bayes via Transformers Anzo Teh Mark Jabbour Yury Polyanskiy 90 0 0 17 Feb 2025
Lost in the Passage: Passage-level In-context Learning Does Not Necessarily Need a "Passage" Hao Sun Chenming Tang Gengyang Li Yunfang Wu AIMat 45 0 0 15 Feb 2025
PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning Yu Feng Yangli-ao Geng Yifan Zhu Zongfu Han Xie Yu Kaiwen Xue Haoran Luo Mengyang Sun Guangwei Zhang Meina Song FedML MoE 60 0 0 01 Feb 2025
Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong Yin Zhaoran Wang LRM ReLM 137 0 0 27 Jan 2025
Evolution and The Knightian Blindspot of Machine Learning Joel Lehman Elliot Meyerson Tarek El-Gaaly Kenneth O. Stanley Tarin Ziyaee 86 1 0 22 Jan 2025
In-Context Learning with Iterative Demonstration Selection Chengwei Qin Aston Zhang C. L. P. Chen Anirudh Dagar Wenming Ye LRM 68 38 0 31 Dec 2024
A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine Hanguang Xiao Feizhong Zhou X. Liu Tianqi Liu Zhipeng Li Xin Liu Xiaoxuan Huang AILaw LM&MA LRM 61 17 0 31 Dec 2024
Pretrained transformer efficiently learns low-dimensional target functions in-context Kazusato Oko Yujin Song Taiji Suzuki Denny Wu 39 4 0 04 Nov 2024
On the Role of Depth and Looping for In-Context Learning with Task Diversity Khashayar Gatmiry Nikunj Saunshi Sashank J. Reddi Stefanie Jegelka Sanjiv Kumar 29 2 0 29 Oct 2024
SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidance Deepak Vungarala Sakila Alam Arnob Ghosh Shaahin Angizi 30 3 0 27 Oct 2024
How Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs Guhao Feng Kai-Bo Yang Yuntian Gu Xinyue Ai Shengjie Luo Jiacheng Sun Di He Z. Li Liwei Wang LRM 35 5 0 17 Oct 2024
On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery Renpu Liu Ruida Zhou Cong Shen Jing Yang 28 0 0 17 Oct 2024
A Theoretical Survey on Foundation Models Shi Fu Yuzhu Chen Yingjie Wang Dacheng Tao 28 0 0 15 Oct 2024
RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answering Zhongwu Chen Chengjin Xu Dingmin Wang Zhen Huang Yong Dou Xuhui Jiang Jian Guo RALM 150 1 0 15 Oct 2024
Inference and Verbalization Functions During In-Context Learning Junyi Tao Xiaoyin Chen Nelson F. Liu ReLM LRM 26 0 0 12 Oct 2024
MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data Mingu Kang Dongseok Lee Woojin Cho Jaehyeon Park Kookjin Lee Anthony Gruber Youngjoon Hong Noseong Park DiffM AI4CE 29 0 0 09 Oct 2024
Transformers learn variable-order Markov chains in-context Ruida Zhou C. Tian Suhas Diggavi 26 0 0 07 Oct 2024
Density estimation with LLMs: a geometric investigation of in-context learning trajectories Toni J. B. Liu Nicolas Boullé Raphael Sarfati Christopher Earls 28 0 0 07 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 46 3 0 06 Oct 2024
Mitigating Copy Bias in In-Context Learning through Neuron Pruning Ameen Ali Lior Wolf Ivan Titov 36 2 0 02 Oct 2024
Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models Can Demircan Tankred Saanum A. Jagadish Marcel Binz Eric Schulz 30 1 0 02 Oct 2024
Automating Traffic Model Enhancement with AI Research Agent Xusen Guo Xinxi Yang Mingxing Peng Hongliang Lu Meixin Zhu Hai Yang 62 0 0 25 Sep 2024
Multimodal Contrastive In-Context Learning Yosuke Miyanishi Minh Le Nguyen 32 2 0 23 Aug 2024
In-Context Learning with Representations: Contextual Generalization of Trained Transformers Tong Yang Yu Huang Yingbin Liang Yuejie Chi MLT 32 5 0 19 Aug 2024
BadRobot: Jailbreaking Embodied LLMs in the Physical World Hangtao Zhang Chenyu Zhu Xianlong Wang Ziqi Zhou Yichen Wang ... Shengshan Hu Leo Yu Zhang Aishan Liu Peijin Guo Leo Yu Zhang LM&Ro 47 7 0 16 Jul 2024
Distributed Rule Vectors is A Key Mechanism in Large Language Models' In-Context Learning Bowen Zheng Ming Ma Zhongqiao Lin Tianming Yang 33 1 0 23 Jun 2024
HCQA @ Ego4D EgoSchema Challenge 2024 Haoyu Zhang Yuquan Xie Yisen Feng Zaijing Li Meng Liu Liqiang Nie 34 2 0 22 Jun 2024
Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problem Sara Court Micha Elsner 32 6 0 21 Jun 2024
Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs Ahmad Mohsin Helge Janicke Adrian Wood Iqbal H. Sarker Leandros A. Maglaras N. Janjua 35 8 0 18 Jun 2024
Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations Rima Hazra Sayan Layek Somnath Banerjee Soujanya Poria KELM LLMSV 29 6 0 17 Jun 2024
Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions Yiming Tang Bin Dong 36 0 0 16 Jun 2024
StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation Weike Fang Zhejian Zhou Junzhou He Weihang Wang LRM 19 1 0 07 Jun 2024
Cycles of Thought: Measuring LLM Confidence through Stable Explanations Evan Becker Stefano Soatto 42 6 0 05 Jun 2024
GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model Ling Li Yu Ye Bingchuan Jiang Wei Zeng VLM LRM 31 7 0 03 Jun 2024
Benchmarking General-Purpose In-Context Learning Fan Wang Chuan Lin Yang Cao Yu Kang 32 1 0 27 May 2024
Evaluating the Adversarial Robustness of Retrieval-Based In-Context Learning for Large Language Models Simon Chi Lok Yu Jie He Pasquale Minervini Jeff Z. Pan 26 0 0 24 May 2024