Code Pretraining Improves Entity Tracking Abilities of Language Models

31 May 2024

Papers citing "Code Pretraining Improves Entity Tracking Abilities of Language Models"

13 / 13 papers shown

Title
Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning Yoichi Ishibashi Taro Yano Masafumi Oyamada SyDa LRM 44 0 0 15 May 2025
Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources Zihao Li Shaoxiong Ji Hengyu Luo Jörg Tiedemann CLL 166 0 0 05 Apr 2025
General Reasoning Requires Learning to Reason from the Get-go Seungwook Han Jyothish Pari Samuel J. Gershman Pulkit Agrawal LRM 175 1 0 26 Feb 2025
MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models Vanya Cohen Raymond J. Mooney 47 0 0 15 Feb 2025
Chain and Causal Attention for Efficient Entity Tracking Erwan Fagnou Paul Caillon Blaise Delattre Alexandre Allauzen 26 2 0 07 Oct 2024
EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models Shaoxiong Ji Zihao Li Indraneil Paul Jaakko Paavola Peiqin Lin ... Dayyán O'Brien Hengyu Luo Hinrich Schütze Jörg Tiedemann Barry Haddow CLL 43 3 0 26 Sep 2024
Representational Analysis of Binding in Language Models Qin Dai Benjamin Heinzerling Kentaro Inui 29 1 0 09 Sep 2024
How Does Code Pretraining Affect Language Model Task Performance? Jackson Petty Sjoerd van Steenkiste Tal Linzen 68 8 0 06 Sep 2024
To Code, or Not To Code? Exploring Impact of Code in Pre-training Viraat Aryabumi Yixuan Su Raymond Ma Adrien Morisot Ivan Zhang Acyr Locatelli Marzieh Fadaee Ahmet Üstün Sara Hooker SyDa AI4CE 48 19 0 20 Aug 2024
A Notion of Complexity for Theory of Mind via Discrete World Models X. A. Huang Emanuele La Malfa Samuele Marro Andrea Asperti Anthony Cohn Michael Wooldridge 45 6 0 16 Jun 2024
Entity Tracking in Language Models Najoung Kim Sebastian Schuster 55 16 0 03 May 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 363 12,003 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 398 8,559 0 28 Jan 2022