Scalable Influence and Fact Tracing for Large Language Model Pretraining

22 October 2024

Papers citing "Scalable Influence and Fact Tracing for Large Language Model Pretraining"

23 / 23 papers shown

Title
Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI Elisa Nguyen Johannes Bertram Evgenii Kortukov Jean Y. Song Seong Joon Oh TDI 509 2 0 25 Sep 2024
Gecko: Versatile Text Embeddings Distilled from Large Language Models Jinhyuk Lee Zhuyun Dai Xiaoqi Ren Blair Chen Daniel Cer ... Aditya Kusupati Prateek Jain Siddhartha Reddy Jonnalagadda Ming-Wei Chang Iftekhar Naim RALM VLM SyDa 103 51 0 29 Mar 2024
Understanding In-Context Learning via Supportive Pretraining Data Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz Tianlu Wang AIMat 106 38 0 26 Jun 2023
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models Zorik Gekhman Jonathan Herzig Roee Aharoni Chen Elkind Idan Szpektor HILM ELM 103 79 0 18 May 2023
Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs Kelvin Guu Albert Webson Ellie Pavlick Lucas Dixon Ian Tenney Tolga Bolukbasi TDI 110 36 0 14 Mar 2023
Large Language Models Struggle to Learn Long-Tail Knowledge Nikhil Kandpal H. Deng Adam Roberts Eric Wallace Colin Raffel RALM KELM 131 419 0 15 Nov 2022
ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data Xiaochuang Han Yulia Tsvetkov 90 31 0 25 May 2022
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 155 45 0 23 May 2022
$Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$$ Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$ Adam Roberts Hyung Won Chung Anselm Levskaya Gaurav Mishra James Bradbury ... Brennan Saeta Ryan Sepassi A. Spiridonov Joshua Newlan Andrea Gesmundo ALM 126 199 0 31 Mar 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
First is Better Than Last for Language Data Influence Chih-Kuan Yeh Ankur Taly Mukund Sundararajan Frederick Liu Pradeep Ravikumar TDI 125 22 0 24 Feb 2022
Datamodels: Predicting Predictions from Training Data Andrew Ilyas Sung Min Park Logan Engstrom Guillaume Leclerc Aleksander Madry TDI 133 142 0 01 Feb 2022
Scaling Up Influence Functions Andrea Schioppa Polina Zablotskaia David Vilar Artem Sokolov TDI 110 106 0 06 Dec 2021
Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates Xiaochuang Han Yulia Tsvetkov TDI 91 31 0 07 Oct 2021
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality William Timkey Marten van Schijndel 298 116 0 09 Sep 2021
Estimating Training Data Influence by Tracing Gradient Descent G. Pruthi Frederick Liu Mukund Sundararajan Satyen Kale TDI 110 419 0 19 Feb 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 192 1,847 0 26 Nov 2019
Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie Adina Williams Emily Dinan Joey Tianyi Zhou Jason Weston Douwe Kiela 127 1,011 0 31 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 506 20,376 0 23 Oct 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 587 2,680 0 03 Sep 2019
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 209 3,534 0 19 Aug 2018
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 86 1,053 0 11 Apr 2018
Understanding Black-box Predictions via Influence Functions Pang Wei Koh Percy Liang TDI 227 2,910 0 14 Mar 2017