Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training

19 December 2022

Papers citing "Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training"

41 / 41 papers shown

Title
EXECUTE: A Multilingual Benchmark for LLM Token Understanding Lukas Edman Helmut Schmid Alexander Fraser 25 0 0 23 May 2025
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models Julie Kallini Shikhar Murty Christopher D. Manning Christopher Potts Róbert Csordás 75 4 0 28 Oct 2024
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 351 2,377 0 09 Nov 2022
Causal Proxy Models for Concept-Based Model Explanations Zhengxuan Wu Karel DÓosterlinck Atticus Geiger Amir Zur Christopher Potts MILM 88 36 0 28 Sep 2022
What do tokens know about their characters and how do they know it? Ayush Kaushal Kyle Mahowald 67 31 0 06 Jun 2022
Automated Crossword Solving Eric Wallace Nicholas Tomlin Albert Xu Kevin Kaichuang Yang Eshaan Pathak Matthew Ginsberg Dan Klein 70 13 0 19 May 2022
AmbiPun: Generating Humorous Puns with Ambiguous Context Anirudh Mittal Yufei Tian Nanyun Peng 47 26 0 04 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 292 3,634 0 02 May 2022
GPT-NeoX-20B: An Open-Source Autoregressive Language Model Sid Black Stella Biderman Eric Hallahan Quentin G. Anthony Leo Gao ... Shivanshu Purohit Laria Reynolds J. Tow Benqi Wang Samuel Weinbach 154 825 0 14 Apr 2022
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP Sabrina J. Mielke Zaid Alyafeai Elizabeth Salesky Colin Raffel Manan Dey ... Arun Raja Chenglei Si Wilson Y. Lee Benoît Sagot Samson Tan 77 148 0 20 Dec 2021
Causal Distillation for Language Models Zhengxuan Wu Atticus Geiger J. Rozner Elisa Kreiss Hanson Lu Thomas Icard Christopher Potts Noah D. Goodman 96 25 0 05 Dec 2021
Inducing Causal Structure for Interpretable Neural Networks Atticus Geiger Zhengxuan Wu Hanson Lu J. Rozner Elisa Kreiss Thomas Icard Noah D. Goodman Christopher Potts CML OOD 56 72 0 01 Dec 2021
Noisy UGC Translation at the Character Level: Revisiting Open-Vocabulary Capabilities and Robustness of Char-Based Models José Carlos Rosales Núnez Guolong Su Djamé Seddah 42 8 0 24 Oct 2021
Why don't people use character-level machine translation? Jindrich Libovický Helmut Schmid Alexander Fraser 102 29 0 15 Oct 2021
Integrating Approaches to Word Representation Yuval Pinter NAI 81 5 0 10 Sep 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 116 3,723 0 03 Sep 2021
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens Itay Itzhak Omer Levy 36 20 0 25 Aug 2021
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization Yi Tay Vinh Q. Tran Sebastian Ruder Jai Gupta Hyung Won Chung Dara Bahri Zhen Qin Simon Baumgartner Cong Yu Donald Metzler 86 158 0 23 Jun 2021
Causal Abstractions of Neural Networks Atticus Geiger Hanson Lu Thomas Icard Christopher Potts NAI CML 66 241 0 06 Jun 2021
ByT5: Towards a token-free future with pre-trained byte-to-byte models Linting Xue Aditya Barua Noah Constant Rami Al-Rfou Sharan Narang Mihir Kale Adam Roberts Colin Raffel 83 501 0 28 May 2021
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation J. Clark Dan Garrette Iulia Turc John Wieting 85 218 0 11 Mar 2021
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language Avia Efrat Uri Shaham D. Kilman Omer Levy ELM 46 18 0 01 Mar 2021
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality Gustavo Aguilar Bryan McCann Tong Niu Nazneen Rajani N. Keskar Thamar Solorio 69 13 0 24 Oct 2020
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters Hicham El Boukkouri Olivier Ferret Thomas Lavergne Hiroshi Noji Pierre Zweigenbaum Junichi Tsujii 101 160 0 20 Oct 2020
Will it Unblend? Yuval Pinter Cassandra L. Jacobs Jacob Eisenstein 38 14 0 18 Sep 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 129 2,724 0 05 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 604 41,736 0 28 May 2020
Injecting Numerical Reasoning Skills into Language Models Mor Geva Ankit Gupta Jonathan Berant AIMat LRM 59 225 0 09 Apr 2020
Byte Pair Encoding is Suboptimal for Language Model Pretraining Kaj Bostrom Greg Durrett 61 209 0 07 Apr 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 367 20,053 0 23 Oct 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 516 24,351 0 26 Jul 2019
Approximate Causal Abstraction Sander Beckers F. Eberhardt Joseph Y. Halpern 52 52 0 27 Jun 2019
Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts Timo Schick Hinrich Schütze 49 48 0 02 Apr 2019
Abstracting Causal Models Sander Beckers Joseph Y. Halpern 67 106 0 10 Dec 2018
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates Taku Kudo 190 1,165 0 29 Apr 2018
Deep contextualized word representations Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer NAI 182 11,542 0 15 Feb 2018
Synthetic and Natural Noise Both Break Neural Machine Translation Yonatan Belinkov Yonatan Bisk 109 741 0 06 Nov 2017
Mimicking Word Embeddings using Subword RNNs Yuval Pinter Robert Guthrie Jacob Eisenstein 58 159 0 21 Jul 2017
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models Minh-Thang Luong Christopher D. Manning VLM 83 373 0 04 Apr 2016
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF Xuezhe Ma Eduard H. Hovy 90 2,651 0 04 Mar 2016
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 195 7,729 0 31 Aug 2015