EXECUTE: A Multilingual Benchmark for LLM Token Understanding

23 May 2025

Papers citing "EXECUTE: A Multilingual Benchmark for LLM Token Understanding"

9 / 9 papers shown

Title
Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier John Dang Shivalika Singh Daniel D'souza Arash Ahmadian Alejandro Salamanca ... Nick Frosst Marzieh Fadaee Beyza Ermis Ahmet Üstün Sara Hooker ELM OSLM MoE 126 41 0 05 Dec 2024
The Impact of Visual Information in Chinese Characters: Evaluating Large Models' Ability to Recognize and Utilize Radicals Xiaofeng Wu Karl Stratos Wei Xu 40 1 0 11 Oct 2024
CUTE: Measuring LLMs' Understanding of Their Tokens Lukas Edman Helmut Schmid Alexander Fraser 47 6 0 23 Sep 2024
Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction S. Kwon Gagan Bhatia El Moatez Billah Nagoudi Muhammad Abdul-Mageed 66 18 0 13 Dec 2023
On the (In)Effectiveness of Large Language Models for Chinese Text Correction Hai-Tao Zheng Haojing Huang Shirong Ma Yong Jiang Yongqian Li F. Zhou Haitao Zheng Qingyu Zhou 57 46 0 18 Jul 2023
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? Ronen Eldan Yuan-Fang Li SyDa LRM 54 258 0 12 May 2023
Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training Jing-ling Huang Zhengxuan Wu Kyle Mahowald Christopher Potts 43 14 0 19 Dec 2022
What do tokens know about their characters and how do they know it? Ayush Kaushal Kyle Mahowald 69 31 0 06 Jun 2022
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens Itay Itzhak Omer Levy 36 20 0 25 Aug 2021