Constructing a BPE Tokenization DFA

13 May 2024

Papers citing "Constructing a BPE Tokenization DFA"

2 / 2 papers shown

Title
Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo João Loula Benjamin LeBrun Li Du Ben Lipkin Clemente Pasti ... Ryan Cotterel Vikash K. Mansinghka Alexander K. Lew Tim Vieira Timothy J. O'Donnell 47 2 0 17 Apr 2025
Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection Yuxi Li Yi Liu Gelei Deng Ying Zhang Wenjia Song Ling Shi Kailong Wang Yuekang Li Yang Liu Haoyu Wang 52 21 0 15 Apr 2024