SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,925 papers shown

Title
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset Sneha Kudugunta Isaac Caswell Biao Zhang Xavier Garcia Christopher A. Choquette-Choo ... Derrick Xin Aditya Kusupati Romi Stella Ankur Bapna Orhan Firat 79 120 0 09 Sep 2023
Mask-CTC-based Encoder Pre-training for Streaming End-to-End Speech Recognition Huaibo Zhao Yosuke Higuchi Yusuke Kida Tetsuji Ogawa Tetsunori Kobayashi 40 1 0 09 Sep 2023
Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen Yilun Huang Zhijian Ma Hesen Chen Xuchen Pan ... Zhaoyang Liu Jinyang Gao Yaliang Li Bolin Ding Jingren Zhou SyDa VLM 42 32 0 05 Sep 2023
TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models Shangguan Yuan Haichuan Yang Danni Li Chunyang Wu Yassir Fathullah ... Junteng Jia Jay Mahadeokar Xin Lei Michael Seltzer Vikas Chandra 55 2 0 05 Sep 2023
One Wide Feedforward is All You Need Telmo Pires António V. Lopes Yannick Assogba Hendra Setiawan 53 12 0 04 Sep 2023
Towards Foundational AI Models for Additive Manufacturing: Language Models for G-Code Debugging, Manipulation, and Comprehension Anushrut Jignasu Kelly O. Marshall Baskar Ganapathysubramanian Aditya Balu Chinmay Hegde A. Krishnamurthy ELM AI4CE 35 6 0 04 Sep 2023
Multilingual Text Representation Fahim Faisal 32 0 0 02 Sep 2023
RepCodec: A Speech Representation Codec for Speech Tokenization Zhichao Huang Chutong Meng Tom Ko 27 25 0 31 Aug 2023
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants Lucas Bandarkar Davis Liang Benjamin Muller Mikel Artetxe Satya Narayan Shukla Don Husa Naman Goyal Abhinandan Krishnan Luke Zettlemoyer Madian Khabsa 39 135 0 31 Aug 2023
Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Texts Thanh Thi Nguyen Campbell Wilson Janis Dalins 32 24 0 28 Aug 2023
ANER: Arabic and Arabizi Named Entity Recognition using Transformer-Based Approach Abdelrahman Boda Sadallah Omar Ahmed Shimaa S. Mohamed Omar Hatem Doaa Hesham A. Yousef 22 2 0 28 Aug 2023
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation Pengzhi Gao Ruiqing Zhang Zhongjun He Hua Wu Haifeng Wang 30 4 0 28 Aug 2023
Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level Daniel Deutsch Juraj Juraska M. Finkelstein and Markus Freitag 83 11 0 25 Aug 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 63 1,941 0 24 Aug 2023
Cabrita: closing the gap for foreign languages Celio H. N. Larcher Marcos Piau Paulo Finardi P. Gengo P. Esposito Vinicius Fernandes Caridá CLL 29 20 0 23 Aug 2023
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge Minsu Kim Jeong Hun Yeo J. Choi Y. Ro 36 16 0 18 Aug 2023
Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We? A. Mastropaolo M. D. Penta Gabriele Bavota 38 7 0 17 Aug 2023
Lightweight Adaptation of Neural Language Models via Subspace Embedding Amit Kumar Jaiswal Haiming Liu 36 2 0 16 Aug 2023
BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition Vera Pavlova M. Makhlouf 26 3 0 16 Aug 2023
Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals Running Zhao Jiang-Tao Luca Yu Haiying Zhao Edith C.H. Ngai 46 4 0 16 Aug 2023
SOTASTREAM: A Streaming Approach to Machine Translation Training Matt Post Thamme Gowda Roman Grundkiewicz Huda Khayrallah Rohit Jain Marcin Junczys-Dowmunt 43 5 0 14 Aug 2023
O-1: Self-training with Oracle and 1-best Hypothesis M. Baskar Andrew Rosenberg Bhuvana Ramabhadran Kartik Audhkhasi VLM 31 0 0 14 Aug 2023
A Novel Ehanced Move Recognition Algorithm Based on Pre-trained Models with Positional Embeddings H. Wen Jie Wang Xiaodong Qiao 28 0 0 14 Aug 2023
A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation Ramakrishna Appicharla Baban Gain Santanu Pal Asif Ekbal 40 1 0 11 Aug 2023
Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT Jing Yang Cong Liu Wendy Deng Dangwei Wu Chunhua Weng Yunyun Zhou Kai Wang 27 20 0 11 Aug 2023
IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer Keqi Fan Xiaohao Cai M. Niranjan MedIm ViT 29 3 0 10 Aug 2023
Exploring Linguistic Similarity and Zero-Shot Learning for Multilingual Translation of Dravidian Languages Danish Ebadulla Rahul Raman S. Natarajan Hridhay Kiran Shetty A. Shenoy 29 1 0 10 Aug 2023
Negative Lexical Constraints in Neural Machine Translation Josef Jon Duvsan Varivs Michal Novák João Paulo Aires Ondrej Bojar 29 1 0 07 Aug 2023
Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining Nour Eddine Zekaoui Siham Yousfi Maryem Rhanoui M. Mikram 32 3 0 07 Aug 2023
Spanish Pre-trained BERT Model and Evaluation Data J. Cañete Gabriel Chaperon Rodrigo Fuentes Jou-Hui Ho Hojin Kang Jorge Pérez 32 659 0 06 Aug 2023
N-gram Boosting: Improving Contextual Biasing with Normalized N-gram Targets Wang Yau Li Shreekantha Nadig K. Chang Zafarullah Mahmood Riqiang Wang Simon Vandieken Jonas Robertson Frederic Mailhot 33 0 0 04 Aug 2023
Federated Representation Learning for Automatic Speech Recognition Guruprasad V Ramesh Gopinath Chennupati Milind Rao Anit Kumar Sahu Ariya Rastrow J. Droppo 37 0 0 03 Aug 2023
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation Minsu Kim J. Choi Dahun Kim Y. Ro 52 12 0 03 Aug 2023
ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders Shawn Xu Ling Yang Christopher J. Kelly M. Sieniek Timo Kohlberger ... Shruthi Prabhakara Daniel Golden Rory Pilgrim Krish Eswaran Andrew Sellergren LM&MA MedIm 39 52 0 02 Aug 2023
CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code Nadezhda Chirkova Sergey Troshin 52 8 0 01 Aug 2023
SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation Haiyue Song Raj Dabre Chenhui Chu Sadao Kurohashi Eiichiro Sumita 21 3 0 31 Jul 2023
BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering Khiem Vinh Tran Kiet Van Nguyen Ngan Luu-Thuy Nguyen ViT 35 2 0 28 Jul 2023
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis Izzeddin Gur Hiroki Furuta Austin Huang Mustafa Safdari Yutaka Matsuo Douglas Eck Aleksandra Faust LM&Ro LLMAG 59 203 0 24 Jul 2023
Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding Suyoun Kim Akshat Shrivastava Duc Le Ju Lin Ozlem Kalinli M. Seltzer AuLLM 38 2 0 22 Jul 2023
Incorporating Human Translator Style into English-Turkish Literary Machine Translation Zeynep Yi̇rmi̇beşoğlu Olgun Dursun Harun Dalli Mehmet Şahin Ena Hodzik Sabri Gürses Tunga Güngör 31 0 0 21 Jul 2023
Topic Identification For Spontaneous Speech: Enriching Audio Features With Embedded Linguistic Information Dejan Porjazovski Tamás Grósz M. Kurimo 18 1 0 21 Jul 2023
Prompting Large Language Models with Speech Recognition Abilities Yassir Fathullah Chunyang Wu Egor Lakomkin Junteng Jia Yuan Shangguan ... Wenhan Xiong Jay Mahadeokar Ozlem Kalinli Christian Fuegen M. Seltzer AuLLM 35 135 0 21 Jul 2023
Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models Michael Gunther Louis Milliken Jonathan Geuter Georgios Mastrapas Bo Wang Han Xiao RALM 50 30 0 20 Jul 2023
Gradient Sparsification For Masked Fine-Tuning of Transformers J. Ó. Neill Sourav Dutta 27 0 0 19 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 150 11,259 0 18 Jul 2023
Gloss Attention for Gloss-free Sign Language Translation Aoxiong Yin Tianyun Zhong Lilian H. Y. Tang Weike Jin Tao Jin Zhou Zhao SLR 34 38 0 14 Jul 2023
Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Filling Hengguan Huang Jagadeesh Balam Boris Ginsburg 31 4 0 13 Jul 2023
Copy Is All You Need Tian Lan Deng Cai Yan Wang Heyan Huang Xian-Ling Mao 35 27 0 13 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 38 41 0 12 Jul 2023
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution Mostafa Dehghani Basil Mustafa Josip Djolonga Jonathan Heek Matthias Minderer ... Avital Oliver Piotr Padlewski A. Gritsenko Mario Luvcić N. Houlsby ViT 31 107 0 12 Jul 2023