LSHBloom: Memory-efficient, Extreme-scale Document Deduplication

v1v2 (latest)

LSHBloom: Memory-efficient, Extreme-scale Document Deduplication

6 November 2024

Robert Underwood

Carlo Siebenschuh

Alexander Brace

ArXiv (abs)PDF HTML

Papers citing "LSHBloom: Memory-efficient, Extreme-scale Document Deduplication"

12 / 12 papers shown

Title
A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery Yu Zhang Xiusi Chen Bowen Jin Sheng Wang Shuiwang Ji Wei Wang Jiawei Han 140 43 0 16 Jun 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 242 513 0 13 Mar 2024
TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs Tanmay Rajore Nishanth Chandran Sunayana Sitaram Divya Gupta Rahul Sharma Kashish Mittal Manohar Swaminathan 113 16 0 01 Mar 2024
Evading Data Contamination Detection for Language Models is (too) Easy Jasper Dekoninck Mark Niklas Muller Maximilian Baader Marc Fischer Martin Vechev 197 20 0 05 Feb 2024
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research Luca Soldaini Rodney Michael Kinney Akshita Bhagia Dustin Schwenk David Atkinson ... Hanna Hajishirzi Iz Beltagy Dirk Groeneveld Jesse Dodge Kyle Lo 127 282 0 31 Jan 2024
Nougat: Neural Optical Understanding for Academic Documents Lukas Blecher Guillem Cucurull Thomas Scialom Robert Stojnic ViT 66 120 0 25 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 537 12,130 0 18 Jul 2023
UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learner Yiwen Ye Yutong Xie Jianpeng Zhang Ziyang Chen Yong-quan Xia SSL 103 43 0 07 Apr 2023
SemDeDup: Data-efficient learning at web-scale through semantic deduplication Amro Abbas Kushal Tirumala Daniel Simig Surya Ganguli Ari S. Morcos 77 183 0 16 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.7K 13,545 0 27 Feb 2023
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 371 639 0 14 Jul 2021
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data Guillaume Wenzek Marie-Anne Lachaux Alexis Conneau Vishrav Chaudhary Francisco Guzmán Armand Joulin Edouard Grave 124 658 0 01 Nov 2019