Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

18 April 2021

Dirk Groeneveld

Papers citing "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus"

50 / 84 papers shown

Title
Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning Shaurya Sharthak Vinayak Pahalwan Adithya Kamath Adarsh Shirawalmath CLL VLM 45 0 0 14 May 2025
Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs Sai Krishna Mendu Harish Yenala Aditi Gulati Shanu Kumar Parag Agrawal 36 0 0 04 May 2025
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference Zhenyu Zhang Zechun Liu Yuandong Tian Harshit Khaitan Zhigang Wang Steven Li 68 0 0 28 Apr 2025
Bias Analysis and Mitigation through Protected Attribute Detection and Regard Classification Takuma Udagawa Yang Zhao H. Kanayama Bishwaranjan Bhattacharjee 33 0 0 19 Apr 2025
ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selection Xiaoxuan Zhu Zhouhong Gu Baiqian Wu Suhang Zheng Tao Wang Tianyu Li Hongwei Feng Yanghua Xiao 42 0 0 01 Apr 2025
Un-Straightening Generative AI: How Queer Artists Surface and Challenge the Normativity of Generative AI Models Jordan Taylor Joel Mire Franchesca Spektor Alicia DeVrio Maarten Sap Haiyi Zhu Sarah E Fox 58 1 0 12 Mar 2025
Re-evaluating Theory of Mind evaluation in large language models Jennifer Hu Felix Sosa T. Ullman 45 0 0 28 Feb 2025
More for Keys, Less for Values: Adaptive KV Cache Quantization Mohsen Hariri Lam Nguyen Sixu Chen Shaochen Zhong Qifan Wang Xia Hu Xiaotian Han V. Chaudhary MQ 48 0 0 24 Feb 2025
Exploring and Controlling Diversity in LLM-Agent Conversation Kuanchao Chu Yi-Pei Chen Hideki Nakayama LLMAG 50 1 0 24 Feb 2025
MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs Andreas Opedal Haruki Shirakami Bernhard Schölkopf Abulhair Saparov Mrinmaya Sachan LRM 57 1 0 17 Feb 2025
Privacy-Preserving Dataset Combination Keren Fuentes Mimee Xu Irene Chen 43 0 0 09 Feb 2025
LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation Eunsu Kim Juyoung Suk Seungone Kim Niklas Muennighoff Dongkwan Kim Alice H. Oh ELM 91 1 0 31 Dec 2024
BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration Yuzong Chen Ahmed F. AbouElhamayed Xilai Dai Yang Wang Marta Andronic George A. Constantinides Mohamed S. Abdelfattah MQ 108 1 0 18 Nov 2024
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training Elia Cunegatti Leonardo Lucio Custode Giovanni Iacca 52 0 0 11 Nov 2024
Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions Yujuan Fu Özlem Uzuner Meliha Yetisgen Fei Xia 61 4 0 24 Oct 2024
Self-calibration for Language Model Quantization and Pruning Miles Williams G. Chrysostomou Nikolaos Aletras MQ 180 0 0 22 Oct 2024
ToW: Thoughts of Words Improve Reasoning in Large Language Models Zhikun Xu Ming shen Jacob Dineen Zhaonan Li Xiao Ye Shijie Lu Aswin Rrv Chitta Baral Ben Zhou LRM 185 1 0 21 Oct 2024
One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks Fangru Lin Shaoguang Mao Emanuele La Malfa Valentin Hofmann Adrian de Wynter Jing Yao Si-Qing Chen Michael Wooldridge Furu Wei Furu Wei 51 2 0 14 Oct 2024
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? HyoJung Han Akiko Eriguchi Haoran Xu Hieu T. Hoang Marine Carpuat Huda Khayrallah VLM 40 2 0 12 Oct 2024
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling David Grangier Simin Fan Skyler Seto Pierre Ablin 44 3 0 30 Sep 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 38 39 0 06 Jun 2024
DEPTH: Discourse Education through Pre-Training Hierarchically Zachary Bamberger Ofek Glick Chaim Baskin Yonatan Belinkov 67 0 0 13 May 2024
Recall Them All: Retrieval-Augmented Language Models for Long Object List Extraction from Long Documents Sneha Singhania Simon Razniewski Gerhard Weikum RALM 34 1 0 04 May 2024
From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets Manuel Tonneau Diyi Liu Samuel Fraiberger Ralph Schroeder Scott A. Hale Paul Röttger 37 5 0 27 Apr 2024
Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? Shayne Longpre Robert Mahari Naana Obeng-Marnu William Brannon Tobin South Katy Gero Sandy Pentland Jad Kabbara 66 5 0 19 Apr 2024
Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models Hyunbyung Park Sukyung Lee Gyoungjin Gim Yungi Kim Dahyun Kim Chanjun Park VLM 42 0 0 28 Mar 2024
Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models Jiahao Ying Yixin Cao Yushi Bai Qianru Sun Bo Wang Wei Tang Zhaojun Ding Yizhe Yang Xuanjing Huang Shuicheng Yan KELM 26 6 0 19 Feb 2024
Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens Jiacheng Liu Sewon Min Luke Zettlemoyer Yejin Choi Hannaneh Hajishirzi 51 50 0 30 Jan 2024
Potential Societal Biases of ChatGPT in Higher Education: A Scoping Review Ming Li Ariunaa Enkhtur B. Yamamoto Fei Cheng Lilan Chen AI4CE 34 3 0 24 Nov 2023
Oasis: Data Curation and Assessment System for Pretraining of Large Language Models Tong Zhou Yubo Chen Pengfei Cao Kang Liu Jun Zhao Shengping Liu 29 3 0 21 Nov 2023
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples Shuo Yang Wei-Lin Chiang Lianmin Zheng Joseph E. Gonzalez Ion Stoica ALM 27 111 0 08 Nov 2023
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models Ziqiao Ma Jacob Sansom Run Peng Joyce Chai 47 17 0 30 Oct 2023
A Comprehensive Evaluation of Tool-Assisted Generation Strategies Alon Jacovi Avi Caciularu Jonathan Herzig Roee Aharoni Bernd Bohnet Mor Geva ELM 43 6 0 16 Oct 2023
Position: Key Claims in LLM Research Have a Long Tail of Footnotes Anna Rogers A. Luccioni 53 19 0 14 Aug 2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore Sewon Min Suchin Gururangan Eric Wallace Hannaneh Hajishirzi Noah A. Smith Luke Zettlemoyer AILaw 22 63 0 08 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 123 11,099 0 18 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 78 304 0 17 Jul 2023
Large Language Models as Batteries-Included Zero-Shot ESCO Skills Matchers Benjamin Clavié Guillaume Soulié 26 11 0 07 Jul 2023
Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extraction Simone Scaboro Beatrice Portelli Emmanuele Chersoni Enrico Santus Giuseppe Serra 24 8 0 08 Jun 2023
Just CHOP: Embarrassingly Simple LLM Compression A. Jha Tom Sherborne Evan Pete Walsh Dirk Groeneveld Emma Strubell Iz Beltagy 30 3 0 24 May 2023
Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models Tim Schott Daniel Furman Shreshta Bhat ELM 35 4 0 23 May 2023
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate Boshi Wang Xiang Yue Huan Sun ELM LRM 46 60 0 22 May 2023
AIwriting: Relations Between Image Generation and Digital Writing S. Rettberg Talan Memmott Jill Walker Rettberg Jason Nelson P. Lichty DiffM 24 1 0 18 May 2023
PaLM 2 Technical Report Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin ... Ce Zheng Wei Zhou Denny Zhou Slav Petrov Yonghui Wu ReLM LRM 128 1,152 0 17 May 2023
Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks Alon Jacovi Avi Caciularu Omer Goldman Yoav Goldberg 17 95 0 17 May 2023
NorBench -- A Benchmark for Norwegian Language Models David Samuel Andrey Kutuzov Samia Touileb Erik Velldal Lilja Ovrelid Egil Rønningstad Elina Sigdel Anna Palatkina 26 23 0 06 May 2023
The MiniPile Challenge for Data-Efficient Language Models Jean Kaddour MoE ALM 26 40 0 17 Apr 2023
Sociocultural knowledge is needed for selection of shots in hate speech detection tasks Antonis Maronikolakis Abdullatif Köksal Hinrich Schütze 43 0 0 04 Apr 2023
BloombergGPT: A Large Language Model for Finance Shijie Wu Ozan Irsoy Steven Lu Vadim Dabravolski Mark Dredze Sebastian Gehrmann P. Kambadur David S. Rosenberg Gideon Mann AIFin 85 789 0 30 Mar 2023
Class Cardinality Comparison as a Fermi Problem Shrestha Ghosh Simon Razniewski Gerhard Weikum 45 2 0 08 Mar 2023