Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

27 March 2020

Joelle Pineau

Philippe Vincent-Lamarre

Papers citing "Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)"

50 / 60 papers shown

Title
Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data Poli A. Nemkova Solomon Ubani Mark V. Albert AILaw 40 0 0 15 May 2025
Rethink Repeatable Measures of Robot Performance with Statistical Query Bowen Weng L. Capito Guillermo A. Castillo Dylan Khor 29 0 0 13 May 2025
Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis Nikita Ravi Abhinav Goel James C. Davis George K. Thiruvathukal 51 0 0 06 May 2025
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning Minju Seo Jinheon Baek Seongyun Lee Sung Ju Hwang AI4CE 44 1 0 24 Apr 2025
LimeSoDa: A Dataset Collection for Benchmarking of Machine Learning Regressors in Digital Soil Mapping J. Schmidinger S. Vogel V. Barkov A.-D. Pham R. Gebbers ... P. Rosso M. M. Costa R. S. Zandonadi J. Wetterlind M. Atzmueller 63 0 0 27 Feb 2025
Beyond Release: Access Considerations for Generative AI Systems Irene Solaiman Rishi Bommasani Dan Hendrycks Ariel Herbert-Voss Yacine Jernite Aviya Skowron Andrew Trask 77 1 0 23 Feb 2025
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark Zachary S. Siegel Sayash Kapoor Nitya Nagdir Benedikt Stroebl Arvind Narayanan 39 9 0 17 Sep 2024
Saliency Detection in Educational Videos: Analyzing the Performance of Current Models, Identifying Limitations and Advancement Directions Evelyn Navarrete Ralph Ewerth Anett Hoppe 31 0 0 08 Aug 2024
Generalizability of experimental studies Federico Matteucci Vadim Arzamasov Jose Cribeiro-Ramallo Marco Heyden Konstantin Ntounas Klemens Bohm 50 0 0 25 Jun 2024
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers Harald Semmelrock Tony Ross-Hellauer Simone Kopeinik Dieter Theiler Armin Haberl Stefan Thalmann Dominik Kowald 65 7 0 20 Jun 2024
Repeatable and Reliable Efforts of Accelerated Risk Assessment L. Capito Guillermo A. Castillo Bowen Weng 37 2 0 30 May 2024
A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden Vishal Purohit Wenxin Jiang Akshath R. Ravikiran James C. Davis 40 1 0 29 Apr 2024
From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap Tianqi Kou 44 0 0 19 Apr 2024
Supervised machine learning for microbiomics: bridging the gap between current and best practices Natasha K. Dudek Mariam Chakhvadze Saba Kobakhidze Omar Kantidze Yuriy Gankin LM&MA 42 2 0 27 Feb 2024
SzCORE: A Seizure Community Open-source Research Evaluation framework for the validation of EEG-based automated seizure detection algorithms Jonathan Dan U. Pale Alireza Amirshahi William Cappelletti T. Ingolfsson ... Adriano Bernini Luca Benini S. Beniczky David Atienza P. Ryvlin 29 7 0 20 Feb 2024
Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization Liang Zhang Junchi Yang Amin Karbasi Niao He 34 2 0 26 Oct 2023
torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP Yoshitomo Matsubara VLM 34 1 0 26 Oct 2023
Reproducibility in Machine Learning-Driven Research Harald Semmelrock Simone Kopeinik Dieter Theiler Tony Ross-Hellauer Dominik Kowald AI4CE 28 15 0 19 Jul 2023
LOB-Based Deep Learning Models for Stock Price Trend Prediction: A Benchmark Study Matteo Prata Giuseppe Masi Leonardo Berti Viviana Arrigoni Andrea Coletta Irene Cannistraci Svitlana Vyetrenko Paola Velardi N. Bartolini 29 8 0 05 Jul 2023
Statistical Indistinguishability of Learning Algorithms Alkis Kalavasis Amin Karbasi Shay Moran Grigoris Velegkas 25 16 0 23 May 2023
List and Certificate Complexities in Replicable Learning P. Dixon A. Pavan Jason Vander Woude N. V. Vinodchandran 32 12 0 05 Apr 2023
When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP Sara Papi Marco Gaido Andrea Pilzer Matteo Negri 59 10 0 28 Mar 2023
An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry Wenxin Jiang Nicholas Synovic Matt Hyatt Taylor R. Schorlemmer R. Sethi Yung-Hsiang Lu George K. Thiruvathukal James C. Davis 33 65 0 05 Mar 2023
Replicable Clustering Hossein Esfandiari Amin Karbasi Vahab Mirrokni Grigoris Velegkas Felix Y. Zhou 37 13 0 20 Feb 2023
Caching and Reproducibility: Making Data Science experiments faster and FAIRer M. Schubotz Ankit Satpute André Greiner-Petter Akiko Aizawa Bela Gipp 14 2 0 08 Nov 2022
Artificial intelligence in government: Concepts, standards, and a unified framework Vince J. Straub Deborah Morgan Jonathan Bright Helen Z. Margetts AI4TS 38 32 0 31 Oct 2022
A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges Mario Alfonso Prado-Romero Bardh Prenkaj Giovanni Stilo F. Giannotti CML 36 30 0 21 Oct 2022
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements Leandro von Werra Lewis Tunstall A. Thakur A. Luccioni Tristan Thrush ... Julien Chaumond Margaret Mitchell Alexander M. Rush Thomas Wolf Douwe Kiela ELM 25 24 0 30 Sep 2022
When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Development Nghia Duong-Trung Stefan Born Jong Woo Kim M. Schermeyer Katharina Paulick ... Thorben Werner Randolf Scholz Lars Schmidt-Thieme Peter Neubauer Ernesto Martinez 36 20 0 02 Sep 2022
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage Kurt Shuster Jing Xu M. Komeili Da Ju Eric Michael Smith ... Naman Goyal Arthur Szlam Y-Lan Boureau Melanie Kambadur Jason Weston LM&Ro KELM 37 235 0 05 Aug 2022
Innovations in Neural Data-to-text Generation: A Survey Mandar Sharma Ajay K. Gogineni Naren Ramakrishnan 36 10 0 25 Jul 2022
Leakage and the Reproducibility Crisis in ML-based Science Sayash Kapoor Arvind Narayanan 25 177 0 14 Jul 2022
Open High-Resolution Satellite Imagery: The WorldStrat Dataset -- With Application to Super-Resolution Julien Cornebise Ivan Orsolic F. Kalaitzis 27 54 0 13 Jul 2022
Long-term Reproducibility for Neural Architecture Search David Towers M. Forshaw Amir Atapour-Abarghouei A. Mcgough 27 1 0 11 Jul 2022
The "Collections as ML Data" Checklist for Machine Learning & Cultural Heritage Benjamin Charles Germain Lee VLM 16 7 0 06 Jul 2022
The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality Rex Chen Fei Fang Norman M. Sadeh 37 8 0 23 Jun 2022
Towards Better User Studies in Computer Graphics and Vision Zoya Bylinskii L. Herman Aaron Hertzmann Stefanie Hutka Yile Zhang 28 13 0 23 Jun 2022
The Fallacy of AI Functionality Inioluwa Deborah Raji Indra Elizabeth Kumar Aaron Horowitz Andrew D. Selbst 34 180 0 20 Jun 2022
SoK: The Impact of Unlabelled Data in Cyberthreat Detection Giovanni Apruzzese Pavel Laskov A.T. Tastemirova 38 29 0 18 May 2022
ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild Chirag Raman Jose Vargas-Quiros Stephanie Tan Ashraful Islam Ekin Gedik Hayley Hung 19 8 0 10 May 2022
Deep Learning Reproducibility and Explainable AI (XAI) Anastasia-Maria Leventi-Peetz T. Östreich 19 9 0 23 Feb 2022
Towards a consistent interpretation of AIOps models Yingzhe Lyu Gopi Krishnan Rajbahadur Dayi Lin Boyuan Chen Zhen Ming Z. Jiang AI4CE 22 20 0 04 Feb 2022
Towards Training Reproducible Deep Learning Models Boyuan Chen Mingzhi Wen Yong Shi Dayi Lin Gopi Krishnan Rajbahadur Zhen Ming Z. Jiang SyDa 23 37 0 04 Feb 2022
Reproducibility in Learning R. Impagliazzo Rex Lei T. Pitassi Jessica Sorrell 32 8 0 20 Jan 2022
Automated Deep Learning: Neural Architecture Search Is Not the End Xuanyi Dong D. Kedziora Katarzyna Musial Bogdan Gabrys 31 26 0 16 Dec 2021
CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability Martin Mundt Steven Braun Quentin Delfosse Kristian Kersting 27 35 0 07 Oct 2021
Trustworthy AI: From Principles to Practices Bo-wen Li Peng Qi Bo Liu Shuai Di Jingen Liu Jiquan Pei Jinfeng Yi Bowen Zhou 119 357 0 04 Oct 2021
Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms Albert Jiménez Sanfiz Mohamed Akrout OOD AAML 22 8 0 30 Aug 2021
Deep Reinforcement Learning at the Edge of the Statistical Precipice Rishabh Agarwal Max Schwarzer Pablo Samuel Castro Aaron Courville Marc G. Bellemare OffRL 61 639 0 30 Aug 2021
Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study Tianlong Chen Kaixiong Zhou Keyu Duan Wenqing Zheng Peihao Wang Xia Hu Zhangyang Wang AAML GNN 32 63 0 24 Aug 2021