"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

25 March 2024

Shreya Shankar

Rolando Garcia

J. M. Hellerstein

Aditya G. Parameswaran

ArXiv (abs)PDF HTML

Papers citing ""We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning"

21 / 21 papers shown

Title
OneLabeler: A Flexible System for Building Data Labeling Tools Yu Zhang Yun Wang Haidong Zhang Bin Zhu Si Chen Dongmei Zhang 139 31 0 27 Mar 2022
The CLEAR Benchmark: Continual LEArning on Real-World Imagery Zhiqiu Lin Jia Shi Deepak Pathak Deva Ramanan CLL VLM 192 93 0 17 Jan 2022
Orienting, Framing, Bridging, Magic, and Counseling: How Data Scientists Navigate the Outer Loop of Client Collaborations in Industry and Academia Sean Kross Philip J. Guo 67 33 0 13 May 2021
A Data Quality-Driven View of MLOps Cédric Renggli Luka Rimanic Nezihe Merve Gürel Bojan Karlavs Wentao Wu Ce Zhang AI4TS 44 65 0 15 Feb 2021
Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows Doris Xin Eva Yiwei Wu D. Lee Niloufar Salehi Aditya G. Parameswaran 118 97 0 13 Jan 2021
Challenges in Deploying Machine Learning: a Survey of Case Studies Andrei Paleyes Raoul-Gabriel Urma Neil D. Lawrence 71 406 0 18 Nov 2020
An Empirical Analysis of Backward Compatibility in Machine Learning Systems Megha Srivastava Besmira Nushi Ece Kamar S. Shah Eric Horvitz AAML 88 47 0 11 Aug 2020
Monitoring and explainability of models in production Janis Klaise A. V. Looveren Clive Cox G. Vacanti Alexandru Coca 99 49 0 13 Jul 2020
Hindsight Logging for Model Training Rolando Garcia Eric Liu Vikram Sreekanti Bobby Yan Anusha Dandamudi Joseph E. Gonzalez J. M. Hellerstein Koushik Sen VLM 75 10 0 12 Jun 2020
Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development Angela Lee Doris Xin D. Lee Aditya G. Parameswaran HAI 105 12 0 04 May 2020
Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology Stefan Studer T. Bui C. Drescher A. Hanuschkin Ludwig Winkler S. Peters Klaus-Robert Muller 113 178 0 11 Mar 2020
How do Data Science Workers Collaborate? Roles, Workflows, and Tools Amy X. Zhang Michael J. Muller Dakuo Wang FedML AI4CE 88 261 0 18 Jan 2020
A Survey of Deep Learning Applications to Autonomous Vehicle Control Sampo Kuutti Richard Bowden Yaochu Jin P. Barber Saber Fallah 108 520 0 23 Dec 2019
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI Dakuo Wang Justin D. Weisz Michael J. Muller Parikshit Ram Werner Geyer Casey Dugan Y. Tausczik Horst Samulowitz Alexander G. Gray 224 313 0 05 Sep 2019
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift Yaniv Ovadia Emily Fertig Jie Jessie Ren Zachary Nado D. Sculley Sebastian Nowozin Joshua V. Dillon Balaji Lakshminarayanan Jasper Snoek UQCV 187 1,705 0 06 Jun 2019
Improving fairness in machine learning systems: What do industry practitioners need? Kenneth Holstein Jennifer Wortman Vaughan Hal Daumé Miroslav Dudík Hanna M. Wallach FaML HAI 255 773 0 13 Dec 2018
A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective Yuji Roh A. Mishra Steven Euijong Whang 84 682 0 08 Nov 2018
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift Stephan Rabanser Stephan Günnemann Zachary Chase Lipton 68 371 0 29 Oct 2018
Datasheets for Datasets Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna M. Wallach Hal Daumé Kate Crawford 292 2,201 0 23 Mar 2018
Snorkel: Rapid Training Data Creation with Weak Supervision Alexander Ratner Stephen H. Bach Henry R. Ehrenberg Jason Alan Fries Sen Wu Christopher Ré 83 1,032 0 28 Nov 2017
Clipper: A Low-Latency Online Prediction Serving System D. Crankshaw Xin Wang Giulio Zhou Michael Franklin Joseph E. Gonzalez Ion Stoica 72 679 0 09 Dec 2016