Jury Learning: Integrating Dissenting Voices into Machine Learning Models

7 February 2022

Kayur Patel

Tatsunori Hashimoto

Papers citing "Jury Learning: Integrating Dissenting Voices into Machine Learning Models"

50 / 89 papers shown

Title
Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest Jakob Schoeffer Maria De-Arteaga Jonathan Elmer 277 0 0 05 Apr 2025
HALO: Human-Aligned End-to-end Image Retargeting with Layered Transformations Yiran Xu Siqi Xie Zhuofang Li Harris Shadmany Yinxiao Li ... Jesse Berent Ming-Hsuan Yang Irfan Essa Jia-Bin Huang Feng Yang VOS 71 0 0 03 Apr 2025
Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification Allison Koenecke Jed Stiglitz David Mimno Matthew Wilkens AILaw ELM 89 0 0 02 Apr 2025
On Benchmarking Human-Like Intelligence in Machines Lance Ying Katherine M. Collins L. Wong Ilia Sucholutsky Ryan Liu Adrian Weller Tianmin Shu Thomas Griffiths Joshua B. Tenenbaum ALM ELM 266 5 0 27 Feb 2025
Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training Mugdha Pandya Nafise Sadat Moosavi Diana Maynard 108 0 0 05 Dec 2024
Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment Andrew Konya Aviv Ovadya K. J. Kevin Feng Quan Ze Chen Lisa Schirch Colin Irwin Amy X. Zhang ALM 71 2 0 15 Nov 2024
LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output Elise Karinshak Amanda Hu Kewen Kong Vishwanatha Rao Jingren Wang Jindong Wang Yi Zeng 75 2 0 09 Nov 2024
Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators Claire Jo Miki Wesołowska Magdalena Wojcieszak 55 5 0 06 Nov 2024
Intuitions of Compromise: Utilitarianism vs. Contractualism Jared Moore Yejin Choi Sydney Levine 43 0 0 07 Oct 2024
Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymaking K. J. Kevin Feng Inyoung Cheong Quan Ze Chen Amy X. Zhang 78 3 0 13 Sep 2024
User-Driven Value Alignment: Understanding Users' Perceptions and Strategies for Addressing Biased and Discriminatory Statements in AI Companions Xianzhe Fan Qing Xiao Xuhui Zhou Jiaxin Pei Maarten Sap Zhicong Lu Hong Shen 75 7 0 01 Sep 2024
Beyond Preferences in AI Alignment Tan Zhi-Xuan Micah Carroll Matija Franklin Hal Ashton 105 17 0 30 Aug 2024
How will advanced AI systems impact democracy? Christopher Summerfield Lisa Argyle Michiel Bakker Teddy Collins Esin Durmus ... Elizabeth Seger Divya Siddarth Henrik Skaug Sætra MH Tessler M. Botvinick 65 2 0 27 Aug 2024
Improving Context-Aware Preference Modeling for Language Models Silviu Pitis Ziang Xiao Nicolas Le Roux Alessandro Sordoni 62 8 0 20 Jul 2024
Position: Measure Dataset Diversity, Don't Just Claim It Dora Zhao Jerone T. A. Andrews Orestis Papakyriakopoulos Alice Xiang 69 16 0 11 Jul 2024
LEMoN: Label Error Detection using Multimodal Neighbors Haoran Zhang Aparna Balagopalan Nassim Oufattole Hyewon Jeong Yan Wu Jiacheng Zhu Marzyeh Ghassemi 75 0 0 10 Jul 2024
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation Katherine M. Collins Najoung Kim Yonatan Bitton Verena Rieser Shayegan Omidshafiei ... Gang Li Adrian Weller Junfeng He Deepak Ramachandran Krishnamurthy Dvijotham EGVM 47 3 0 24 Jun 2024
Pareto-Optimal Learning from Preferences with Hidden Context Ryan Boldi Li Ding Lee Spector S. Niekum 75 6 0 21 Jun 2024
Supporting Human Raters with the Detection of Harmful Content using Large Language Models Kurt Thomas Patrick Gage Kelley David Tao Sarah Meiklejohn Owen Vallis Shunwen Tan Blaz Bratanic Felipe Tiengo Ferreira Vijay Eranti Elie Bursztein 65 2 0 18 Jun 2024
Collective Constitutional AI: Aligning a Language Model with Public Input Saffron Huang Divya Siddarth Liane Lovitt Thomas I. Liao Esin Durmus Alex Tamkin Deep Ganguli ELM 86 72 0 12 Jun 2024
Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks Justin Zhao Flor Miriam Plaza del Arco Amanda Cercas Curry Amanda Cercas Curry ELM ALM 60 0 0 12 Jun 2024
A Taxonomy of Challenges to Curating Fair Datasets Dora Zhao M. Scheuerman Pooja Chitre Jerone T. A. Andrews Georgia Panagiotidou Shawn Walker Kathleen H. Pine Alice Xiang 64 2 0 10 Jun 2024
Counterspeakers' Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hate Jimin Mun Cathy Buerger Jenny T Liang Joshua Garland Maarten Sap 57 10 0 29 Feb 2024
DISCERN: Designing Decision Support Interfaces to Investigate the Complexities of Workplace Social Decision-Making With Line Managers Pranav Khadpe Lindy Le Kate Nowak Shamsi T. Iqbal Jina Suh 51 7 0 29 Feb 2024
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping Zijie J. Wang Chinmay Kulkarni Lauren Wilcox Michael Terry Michael A. Madaio 49 44 0 23 Feb 2024
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Tzu-Sheng Kuo Aaron L Halfaker Zirui Cheng Jiwoo Kim Meng-Hsin Wu Tongshuang Wu Kenneth Holstein Haiyi Zhu 69 22 0 21 Feb 2024
A Roadmap to Pluralistic Alignment Taylor Sorensen Jared Moore Jillian R. Fisher Mitchell L. Gordon Niloofar Mireshghallah ... Liwei Jiang Ximing Lu Nouha Dziri Tim Althoff Yejin Choi 77 83 0 07 Feb 2024
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget Florian E. Dorner Moritz Hardt 39 4 0 03 Feb 2024
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making Zhuoran Lu Dakuo Wang Ming Yin 36 16 0 13 Jan 2024
Inclusive Portraits: Race-Aware Human-in-the-Loop Technology Claudia Flores-Saviaga Christopher Curtis Saiph Savage 54 2 0 06 Nov 2023
Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities Senjuti Dutta Sid Mittal Sherol Chen Deepak Ramachandran Ravi Rajakumar Ian D Kivlichan Sunny Mak Alena Butryna Praveen Paritosh University of Tennessee 61 7 0 01 Nov 2023
Eliciting Model Steering Interactions from Users via Data and Visual Design Probes Anamaria Crisan Maddie Shang Eric Brochu 43 3 0 12 Oct 2023
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations Zhuoyan Li Hangxiao Zhu Zhuoran Lu Ming Yin SyDa 77 70 0 11 Oct 2023
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values Hannah Rose Kirk Andrew M. Bean Bertie Vidgen Paul Röttger Scott A. Hale ALM 43 42 0 11 Oct 2023
Case Law Grounding: Aligning Judgments of Humans and AI on Socially-Constructed Concepts Quan Ze Chen Amy X. Zhang ELM 72 6 0 10 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 71 2 0 03 Oct 2023
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice Fernando Delgado Stephen Yang Michael A. Madaio Qian Yang 80 102 0 02 Oct 2023
Human Feedback is not Gold Standard Tom Hosking Phil Blunsom Max Bartolo ALM 48 51 0 28 Sep 2023
On the Challenges of Building Datasets for Hate Speech Detection Vitthal Bhandari 52 1 0 06 Sep 2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets Danula Hettiachchi I. Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson 55 11 0 03 Sep 2023
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties Taylor Sorensen Liwei Jiang Jena D. Hwang Sydney Levine Valentina Pyatkin ... Kavel Rao Chandra Bhagavatula Maarten Sap J. Tasioulas Yejin Choi SLR 60 52 0 02 Sep 2023
Cura: Curation at Social Media Scale Wanrong He Mitchell L. Gordon Lindsay Popowski Michael S. Bernstein 43 10 0 26 Aug 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 52 483 0 27 Jul 2023
Mitigating Voter Attribute Bias for Fair Opinion Aggregation Ryosuke Ueda Koh Takeuchi H. Kashima 54 1 0 20 Jul 2023
Conformal prediction under ambiguous ground truth David Stutz Abhijit Guha Roy Tatiana Matejovicova Patricia Strachan A. Cemgil Arnaud Doucet 50 18 0 18 Jul 2023
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning Tharindu Cyril Weerasooriya Sarah K. K. Luger Saloni Poddar Ashiqur R. KhudaBukhsh Christopher Homan 43 5 0 07 Jul 2023
Scaling Laws Do Not Scale Fernando Diaz Michael A. Madaio 52 9 0 05 Jul 2023
Evaluating AI systems under uncertain ground truth: a case study in dermatology David Stutz A. Cemgil Abhijit Guha Roy Tatiana Matejovicova Melih Barsbey ... Yossi Matias Pushmeet Kohli Yun-Hui Liu Arnaud Doucet Alan Karthikesalingam 59 3 0 05 Jul 2023
Towards Measuring the Representation of Subjective Global Opinions in Language Models Esin Durmus Karina Nyugen Thomas I. Liao Nicholas Schiefer Amanda Askell ... Alex Tamkin Janel Thamkul Jared Kaplan Jack Clark Deep Ganguli 49 218 0 28 Jun 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 45 27 0 20 Jun 2023