Eliciting and Learning with Soft Labels from Every Annotator

2 July 2022

Papers citing "Eliciting and Learning with Soft Labels from Every Annotator"

38 / 38 papers shown

Title
Validating LLM-as-a-Judge Systems in the Absence of Gold Labels Luke M. Guerdan Solon Barocas Kenneth Holstein Hanna M. Wallach Zhiwei Steven Wu Alexandra Chouldechova ALM ELM 209 0 0 13 Mar 2025
Embracing Diversity: A Multi-Perspective Approach with Soft Labels Benedetta Muscato Praveen Bushipaka Gizem Gezici Lucia Passaro F. Giannotti Tommaso Cucinotta 36 0 0 01 Mar 2025
Subjective Logic Encodings Jake Vasilakes Chrysoula Zerva Sophia Ananiadou 48 0 0 17 Feb 2025
Revisiting Rogers' Paradox in the Context of Human-AI Interaction K. M. Collins Umang Bhatt Ilia Sucholutsky 46 1 0 16 Jan 2025
CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults Mohit Prabhushankar Kiran Kokilepersaud Jorge Quesada Yavuz Yarici Chen Zhou Mohammad Alotaibi Ghassan AlRegib Ahmad Mustafa Yusufjon Kumakov 23 1 0 20 Aug 2024
No Need to Sacrifice Data Quality for Quantity: Crowd-Informed Machine Annotation for Cost-Effective Understanding of Visual Data Christopher Klugmann Rafid Mahmood Guruprasad Hegde Amit Kale Daniel Kondermann 38 2 0 19 Aug 2024
Fairness and Bias Mitigation in Computer Vision: A Survey Sepehr Dehdashtian Ruozhen He Yi Li Guha Balakrishnan Nuno Vasconcelos Vicente Ordonez Vishnu Naresh Boddeti 37 4 0 05 Aug 2024
dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans M. Herde Denis Huseljic Lukas Rauch Bernhard Sick 31 1 0 30 Jul 2024
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation Katherine M. Collins Najoung Kim Yonatan Bitton Verena Rieser Shayegan Omidshafiei ... Gang Li Adrian Weller Junfeng He Deepak Ramachandran Krishnamurthy Dvijotham EGVM 47 3 0 24 Jun 2024
A Taxonomy of Challenges to Curating Fair Datasets Dora Zhao M. Scheuerman Pooja Chitre Jerone T. A. Andrews Georgia Panagiotidou Shawn Walker Kathleen H. Pine Alice Xiang 47 2 0 10 Jun 2024
Annotator-Centric Active Learning for Subjective NLP Tasks Michiel van der Meer Neele Falk P. Murukannaiah Enrico Liscio 36 8 0 24 Apr 2024
Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty Changbin Li Kangshuo Li Yuzhe Ou Lance M. Kaplan A. Jøsang Jin-Hee Cho Dong Hyun. Jeong Feng Chen UQCV BDL EDL 38 5 0 17 Apr 2024
ReffAKD: Resource-efficient Autoencoder-based Knowledge Distillation Divyang Doshi Jung-Eun Kim 26 1 0 15 Apr 2024
Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks Bálint Mucsányi Michael Kirchhof Seong Joon Oh UQCV BDL OODD 417 20 1 29 Feb 2024
Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction Senjuti Dutta Sherol Chen Sunny Mak Amnah Ahmad Katherine M. Collins Alena Butryna Deepak Ramachandran Krishnamurthy Dvijotham Ellie Pavlick Ravi Rajakumar EGVM 24 1 0 27 Feb 2024
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? Joris Baan Raquel Fernández Barbara Plank Wilker Aziz 45 10 0 25 Feb 2024
Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labels Minoh Jeong Martina Cardone Alex Dytso 29 0 0 27 Jan 2024
Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels Ben Wu Yue Li Yida Mu Carolina Scarton Kalina Bontcheva Xingyi Song 20 10 0 09 Nov 2023
Getting aligned on representational alignment Ilia Sucholutsky Lukas Muttenthaler Adrian Weller Andi Peng Andreea Bobu ... Thomas Unterthiner Andrew Kyle Lampinen Klaus-Robert Muller M. Toneva Thomas L. Griffiths 61 74 0 18 Oct 2023
Binary Classification with Confidence Difference Wei Wang Lei Feng Yuchen Jiang Gang Niu Min-Ling Zhang Masashi Sugiyama 36 5 0 09 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 44 2 0 03 Oct 2023
Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation Andre Ye Quan Ze Chen Amy X. Zhang UQCV 22 1 0 15 Aug 2023
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning Tharindu Cyril Weerasooriya Sarah K. K. Luger Saloni Poddar Ashiqur R. KhudaBukhsh Christopher Homan 15 5 0 07 Jul 2023
Evaluating AI systems under uncertain ground truth: a case study in dermatology David Stutz A. Cemgil Abhijit Guha Roy Tatiana Matejovicova Melih Barsbey ... Yossi Matias Pushmeet Kohli Yun-hui Liu Arnaud Doucet Alan Karthikesalingam 30 4 0 05 Jul 2023
Selective Concept Models: Permitting Stakeholder Customisation at Test-Time Matthew Barker Katherine M. Collins Krishnamurthy Dvijotham Adrian Weller Umang Bhatt 22 2 0 14 Jun 2023
Handling Label Uncertainty on the Example of Automatic Detection of Shepherd's Crook RCA in Coronary CT Angiography Felix Denzinger M. Wels O. Taubmann Florian Kordon Fabian Wagner ... F. André S. Buss Johannes Görich M. Sühling Andreas Maier 15 0 0 22 May 2023
Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality Lars Schmarje Vasco Grossmann Tim Michels Jakob Nazarenus M. Santarossa Claudius Zelenka Reinhard Koch 39 3 0 22 May 2023
Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement Quan Ze Chen Amy X. Zhang 32 7 0 02 May 2023
Learning Personalized Decision Support Policies Umang Bhatt Valerie Chen Katherine M. Collins Parameswaran Kamalaruban Emma Kallina Adrian Weller Ameet Talwalkar OffRL 53 10 0 13 Apr 2023
Human Uncertainty in Concept-Based AI Systems Katherine M. Collins Matthew Barker M. Zarlenga Naveen Raman Umang Bhatt M. Jamnik Ilia Sucholutsky Adrian Weller Krishnamurthy Dvijotham 66 39 0 22 Mar 2023
On the Informativeness of Supervision Signals Ilia Sucholutsky Ruairidh M. Battleday Katherine M. Collins Raja Marjieh Joshua C. Peterson Pulkit Singh Umang Bhatt Nori Jacoby Adrian Weller Thomas L. Griffiths 27 12 0 02 Nov 2022
Human-in-the-Loop Mixup Katherine M. Collins Umang Bhatt Weiyang Liu Vihari Piratla Ilia Sucholutsky Bradley C. Love Adrian Weller 27 7 0 02 Nov 2022
Ambiguous Images With Human Judgments for Robust Visual Event Classification Kate Sanders Reno Kriz Anqi Liu Benjamin Van Durme 65 12 0 06 Oct 2022
Perspectives on Incorporating Expert Feedback into Model Updates Valerie Chen Umang Bhatt Hoda Heidari Adrian Weller Ameet Talwalkar 30 11 0 13 May 2022
DiverseNet: When One Right Answer is not Enough Michael Firman Neill D. F. Campbell Lourdes Agapito Gabriel J. Brostow 3DH OOD 42 24 0 24 Aug 2020
Building machines that adapt and compute like brains Brenden Lake J. Tenenbaum AI4CE FedML NAI AILaw 254 890 0 11 Nov 2017
Adversarial Machine Learning at Scale Alexey Kurakin Ian Goodfellow Samy Bengio AAML 261 3,110 0 04 Nov 2016
Improving neural networks by preventing co-adaptation of feature detectors Geoffrey E. Hinton Nitish Srivastava A. Krizhevsky Ilya Sutskever Ruslan Salakhutdinov VLM 266 7,636 0 03 Jul 2012