PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics
Capabilities

PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities

13 January 2024

Tankala Pavan Kalyan

Pushpak Bhattacharyya

Papers citing "PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities"

11 / 11 papers shown

Title
A fine-grained comparison of pragmatic language understanding in humans and language models Jennifer Hu Sammy Floyd Olessia Jouravlev Evelina Fedorenko E. Gibson 42 58 0 13 Dec 2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 287 2,364 0 09 Nov 2022
The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs Laura Ruis Akbir Khan Stella Biderman Sara Hooker Tim Rocktaschel Edward Grefenstette ReLM 29 38 0 26 Oct 2022
Testing the Ability of Language Models to Interpret Figurative Language Emmy Liu Chenxuan Cui Kenneth Zheng Graham Neubig ELM LRM 43 69 0 26 Apr 2022
NOPE: A Corpus of Naturally-Occurring Presuppositions in English Alicia Parrish Sebastian Schuster Alex Warstadt Omar Agha Soo-hwan Lee Zhuoye Zhao Sam Bowman Tal Linzen LRM 60 24 0 14 Sep 2021
Do Prompt-Based Models Really Understand the Meaning of their Prompts? Albert Webson Ellie Pavlick LRM 84 361 0 02 Sep 2021
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 229 657 0 20 May 2021
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 192 2,296 0 02 May 2019
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 200 2,830 0 11 Jun 2018
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 434 4,444 0 18 Apr 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 187 2,814 0 26 Sep 2016