Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models

26 October 2023

Dingli Yu

Papers citing "Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models"

33 / 33 papers shown

Title
Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning Jikai Jin Vasilis Syrgkanis Sham Kakade Hanlin Zhang ELM 125 1 0 12 Jun 2025
Extrapolation by Association: Length Generalization Transfer in Transformers Ziyang Cai Nayoung Lee Avi Schwarzschild Samet Oymak Dimitris Papailiopoulos 37 0 0 10 Jun 2025
Circuit Stability Characterizes Language Model Generalization Alan Sun LRM 31 0 0 30 May 2025
A Mathematical Framework for AI-Human Integration in Work Elisa Celis Lingxiao Huang Nisheeth K. Vishnoi 75 0 0 29 May 2025
Concealment of Intent: A Game-Theoretic Analysis Xinbo Wu A. Umrawal Lav Varshney 23 0 0 27 May 2025
The Role of Diversity in In-Context Learning for Large Language Models Wenyang Xiao Haoyu Zhao Lingxiao Huang 90 0 0 26 May 2025
Understanding Task Representations in Neural Networks via Bayesian Ablation Andrew Nam Declan Campbell Thomas Griffiths Jonathan Cohen Sarah-Jane Leslie BDL 47 0 0 19 May 2025
In-Context Learning can distort the relationship between sequence likelihoods and biological fitness Pranav Kantroo Günter P. Wagner Benjamin B. Machta 131 0 0 23 Apr 2025
MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks Jaime Raldua Veuthey Zainab Ali Majid Suhas Hariharan Jacob Haimes ELM 72 0 0 18 Apr 2025
Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models Nariman Naderi Seyed Amir Ahmad Safavi-Naini Thomas Savage Zahra Atf Peter Lewis Girish Nadkarni Ali Soroush ELM 132 2 0 24 Mar 2025
Compute Optimal Scaling of Skills: Knowledge vs Reasoning Nicholas Roberts Niladri S. Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes 117 2 0 13 Mar 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 218 1 0 28 Feb 2025
Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong Yin Zhaoran Wang LRM ReLM 474 1 0 27 Jan 2025
A ghost mechanism: An analytical model of abrupt learning Fatih Dinc Ege Cirakman Yiqi Jiang Mert Yuksekgonul Mark J. Schnitzer Hidenori Tanaka 121 3 0 04 Jan 2025
Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models Mazda Moayeri Vidhisha Balachandran Varun Chandrasekaran Safoora Yousefi Thomas Fel Soheil Feizi Besmira Nushi Neel Joshi Vibhav Vineet 71 5 0 17 Oct 2024
Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity Jinze Zhao Peihao Wang J. Yang Ruisi Cai Gaowen Liu Jayanth Srinivasa Ramana Rao Kompella Yingbin Liang Zhangyang Wang MoE 75 0 0 17 Oct 2024
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 121 10 0 17 Oct 2024
Can LLMs Reliably Simulate Human Learner Actions? A Simulation Authoring Framework for Open-Ended Learning Environments Amogh Mannekote Adam Davies Jina Kang K. Boyer 92 1 0 03 Oct 2024
Can Models Learn Skill Composition from Examples? Haoyu Zhao Simran Kaur Dingli Yu Anirudh Goyal Sanjeev Arora CoGe MoE 126 8 0 29 Sep 2024
ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty Xindi Wu Dingli Yu Yangsibo Huang Olga Russakovsky Sanjeev Arora CoGe EGVM 102 22 0 26 Aug 2024
AI-Assisted Generation of Difficult Math Questions Vedant Shah Dingli Yu Kaifeng Lyu Simon Park Nan Rosemary Ke ... Yoshua Bengio Sanjeev Arora Anirudh Goyal Sanjeev Arora Anirudh Goyal 123 18 0 30 Jul 2024
Training on the Test Task Confounds Evaluation and Emergence Ricardo Dominguez-Olmedo Florian E. Dorner Moritz Hardt ELM 154 9 1 10 Jul 2024
DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph Zhehao Zhang Jiaao Chen Diyi Yang LRM 73 13 0 25 Jun 2024
Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models Lynn Chua Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chulin Xie Chiyuan Zhang 156 2 0 23 Jun 2024
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving Aniket Didolkar Anirudh Goyal Nan Rosemary Ke Siyuan Guo Michal Valko Timothy Lillicrap Danilo Jimenez Rezende Yoshua Bengio Michael C. Mozer Sanjeev Arora LRM 72 30 0 20 May 2024
TEL'M: Test and Evaluation of Language Models G. Cybenko Joshua Ackerman Paul Lintilhac ALM ELM 81 0 0 16 Apr 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 283 29 0 22 Mar 2024
MERA: A Comprehensive LLM Evaluation in Russian Alena Fenogenova Artem Chervyakov Nikita Martynov Anastasia Kozlova Maria Tikhonova ... Nikita Savushkin Polina Mikhailova Denis Dimitrov Alexander Panchenko Sergey Markov ELM 97 12 0 09 Jan 2024
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks Rahul Ramesh Ekdeep Singh Lubana Mikail Khona Robert P. Dick Hidenori Tanaka CoGe 87 12 0 21 Nov 2023
Large Language Models' Understanding of Math: Source Criticism and Extrapolation Roozbeh Yousefzadeh Xuenan Cao ELM LRM 29 3 0 12 Nov 2023
ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models Julien Pourcel Cédric Colas Gaia Molinaro Pierre-Yves Oudeyer Laetitia Teodorescu 97 3 0 15 Oct 2023
Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure Haotong Yang Fanxu Meng Zhouchen Lin Muhan Zhang LRM 109 3 0 09 Oct 2023
Language Models as a Service: Overview of a New Paradigm and its Challenges Emanuele La Malfa Aleksandar Petrov Simon Frieder Christoph Weinhuber Ryan Burnell Raza Nazar Anthony Cohn Nigel Shadbolt Michael Wooldridge ALM ELM 79 6 0 28 Sep 2023