Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research

17 April 2025

Papers citing "Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research"

50 / 89 papers shown

Title
Interaction Asymmetry: A General Principle for Learning Composable Abstractions Jack Brady Julius von Kügelgen Sébastien Lachapelle Simon Buchholz Thomas Kipf Wieland Brendel DRL CoGe 61 3 0 12 Nov 2024
Objective drives the consistency of representational similarity across datasets Laure Ciernik Lorenz Linhardt Marco Morik Jonas Dippel Simon Kornblith Lukas Muttenthaler 48 5 0 08 Nov 2024
All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling Emanuele Marconato Sébastien Lachapelle Sebastian Weichwald Luigi Gresele 88 4 0 30 Oct 2024
Cross-Entropy Is All You Need To Invert the Data Generating Process Patrik Reizinger Alice Bizeul Attila Juhos Julia E. Vogt Randall Balestriero Wieland Brendel David Klindt SSL OOD BDL DRL 166 6 0 29 Oct 2024
In Search of Forgotten Domain Generalization Prasanna Mayilvahanan Roland S. Zimmermann Thaddäus Wiedemer E. Rusak Attila Juhos Matthias Bethge Wieland Brendel CLL VLM OOD 55 4 0 10 Oct 2024
Compositional Risk Minimization Divyat Mahajan Mohammad Pezeshki Ioannis Mitliagkas Kartik Ahuja Pascal Vincent Pascal Vincent 71 4 0 08 Oct 2024
Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling Hritik Bansal Arian Hosseini Rishabh Agarwal Vinh Q. Tran Mehran Kazemi SyDa OffRL LRM 63 47 0 29 Aug 2024
InfoNCE: Identifying the Gap Between Theory and Practice E. Rusak Patrik Reizinger Attila Juhos Oliver Bringmann Roland S. Zimmermann Wieland Brendel 87 10 0 28 Jun 2024
Latent Space Translation via Inverse Relative Projection Valentino Maiorca Luca Moschella Marco Fumero Francesco Locatello Emanuele Rodolà 63 2 0 21 Jun 2024
Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environment Julius von Kügelgen 49 4 0 19 Jun 2024
Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations? Mark Ibrahim David Klindt Randall Balestriero SSL 76 5 1 15 Jun 2024
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning D. Kunin Allan Raventós Clémentine Dominé Feng Chen David Klindt Andrew M. Saxe Surya Ganguli MLT 87 18 0 10 Jun 2024
Marrying Causal Representation Learning with Dynamical Systems for Science Dingling Yao Caroline Muller Francesco Locatello CML AI4CE 109 9 0 22 May 2024
Position: Understanding LLMs Requires More Than Statistical Generalization Patrik Reizinger Szilvia Ujváry Anna Mészáros A. Kerekes Wieland Brendel Ferenc Huszár 82 14 0 03 May 2024
Investigating the Benefits of Projection Head for Representation Learning Yihao Xue Eric Gan Jiayi Ni Siddharth Joshi Baharan Mirzasoleiman 68 12 0 18 Mar 2024
Augmentations vs Algorithms: What Works in Self-Supervised Learning Warren Morningstar Alex Bijamov Chris Duvarney Luke Friedman Neha Kalibhat ... Philip Mansfield Renan A. Rojas-Gomez Karan Singhal Bradley Green Sushant Prakash SSL 50 11 0 08 Mar 2024
On Provable Length and Compositional Generalization Kartik Ahuja Amin Mansouri OODD 53 7 0 07 Feb 2024
An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysis Goutham Rajendran Patrik Reizinger Wieland Brendel Pradeep Ravikumar CML 108 8 0 29 Nov 2023
Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations Cian Eastwood Julius von Kügelgen Linus Ericsson Diane Bouchacourt Pascal Vincent Bernhard Schölkopf Mark Ibrahim 86 11 0 15 Nov 2023
The Linear Representation Hypothesis and the Geometry of Large Language Models Kiho Park Yo Joong Choe Victor Veitch LLMSV MILM 95 177 0 07 Nov 2023
Causal Representation Learning Made Identifiable by Grouping of Observational Variables H. Morioka Aapo Hyvarinen OOD CML BDL 90 10 0 24 Oct 2023
Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity? Prasanna Mayilvahanan Thaddäus Wiedemer E. Rusak Matthias Bethge Wieland Brendel OODD 71 23 0 14 Oct 2023
Provable Compositional Generalization for Object-Centric Learning Thaddäus Wiedemer Jack Brady Alexander Panfilov Attila Juhos Matthias Bethge Wieland Brendel OCL 69 20 0 09 Oct 2023
Generalization in diffusion models arises from geometry-adaptive harmonic representations Zahra Kadkhodaie Florentin Guth Eero P. Simoncelli Stéphane Mallat AI4CE DiffM 96 81 0 04 Oct 2023
From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication Irene Cannistraci Luca Moschella Marco Fumero Valentino Maiorca Emanuele Rodolà 67 14 0 02 Oct 2023
Towards the Sparseness of Projection Head in Self-Supervised Learning Changwen Zheng Xingzhe Su Wenwen Qiang Jingyao Wang Changwen Zheng Gang Hua 80 3 0 18 Jul 2023
Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation Sébastien Lachapelle Divyat Mahajan Ioannis Mitliagkas Simon Lacoste-Julien 58 28 0 05 Jul 2023
Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression Allan Raventós Mansheej Paul F. Chen Surya Ganguli 84 86 0 26 Jun 2023
BISCUIT: Causal Representation Learning from Binary Interactions Phillip Lippe Sara Magliacane Sindy Löwe Yuki M. Asano Taco S. Cohen E. Gavves CML 55 31 0 16 Jun 2023
Nonparametric Identifiability of Causal Representations from Unknown Interventions Julius von Kügelgen M. Besserve Wendong Liang Luigi Gresele Armin Kekić Elias Bareinboim David M. Blei Bernhard Schölkopf CML 99 63 0 01 Jun 2023
Causal Component Analysis Wendong Liang Armin Kekić Julius von Kügelgen Simon Buchholz M. Besserve Luigi Gresele Bernhard Schölkopf CML 84 38 0 26 May 2023
Randomized Positional Encodings Boost Length Generalization of Transformers Anian Ruoss Grégoire Delétang Tim Genewein Jordi Grau-Moya Róbert Csordás Mehdi Abbana Bennani Shane Legg J. Veness LLMAG 57 103 0 26 May 2023
On the Stepwise Nature of Self-Supervised Learning James B. Simon Maksis Knutins Liu Ziyin Daniel Geisz Abraham J. Fetterman Joshua Albrecht SSL 55 34 0 27 Mar 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 128 1,131 0 27 Mar 2023
Identifiability Results for Multimodal Contrastive Learning Imant Daunhawer Alice Bizeul Emanuele Palumbo Alexander Marx Julia E. Vogt 52 42 0 16 Mar 2023
The SSL Interplay: Augmentations, Inductive Bias, and Generalization Vivien A. Cabannes B. Kiani Randall Balestriero Yann LeCun A. Bietti SSL 63 32 0 06 Feb 2023
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 100 52 0 25 Oct 2022
Self-Supervised Learning via Maximum Entropy Coding Xin Liu Zhongdao Wang Yali Li Shengjin Wang SSL 85 43 0 20 Oct 2022
Unsupervised visualization of image datasets using contrastive learning Jan Boehm Philipp Berens D. Kobak SSL 62 15 0 18 Oct 2022
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank Q. Garrido Randall Balestriero Laurent Najman Yann LeCun SSL 90 76 0 05 Oct 2022
ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training Antonio Norelli Marco Fumero Valentino Maiorca Luca Moschella Emanuele Rodolà Francesco Locatello VLM 105 36 0 04 Oct 2022
Relative representations enable zero-shot latent space communication Luca Moschella Valentino Maiorca Marco Fumero Antonio Norelli Francesco Locatello Emanuele Rodolà 66 105 0 30 Sep 2022
Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations Grégoire Mialon Randall Balestriero Yann LeCun 57 9 0 29 Sep 2022
Function Classes for Identifiable Nonlinear Independent Component Analysis Simon Buchholz M. Besserve Bernhard Schölkopf 61 40 0 12 Aug 2022
What Do We Maximize in Self-Supervised Learning? Ravid Shwartz-Ziv Randall Balestriero Yann LeCun SSL 44 17 0 20 Jul 2022
Beyond neural scaling laws: beating power law scaling via data pruning Ben Sorscher Robert Geirhos Shashank Shekhar Surya Ganguli Ari S. Morcos 85 439 0 29 Jun 2022
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning Florian Bordes Randall Balestriero Q. Garrido Adrien Bardes Pascal Vincent 80 22 0 27 Jun 2022
Self-supervised Learning in Remote Sensing: A Review Yi Wang C. Albrecht Nassim Ait Ali Braham Lichao Mou Xiao Xiang Zhu 71 223 0 27 Jun 2022
Embrace the Gap: VAEs Perform Independent Mechanism Analysis Patrik Reizinger Luigi Gresele Jack Brady Julius von Kügelgen Dominik Zietlow Bernhard Schölkopf Georg Martius Wieland Brendel M. Besserve DRL 59 20 0 06 Jun 2022
On the duality between contrastive and non-contrastive self-supervised learning Q. Garrido Yubei Chen Adrien Bardes Laurent Najman Yann LeCun SSL 51 90 0 03 Jun 2022