The cornerstone of cognitive intelligence lies in extracting hidden patterns from observations and leveraging these principles to systematically predict future outcomes. However, current image tokenization methods demonstrate significant limitations in tasks requiring symbolic abstraction and logical reasoning capabilities essential for systematic inference. To address this challenge, we propose Discrete-JEPA, extending the latent predictive coding framework with semantic tokenization and novel complementary objectives to create robust tokenization for symbolic reasoning tasks. Discrete-JEPA dramatically outperforms baselines on visual symbolic prediction tasks, while striking visual evidence reveals the spontaneous emergence of deliberate systematic patterns within the learned semantic token space. Though an initial model, our approach promises a significant impact for advancing Symbolic world modeling and planning capabilities in artificial intelligence systems.
View on arXiv@article{baek2025_2506.14373,
title={ Discrete JEPA: Learning Discrete Token Representations without Reconstruction },
author={ Junyeob Baek and Hosung Lee and Christopher Hoang and Mengye Ren and Sungjin Ahn },
journal={arXiv preprint arXiv:2506.14373},
year={ 2025 }
}