OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation

13 June 2024

Junke Wang

Yu-Gang Jiang

ArXiv (abs)PDF HTML Github (295★)

Papers citing "OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation"

46 / 46 papers shown

Title
Video-GPT via Next Clip Diffusion Shaobin Zhuang Zhipeng Huang Ying Zhang Fangyikang Wang Canmiao Fu Binxin Yang Chong Sun Chen Li Yali Wang DiffM VGen 223 0 0 18 May 2025
Generative Pre-trained Autoregressive Diffusion Transformer Yuan Zhang Jiacheng Jiang Guoqing Ma Zhiying Lu Haoyang Huang Jianlong Yuan Nan Duan VGen 112 1 0 12 May 2025
Long-Context Autoregressive Video Modeling with Next-Frame Prediction Yuchao Gu Weijia Mao Mike Zheng Shou VGen 159 6 0 25 Mar 2025
Parallelized Autoregressive Visual Generation Yanjie Wang Shuhuai Ren Zhijie Lin Yujin Han Haoyuan Guo Zhenheng Yang Difan Zou Jiashi Feng Xihui Liu VGen 163 12 0 19 Dec 2024
Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation Zhuoyan Luo Fengyuan Shi Yixiao Ge Yujiu Yang Limin Wang Ying Shan VLM 119 55 0 06 Sep 2024
Latte: Latent Diffusion Transformer for Video Generation Xin Ma Yaohui Wang Gengyun Jia Xinyuan Chen Ziqiang Liu Yuan-Fang Li Cunjian Chen Yu Qiao DiffM VGen 238 270 0 05 Jan 2024
Video Probabilistic Diffusion Models in Projected Latent Space Sihyun Yu Kihyuk Sohn Subin Kim Jinwoo Shin VGen DiffM 88 169 0 15 Feb 2023
Adding Conditional Control to Text-to-Image Diffusion Models Lvmin Zhang Anyi Rao Maneesh Agrawala AI4CE 148 4,113 1 10 Feb 2023
Scalable Diffusion Models with Transformers William S. Peebles Saining Xie GNN 93 2,301 0 19 Dec 2022
Look Before You Match: Instance Understanding Matters in Video Object Segmentation Junke Wang Dongdong Chen Zuxuan Wu Chong Luo Chuanxin Tang Xiyang Dai Yucheng Zhao Yujia Xie Lu Yuan Yu-Gang Jiang VOS 86 41 0 13 Dec 2022
ResFormer: Scaling ViTs with Multi-Resolution Training Rui Tian Zuxuan Wu Qiuju Dai Hang-Rui Hu Yu Qiao Yu-Gang Jiang ViT 77 35 0 01 Dec 2022
MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis Tianhong Li Huiwen Chang Shlok Kumar Mishra Han Zhang Dina Katabi Dilip Krishnan 70 166 0 16 Nov 2022
Phenaki: Variable Length Video Generation From Open Domain Textual Description Ruben Villegas Mohammad Babaeizadeh Pieter-Jan Kindermans Hernan Moraldo Han Zhang M. Saffar Santiago Castro Julius Kunze D. Erhan DiffM VGen 127 389 0 05 Oct 2022
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks Junke Wang Dongdong Chen Zuxuan Wu Chong Luo Luowei Zhou Yucheng Zhao Yujia Xie Ce Liu Yu-Gang Jiang Lu Yuan MLLM VLM 83 153 0 15 Sep 2022
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation Jiahui Yu Yuanzhong Xu Jing Yu Koh Thang Luong Gunjan Baid ... Zarana Parekh Xin Li Han Zhang Jason Baldridge Yonghui Wu EGVM 187 1,121 0 22 Jun 2022
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers Wenyi Hong Ming Ding Wendi Zheng Xinghan Liu Jie Tang DiffM 308 608 0 29 May 2022
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer Songwei Ge Thomas Hayes Harry Yang Xiaoyue Yin Guan Pang David Jacobs Jia-Bin Huang Devi Parikh ViT 123 220 0 07 Apr 2022
ObjectFormer for Image Manipulation Detection and Localization Junke Wang Zuxuan Wu Jingjing Chen Xintong Han Abhinav Shrivastava Ser-Nam Lim Yu-Gang Jiang 82 113 0 28 Mar 2022
Autoregressive Image Generation using Residual Quantization Doyup Lee Chiheon Kim Saehoon Kim Minsu Cho Wook-Shin Han VGen 269 361 0 03 Mar 2022
Generative Adversarial Networks Gilad Cohen Raja Giryes GAN 280 30,103 0 01 Mar 2022
Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks Sihyun Yu Jihoon Tack Sangwoo Mo Hyunsu Kim Junho Kim Jung-Woo Ha Jinwoo Shin DiffM VGen 102 201 0 21 Feb 2022
MaskGIT: Masked Generative Image Transformer Huiwen Chang Han Zhang Lu Jiang Ce Liu William T. Freeman ViT 153 678 0 08 Feb 2022
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets Axel Sauer Katja Schwarz Andreas Geiger 266 509 0 01 Feb 2022
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 Ivan Skorokhodov Sergey Tulyakov Mohamed Elhoseiny VGen 85 287 0 29 Dec 2021
High-Resolution Image Synthesis with Latent Diffusion Models Robin Rombach A. Blattmann Dominik Lorenz Patrick Esser Bjorn Ommer 3DV 428 15,515 0 20 Dec 2021
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models Alex Nichol Prafulla Dhariwal Aditya A. Ramesh Pranav Shyam Pamela Mishkin Bob McGrew Ilya Sutskever Mark Chen 347 3,605 0 20 Dec 2021
Vector-quantized Image Modeling with Improved VQGAN Jiahui Yu Xin Li Jing Yu Koh Han Zhang Ruoming Pang James Qin Alexander Ku Yuanzhong Xu Jason Baldridge Yonghui Wu ViT VLM DRL 117 519 0 09 Oct 2021
Cascaded Diffusion Models for High Fidelity Image Generation Jonathan Ho Chitwan Saharia William Chan David J. Fleet Mohammad Norouzi Tim Salimans 161 1,222 0 30 May 2021
Diffusion Models Beat GANs on Image Synthesis Prafulla Dhariwal Alex Nichol 227 7,857 0 11 May 2021
A Good Image Generator Is What You Need for High-Resolution Video Synthesis Yu Tian Jian Ren Menglei Chai Kyle Olszewski Xi Peng Dimitris N. Metaxas Sergey Tulyakov VGen 98 186 0 30 Apr 2021
VideoGPT: Video Generation using VQ-VAE and Transformers Wilson Yan Yunzhi Zhang Pieter Abbeel A. Srinivas ViT VGen 303 502 0 20 Apr 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 447 21,439 0 25 Mar 2021
Predicting Video with VQVAE Jacob Walker Ali Razavi Aaron van den Oord DRL 93 68 0 02 Mar 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 415 4,953 0 24 Feb 2021
Improved Denoising Diffusion Probabilistic Models Alex Nichol Prafulla Dhariwal DiffM 337 3,686 0 18 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 372 2,053 0 09 Feb 2021
Denoising Diffusion Implicit Models Jiaming Song Chenlin Meng Stefano Ermon VLM DiffM 283 7,384 0 06 Oct 2020
Latent Video Transformer Ruslan Rakhimov Denis Volkhonskiy Alexey Artemov Denis Zorin Evgeny Burnaev VGen 95 120 0 18 Jun 2020
End-to-End Object Detection with Transformers Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko ViT 3DV PINN 415 13,048 0 26 May 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 605 4,822 0 23 Jan 2020
A Style-Based Generator Architecture for Generative Adversarial Networks Tero Karras S. Laine Timo Aila 583 10,561 0 12 Dec 2018
Moments in Time Dataset: one million videos for event understanding Mathew Monfort A. Andonian Bolei Zhou K. Ramakrishnan Sarah Adel Bargal ... L. Brown Quanfu Fan Dan Gutfreund Carl Vondrick A. Oliva 92 549 0 09 Jan 2018
Neural Discrete Representation Learning Aaron van den Oord Oriol Vinyals Koray Kavukcuoglu BDL SSL OCL 226 5,019 0 02 Nov 2017
Progressive Growing of GANs for Improved Quality, Stability, and Variation Tero Karras Timo Aila S. Laine J. Lehtinen GAN 137 7,361 0 27 Oct 2017
MoCoGAN: Decomposing Motion and Content for Video Generation Sergey Tulyakov Ming-Yuan Liu Xiaodong Yang Jan Kautz GAN 131 1,147 0 17 Jul 2017
UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild K. Soomro Amir Zamir M. Shah CLIP VGen 152 6,148 0 03 Dec 2012