Learning Interactive Real-World Simulators

9 October 2023

Pieter Abbeel

Papers citing "Learning Interactive Real-World Simulators"

50 / 139 papers shown

Title
Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling Mingtong Zhang Kaifeng Zhang Yunzhu Li 3DGS AI4CE 31 5 0 24 Oct 2024
WorldSimBench: Towards Video Generation Models as World Simulators Yiran Qin Zhelun Shi Jiwen Yu Xijun Wang Enshen Zhou ... Lu Sheng Jing Shao Junlin Wu Wanli Ouyang Ruimao Zhang EGVM VGen 126 381 0 23 Oct 2024
EVA: An Embodied World Model for Future Video Anticipation Xiaowei Chi Hengyuan Zhang Chun-Kai Fan Xingqun Qi Rongyu Zhang ... Chi-Min Chan Wei Xue Wenhan Luo Shanghang Zhang Yike Guo VGen 38 5 0 20 Oct 2024
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation Hyungjoo Chae Namyoung Kim Kai Tzu-iunn Ong Minju Gwak Gwanwoo Song Jihoon Kim S. Kim Dongha Lee Jinyoung Yeo LLMAG 33 14 0 17 Oct 2024
Latent Action Pretraining from Videos Seonghyeon Ye Joel Jang Byeongguk Jeon Sejune Joo Jianwei Yang ... Kimin Lee J. Gao Luke Zettlemoyer Dieter Fox Minjoon Seo 35 27 0 15 Oct 2024
Asymptotic Analysis of Sample-averaged Q-learning Saunak Kumar Panda Ruiqi Liu Yisha Xiang OnRL 64 0 0 14 Oct 2024
VideoAgent: Self-Improving Video Generation Achint Soni Sreyas Venkataraman Abhranil Chandra Sebastian Fischmeister Percy Liang Bo Dai Sherry Yang LM&Ro VGen 58 7 0 14 Oct 2024
Retrieval-Augmented Decision Transformer: External Memory for In-context RL Thomas Schmied Fabian Paischer Vihang Patil M. Hofmarcher Razvan Pascanu Sepp Hochreiter OffRL 44 6 0 09 Oct 2024
GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation Chi-Lam Cheang Guangzeng Chen Ya Jing Tao Kong Hang Li ... Hongtao Wu Jiafeng Xu Yichu Yang Hanbo Zhang Minzhao Zhu VGen LM&Ro 61 52 0 08 Oct 2024
Diffusion Model Predictive Control Guangyao Zhou Sivaramakrishnan Swaminathan Rajkumar Vasudeva Raju J. S. Guntupalli Wolfgang Lehrach Joseph Ortiz Antoine Dedieu Miguel Lázaro-Gredilla Kevin P. Murphy 31 6 0 07 Oct 2024
OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction Leheng Li Weichao Qiu Xu Yan Jing He Kaiqiang Zhou Yingjie Cai Qing Lian Bingbing Liu Ying-Cong Chen SyDa DiffM 47 1 0 07 Oct 2024
LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting Qifeng Chen Sheng Yang Sicong Du Tao Tang Peng Chen Yuchi Huo 3DGS AI4CE 61 6 0 07 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 84 26 0 04 Oct 2024
Grounded Answers for Multi-agent Decision-making Problem through Generative World Model Zeyang Liu Xinrui Yang Shiguang Sun Long Qian Lipeng Wan Xingyu Chen Xuguang Lan 22 2 0 03 Oct 2024
Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling Jinghan Li Zhicheng Sun Fei Li 102 1 0 02 Oct 2024
AVID: Adapting Video Diffusion Models to World Models Marc Rigter Tarun Gupta Agrin Hilmkil Chao Ma VGen 19 3 0 01 Oct 2024
SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs Leheng Li Weichao Qiu Yingjie Cai Xu Yan Qing Lian Bingbing Liu Ying-Cong Chen DiffM 31 3 0 01 Oct 2024
Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization Haoran Li Zhennan Jiang Yuhui Chen Dongbin Zhao OffRL 23 2 0 28 Sep 2024
One-shot World Models Using a Transformer Trained on a Synthetic Prior Fabio Ferreira Moreno Schlageter Raghu Rajan André Biedenkapp Frank Hutter 36 0 0 21 Sep 2024
KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data Grace Tang Swetha Rajkumar Yifei Zhou Homer Walke Sergey Levine Kuan Fang LM&Ro VLM 29 7 0 21 Sep 2024
$DexSim2Real$^{2}$: Building Explicit World Model for Precise Articulated Object Dexterous Manipulation$ DexSim2Real $^{2}$ : Building Explicit World Model for Precise Articulated Object Dexterous Manipulation Taoran Jiang Liqian Ma Yixuan Guan Jiaojiao Meng Weihang Chen Zecui Zeng Lusong Li Dan Wu Jing Xu Rui Chen 30 0 0 13 Sep 2024
xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing Haoyi Niu Qimao Chen Tenglong Liu Jianxiong Li Guyue Zhou Yi Zhang Jianming Hu Xianyuan Zhan 34 0 0 13 Sep 2024
Generative Hierarchical Materials Search Sherry Yang Simon L. Batzner Ruiqi Gao Muratahan Aykol Alexander L. Gaunt Brendan McMorrow Danilo J. Rezende Dale Schuurmans Igor Mordatch E. D. Cubuk AI4CE 40 6 0 10 Sep 2024
Forward KL Regularized Preference Optimization for Aligning Diffusion Policies Zhao Shan Chenyou Fan Shuang Qiu Jiyuan Shi Chenjia Bai 40 4 0 09 Sep 2024
MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model Junjie Li Yang Liu Weiqing Liu Shikai Fang Lewen Wang Chang Xu Jiang Bian VGen 43 4 0 04 Sep 2024
Diffusion Policy Policy Optimization Allen Z. Ren Justin Lidard Lars L. Ankile Anthony Simeonov Pulkit Agrawal Anirudha Majumdar Benjamin Burchfiel Hongkai Dai Max Simchowitz 45 36 0 01 Sep 2024
MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning Yifu Yuan Zhenrui Zheng Zibin Dong Jianye Hao OffRL 52 1 0 28 Aug 2024
Diffusion Models Are Real-Time Game Engines Dani Valevski Yaniv Leviathan Moab Arar Shlomi Fruchter DiffM VGen AI4CE 33 59 0 27 Aug 2024
GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy Peiyan Li Hongtao Wu Yan Huang Chilam Cheang Liang Wang Tao Kong VGen 54 11 0 26 Aug 2024
Controlling the World by Sleight of Hand Sruthi Sudhakar Ruoshi Liu Basile Van Hoorick Carl Vondrick Richard Zemel 41 4 0 13 Aug 2024
Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators Wentao Zhang Junliang Guo Tianyu He Li Zhao Linli Xu Jiang Bian 47 3 0 10 Jul 2024
Learning Action and Reasoning-Centric Image Editing from Videos and Simulations Benno Krojer Dheeraj Vattikonda Luis Lara Varun Jampani Eva Portelance Christopher Pal Siva Reddy EGVM VGen 49 3 0 03 Jul 2024
Modeling the Real World with High-Density Visual Particle Dynamics William F. Whitney Jacob Varley Deepali Jain Krzysztof Choromanski Sumeet Singh Vikas Sindhwani 35 2 0 28 Jun 2024
Human-centered In-building Embodied Delivery Benchmark Zhuoqun Xu Yang Liu Xiaoqi Li Jiyao Zhang Hao Dong 42 0 0 25 Jun 2024
Diffusion Spectral Representation for Reinforcement Learning Dmitry Shribak Chen-Xiao Gao Yitong Li Chenjun Xiao Bo Dai DiffM 26 3 0 23 Jun 2024
IRASim: Learning Interactive Real-Robot Action Simulators Fangqi Zhu Hongtao Wu Song Guo Yuxiao Liu Chilam Cheang Tao Kong 80 13 0 20 Jun 2024
CityBench: Evaluating the Capabilities of Large Language Model as World Model Jie Feng Jun Zhang Junbo Yan Xin Zhang Tianjian Ouyang Tianhui Liu Yuwei Du Siqi Guo Yong Li ELM 56 0 0 20 Jun 2024
CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making Zibin Dong Yifu Yuan Jianye Hao Fei Ni Yi Ma Pengyi Li Yan Zheng DiffM 58 9 0 13 Jun 2024
Pandora: Towards General World Model with Natural Language Actions and Video States Jiannan Xiang Guangyi Liu Yi Gu Qiyue Gao Yuting Ning ... Shibo Hao Yemin Shi Zhengzhong Liu Eric P. Xing Zhiting Hu VGen 62 36 0 12 Jun 2024
HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness Zihui Xue Mi Luo Changan Chen Kristen Grauman DiffM 32 6 0 11 Jun 2024
Coherent Zero-Shot Visual Instruction Generation Quynh Phung Songwei Ge Jia-Bin Huang 54 2 0 06 Jun 2024
Open-Endedness is Essential for Artificial Superhuman Intelligence Edward Hughes Michael Dennis Jack Parker-Holder Feryal M. P. Behbahani Aditi Mavalankar Yuge Shi Tom Schaul Tim Rocktaschel LRM 34 21 0 06 Jun 2024
Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting Jincheng Zhong Xingzhuo Guo Jiaxiang Dong Mingsheng Long DiffM 40 2 0 02 Jun 2024
Learning Manipulation by Predicting Interaction Jia Zeng Qingwen Bu Bangjun Wang Wenke Xia Li Chen ... Heming Cui Bin Zhao Xuelong Li Yu Qiao Hongyang Li 53 20 0 01 Jun 2024
Do's and Don'ts: Learning Desirable Skills with Instruction Videos Hyunseung Kim ByungKun Lee Hojoon Lee Dongyoon Hwang Donghu Kim Jaegul Choo 39 1 0 01 Jun 2024
Instruction-Guided Visual Masking Jinliang Zheng Jianxiong Li Si Cheng Yinan Zheng Jiaming Li Jihao Liu Yu Liu Jingjing Liu Xianyuan Zhan 53 5 0 30 May 2024
In-Context Symmetries: Self-Supervised Learning through Contextual World Models Sharut Gupta Chenyu Wang Yifei Wang Tommi Jaakkola Stefanie Jegelka 37 1 0 28 May 2024
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability Shenyuan Gao Jiazhi Yang Li Chen Kashyap Chitta Yihang Qiu Andreas Geiger Jun Zhang Hongyang Li 71 75 0 27 May 2024
Ego-Foresight: Agent Visuomotor Prediction as Regularization for RL Manuel S. Nunes Atabak Dehban Y. Demiris J. Santos-Victor 48 0 0 27 May 2024
Position: Foundation Agents as the Paradigm Shift for Decision Making Xiaoqian Liu Xingzhou Lou Jianbin Jiao Junge Zhang OffRL LLMAG 42 6 0 27 May 2024