v1v2 (latest)

Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning

1 April 2025

Papers citing "Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning"

26 / 26 papers shown

Title
Scaling Autonomous Agents via Automatic Reward Modeling And Planning Zhenfang Chen Delin Chen Rui Sun Wenjun Liu Chuang Gan LLMAG 90 4 0 17 Feb 2025
Flexible and Efficient Grammar-Constrained Decoding Kanghee Park Timothy Zhou Loris Dántoni OffRL 68 4 0 07 Feb 2025
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons Andrew Szot Bogdan Mazoure Omar Attia Aleksei Timofeev Harsh Agrawal Devon Hjelm Zhe Gan Z. Kira Alexander Toshev 3DGS 86 7 0 11 Dec 2024
Automated Rewards via LLM-Generated Progress Functions Vishnu Sarukkai Brennan Shacklett Zander Majercik Kush S. Bhatia Christopher Ré Kayvon Fatahalian 60 1 0 11 Oct 2024
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning Amrith Rajagopal Setlur Chirag Nagpal Adam Fisch Xinyang Geng Jacob Eisenstein Rishabh Agarwal Alekh Agarwal Jonathan Berant Aviral Kumar OffRL LRM 96 75 0 10 Oct 2024
Situated Instruction Following So Yeon Min Xavi Puig Devendra Singh Chaplot Tsung-Yen Yang Akshara Rai Priyam Parashar Ruslan Salakhutdinov Yonatan Bisk Roozbeh Mottaghi 59 2 0 15 Jul 2024
Robotic Control via Embodied Chain-of-Thought Reasoning Michał Zawalski William Chen Karl Pertsch Oier Mees Chelsea Finn Sergey Levine LRM LM&Ro 125 88 0 11 Jul 2024
Learning Reward for Physical Skills using Large Language Model Yuwei Zeng Yiqing Xu 55 6 0 21 Oct 2023
Language to Rewards for Robotic Skill Synthesis Wenhao Yu Nimrod Gileadi Chuyuan Fu Sean Kirmani Kuang-Huei Lee ... N. Heess Dorsa Sadigh Jie Tan Yuval Tassa F. Xia LM&Ro 92 282 0 14 Jun 2023
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second Vincent-Pierre Berges Andrew Szot Devendra Singh Chaplot Aaron Gokaslan Roozbeh Mottaghi Dhruv Batra Eric Undersander LRM LM&Ro 79 5 0 13 Jun 2023
Ask4Help: Learning to Leverage an Expert for Embodied Tasks Kunal Pratap Singh Luca Weihs Alvaro Herrasti Jonghyun Choi Aniruddha Kemhavi Roozbeh Mottaghi 63 19 0 18 Nov 2022
ReAct: Synergizing Reasoning and Acting in Language Models Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan Yuan Cao LLMAG ReLM LRM 436 2,955 0 06 Oct 2022
Inner Monologue: Embodied Reasoning through Planning with Language Models Wenlong Huang F. Xia Ted Xiao Harris Chan Jacky Liang ... Tomas Jackson Linda Luu Sergey Levine Karol Hausman Brian Ichter LLMAG LM&Ro LRM 134 920 0 12 Jul 2022
Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items Laura Downs Anthony G. Francis Nate Koenig Brandon Kinman R. Hickman Krista Reymann T. B. McHugh Vincent Vanhoucke LM&Ro 109 500 0 25 Apr 2022
Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale Ram Ramrakhya Eric Undersander Dhruv Batra Abhishek Das LM&Ro 122 119 0 07 Apr 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 192 1,984 0 04 Apr 2022
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following Xiaofeng Gao Qiaozi Gao Ran Gong Kaixiang Lin Govind Thattai Gaurav Sukhatme LM&Ro 130 71 0 27 Feb 2022
Perceiver IO: A General Architecture for Structured Inputs & Outputs Andrew Jaegle Sebastian Borgeaud Jean-Baptiste Alayrac Carl Doersch Catalin Ionescu ... Olivier J. Hénaff M. Botvinick Andrew Zisserman Oriol Vinyals João Carreira MLLM VLM GNN 86 584 0 30 Jul 2021
Habitat 2.0: Training Home Assistants to Rearrange their Habitat Andrew Szot Alexander Clegg Eric Undersander Erik Wijmans Yili Zhao ... Z. Kira V. Koltun Jitendra Malik Manolis Savva Dhruv Batra LM&Ro 109 526 0 28 Jun 2021
ManipulaTHOR: A Framework for Visual Object Manipulation Kiana Ehsani Winson Han Alvaro Herrasti Eli VanderBilt Luca Weihs Eric Kolve Aniruddha Kembhavi Roozbeh Mottaghi LM&Ro 247 130 0 22 Apr 2021
Visual Room Rearrangement Luca Weihs Matt Deitke Aniruddha Kembhavi Roozbeh Mottaghi LM&Ro 64 139 0 30 Mar 2021
The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation Shurjo Banerjee Jesse Thomason Jason J. Corso LM&Ro 120 30 0 23 Oct 2020
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames Erik Wijmans Abhishek Kadian Ari S. Morcos Stefan Lee Irfan Essa Devi Parikh Manolis Savva Dhruv Batra 85 484 0 01 Nov 2019
Learning Dexterous In-Hand Manipulation OpenAI OpenAI Marcin Andrychowicz Bowen Baker Maciek Chociej Rafal Jozefowicz ... Szymon Sidor Joshua Tobin Peter Welinder Lilian Weng Wojciech Zaremba 159 1,884 0 01 Aug 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 538 19,265 0 20 Jul 2017
High-Dimensional Continuous Control Using Generalized Advantage Estimation John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel OffRL 129 3,438 0 08 Jun 2015