Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

26 September 2024

Matthew Johnson-Roberson

Yonatan Bisk

Matthew Johnson-Roberson

Yonatan Bisk

LM&Ro

ArXiv PDF HTML

Papers citing "Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation"

50 / 50 papers shown

Title
3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model Wenbo Hu Yining Hong Yanjun Wang Leison Gao Zibu Wei Xingcheng Yao Nanyun Peng Yonatan Bitton Idan Szpektor Kai-Wei Chang 21 0 0 28 May 2025
A Grounded Memory System For Smart Personal Assistants Felix Ocker J. Deigmöller Pavel Smirnov Julian Eggert 48 0 0 09 May 2025
RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration Omar Alama A. Bhattacharya Haoyang He Seungchan Kim Yuheng Qiu Wenshan Wang Cherie Ho Nikhil Varma Keetha Sebastian A. Scherer 51 0 0 09 Apr 2025
Dexterous Manipulation through Imitation Learning: A Survey Shan An Ziyu Meng Chao Tang Yimiao Zhou Tengyu Liu ... Yao Mu Ran Song Wei Zhang Zeng-Guang Hou Haoyang Zhang 113 0 0 04 Apr 2025
Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook Xu Zheng Ziqiao Weng Yuanhuiyi Lyu Lutao Jiang Haiwei Xue Bin Ren Danda Pani Paudel N. Sebe Luc Van Gool Xuming Hu 3DV 102 5 0 23 Mar 2025
A Survey of Model Architectures in Information Retrieval Zhichao Xu Fengran Mo Zhiqi Huang Crystina Zhang Puxuan Yu Bei Wang Jimmy J. Lin Vivek Srikumar KELM 3DV 103 2 0 21 Feb 2025
From Local to Global: A Graph RAG Approach to Query-Focused Summarization Darren Edge Ha Trinh Newman Cheng Joshua Bradley Alex Chao Apurva Mody Steven Truitt Dasha Metropolitansky Robert Osazuwa Ness Jonathan Larson RALM 167 387 0 20 Feb 2025
Retrieval-Enhanced Machine Learning: Synthesis and Opportunities To Eun Kim Alireza Salemi Andrew Drozdov Fernando Diaz Hamed Zamani 81 7 0 17 Jul 2024
OpenVLA: An Open-Source Vision-Language-Action Model Moo Jin Kim Karl Pertsch Siddharth Karamcheti Ted Xiao Ashwin Balakrishna ... Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn LM&Ro VLM 136 425 0 13 Jun 2024
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation Mukul Khanna Ram Ramrakhya Gunjan Chhablani Sriram Yenamandra Théophile Gervet Matthew Chang Z. Kira Devendra Singh Chaplot Dhruv Batra Roozbeh Mottaghi LM&Ro 73 30 0 09 Apr 2024
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs Soroush Nasiriany Fei Xia Wenhao Yu Ted Xiao Jacky Liang ... Karol Hausman N. Heess Chelsea Finn Sergey Levine Brian Ichter LM&Ro LRM 40 98 0 12 Feb 2024
Agent AI: Surveying the Horizons of Multimodal Interaction Zane Durante Qiuyuan Huang Naoki Wake Ran Gong Jinho Park ... Yejin Choi Katsushi Ikeuchi Hoi Vo Fei-Fei Li Jianfeng Gao LM&Ro 121 83 0 07 Jan 2024
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis Yafei Hu Quanting Xie Vidhi Jain Jonathan M Francis Jay Patrikar ... Xiaolong Wang Sebastian A. Scherer Z. Kira Fei Xia Yonatan Bisk LM&Ro AI4CE 61 69 0 14 Dec 2023
GOAT: GO to Any Thing Matthew Chang Théophile Gervet Mukul Khanna Sriram Yenamandra Dhruv Shah ... Saurabh Gupta Dhruv Batra Roozbeh Mottaghi Jitendra Malik Devendra Singh Chaplot 48 68 0 10 Nov 2023
Open X-Embodiment: Robotic Learning Datasets and RT-X Models Open X-Embodiment Collaboration Abby OÑeill Abdul Rehman Abhinav Gupta Abhiram Maddukuri ... Zhuo Xu Zichen Jeff Cui Zichen Zhang Zipeng Fu Zipeng Lin LM&Ro 111 487 0 13 Oct 2023
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models Yiyang Zhou Chenhang Cui Jaehong Yoon Linjun Zhang Zhun Deng Chelsea Finn Mohit Bansal Huaxiu Yao MLLM 93 174 0 01 Oct 2023
Benchmarking Large Language Models in Retrieval-Augmented Generation Jiawei Chen Hongyu Lin Xianpei Han Le Sun 3DV RALM 29 282 0 04 Sep 2023
A Survey on Large Language Model based Autonomous Agents Lei Wang Chengbang Ma Xueyang Feng Zeyu Zhang Hao-ran Yang ... Xu Chen Yankai Lin Wayne Xin Zhao Zhewei Wei Ji-Rong Wen LLMAG AI4CE LM&Ro 56 1,225 0 22 Aug 2023
SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning Krishan Rana Jesse Haviland Sourav Garg Jad Abou-Chakra Ian Reid Niko Sünderhauf LM&Ro 56 230 0 12 Jul 2023
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models Gengze Zhou Yicong Hong Qi Wu ELM LM&Ro LLMAG LRM 66 146 0 26 May 2023
Tree of Thoughts: Deliberate Problem Solving with Large Language Models Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas Griffiths Yuan Cao Karthik Narasimhan LM&Ro LRM AI4CE 106 1,850 0 17 May 2023
A System for Generalized 3D Multi-Object Search Kaiyu Zheng Anirudha Paul Stefanie Tellex 3DPC 64 10 0 06 Mar 2023
Self-Supervised Object Goal Navigation with In-Situ Finetuning So Yeon Min Yao-Hung Hubert Tsai Wei Ding Ali Farhadi Ruslan Salakhutdinov Yonatan Bisk Jian Zhang SSL 54 7 0 09 Dec 2022
Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances Jacob Krantz Stefan Lee Jitendra Malik Dhruv Batra Devendra Singh Chaplot LM&Ro VGen 52 30 0 29 Nov 2022
Visual Language Maps for Robot Navigation Chen Huang Oier Mees Andy Zeng Wolfram Burgard LM&Ro 191 355 0 11 Oct 2022
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory Nur Muhammad (Mahi) Shafiullah Chris Paxton Lerrel Pinto Soumith Chintala Arthur Szlam VLM LM&Ro CLIP 110 161 0 11 Oct 2022
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue So Yeon Min Hao Zhu Ruslan Salakhutdinov Yonatan Bisk LM&Ro 76 12 0 10 Oct 2022
Open-vocabulary Queryable Scene Representations for Real World Planning Boyuan Chen F. Xia Brian Ichter Kanishka Rao K. Gopalakrishnan Michael S. Ryoo Austin Stone Daniel Kappler LM&Ro 171 183 0 20 Sep 2022
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions Jing Gu Eliana Stefani Qi Wu Jesse Thomason Xinze Wang LM&Ro 86 105 0 22 Mar 2022
Video Question Answering: Datasets, Algorithms and Challenges Yaoyao Zhong Junbin Xiao Wei Ji Yicong Li Wei Deng Tat-Seng Chua 68 89 0 02 Mar 2022
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following Xiaofeng Gao Qiaozi Gao Ran Gong Kaixiang Lin Govind Thattai Gaurav Sukhatme LM&Ro 107 70 0 27 Feb 2022
Survey of Hallucination in Natural Language Generation Ziwei Ji Nayeon Lee Rita Frieske Tiezheng Yu D. Su ... Delong Chen Wenliang Dai Ho Shu Chan Andrea Madotto Pascale Fung HILM LRM 172 2,327 0 08 Feb 2022
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization Nathan Hughes Yun Chang Luca Carlone 3DPC 143 143 0 31 Jan 2022
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning Santhosh Kumar Ramakrishnan Devendra Singh Chaplot Ziad Al-Halah Jitendra Malik Kristen Grauman 82 155 0 25 Jan 2022
FILM: Following Instructions in Language with Modular Methods So Yeon Min Devendra Singh Chaplot Pradeep Ravikumar Yonatan Bisk Ruslan Salakhutdinov LM&Ro 235 161 0 12 Oct 2021
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI Santhosh Kumar Ramakrishnan Aaron Gokaslan Erik Wijmans Oleksandr Maksymets Alexander Clegg ... Andrew Westbury Angel X. Chang Manolis Savva Yili Zhao Dhruv Batra 43 375 0 16 Sep 2021
Knowledge-based Embodied Question Answering Sinan Tan Mengmeng Ge Di Guo Huaping Liu F. Sun 60 20 0 16 Sep 2021
NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions Junbin Xiao Xindi Shang Angela Yao Tat-Seng Chua 65 459 0 18 May 2021
Retrieval Augmentation Reduces Hallucination in Conversation Kurt Shuster Spencer Poff Moya Chen Douwe Kiela Jason Weston HILM 69 721 0 15 Apr 2021
Memory-Augmented Reinforcement Learning for Image-Goal Navigation Lina Mezghani Sainbayar Sukhbaatar Thibaut Lavril Oleksandr Maksymets Dhruv Batra Piotr Bojanowski Alahari Karteek 49 70 0 13 Jan 2021
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding Alexander Ku Peter Anderson Roma Patel Eugene Ie Jason Baldridge 62 305 0 15 Oct 2020
Object Goal Navigation using Goal-Oriented Semantic Exploration Devendra Singh Chaplot Dhiraj Gandhi Abhinav Gupta Ruslan Salakhutdinov 83 506 0 01 Jul 2020
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks Mohit Shridhar Jesse Thomason Daniel Gordon Yonatan Bisk Winson Han Roozbeh Mottaghi Luke Zettlemoyer Dieter Fox LM&Ro 77 760 0 03 Dec 2019
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments Yuankai Qi Qi Wu Peter Anderson Xinze Wang Wenjie Wang Chunhua Shen Anton Van Den Hengel LM&Ro 76 320 0 23 Apr 2019
Multi-Target Embodied Question Answering Licheng Yu Xinlei Chen Georgia Gkioxari Joey Tianyi Zhou Tamara L. Berg Dhruv Batra 47 103 0 09 Apr 2019
On Evaluation of Embodied Navigation Agents Peter Anderson Angel X. Chang Devendra Singh Chaplot Alexey Dosovitskiy Saurabh Gupta ... Jana Kosecka Jitendra Malik Roozbeh Mottaghi Manolis Savva Amir Zamir 97 791 0 18 Jul 2018
Embodied Question Answering Abhishek Das Samyak Datta Georgia Gkioxari Stefan Lee Devi Parikh Dhruv Batra LM&Ro 71 644 0 30 Nov 2017
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments Peter Anderson Qi Wu Damien Teney Jake Bruce Mark Johnson Niko Sünderhauf Ian Reid Stephen Gould Anton Van Den Hengel LM&Ro 79 1,299 0 20 Nov 2017
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles S. Shah Debadeepta Dey Chris Lovett Ashish Kapoor 77 1,976 0 15 May 2017
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning Yuke Zhu Roozbeh Mottaghi Eric Kolve Joseph J. Lim Abhinav Gupta Li Fei-Fei Ali Farhadi VGen 54 1,516 0 16 Sep 2016