The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning

24 February 2025

Papers citing "The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning"

38 / 38 papers shown

Title
HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving Zhiwen Chen Bo Leng Zhuoren Li Hanming Deng Guizhe Jin Ran Yu Huanxi Wen 106 0 0 21 May 2025
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning Yuexiang Zhai Hao Bai Zipeng Lin Jiayi Pan Shengbang Tong ... Alane Suhr Saining Xie Yann LeCun Yi-An Ma Sergey Levine LLMAG LRM 68 72 0 16 May 2024
Learning Reward for Robot Skills Using Large Language Models via Self-Alignment Yuwei Zeng Yao Mu Lin Shao 52 12 0 12 May 2024
Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks Murtaza Dalal Tarun Chiruvolu Devendra Singh Chaplot Ruslan Salakhutdinov LM&Ro 81 42 0 02 May 2024
Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts Jing-Cheng Pang Si-Hang Yang Kaiyuan Li Jiaji Zhang Xiong-Hui Chen Nan Tang Yang Yu OffRL KELM LLMAG 51 5 0 14 Apr 2024
Language-guided Skill Learning with Temporal Variational Inference Haotian Fu Pratyusha Sharma Elias Stengel-Eskin George Konidaris Nicolas Le Roux Marc-Alexandre Côté Xingdi Yuan 63 7 0 26 Feb 2024
PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning Simon Holk Daniel Marta Iolanda Leite 73 12 0 23 Feb 2024
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback Yufei Wang Zhanyi Sun Jesse Zhang Zhou Xian Erdem Biyik David Held Zackory M. Erickson VLM 63 55 0 06 Feb 2024
Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents Zihao Zhou Bin-Bin Hu Chenyang Zhao Pu Zhang Bin Liu LLMAG 54 10 0 22 Nov 2023
Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game Zelai Xu Chao Yu Fei Fang Yu Wang Yi Wu LLMAG 68 84 0 29 Oct 2023
Large Language Models as Generalizable Policies for Embodied Tasks Andrew Szot Max Schwarzer Harsh Agrawal Bogdan Mazoure Walter A. Talbott Katherine Metcalf Natalie Mackraz Devon Hjelm Alexander Toshev LM&Ro 56 62 0 26 Oct 2023
Eureka: Human-Level Reward Design via Coding Large Language Models Yecheng Jason Ma William Liang Guanzhi Wang De-An Huang Osbert Bastani Dinesh Jayaraman Yuke Zhu Linxi Fan A. Anandkumar 47 304 0 19 Oct 2023
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning Juan Rocamonde Victoriano Montesinos Elvis Nava Ethan Perez David Lindner VLM 42 81 0 19 Oct 2023
Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance Jesse Zhang Jiahui Zhang Karl Pertsch Ziyi Liu Xiang Ren Minsuk Chang Shao-Hua Sun Joseph J Lim LLMAG LM&Ro 134 62 0 16 Oct 2023
LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents Yash Shukla Wenchang Gao Vasanth Sarathy Alvaro Velasquez Robert Wright Jivko Sinapov 69 9 0 14 Oct 2023
AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback Wanpeng Zhang Zongqing Lu LLMAG 104 6 0 29 Sep 2023
ExpeL: LLM Agents Are Experiential Learners Andrew Zhao Daniel Huang Quentin Xu Matthieu Lin Yang Liu Gao Huang LLMAG 46 205 0 20 Aug 2023
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization Weiran Yao Shelby Heinecke Juan Carlos Niebles Zhiwei Liu Yihao Feng ... Ran Xu P. Mùi Haiquan Wang Caiming Xiong Silvio Savarese LLMAG LM&Ro 41 77 0 04 Aug 2023
SayTap: Language to Quadrupedal Locomotion Yujin Tang Wenhao Yu Jie Tan Heiga Zen Aleksandra Faust Tatsuya Harada 49 41 0 13 Jun 2023
Large Language Models Are Semi-Parametric Reinforcement Learning Agents Danyang Zhang Lu Chen Situo Zhang Hongshen Xu Zihan Zhao Kai Yu LM&Ro KELM LLMAG 47 21 0 09 Jun 2023
Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach Bin-Bin Hu Chenyang Zhao Pushi Zhang Zihao Zhou Yuanhang Yang Zenglin Xu Bin Liu LM&Ro LLMAG 60 22 0 06 Jun 2023
Augmenting Autotelic Agents with Large Language Models Cédric Colas Laetitia Teodorescu Pierre-Yves Oudeyer Xingdi Yuan Marc-Alexandre Côté LLMAG LM&Ro 35 22 0 21 May 2023
Language Instructed Reinforcement Learning for Human-AI Coordination Hengyuan Hu Dorsa Sadigh LM&Ro 49 61 0 13 Apr 2023
Text2Motion: From Natural Language Instructions to Feasible Plans Kevin Qinghong Lin Christopher Agia Toki Migimatsu Marco Pavone Jeannette Bohg LM&Ro 49 272 0 21 Mar 2023
Reflexion: Language Agents with Verbal Reinforcement Learning Noah Shinn Federico Cassano Beck Labash A. Gopinath Karthik Narasimhan Shunyu Yao LLMAG KELM 32 1,190 0 20 Mar 2023
PaLM-E: An Embodied Multimodal Language Model Danny Driess F. Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery ... Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch Peter R. Florence LM&Ro 32 1,594 0 06 Mar 2023
Reward Design with Language Models Minae Kwon Sang Michael Xie Kalesha Bullard Dorsa Sadigh LM&Ro 84 209 0 27 Feb 2023
Guiding Pretraining in Reinforcement Learning with Large Language Models Yuqing Du Olivia Watkins Zihan Wang Cédric Colas Trevor Darrell Pieter Abbeel Abhishek Gupta Jacob Andreas LM&Ro 44 177 0 13 Feb 2023
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning Thomas Carta Clément Romac Thomas Wolf Sylvain Lamprier Olivier Sigaud Pierre-Yves Oudeyer LM&Ro LLMAG 37 185 0 06 Feb 2023
Large Language Models can Implement Policy Iteration Ethan A. Brooks Logan Walls Richard L. Lewis Satinder Singh LM&Ro OffRL 142 21 0 07 Oct 2022
Inner Monologue: Embodied Reasoning through Planning with Language Models Wenlong Huang F. Xia Ted Xiao Harris Chan Jacky Liang ... Tomas Jackson Linda Luu Sergey Levine Karol Hausman Brian Ichter LLMAG LM&Ro LRM 65 880 0 12 Jul 2022
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge Linxi Fan Guanzhi Wang Yunfan Jiang Ajay Mandlekar Yuncong Yang Haoyi Zhu Andrew Tang De-An Huang Yuke Zhu Anima Anandkumar LM&Ro 88 366 0 17 Jun 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 105 1,901 0 04 Apr 2022
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 592 28,659 0 26 Feb 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 432 41,106 0 28 May 2020
Leveraging human knowledge in tabular reinforcement learning: A study of human subjects Ariel Rosenfeld Moshe Cohen Matthew E. Taylor Sarit Kraus OffRL 24 31 0 15 May 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 201 18,685 0 20 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 338 129,831 0 12 Jun 2017