OneLLM: One Framework to Align All Modalities with Language

10 January 2025

Papers citing "OneLLM: One Framework to Align All Modalities with Language"

50 / 168 papers shown

Title
Surprisingly Fragile: Assessing and Addressing Prompt Instability in Multimodal Foundation Models Ian Stewart Sameera Horawalavithana Brendan Kennedy Sai Munikoti Karl Pazdernik AAML 56 2 0 26 Aug 2024
On-Device Language Models: A Comprehensive Review Jiajun Xu Zhiyuan Li Wei Chen Qun Wang Xin Gao Qi Cai Ziyuan Ling 113 35 0 26 Aug 2024
EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning Bohao Xing Zitong Yu Xin Liu Kaishen Yuan Qilang Ye Weicheng Xie Huanjing Yue Jingyu Yang Heikki Kälviäinen 72 13 0 21 Aug 2024
SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data Yichen Lu Álvaro Huertas-García Xuankai Chang Hengwei Bian Soumi Maiti Shinji Watanabe 79 2 0 01 Aug 2024
Decoding Linguistic Representations of Human Brain Yu Wang Heyang Liu Yuhao Wang Chuan Xuan Yixuan Hou Sheng Feng Hongcheng Liu Yusheng Liao Yanfeng Wang AI4CE 67 1 0 30 Jul 2024
Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment Shenghong Dai Shiqi Jiang Yifan Yang Ting Cao Mo Li Suman Banerjee Lili Qiu 54 2 0 25 Jul 2024
MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues Liyun Zhang 44 1 0 23 Jul 2024
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model Yiwei Ma Zhibin Wang Xiaoshuai Sun Weihuang Lin Qiang-feng Zhou Jiayi Ji Rongrong Ji MLLM VLM 73 1 0 23 Jul 2024
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models Haodong Duan Junming Yang Junming Yang Xinyu Fang Lin Chen ... Yuhang Zang Pan Zhang Jiaqi Wang Dahua Lin Kai Chen LM&MA VLM 122 161 0 16 Jul 2024
AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition Zheng Lian Haiyang Sun Guoying Zhao Jiangyan Yi Bin Liu Jianhua Tao 72 2 0 10 Jul 2024
Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review Moseli Motsóehli VLM 3DV 78 0 0 28 Jun 2024
Learning Modality Knowledge Alignment for Cross-Modality Transfer Wenxuan Ma Shuang Li Lincan Cai Jingxuan Kang 65 2 0 27 Jun 2024
Explore the Limits of Omni-modal Pretraining at Scale Yiyuan Zhang Handong Li Jing Liu Xiangyu Yue VLM LRM 73 1 0 13 Jun 2024
Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generation Lincan Cai Shuang Li Wenxuan Ma Jingxuan Kang Binhui Xie Zixun Sun Chengwei Zhu MoE MoMe 72 1 0 13 Jun 2024
Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunities Sai Munikoti Ian Stewart Sameera Horawalavithana Henry Kvinge Tegan H. Emerson Sandra E Thompson Karl Pazdernik 70 2 0 08 Jun 2024
Cross-Modal Safety Alignment: Is textual unlearning all you need? Trishna Chakraborty Erfan Shayegani Zikui Cai Nael B. Abu-Ghazaleh M. Salman Asif Yue Dong Amit K. Roy-Chowdhury Chengyu Song 55 17 0 27 May 2024
Safety Alignment for Vision Language Models Zhendong Liu Yuanbi Nie Yingshui Tan Xiangyu Yue Qiushi Cui Chongjun Wang Xiaoyong Zhu Bo Zheng VLM MLLM 108 12 0 22 May 2024
Creative Problem Solving in Large Language and Vision Models -- What Would it Take? Lakshmi Nair Evana Gizzi Jivko Sinapov MLLM 87 4 0 02 May 2024
MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors Yuan Tang Xu Han Xianzhi Li Qiao Yu Yixue Hao Long Hu Min Chen 49 17 0 02 May 2024
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition Zheng Lian Haiyang Sun Guoying Zhao Zhuofan Wen Siyuan Zhang ... Bin Liu Min Zhang Guoying Zhao Björn W. Schuller Jianhua Tao VLM 86 11 0 26 Apr 2024
UMBRAE: Unified Multimodal Brain Decoding Weihao Xia Raoul de Charette Cengiz Öztireli Jing-Hao Xue 50 9 0 10 Apr 2024
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios Qilang Ye Zitong Yu Rui Shao Xinyu Xie Philip Torr Xiaochun Cao MLLM 91 26 0 07 Mar 2024
User-LLM: Efficient LLM Contextualization with User Embeddings Lin Ning Luyang Liu Jiaxing Wu Neo Wu D. Berlowitz Sushant Prakash Bradley Green S. O’Banion Jun Xie 75 36 0 21 Feb 2024
The Revolution of Multimodal Large Language Models: A Survey Davide Caffagni Federico Cocchi Luca Barsellotti Nicholas Moratelli Sara Sarto Lorenzo Baraldi Lorenzo Baraldi Marcella Cornia Rita Cucchiara LRM VLM 80 57 0 19 Feb 2024
Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing? Marco Gaido Sara Papi Matteo Negri L. Bentivogli 89 16 0 19 Feb 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 113 6 0 08 Feb 2024
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities Yiyuan Zhang Xiaohan Ding Kaixiong Gong Yixiao Ge Ying Shan Xiangyu Yue ViT 95 7 0 25 Jan 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Ping Luo Jiebo Luo Chenliang Xu VLM 104 95 0 29 Dec 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Shafiq Joty Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 99 55 0 30 Nov 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 114 2,695 0 05 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 240 1,805 0 28 Sep 2023
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model Avamarie Brueggeman Andrea Madotto Zhaojiang Lin Tushar Nagarajan Matt Smith ... Peyman Heidari Yue Liu Kavya Srinet Babak Damavandi Anuj Kumar MLLM 58 94 0 27 Sep 2023
ImageBind-LLM: Multi-modality Instruction Tuning Jiaming Han Renrui Zhang Wenqi Shao Peng Gao Peng Xu ... Yafei Wen Xiaoxin Chen Xiangyu Yue Hongsheng Li Yu Qiao MLLM 73 122 0 07 Sep 2023
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following Ziyu Guo Renrui Zhang Xiangyang Zhu Yiwen Tang Xianzheng Ma ... Ke Chen Peng Gao Xianzhi Li Hongsheng Li Pheng-Ann Heng MLLM 75 139 0 01 Sep 2023
PointLLM: Empowering Large Language Models to Understand Point Clouds Runsen Xu Xiaolong Wang Tai Wang Yilun Chen Jiangmiao Pang Dahua Lin MLLM 66 171 0 31 Aug 2023
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities Weihao Yu Zhengyuan Yang Linjie Li Jianfeng Wang Kevin Qinghong Lin Zicheng Liu Xinchao Wang Lijuan Wang MLLM 98 684 0 04 Aug 2023
From Sparse to Soft Mixtures of Experts J. Puigcerver C. Riquelme Basil Mustafa N. Houlsby MoE 158 121 0 02 Aug 2023
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension Bohao Li Rui Wang Guangzhi Wang Yuying Ge Yixiao Ge Ying Shan MLLM ELM 101 549 0 30 Jul 2023
Meta-Transformer: A Unified Framework for Multimodal Learning Yiyuan Zhang Kaixiong Gong Kaipeng Zhang Hongsheng Li Yu Qiao Wanli Ouyang Xiangyu Yue 60 147 0 20 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 280 11,828 0 18 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Jiaqi Wang Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 91 1,013 0 12 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 97 195 0 10 Jul 2023
Explainable Multimodal Emotion Recognition Zheng Lian Haiyang Sun Guoying Zhao Hao Gu Zhuofan Wen ... Shan Liang Ya Li Jiangyan Yi B. Liu Jianhua Tao MLLM 48 6 0 27 Jun 2023
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang ... Xiawu Zheng Ke Li Xing Sun Zhenyu Qiu Rongrong Ji ELM MLLM 83 828 0 23 Jun 2023
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurenccon Lucile Saulnier Léo Tronchon Stas Bekman Amanpreet Singh ... Siddharth Karamcheti Alexander M. Rush Douwe Kiela Matthieu Cord Victor Sanh 105 240 0 21 Jun 2023
Scalable 3D Captioning with Pretrained Models Tiange Luo C. Rockwell Honglak Lee Justin Johnson 45 159 0 12 Jun 2023
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding Hang Zhang Xin Li Lidong Bing MLLM 135 1,030 0 05 Jun 2023
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors Paul S. Scotti Atmadeep Banerjee J. Goode Stepan Shabalin A. Nguyen ... Nathalie Verlinde Elad Yundler David Weisberg K. A. Norman Tanishq Mathew Abraham DiffM 77 118 0 29 May 2023
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Sihan Chen Handong Li Qunbo Wang Zijia Zhao Ming-Ting Sun Xinxin Zhu Qingbin Liu 129 105 0 29 May 2023
Multi-Scale Attention for Audio Question Answering Guangyao Li Yixin Xu Di Hu 32 16 0 29 May 2023