ICLR 2024 accepted papers on Vision-language and multimodal

CLIP and zero-shot · 45 papers

• Emu: Generative Pretraining in Multimodality (28 citations)
• Demystifying CLIP Data (22 citations)
• UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling (20 citations)
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (18 citations)
• Data Filtering Networks (15 citations)
• Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching (15 citations)
• Prompt Learning with Quaternion Networks (14 citations)
• Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages (13 citations)
• Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions (11 citations)
• Tag2Text: Guiding Vision-Language Model via Image Tagging (11 citations)
• Consistency-guided Prompt Learning for Vision-Language Models (10 citations)
• Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (10 citations)

and 33 more in the interactive map.

Human and embodied modeling · 25 papers

• Nougat: Neural Optical Understanding for Academic Documents (20 citations)
• Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization (12 citations)
• Frozen Transformers in Language Models Are Effective Visual Encoder Layers (11 citations)
• Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition (10 citations)
• Towards 3D Molecule-Text Interpretation in Language Models (8 citations)
• Sentence-level Prompts Benefit Composed Image Retrieval (5 citations)
• BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (3 citations)
• CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (3 citations)
• Image Clustering Conditioned on Text Criteria (3 citations)
• Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (3 citations)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (3 citations)
• AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ (2 citations)

and 13 more in the interactive map.

Multimodal LLMs · 19 papers

• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (480 citations)
• Grounding Multimodal Large Language Models to the World (131 citations)
• Ferret: Refer and Ground Anything Anywhere at Any Granularity (43 citations)
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (29 citations)
• Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (23 citations)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (20 citations)
• DreamLLM: Synergistic Multimodal Comprehension and Creation (15 citations)
• Guiding Instruction-based Image Editing via Multimodal Large Language Models (11 citations)
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (11 citations)
• LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (9 citations)
• Kosmos-G: Generating Images in Context with Multimodal Large Language Models (6 citations)
• Jointly Training Large Autoregressive Multimodal Models (5 citations)

and 7 more in the interactive map.

Cross-modal learning · 11 papers

• LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (26 citations)
• Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction (8 citations)
• Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (5 citations)
• Quantifying and Enhancing Multi-modal Robustness with Modality Preference (4 citations)
• Multimodal Molecular Pretraining via Modality Blending (3 citations)
• BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs (2 citations)
• Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond (2 citations)
• Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (1 citation)
• RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (1 citation)
• Bridging Vision and Language Spaces with Assignment Prediction
• MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations

Visual question answering · 9 papers

• The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World (13 citations)
• Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (8 citations)
• Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (2 citations)
• VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models (2 citations)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (1 citation)
• Bootstrapping Variational Information Pursuit with Large Language and Vision Models for Interpretable Image Classification
• EQA-MX: Embodied Question Answering using Multimodal Expression
• LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer
• RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

Robot action and navigation · 7 papers

• Vision-Language Foundation Models as Effective Robot Imitators (19 citations)
• Learning Interactive Real-World Simulators (12 citations)
• Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (4 citations)
• Selective Visual Representations Improve Convergence and Generalization for Embodied AI (2 citations)
• End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon (1 citation)
• What Matters to You? Towards Visual Representation Alignment for Robot Learning
• Learning to Jointly Understand Visual and Tactile Signals