NeurIPS 2024 accepted papers on Vision-language and multimodal
318 of the 4,560 papers accepted at NeurIPS 2024 (7.0%) are on Vision-language and multimodal, compared with 4.5% at NeurIPS 2023. They fall into 6 topics.
See NeurIPS 2024 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
Multimodal LLMs · 97 papers
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (98 citations)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (93 citations)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (75 citations)
- What matters when building vision-language models? (41 citations)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (37 citations)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (36 citations)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (23 citations)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (22 citations)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (21 citations)
- Biomedical Visual Instruction Tuning with Clinician Preference Alignment (20 citations)
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (20 citations)
- Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing (19 citations)
and 85 more in the interactive map.
Human and embodied modeling · 96 papers
- VMamba: Visual State Space Model (830 citations)
- CogVLM: Visual Expert for Pretrained Language Models (77 citations)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (29 citations)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (28 citations)
- Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts (27 citations)
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (21 citations)
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token (18 citations)
- STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomics (17 citations)
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (15 citations)
- MmCows: A Multimodal Dataset for Dairy Cattle Monitoring (12 citations)
- Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models (12 citations)
- CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation (11 citations)
and 84 more in the interactive map.
CLIP and zero-shot · 48 papers
- PLIP: Language-Image Pre-training for Person Representation Learning (20 citations)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (15 citations)
- Visual Fourier Prompt Tuning (15 citations)
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection (13 citations)
- Revisiting Few-Shot Object Detection with Vision-Language Models (11 citations)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (10 citations)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (9 citations)
- Scene Graph Generation with Role-Playing Large Language Models (9 citations)
- Boosting Vision-Language Models with Transduction (8 citations)
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (7 citations)
- Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective (7 citations)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (7 citations)
and 36 more in the interactive map.
Cross-modal learning · 45 papers
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion (43 citations)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (39 citations)
- Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model (37 citations)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (29 citations)
- Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry (19 citations)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (16 citations)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (15 citations)
- Cross-Modality Perturbation Synergy Attack for Person Re-identification (14 citations)
- Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (10 citations)
- Humanoid Locomotion as Next Token Prediction (9 citations)
- Mixtures of Experts for Audio-Visual Learning (9 citations)
- Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (9 citations)
and 33 more in the interactive map.
Visual question answering · 18 papers
- SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers (12 citations)
- Right this way: Can VLMs Guide Us to See More to Answer Questions? (5 citations)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (4 citations)
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (4 citations)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (4 citations)
- CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes (2 citations)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (2 citations)
- Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios (2 citations)
- LIVE: Learnable In-Context Vector for Visual Question Answering (2 citations)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (2 citations)
- SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors (2 citations)
- VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (2 citations)
and 6 more in the interactive map.
Robot action and navigation · 14 papers
- DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution (14 citations)
- RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation (13 citations)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (8 citations)
- VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (6 citations)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (4 citations)
- Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control (4 citations)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (3 citations)
- HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid (3 citations)
- PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation (3 citations)
- PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation (3 citations)
- Grounding Multimodal Large Language Models in Actions (1 citation)
- Any2Policy: Learning Visuomotor Policy with Any-Modality
and 2 more in the interactive map.