CVPR 2024 accepted papers on Vision-language and multimodal
329 of the 2,716 papers accepted at CVPR 2024 (12.1%) are on Vision-language and multimodal, compared with 7.5% at CVPR 2023. They fall into 6 topics.
See CVPR 2024 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
Human and embodied modeling · 94 papers
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (251 citations)
- GLaMM: Pixel Grounding Large Multimodal Model (175 citations)
- Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (143 citations)
- BioCLIP: A Vision Foundation Model for the Tree of Life (124 citations)
- HIPTrack: Visual Tracking with Historical Prompts (105 citations)
- Sequential Modeling Enables Scalable Learning for Large Vision Models (102 citations)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (78 citations)
- UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity (68 citations)
- Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery (66 citations)
- InstructDiffusion: A Generalist Modeling Interface for Vision Tasks (63 citations)
- Context-Aware Integration of Language and Visual References for Natural Language Tracking (58 citations)
- FairCLIP: Harnessing Fairness in Vision-Language Learning (52 citations)
and 82 more in the interactive map.
CLIP and zero-shot · 88 papers
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (264 citations)
- VILA: On Pre-training for Visual Language Models (224 citations)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (135 citations)
- ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (122 citations)
- Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images (113 citations)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (106 citations)
- Alpha-CLIP: A CLIP Model Focusing on Wherever You Want (91 citations)
- MMA: Multi-Modal Adapter for Vision-Language Models (76 citations)
- TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model (70 citations)
- Language-only Training of Zero-shot Composed Image Retrieval (58 citations)
- Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning (52 citations)
- ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification (52 citations)
and 76 more in the interactive map.
Cross-modal learning · 59 papers
- Equivariant Multi-Modality Image Fusion (233 citations)
- Noisy-Correspondence Learning for Text-to-Image Person Re-identification (126 citations)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (115 citations)
- MRFS: Mutually Reinforcing Image Fusion and Segmentation (87 citations)
- Task-Customized Mixture of Adapters for General Image Fusion (85 citations)
- CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition (78 citations)
- AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (76 citations)
- Probing Synergistic High-Order Interaction in Infrared and Visible Image Fusion (67 citations)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (58 citations)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (57 citations)
- Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (55 citations)
- Multimodal Representation Learning by Alternating Unimodal Adaptation (54 citations)
and 47 more in the interactive map.
Multimodal LLMs · 56 papers
- Improved Baselines with Visual Instruction Tuning (1,490 citations)
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (355 citations)
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (234 citations)
- CogAgent: A Visual Language Model for GUI Agents (173 citations)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (172 citations)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (166 citations)
- SEED-Bench: Benchmarking Multimodal Large Language Models (145 citations)
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (138 citations)
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (124 citations)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (111 citations)
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback (103 citations)
- Compositional Chain-of-Thought Prompting for Large Multimodal Models (92 citations)
and 44 more in the interactive map.
Visual question answering · 17 papers
- Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models (120 citations)
- Generative Multimodal Models are In-Context Learners (97 citations)
- Can I Trust Your Answer? Visually Grounded Video Question Answering (62 citations)
- Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction (47 citations)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (43 citations)
- On Scaling Up a Multilingual Vision and Language Model (43 citations)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (26 citations)
- Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (20 citations)
- Situational Awareness Matters in 3D Vision Language Reasoning (15 citations)
- GRAM: Global Reasoning for Multi-Page VQA (13 citations)
- How to Configure Good In-Context Sequence for Visual Question Answering (13 citations)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (12 citations)
and 5 more in the interactive map.
Robot action and navigation · 15 papers
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (82 citations)
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action (77 citations)
- Vision-and-Language Navigation via Causal Learning (40 citations)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (32 citations)
- Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation (28 citations)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (27 citations)
- UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (20 citations)
- GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation (18 citations)
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (18 citations)
- CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation (15 citations)
- OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation (13 citations)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (7 citations)
and 3 more in the interactive map.