ECCV 2024 accepted papers on Vision-language and multimodal
243 of the 2,387 papers accepted at ECCV 2024 (10.2%) are on Vision-language and multimodal, compared with 4.9% at ECCV 2022. They fall into 6 topics.
See ECCV 2024 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Multimodal LLMs · 80 papers
- MMBENCH: Is Your Multi-Modal Model an All-around Player? (1,494 citations)
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (816 citations)
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (362 citations)
- BLINK: Multimodal Large Language Models Can See but Not Perceive (349 citations)
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (339 citations)
- MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models (232 citations)
- LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents (187 citations)
- LLaVA-UHD: an LMM Perceiving any Aspect Ratio and High-Resolution Images (172 citations)
- LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model (152 citations)
- Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs (129 citations)
- Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models (115 citations)
- How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs (110 citations)
and 68 more in the interactive map.
CLIP and zero-shot · 61 papers
- Long-CLIP: Unlocking the Long-Text Capability of CLIP (265 citations)
- AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection (195 citations)
- SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference (141 citations)
- UniIR: Training and Benchmarking Universal Multimodal Information Retrievers (95 citations)
- VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation (82 citations)
- VeCLIP: Improving CLIP Training via Visual-enriched Captions (65 citations)
- When Do We Not Need Larger Vision Models? (55 citations)
- SILC: Improving Vision Language Pretraining with Self-Distillation (45 citations)
- Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval (45 citations)
- EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding (40 citations)
- Prompting Language-Informed Distribution for Compositional Zero-Shot Learning (36 citations)
- Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models (36 citations)
and 49 more in the interactive map.
Cross-modal learning · 40 papers
- Decoupling Common and Unique Representations for Multimodal Self-supervised Learning (80 citations)
- MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning (69 citations)
- ReMamber: Referring Image Segmentation with Mamba Twister (63 citations)
- Diagnosing and Re-learning for Balanced Multimodal Learning (44 citations)
- UMBRAE: Unified Multimodal Brain Decoding (31 citations)
- GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation (29 citations)
- Improving Medical Multi-modal Contrastive Learning with Expert Annotations (29 citations)
- Robust Multimodal Learning via Representation Decoupling (26 citations)
- When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset (26 citations)
- Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes (25 citations)
- TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data (24 citations)
- Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation (23 citations)
and 28 more in the interactive map.
Human and embodied modeling · 32 papers
- SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation (53 citations)
- MotionChain: Conversational Motion Controllers via Multimodal Prompts (20 citations)
- SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs (20 citations)
- PLOT: Text-based Person Search with Part Slot Attention for Corresponding Part Discovery (19 citations)
- Multi-branch Collaborative Learning Network for 3D Visual Grounding (17 citations)
- Finding Visual Task Vectors (14 citations)
- Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment (12 citations)
- DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation (11 citations)
- GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths (11 citations)
- Parrot Captions Teach CLIP to Spot Text (11 citations)
- Seeing Faces in Things: A Model and Dataset for Pareidolia (11 citations)
- FreestyleRet: Retrieving Images from Style-Diversified Queries (9 citations)
and 20 more in the interactive map.
Visual question answering · 16 papers
- DriveLM: Driving with Graph Visual Question Answering (465 citations)
- Evaluating Text-to-Visual Generation with Image-to-Text Generation (282 citations)
- CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios (37 citations)
- MyVLM: Personalizing VLMs for User-Specific Queries (37 citations)
- WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering (33 citations)
- Fully Authentic Visual Question Answering Dataset from Online Communities (10 citations)
- Grounding Language Models for Visual Entity Recognition (9 citations)
- Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge (9 citations)
- Recursive Visual Programming (9 citations)
- Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations (7 citations)
- Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality (7 citations)
- Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following (3 citations)
and 4 more in the interactive map.
Robot action and navigation · 14 papers
- NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models (88 citations)
- Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation (75 citations)
- Octopus: Embodied Vision-Language Programmer from Environmental Feedback (64 citations)
- QUAR-VLA: Vision-Language-Action Model for Quadruped Robots (41 citations)
- BlenderAlchemy: Editing 3D Graphics with Vision-Language Models (35 citations)
- Prioritized Semantic Learning for Zero-shot Instance Navigation (23 citations)
- Navigation Instruction Generation with BEV Perception and Large Language Models (18 citations)
- LLM as Copilot for Coarse-grained Vision-and-Language Navigation (12 citations)
- PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigation (9 citations)
- "Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos" (7 citations)
- Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts (7 citations)
- DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control (7 citations)
and 2 more in the interactive map.
