AAAI 2024 accepted papers on Vision-language and multimodal

Human and embodied modeling · 78 papers

• PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation (123 citations)
• Reinforced Adaptive Knowledge Learning for Multimodal Fake News Detection (56 citations)
• DocFormerv2: Local Features for Document Understanding (52 citations)
• Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (51 citations)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations (40 citations)
• Multi-Prompts Learning with Cross-Modal Alignment for Attribute-Based Person Re-identification (39 citations)
• SECap: Speech Emotion Captioning with Large Language Model (38 citations)
• CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare (36 citations)
• Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (32 citations)
• Automatic Radiology Reports Generation via Memory Alignment Network (29 citations)
• Parallel Vertex Diffusion for Unified Visual Grounding (28 citations)
• Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (28 citations)

and 66 more in the interactive map.

Cross-modal learning · 48 papers

• Bi-directional Adapter for Multimodal Tracking (132 citations)
• AVSegFormer: Audio-Visual Segmentation with Transformer (67 citations)
• A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (59 citations)
• Generative-Based Fusion Mechanism for Multi-Modal Tracking (59 citations)
• TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (58 citations)
• DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency (56 citations)
• Unifying Visual and Vision-Language Tracking via Contrastive Learning (56 citations)
• Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (52 citations)
• Frequency Spectrum Is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector (46 citations)
• Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (44 citations)
• Unifying Multi-Modal Uncertainty Modeling and Semantic Alignment for Text-to-Image Person Re-identification (39 citations)
• Data Roaming and Quality Assessment for Composed Image Retrieval (36 citations)

and 36 more in the interactive map.

CLIP and zero-shot · 41 papers

• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (279 citations)
• VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection (206 citations)
• An Empirical Study of CLIP for Text-Based Person Search (99 citations)
• SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (91 citations)
• MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (79 citations)
• Delving into Multimodal Prompting for Fine-Grained Visual Classification (70 citations)
• VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting (43 citations)
• SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger (42 citations)
• ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization (41 citations)
• A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (38 citations)
• DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval (34 citations)
• CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model (24 citations)

and 29 more in the interactive map.

Visual question answering · 20 papers

• NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario (157 citations)
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (94 citations)
• EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering (50 citations)
• UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (24 citations)
• Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (20 citations)
• Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (20 citations)
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models (10 citations)
• Object Attribute Matters in Visual Question Answering (9 citations)
• Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA (9 citations)
• Exploiting the Social-Like Prior in Transformer for Visual Reasoning (8 citations)
• Cross-Modal Feature Distribution Calibration for Few-Shot Visual Question Answering (7 citations)
• BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining (5 citations)

and 8 more in the interactive map.

Multimodal LLMs · 7 papers

• Detecting and Preventing Hallucinations in Large Vision Language Models (145 citations)
• PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology (59 citations)
• Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning (43 citations)
• VIGC: Visual Instruction Generation and Correction (28 citations)
• UMIE: Unified Multimodal Information Extraction with Instruction Tuning (13 citations)
• Visual Instruction Tuning with Polite Flamingo (13 citations)
• Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models (9 citations)

Robot action and navigation · 4 papers

• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (184 citations)
• Augmented Commonsense Knowledge for Remote Object Grounding (18 citations)
• WebVLN: Vision-and-Language Navigation on Websites (11 citations)
• Angle Robustness Unmanned Aerial Vehicle Navigation in GNSS-Denied Scenarios (2 citations)