NeurIPS 2023 accepted papers on Vision-language and multimodal

Human and embodied modeling · 74 papers

• MotionGPT: Human Motion as a Foreign Language (55 citations)
• OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (48 citations)
• Generating Images with Multimodal Language Models (39 citations)
• Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias (29 citations)
• SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality (25 citations)
• Improving multimodal datasets with image captioning (22 citations)
• Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (21 citations)
• What Makes Good Examples for Visual In-Context Learning? (18 citations)
• Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text (17 citations)
• Brain encoding models based on multimodal transformers can transfer across language and vision (16 citations)
• Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling (15 citations)
• Consensus and Subjectivity of Skin Tone Annotation for ML Fairness (14 citations)

and 62 more in the interactive map.

CLIP and zero-shot · 38 papers

• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (409 citations)
• DataComp: In search of the next generation of multimodal datasets (73 citations)
• Quilt-1M: One Million Image-Text Pairs for Histopathology (55 citations)
• Improving CLIP Training with Language Rewrites (35 citations)
• Pengi: An Audio Language Model for Audio Tasks (20 citations)
• Text-to-Image Diffusion Models are Zero Shot Classifiers (17 citations)
• Towards Label-free Scene Understanding by Vision Foundation Models (15 citations)
• ChatGPT-Powered Hierarchical Comparisons for Image Classification (11 citations)
• Geodesic Multi-Modal Mixup for Robust Fine-Tuning (11 citations)
• Meta-Adapter: An Online Few-shot Learner for Vision-Language Model (10 citations)
• Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models (9 citations)
• An Inverse Scaling Law for CLIP Training (8 citations)

and 26 more in the interactive map.

Cross-modal learning · 22 papers

• VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset (28 citations)
• Incomplete Multimodality-Diffused Emotion Recognition (27 citations)
• MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing (25 citations)
• GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph (21 citations)
• Factorized Contrastive Learning: Going Beyond Multi-view Redundancy (15 citations)
• Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks (9 citations)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval (8 citations)
• SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization (8 citations)
• Cross-modal Active Complementary Learning with Self-refining Correspondence (7 citations)
• Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception (6 citations)
• Achieving Cross Modal Generalization with Multimodal Unified Representation (5 citations)
• Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning (3 citations)

and 10 more in the interactive map.

Visual question answering · 11 papers

• Large Language Models are Visual Reasoning Coordinators (14 citations)
• What You See is What You Read? Improving Text-Image Alignment Evaluation (14 citations)
• EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images (7 citations)
• Foundation Model is Efficient Multimodal Multitask Model Selector (4 citations)
• 3D-Aware Visual Question Answering about Parts, Poses and Occlusions (3 citations)
• Exploring Question Decomposition for Zero-Shot VQA (3 citations)
• Emergent Communication in Interactive Sketch Question Answering (1 citation)
• Implicit Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis (1 citation)
• TOA: Task-oriented Active VQA (1 citation)
• LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering
• Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

Robot action and navigation · 9 papers

• PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation (18 citations)
• AndroidInTheWild: A Large-Scale Dataset For Android Device Control (13 citations)
• Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation (10 citations)
• ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation (10 citations)
• FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation (4 citations)
• Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation (2 citations)
• Guide Your Agent with Adaptive Multimodal Rewards (1 citation)
• DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
• EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

Multimodal LLMs · 7 papers

• Visual Instruction Tuning (1,198 citations)
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (232 citations)
• Language Is Not All You Need: Aligning Perception with Language Models (160 citations)
• Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models (45 citations)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark (40 citations)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective (3 citations)
• VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models (2 citations)