NeurIPS 2021 accepted papers on Vision-language and multimodal
60 of the 2,508 papers accepted at NeurIPS 2021 (2.4%) are on Vision-language and multimodal, compared with 1.4% at NeurIPS 2020. They fall into 6 topics.
See NeurIPS 2021 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Human and embodied modeling · 28 papers
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (812 citations)
- A sandbox for prediction and integration of DNA, RNA, and proteins in single cells (91 citations)
- Referring Transformer: A One-step Approach to Multi-task Visual Grounding (72 citations)
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports (67 citations)
- UniDoc: Unified Pretraining Framework for Document Understanding (45 citations)
- Lip to Speech Synthesis with Visual Context Attentional GAN (24 citations)
- Low-dimensional Structure in the Space of Language Representations is Reflected in Brain Responses (24 citations)
- Chest ImaGenome Dataset for Clinical Reasoning (23 citations)
- ProTo: Program-Guided Transformer for Program-Guided Tasks (15 citations)
- Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model (13 citations)
- Physion: Evaluating Physical Prediction from Vision in Humans and Machines (12 citations)
- Learning to Compose Visual Relations (11 citations)
and 16 more in the interactive map.
Visual question answering · 14 papers
- Debiased Visual Question Answering from Feature and Sample Perspectives (39 citations)
- Introspective Distillation for Robust Question Answering (33 citations)
- Human-Adversarial Visual Question Answering (21 citations)
- Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning (17 citations)
- LiRo: Benchmark and leaderboard for Romanian language tasks (16 citations)
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (10 citations)
- One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (7 citations)
- Robust Visual Reasoning via Language Guided Neural Module Networks (5 citations)
- Supervising the Transfer of Reasoning Patterns in VQA (5 citations)
- Learning to Generate Visual Questions with Noisy Supervision (4 citations)
- Alignment Attention by Matching Key and Query Distributions (2 citations)
- Perceptual Score: What Data Modalities Does Your Model Perceive? (1 citation)
and 2 more in the interactive map.
Cross-modal learning · 11 papers
- Attention Bottlenecks for Multimodal Fusion (265 citations)
- Learning with Noisy Correspondence for Cross-modal Matching (61 citations)
- What Makes Multi-Modal Learning Better than Single (Provably) (43 citations)
- Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training (32 citations)
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (22 citations)
- Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions (16 citations)
- Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos (13 citations)
- TriBERT: Human-centric Audio-visual Representation Learning (7 citations)
- Cross-modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning (5 citations)
- Modality-Agnostic Topology Aware Localization (4 citations)
- Learning Signal-Agnostic Manifolds of Neural Fields (1 citation)
Robot action and navigation · 5 papers
- History Aware Multimodal Transformer for Vision-and-Language Navigation (21 citations)
- Landmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision (18 citations)
- SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark (5 citations)
- SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation (4 citations)
- Grounding Spatio-Temporal Language with Transformers (3 citations)
Multimodal LLMs · 1 paper
- ReaSCAN: Compositional Reasoning in Language Grounding (5 citations)
CLIP and zero-shot · 1 paper
- Multimodal Few-Shot Learning with Frozen Language Models (86 citations)
