NeurIPS 2022 accepted papers on Vision-language and multimodal
78 of the 2,901 papers accepted at NeurIPS 2022 (2.7%) are on Vision-language and multimodal, compared with 2.4% at NeurIPS 2021. They fall into 5 topics.
See NeurIPS 2022 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
Human and embodied modeling · 30 papers
- Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (67 citations)
- PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding (59 citations)
- SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders (48 citations)
- Harmonizing the object recognition strategies of deep neural networks with humans (39 citations)
- Mind Reader: Reconstructing complex images from brain activities (35 citations)
- Fine-Grained Semantically Aligned Vision-Language Pre-Training (29 citations)
- Divert More Attention to Vision-Language Tracking (23 citations)
- UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (23 citations)
- Touch and Go: Learning from Human-Collected Vision and Touch (20 citations)
- OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics (18 citations)
- Peripheral Vision Transformer (18 citations)
- 3DB: A Framework for Debugging Computer Vision Models (14 citations)
and 18 more in the interactive map.
CLIP and zero-shot · 22 papers
- LAION-5B: An open large-scale dataset for training next generation image-text models (1,032 citations)
- GLIPv2: Unifying Localization and Vision-Language Understanding (126 citations)
- Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (111 citations)
- Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (71 citations)
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (63 citations)
- CyCLIP: Cyclic Contrastive Language-Image Pretraining (46 citations)
- K-LITE: Learning Transferable Visual Models with External Knowledge (44 citations)
- PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (42 citations)
- DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations (41 citations)
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (40 citations)
- Delving into Out-of-Distribution Detection with Vision-Language Representations (31 citations)
- Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark (29 citations)
and 10 more in the interactive map.
Cross-modal learning · 12 papers
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (285 citations)
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (100 citations)
- Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning (61 citations)
- OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport (34 citations)
- Few-Shot Audio-Visual Learning of Environment Acoustics (17 citations)
- TVLT: Textless Vision-Language Transformer (16 citations)
- u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality (15 citations)
- Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings (6 citations)
- Cross-Linked Unified Embedding for cross-modality representation learning (5 citations)
- A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval (1 citation)
- Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval
- M$^4$I: Multi-modal Models Membership Inference
Visual question answering · 8 papers
- Flamingo: a Visual Language Model for Few-Shot Learning (1,238 citations)
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (215 citations)
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (64 citations)
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (44 citations)
- Towards Improving Faithfulness in Abstractive Summarization (14 citations)
- WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models (6 citations)
- VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives (4 citations)
- Towards Video Text Visual Question Answering: Benchmark and Baseline (1 citation)
Robot action and navigation · 6 papers
- ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings (41 citations)
- Behavior Transformers: Cloning $k$ modes with one stone (31 citations)
- Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation (16 citations)
- VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation (15 citations)
- ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment (9 citations)
- AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments (8 citations)
