ICLR 2023 accepted papers on Vision-language and multimodal

53 of the 1,575 papers accepted at ICLR 2023 (3.4%) are on Vision-language and multimodal, compared with 1.9% at ICLR 2022. They fall into 6 topics.

See ICLR 2023 in the interactive map

Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.

Topics and papers

Human and embodied modeling · 18 papers

PaLI: A Jointly-Scaled Multilingual Language-Image Model (193 citations)
UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks (110 citations)
When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? (37 citations)
StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training (17 citations)
Human alignment of neural network representations (16 citations)
Multimodal Analogical Reasoning over Knowledge Graphs (10 citations)
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus (10 citations)
CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks (9 citations)
Unified Discrete Diffusion for Simultaneous Vision-Language Generation (8 citations)
Words are all you need? Language as an approximation for human similarity judgments (8 citations)
Learning to reason over visual objects (7 citations)
Understanding Embodied Reference with Touch-Line Transformer (6 citations)

and 6 more in the interactive map.

CLIP and zero-shot · 15 papers

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (169 citations)
Visual Classification via Description from Large Language Models (56 citations)
Learning to Compose Soft Prompts for Compositional Zero-Shot Learning (41 citations)
Linearly Mapping from Image to Text Space (26 citations)
DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training (24 citations)
MEDICAL IMAGE UNDERSTANDING WITH PRETRAINED VISION LANGUAGE MODELS: A COMPREHENSIVE STUDY (24 citations)
PLOT: Prompt Learning with Optimal Transport for Vision-Language Models (23 citations)
LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval (13 citations)
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention (11 citations)
Is a Caption Worth a Thousand Images? A Study on Representation Learning (9 citations)
Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning (8 citations)
Composing Ensembles of Pre-trained Models via Iterative Consensus (7 citations)

and 3 more in the interactive map.

Cross-modal learning · 11 papers

Masked Vision and Language Modeling for Multi-modal Representation Learning (23 citations)
Contrastive Audio-Visual Masked Autoencoder (16 citations)
Jointly Learning Visual and Auditory Speech Representations from Raw Data (13 citations)
MultiViz: Towards Visualizing and Understanding Multimodal Models (13 citations)
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation (12 citations)
Learning Multimodal Data Augmentation in Feature Space (9 citations)
Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval (6 citations)
Identifiability Results for Multimodal Contrastive Learning (4 citations)
CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos (2 citations)
Diagnosing and Rectifying Vision Models using Language (2 citations)
Filter-Recovery Network for Multi-Speaker Audio-Visual Speech Separation

Visual question answering · 5 papers

Robot action and navigation · 3 papers

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation (3 citations)
Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation (2 citations)
Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks (2 citations)

Multimodal LLMs · 1 paper

GAMR: A Guided Attention Model for (visual) Reasoning (4 citations)