ICML 2024 accepted papers on Vision-language and multimodal
106 of the 2,610 papers accepted at ICML 2024 (4.1%) are on Vision-language and multimodal, compared with 2.2% at ICML 2023. They fall into 6 topics.
See ICML 2024 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
Human and embodied modeling · 26 papers
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels (33 citations)
- ESM All-Atom: Multi-Scale Protein Language Model for Unified Molecular Modeling (10 citations)
- Image Fusion via Vision-Language Model (9 citations)
- A Touch, Vision, and Language Dataset for Multimodal Alignment (8 citations)
- The Good, The Bad, and Why: Unveiling Emotions in Generative AI (7 citations)
- SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals (6 citations)
- Unveiling the Potential of AI for Nanomaterial Morphology Prediction (3 citations)
- Creative Text-to-Audio Generation via Synthesizer Programming (2 citations)
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (2 citations)
- A Multimodal Automated Interpretability Agent (1 citation)
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (1 citation)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (1 citation)
and 14 more in the interactive map.
CLIP and zero-shot · 24 papers
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (6 citations)
- Scalable Pre-training of Large Autoregressive Image Models (6 citations)
- LangCell: Language-Cell Pre-training for Cell Identity Understanding (5 citations)
- Revisiting the Role of Language Priors in Vision-Language Models (4 citations)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (3 citations)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (3 citations)
- Improving fine-grained understanding in image-text pre-training (1 citation)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (1 citation)
- Non-confusing Generation of Customized Concepts in Diffusion Models (1 citation)
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (1 citation)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models
- Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data
and 12 more in the interactive map.
Multimodal LLMs · 20 papers
- NExT-GPT: Any-to-Any Multimodal LLM (94 citations)
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (59 citations)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (10 citations)
- GPT-4V(ision) is a Generalist Web Agent, if Grounded (9 citations)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (6 citations)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (6 citations)
- LLark: A Multimodal Instruction-Following Language Model for Music (4 citations)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (4 citations)
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding (3 citations)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (1 citation)
- Data-efficient Large Vision Models through Sequential Autoregression (1 citation)
- Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View (1 citation)
and 8 more in the interactive map.
Cross-modal learning · 19 papers
- GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer (7 citations)
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance (5 citations)
- Position: Why Tabular Foundation Models Should Be a Research Priority (5 citations)
- Predictive Dynamic Fusion (5 citations)
- Image Clustering with External Guidance (4 citations)
- Learning Modality Knowledge Alignment for Cross-Modality Transfer (2 citations)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (2 citations)
- Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generation (1 citation)
- EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning (1 citation)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion
- FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation
and 7 more in the interactive map.
Robot action and navigation · 13 papers
- An Embodied Generalist Agent in 3D World (15 citations)
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (14 citations)
- VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model (8 citations)
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs (5 citations)
- RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis (3 citations)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (1 citation)
- SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation (1 citation)
- DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning
- Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
- VinT-6D: A Large-Scale Object-in-hand Dataset from Vision, Touch and Proprioception
- Bridging Environments and Language with Rendering Functions and Vision-Language Models
and 1 more in the interactive map.
Visual question answering · 4 papers
- Unifying Image Processing as Visual Prompting Question Answering (6 citations)
- Fool Your (Vision and) Language Model with Embarrassingly Simple Permutations (4 citations)
- Extracting Training Data From Document-Based VQA Models
- Detecting Any instruction-to-answer interaction relationship:Universal Instruction-to-Answer Navigator for Med-VQA