Research map / Vision-language and multimodal / CLIP and zero-shot

LLMs can see and hear without any training

ICML 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MILS, a training-free test-time loop where an LLM generator proposes candidates and an off-the-shelf multimodal scorer ranks them, iterating to solve captioning, generation, style transfer and cross-modal arithmetic tasks.

Problem

Enabling multimodal understanding and generation capabilities in text-only LLMs without any task-specific training or data curation, i.e., emergent zero-shot performance on new tasks and modalities.

Approach

An LLM (Generator) proposes candidate outputs, an off-the-shelf multimodal model (Scorer such as CLIP/SigLIP/ViCLIP/ImageBind/PickScore) scores them, and the scores are fed back in text to iteratively refine candidates; no weights are updated and the method is gradient-free, so generator and scorer can be swapped across tasks and modalities.

Key results

• On MSCOCO zero-shot image captioning, MILS reaches CIDEr 33.3 and METEOR 15.0 with no captioning training, competitive with MeaCap (CIDEr 42.5 and METEOR 16.6) and above MeaCap re-run (CIDEr 26.0, METEOR 14.1).
• On MSR-VTT zero-shot video captioning, MILS (METEOR 14.4) outperforms the HowTo100M-trained baseline on METEOR and exceeds the VideoCC3M-trained baseline on METEOR (11.3).
• On Clotho audio captioning, MILS gets METEOR 12.4 and SPICE 7.6, beating ZerAuCap (9.4 and 5.3) on these semantic metrics.

Measured effects reported in the paper

Surprising findings

• SigLIP alone cannot caption, yet paired with the LLM generator in MILS it becomes an effective zero-shot captioner.
• MILS can invert multimodal embeddings (images, audio) into text, enabling cross-modal arithmetic without gradients.

Limitations

• Speed of the iterative optimization process is a bottleneck.
• Performance is bounded by the Generator's ability to produce diverse candidates and the Scorer's feedback accuracy.
• Style transfer is limited by Gram-matrix texture resolution and the LLM's ability to describe styles.

Experimental setup

Datasets: ImageNet class labels (for bootstrap candidate captions) AudioSet class labels (for audio bootstrap captions) DrawBench prompts

Benchmarks: MSCOCO captioning test set MSR-VTT Clotho DrawBench

Metrics: BLEU-4 CIDEr METEOR SPICE CLIP-S BLIP2-S ROUGE-L human win rate (quality and faithfulness)

Baselines: ZeroCap ConZIC CLIPRe MeaCap ZerAuCap Nagrani et al. video captioning

Builds on: ZeroCap ImageBind chain-of-thought

Closest papers by meaning

• Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.88)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs (CVPR 2024 · similarity 0.88)
• Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.88)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
• Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.87)