Research map / Vision-language and multimodal / CLIP and zero-shot
One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
Lorenzo Bianchi, Giacomo Pacini, Fabio Carrara, Nicola Messina, Giuseppe Amato, Fabrizio Falchi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla CLIP backbone with Talk2DINO (ablation) | Patch-ioner · Trace captioning (COCO) | CIDEr: 10.9 -> 27.9 | Improved |
| replace vanilla CLIP backbone with Talk2DINO (ablation) | Patch-ioner · Dense captioning (VG v1.2) | CIDEr: 10.9 -> 31.9 | Improved |
| replace vanilla CLIP backbone with Talk2DINO (ablation) | Patch-ioner · Region-set captioning (COCO Entities) | CIDEr: 41.6 -> 109.1 | Improved |
| use dense-adapted CLIP (INViTE, DenseCLIP) instead of vanilla CLIP (ablation) | Patch-ioner · Trace captioning (COCO) | CIDEr: 10.9 -> 13.8 (INViTE), 18.6 (DenseCLIP) | Improved |
| Talk2DINO instead of DINO.txt backbone (ablation) | Patch-ioner · Trace captioning (COCO) | CIDEr: 23.2 -> 27.9 | Improved |
| patch-centric framework vs region-supervised AlphaCLIP | Patch-ioner T2D + Mem. · Region-set captioning (COCO Entities) | CIDEr: 95.1 -> 109.1 | Improved |
| patch-centric framework vs whole-image zero-shot captioners | Patch-ioner T2D + Mem. · Trace captioning (COCO) | CIDEr: 24.3 (ViECap best) -> 27.9 | Improved |
| patch-centric framework vs dedicated image captioners | Patch-ioner T2D + Mem. · Image captioning (COCO) | CIDEr: 96.0 (MERCap) -> 88.5 | Hurt |
| change modality-gap mitigation (noise, external knowledge, diffusion) instead of memory (ablation) | Patch-ioner T2D · region-set and trace captioning | CIDEr: Region-set CIDEr 109.1 -> 114.4 (diffusion), 97.5 (noise) | Mixed |
| switch patch aggregation operator (uniform, gaussian, attention) (ablation) | Patch-ioner · regional captioning | captioning metrics | No clear effect |
| use attention-based weighting instead of mean aggregation (ablation) | Patch-ioner T2D + Mem. · Image captioning (COCO) | CIDEr: 69.2 -> 88.5 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DeCap — DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training (ICLR 2023) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.93)
- A Simple Framework for Open-Vocabulary Zero-Shot Segmentation (ICLR 2025 · similarity 0.89)
- MeaCap: Memory-Augmented Zero-shot Image Captioning (CVPR 2024 · similarity 0.89)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.89)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.89)
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
