Research map / Vision-language and multimodal / CLIP and zero-shot

Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAFT, a hierarchical vision-language model trained from scratch on 30M long-caption image-text pairs that aligns part-level sub-captions with intermediate visual segments and whole-caption with final image embedding, achieving SOTA long-caption retrieval and zero-shot referring segmentation.

Problem

CLIP-style models struggle to faithfully ground fine-grained details in long, detail-rich captions and rely on dominant global cues, with long-caption retrieval requiring composition of local evidence into scene understanding.

Approach

A part-to-whole hierarchical learning principle: align sub-caption parts with intermediate superpixel-based visual segments via text-grounded attention pooling, and align the full caption with the final image representation, using a fine-to-coarse image encoder and a part-whole text encoder.

Key results

• On long-text retrieval, CAFT reaches 69.7 I2T / 72.4 T2I on DCI and 95.5 / 94.8 on SV-10k, outperforming prior SOTA by up to +5.4 R@1 on SV-10k T2I.
• Zero-shot referring segmentation on RefCOCO val mIoU 28.9 vs. 19.8 for FLAIR and 8.0 for GroupViT.
• Placing part-level alignment at intermediate layers improves average score by +5.6 over whole-level alignment alone (ablation).

Surprising findings

• Adding part-level alignment at the final layer yields only marginal change over whole-level alignment, while placing it at intermediate layers gives a large gain.
• Whole-only inference (alpha=0) is competitive with the default weighted combination, suggesting part-level correspondences are reflected in the global representation.

Limitations

• Trained from scratch only on 30M synthetic MLLM-recaptioned pairs; scaling beyond 30M not tested.
• Short-text retrieval trade-off: CAFT is competitive but not best on some short-caption benchmarks (e.g., SigLIP 2 and FLAIR on COCO/Flickr T2I).
• Hierarchy is limited to two levels (sentence chunks to whole caption); deeper hierarchies are left to future work.

Experimental setup

Datasets: DreamLIP CC3M-recap DreamLIP CC12M-recap DreamLIP YFCC15M-recap Merged-30M

Benchmarks: DCI DOCCI ShareGPT4V-1k ShareGPT4V-10k Urban-1k IIW RefCOCO RefCOCO+ GRef MSCOCO Flickr30k FG-OVD

Metrics: R@1 mIoU

Baselines: OpenCLIP SigLIP Long-CLIP TULIP FineLIP DeBias-CLIP FG-CLIP FG-CLIP 2 LoTLIP FLAIR GroupViT MaskCLIP

Builds on: CLIP SigLIP Long-CLIP FLAIR DreamLIP

Closest papers by meaning

• GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.91)
• Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.90)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
• Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.89)
• Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.89)
• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.89)