Research map / Vision-language and multimodal / CLIP and zero-shot

Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Text as Partial Constraint (TPC), a core–residual vision–language alignment method that aligns images to a multi-caption consensus core, learns a single-view core filter, penalizes residual correlation, and uses disagreement-aware temperature, reporting gains in zero-shot robustness and LVLM transfer.

Problem

Natural captions are underspecified, so contrastive vision-language alignment is brittle under paraphrase and omitted details and becomes overconfident, hurting zero-shot robustness and LVLM hallucination.

Approach

Treat multi-view captions as partial supervision: align images to a consensus core of caption views, distill that core into a single-view linear filter for one-query inference, penalize image correlation with the orthogonal unsaid residual, and soften InfoNCE temperature with caption-view disagreement.

Key results

• TPC reaches 81.42/64.05 clean/robust Top-1 on ImageNet, versus 80.26/61.46 for the best prior robust baseline TGA-ZSR.
• On Avg-14, TPC reaches 76.19/52.03 clean/robust, outperforming Adv-W2S (68.75/48.38).
• Used as a drop-in vision encoder in LLaVA-1.5-7B, TPC reaches 85.16 average POPE F1 and 59.57 OKVQA accuracy.

Limitations

• Multi-view supervision requires K captions per image, with synthetic views needed for single-caption corpora.
• Reported gains are from a single paper with the method's own hyperparameters; no code or weights release stated.

Experimental setup

Datasets: MS-COCO Flickr30K ImageNet-1K

Benchmarks: ImageNet-1K Avg-14 transfer suite POPE GQA SciQA VizWiz OKVQA MS-COCO Flickr30K

Metrics: Top-1 accuracy robust accuracy POPE F1 VQA accuracy Recall@K ECE NLL

Baselines: Standard CLIP TeCoA PMG-AFT FARE TGA-ZSR Adv-W2S VL-SAE VCD Patch-Aligned Training LLaVA-1.5

Builds on: CLIP TeCoA FARE Adv-W2S VCD

Closest papers by meaning

• Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.90)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.90)
• DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.90)
• DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.90)
• Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)