Research map / Vision-language and multimodal / Cross-modal learning

ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ITO, an image-text contrastive pretraining method that pairs multiple augmented image views with the same caption (many-to-one cross-modal alignment) plus a train-only two-layer self-attention fusion regularizer, discarded at inference.

Problem

Image-text contrastive pretraining (CLIP) yields embeddings organized partly by modality rather than semantics, with only instance-level one-to-one pairing constraints.

Approach

Restructure cross-modal supervision into many-to-one correspondences by pairing multiple augmented image views with the same caption inside InfoNCE, and add a training-only fusion module whose contrastive loss on fused image-text [CLS] tokens backpropagates into both dual encoders; the fusion module is dropped at inference.

Key results

• ITO reaches 24.9 vs CLIP 19.3 zero-shot average on CC3M and 56.1 vs 53.5 on Laion100M (26 benchmarks, ViT-B/16, 30 epochs).
• On YFCC15M, ITO improves linear probing on ImageNet-1K from 51.82 (CLIP) to 60.52 and average over five datasets by +7.9.
• Multi-view alignment alone (lambda=0) gives most gain; fusion adds +0.6 zero-shot at 15M and +1.1 at 1B, but CLIP trained for 2x epochs remains 7.0 points behind ITO on YFCC15M zero-shot.

Surprising findings

• Cross-attention fusion with an EOS query underperforms self-attention and, on YFCC15M, falls below the no-fusion baseline on zero-shot accuracy.
• Sub-description text augmentation (ITO_sub2) slightly hurts on DataComp-1B at 10 epochs (63.2 vs 63.7 for CLIP) and is therefore not used for large-scale results.

Limitations

• Training cost roughly 2.2x time and 2.1-2.3x memory of CLIP, mostly from multi-view alignment.
• At 100M-1B scale only CLIP (and SLIP on Laion100M) is compared; superiority over stronger baselines is not established there.
• Fusion's accuracy contribution is modest and its effect on global geometry is small and scale-dependent; no controlled curated-vs-noisy ablation.

Experimental setup

Datasets: CC3M CC12M YFCC15M Laion100M DataComp-1B CC3M-recap

Benchmarks: 26-dataset zero-shot classification (EVA-CLIP protocol) ImageNet-1K linear probing MSCOCO retrieval Flickr30k retrieval DOCCI LLaVA-1.5 13 MLLM benchmarks (VQAv2, GQA, MMBench, POPE, MMStar, etc.)

Metrics: top-1 accuracy linear probe accuracy Recall@1/5/10 MLLM benchmark scores

Baselines: CLIP SLIP SigLIP FLAIR OpenCLIP

Builds on: CLIP SLIP FLAIR OpenCLIP CyCLIP

Closest papers by meaning

• Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.89)
• Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
• MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.89)
• Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.89)
• CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.88)
• BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning (AAAI 2025 · similarity 0.88)