Research map / Vision-language and multimodal / Human and embodied modeling
Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NAS, an ALBEF-based VLP model that uses a visual dictionary to quantize image patches and generates token-level negative image samples (NVA) alongside text negatives (NTA), trained with a fine-grained ITM objective, to improve fine-grained vision-language understanding.
Problem
Vision-language pretraining models capture coarse-grained correspondences but struggle with fine-grained understanding, partly because hard negative construction has been limited to text while visual negatives are hard to build.
Approach
Use a learned visual dictionary (VQ-style quantization with EMA updates) as a semantic bridge, then replace top-k similar dictionary tokens on the primary-object patches (chosen by combining text-CLS and image-CLS similarity) to create hard negative images end-to-end, and train a fine-grained ITM task with them together with text negatives.
Key results
- NAS(NTVA) reaches 93.3 average on ARO, 25.3 average on Winoground and 74.4 on VALSE, versus ALBEF_COCO at 74.5, 18.1 and 71.5 respectively.
- NAS(NVA) achieves 90.8 R@1 text retrieval on zero-shot Flickr30K, surpassing ALBEF's 88.1.
- Ablation: adding the visual dictionary alone raises the average from 45.5 to 48.3, and the full NTVA model reaches 56.1 average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VD (ablation) | NAS · ARO | Relation: 59.2 -> 64.7 | Helped |
| Add VD (ablation) | NAS · VALSE | Avg.: 71.5 -> 72.2 | Helped |
| Add NTA (ablation) | NAS with VD · ARO | Relation: 64.7 -> 93.1 | Helped |
| Add NTA (ablation) | NAS with VD · Winoground | Image: 15.3 -> 17.3 | Helped |
| Add NVA to NTA (ablation) | NAS · Winoground | Group: 13.0 -> 18.5 | Helped |
| Add NVA to NTA (ablation) | NAS · VALSE | Avg.: 73.5 -> 74.4 | Helped |
| Use NTA instead of NVA (ablation) | NAS · ARO | Relation: 67.8 -> 93.1 | Helped |
| Use NVA instead of NTA (ablation) | NAS · Winoground | Avg.: 20.8 -> 22.5 | Helped |
| Set VD size to 2048 rather than 1024, 4096, or 8192 (ablation) | NAS · Winoground | Avg.: 35.3, 35.0, 33.5 -> 37.6 | Helped |
| Set visual token replacement ratio to 30% rather than 10%, 50%, or 70% (ablation) | NAS · Winoground | Avg.: 35.6, 37.2, 35.9 -> 37.6 | Helped |
| Set λ to 0.5 rather than 0.0 or 1.0 (ablation) | NAS · Winoground | Avg.: 35.9 -> 37.6 | Helped |
| Apply NVA | NAS · MSCOCO 5K test set fine-tuned text retrieval | R@1: 72.6 -> 74.0 | Helped |
Surprising findings
- NVA alone performs poorly on ARO relation (67.8) compared with NTA (93.1), indicating visual negatives and text negatives address different skills.
- The NTA variant fine-tuned for retrieval underperforms ALBEF on COCO (62.7 vs 72.6 text R@1), which the authors attribute to loss of global information.
Limitations
- Negative image samples are selected by replacing tokens of the primary object without segmentation, which may be imprecise.
- NTA and NTVA variants underperform on retrieval tasks compared with the NVA variant and ALBEF.
- Pretraining used only 2.9M of the stated 4M images because part of the data was unavailable.
Experimental setup
Datasets: MSCOCO Visual Genome SBU Captions Conceptual Captions text-augmented COCO ARO Winoground VALSE Flickr30K
Benchmarks: ARO Winoground VALSE Flickr30K MSCOCO
Metrics: accuracy text score image score group score R@1 R@5 R@10
Baselines: ALBEF NegCLIP syn-CLIP SPEC FDT XVLM BLIP2 MiniGPT-4 CLIP METER
Builds on: ALBEF BLIP VQ-VAE NegCLIP
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training (CVPR 2024 · similarity 0.89)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.