Research map / Vision-language and multimodal / Human and embodied modeling
Contrastive Visual Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoDA, which extracts contrastive textual and visual features of a target concept against the concept an LMM confuses it with, filters them, and uses text-to-image generation to create augmented training images, plus a new NovelSpecies benchmark of post-cutoff animal species.
Problem
Large multimodal models struggle to recognize novel (recently discovered) and confusing, low-resource, or commonly misrepresented visual concepts, and existing data augmentation methods either lack real images for such concepts or are tied to existing images.
Approach
Identify the confusable concept the LMM predicts for the target, extract features that discriminate target from confusable concept, filter them by CLIP-based discriminability and generability, generate images with text-to-image models conditioned on these features, and keep only images the LMM itself recognizes via feature satisfaction, then LoRA-update the LMM.
Key results
- On NovelSpecies with LLaVA-1.6 34B under 5:1 fixed real data, CoDA visual features reach 73.0% accuracy versus 61.2% for all-real training and 60.7% for the best existing augmentation baseline (ARMADA/flipping).
- Human annotators confirmed extracted features present in 92.51% of real target images and 83.97% of synthetic images (kappa 0.82).
- CoDA with LoRA on GPT-4o-mini gives a 9.5% gain on NovelSpecies and CoDA on ViT gives 9.1% for single-shot augmentation (as reported in the paper).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive visual features instead of using non-contrastive visual features (ablation) | CoDA · NovelSpecies, Fixed Real Data 5:1 | accuracy: 71.8 -> 73.0 | Helped |
| add contrastive textual and visual features instead of using non-contrastive features (ablation) | CoDA · SUN, Fixed Real Data 5:1 | accuracy: 82.7 -> 83.4 | Helped |
| add contrastive visual features instead of using non-contrastive visual features (ablation) | CoDA · iNaturalist, Fixed Real Data 5:1 | accuracy: 65.0 -> 67.0 | Helped |
| use CoDA visual augmentation rather than Flipping | LLaVA-1.6 34B · NovelSpecies, Fixed Real Data 5:1 | accuracy: 60.7 -> 73.0 | Helped |
| use CoDA textual and visual augmentation rather than Cropping | LLaVA-1.6 34B · SUN, Fixed Real Data 5:1 | accuracy: 78.3 -> 83.4 | Helped |
| use CoDA visual augmentation rather than Flipping | LLaVA-1.6 34B · iNaturalist, Fixed Real Data 5:1 | accuracy: 61.0 -> 67.0 | Helped |
| increase synthetic image count from 1 to 3 (ablation) | CoDA · NovelSpecies, Fixed Real Data | accuracy: 73.0 -> 72.8 | Mixed |
| replace Stable Diffusion 3.5 Large Turbo with Recraft V3 for synthetic image generation (ablation) | CoDA · NovelSpecies | accuracy: 7.9% | Helped |
Surprising findings
- Mixing real and synthetic CoDA images (50-50) can outperform training on all real data under the fixed compute setting on iNaturalist.
- Adding more synthetic images does not necessarily improve performance; the first generated image is typically highest quality.
Limitations
- Performance is bounded by the instruction-following ability of the text-to-image backbone; SD-3.5 misses some features that Recraft V3 captures.
- Evaluation focuses on fine-tuning use case only; other uses such as in-context or test-time augmentation are left for future work.
- Feature extraction and verification rely on proprietary GPT-4o-mini and LLaVA verification with a threshold that is heuristic.
Experimental setup
Datasets: iNaturalist SUN NovelSpecies Wikidata
Benchmarks: NovelSpecies SUN iNaturalist
Metrics: accuracy human agreement (Fleiss' kappa)
Baselines: All Real Cropping Flipping ARMADA CoDA (w/o contrastive)
Builds on: ARMADA CLIP LoRA LLaVA-NeXT
Closest papers by meaning
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.86)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.86)
- Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models (CVPR 2024 · similarity 0.85)
- Reconstruction Alignment Improves Unified Multimodal Models (ICLR 2026 · similarity 0.85)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.85)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.