Research map / Vision-language and multimodal / Human and embodied modeling
Visual Data Diagnosis and Debiasing with Concept Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConBias, which represents a visual dataset as a concept co-occurrence graph, diagnoses class-concept imbalances via clique enumeration, and debiases by generating inpainted images of under-represented combinations for augmentation.
Problem
Visual datasets contain spurious object/concept co-occurrence biases that hurt generalization, and existing pipelines either lack dataset diagnosis or rely on LLM-generated descriptions (ALIA) that may carry their own biases.
Approach
Model the dataset as a knowledge graph of class and concept nodes weighted by co-occurrence counts, use clique-based enumeration of common class-concept combinations to find imbalances across classes, and generate under-represented combinations with Stable Diffusion inpainting that preserves the labeled object.
Key results
- On Waterbirds, ConBias reaches 77.9% CB and 69.3% OOD accuracy versus 69.6% / 48.2% for ALIA and 67.1% / 44.9% for the baseline.
- On UrbanCars, ConBias reaches 78.3% CB and 52.9% OOD, outperforming RandAug (76.3 / 46.1) and ALIA (74.0 / 42.5).
- On COCO-GB, gains are marginal at the default augmentation size (58.8 CB, 51.4 OOD vs ALIA 58.7 / 52.4), but performance improves with more augmented images up to about 1000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply clique-based ConBias augmentation | ResNet-50 · Waterbirds CB | accuracy: 73.7 -> 77.9 | Helped |
| Apply clique-based ConBias augmentation | ResNet-50 · Waterbirds OOD | accuracy: 60.2 -> 69.3 | Helped |
| Apply clique-based ConBias augmentation | ResNet-50 · UrbanCars CB | accuracy: 76.3 -> 78.3 | Helped |
| Apply clique-based ConBias augmentation | ResNet-50 · UrbanCars OOD | accuracy: 46.1 -> 52.9 | Helped |
| Apply clique-based ConBias augmentation | ResNet-50 · COCO-GB CB | accuracy: 58.7 -> 58.8 | No clear effect |
| Apply clique-based ConBias augmentation | ResNet-50 · COCO-GB OOD | accuracy: 52.4 -> 51.4 | Hurt |
| Use larger concept cliques rather than only single-hop class-concept frequencies (ablation) | ConBias on UrbanCars · UrbanCars CB | accuracy: 73.4 -> 79.4 | Helped |
| Use larger concept cliques rather than only single-hop class-concept frequencies (ablation) | ConBias on UrbanCars · UrbanCars OOD | accuracy: 40.4 -> 53.2 | Helped |
| Replace Stable Diffusion inpainting with IP2P generation (ablation) | ConBias on Waterbirds · Waterbirds OOD | accuracy: 69.3 -> 60.5 | Hurt |
| Increase the number of generated augmentation images up to 1000 (ablation) | ConBias on COCO-GB · COCO-GB | accuracy | Helped |
| Continue adding synthetic images beyond the performance inflection point (ablation) | ConBias on COCO-GB · COCO-GB | accuracy | Hurt |
Surprising findings
- On COCO-GB, ConBias does not beat ALIA on OOD accuracy at the default augmentation size (51.4 vs 52.4).
- Using InstructPix2Pix instead of inpainting still beats ALIA on Waterbirds (72.9 / 60.5 vs 69.6 / 48.2), suggesting concept discovery drives most of the gain.
Limitations
- Clique enumeration grows exponentially with graph size, limiting scalability to larger real-world graphs.
- Focuses only on object co-occurrence biases, not other bias types.
- Assumes reliable ground-truth concept sets (captions or segmentation annotations) are available.
Experimental setup
Datasets: Waterbirds UrbanCars COCO-GB ImageNet-1k
Benchmarks: Waterbirds UrbanCars COCO-GB
Metrics: class-balanced accuracy out-of-distribution accuracy BG-GAP CoObj-GAP BG+CoObj GAP
Baselines: ERM baseline ResNet-50 RandAugment CutMix ALIA
Builds on: ALIA REVISE CutMix RandAugment
Closest papers by meaning
- ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts (NeurIPS 2025 · similarity 0.88)
- Understanding Bias in Large-Scale Visual Datasets (NeurIPS 2024 · similarity 0.87)
- Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers (CVPR 2025 · similarity 0.84)
- Label-Efficient Group Robustness via Out-of-Distribution Concept Curation (CVPR 2024 · similarity 0.84)
- DCBM: Data-Efficient Visual Concept Bottleneck Models (ICML 2025 · similarity 0.84)
- Common Sense Bias Modeling for Classification Tasks (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.