Research map / Image generation / Text-to-image diffusion
Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaSPA, a fine-tuning-free generative augmentation pipeline for fine-grained visual classification that conditions ControlNet (Canny edges) with BLIP-Diffusion subject references and filters outputs with a dataset-trained model and CLIP.
Problem
Generative data augmentation for full-dataset fine-grained visual classification (FGVC) struggles to balance class fidelity and diversity; Img2Img methods anchored on real images limit diversity, while text-to-image methods lose sub-class fidelity.
Approach
Free generation from real-image guidance by conditioning on abstract representations: Canny edge maps from one real image for structure, plus a subject reference image from a different same-sub-class image for sub-class appearance, with GPT-4-generated meta-class prompts and two-stage filtering (CLIP semantic filter and top-k confidence filter from a dataset-trained model).
Key results
- On full FGVC datasets, SaSPA reaches 86.6% on Aircraft, 76.2% on CompCars, 93.8% on Cars, 83.2% on CUB and 71.9% on DTD, outperforming the best traditional and generative baselines (e.g., Real Guidance 84.8% on Aircraft).
- In the contextual bias Airbus vs Boeing split, SaSPA w/o BLIP-diffusion obtains the best overall accuracy (73.0%) and OOD accuracy (41.5%) versus 25.1% for ALIA.
- Few-shot on Cars at 16 shots, SaSPA reaches 91.0% versus 88.3% for the second-best method (Real Guidance).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SaSPA augmentation instead of the strongest reported baseline | ResNet50 FGVC training · Aircraft, CompCars, Cars, CUB, DTD | test accuracy: 84.8 -> 86.6; 73.1 -> 76.2; 92.9 -> 93.8; 82.8 -> 83.2; 69.7 -> 71.9 | Helped |
| Remove BLIP-diffusion subject representation from SaSPA (ablation) | SaSPA · Aircraft | test accuracy: 87.4 vs 86.6 | Helped |
| Add BLIP-diffusion subject representation (ablation) | SaSPA · CompCars and CUB | test accuracy: 74.8 -> 76.2; 83.0 -> 83.2 | Helped |
| Add BLIP-diffusion subject representation (ablation) | SaSPA · Cars | test accuracy: 93.7 -> 93.8 | No clear effect |
| Add edge guidance (ablation) | SaSPA generation · Aircraft | validation accuracy: 83.3 -> 85.7 | Helped |
| Replace text-to-image generation with Img2Img (ablation) | SaSPA generation · Aircraft | validation accuracy: 83.3 -> 83.0 | Hurt |
| Add subject representation without structural conditioning (ablation) | SaSPA generation · Aircraft | validation accuracy: 83.3 -> 81.5 | Hurt |
| Use different source images for edge extraction and subject reference (ablation) | SaSPA with edge and subject conditioning · Aircraft, Cars, CUB, DTD | validation accuracy: 85.2 -> 85.4; 93.1 -> 93.9; 81.3 -> 83.0; 68.7 -> 69.9 | Helped |
| Append artistic styles to half the prompts (ablation) | SaSPA with edge guidance and no subject representation · Aircraft, Cars, CUB, DTD | validation accuracy: 85.7 -> 86.2; 93.4 -> 93.8; 81.8 -> 81.6; 68.4 -> 68.6 | Mixed |
| Append artistic styles to half the prompts (ablation) | SaSPA with edge and subject conditioning · Aircraft, CUB, DTD | validation accuracy: 85.4 -> 85.5; 83.0 -> 82.6; 69.9 -> 69.2 | Mixed |
| Use SaSPA with CutMix instead of SaSPA alone (ablation) | SaSPA · Aircraft and Cars, ResNet50@448 and ViT-B/16@384 | test accuracy: 90.59 -> 90.79; 95.29 -> 95.34; 85.48 -> 85.72; 95.12 -> 95.37 | Helped |
| Increase the synthetic augmentation proportion when real data is reduced (ablation) | SaSPA training · Aircraft, Cars, CUB | validation accuracy: Cars: 89.3 -> 89.5 (0.5 real data); 93.9 -> 93.6 (1.0 real data) | Mixed |
Surprising findings
- Text-to-image generation without real-image guidance outperforms SDEdit-style Img2Img in the ablation, despite lower fidelity.
- As the amount of real data decreases, the optimal augmentation ratio alpha increases.
Limitations
- Performance depends on the underlying generation models; higher resolution did not improve CUB results.
- Relies on LLM-generated prompts, which may be suboptimal if the LLM lacks knowledge of the meta-class.
- BLIP-Diffusion subject conditioning does not help on Aircraft and Cars and reduces diversity in some settings.
Experimental setup
Datasets: Aircraft Stanford Cars CUB DTD CompCars Stanford Dogs Oxford-IIIT Pet Airbus vs Boeing contextual bias split
Benchmarks: Aircraft CompCars Stanford Cars CUB DTD Stanford Dogs Oxford-IIIT Pet
Metrics: test accuracy few-shot accuracy in-domain accuracy out-of-domain accuracy FID LPIPS diversity
Baselines: CAL-Aug RandAugment CutMix Real Guidance ALIA diff-mix DA-Fusion
Builds on: ControlNet BLIP-Diffusion CAL ALIA SDEdit
Closest papers by meaning
- Novel Class Discovery for Ultra-Fine-Grained Visual Categorization (CVPR 2024 · similarity 0.88)
- Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model (CVPR 2024 · similarity 0.88)
- Fine-grained Classes and How to Find Them (ICML 2024 · similarity 0.87)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.86)
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.86)
- Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.