Research map / Label-efficient and robust learning / Recognition and long-tail
Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a causal counterfactual (total direct effect) framework for multi-label image classification, implemented via patch-based inference (PAT-I) and patch-based training (PAT-T) to suppress overfitting to label co-occurrence.
Problem
Multi-label image classification models overfit to label co-occurrence relationships, which causes false positive predictions for absent objects whose co-occurring objects are present.
Approach
Treats correlative features from the target and co-occurring objects as a mediator; estimates the total direct effect by enhancing the target object's direct effect via patch-level predictions, identifying the pivot patch containing the target without knowing its location, and training with both image and patch logits.
Key results
- On MS-COCO with TResNetL (OI) backbone, PAT-T + Q2L-TResL reaches 91.7% mAP, above the prior best 91.3% (Q2L-CvT, 277M-param backbone).
- PAT-T improves ASL with the same TResNetL backbone from 86.6% to 88.7% mAP on MS-COCO and reaches 85.0% mAP with ResNet101 vs 78.3% for the ResNet-101 baseline.
- On VG-256, PAT-T with ResNet101 improves mAP from 47.7% to 49.9%, and PAT-T + Q2L-TResL reaches 59.5% mAP vs 57.7% for Q2L-TResL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add patch-based inference (PAT-I) to the TResNetL baseline | TResNetL · MS-COCO 2014 | mAP: 86.6 -> 87.6 | Helped |
| Add patch-based training (PAT-T) to TResNetL | TResNetL · MS-COCO 2014 | mAP: 86.6 -> 88.7 | Helped |
| Add PAT-T to Q2L-TResL | Q2L-TResL · MS-COCO 2014 | mAP: 89.2 -> 91.0 | Helped |
| Add PAT-T to ML-Decoder | ML-Decoder · MS-COCO 2014 | mAP: 90.0 -> 91.4 | Helped |
| Add PAT-T to Q2L-CvT | Q2L-CvT · MS-COCO 2014 | mAP: 91.3 -> 91.7 | Helped |
| Add PAT-T to TResL | TResL · VOC 2007 test | mAP: 96.1 -> 96.2 | Helped |
| Add PAT-I to TResNetL | TResNetL · VG-256 test | mAP: 53.9 -> 54.8 | Helped |
| Add PAT-T to TResNetL | TResNetL · VG-256 test | mAP: 53.9 -> 56.2 | Helped |
| Add PAT-T to Q2L-TResL | Q2L-TResL · VG-256 test | mAP: 57.7 -> 59.5 | Helped |
| Replace patch logits with TDE logits (ablation) | PAT-T, TResNetL · MS-COCO 2014 | mAP: 87.1 -> 88.7 | Helped |
| Replace patch logits with TDE logits (ablation) | PAT-T, TResNetL (21k) · MS-COCO 2014 | mAP: 88.8 -> 89.2 | Helped |
| Replace patch logits with TDE logits (ablation) | PAT-T, TResNetL (OI) · MS-COCO 2014 | mAP: 89.5 -> 90.8 | Helped |
Surprising findings
- Patch logits alone perform roughly on par with full-image logits when the backbone is strong, so the gain comes mainly from the TDE combination rather than the patch scale.
- DeT (shared backbone) achieves higher conditional TPR and also higher conditional FPR than InT (per-class backbones), revealing that co-occurrence capture comes with overfitting.
Limitations
- PAT-I/PAT-T gains are smaller on VOC 2007, where label co-occurrence is weaker (1.5 labels per image on average).
- Patch-based training with full-size images and patches causes performance decline without resizing patches, requiring an extra scale-correction step.
- The method relies on a dataset-dependent parameter lambda and a temperature tau for patch weighting.
Experimental setup
Datasets: MS-COCO 2014 Pascal VOC 2007 Visual Genome (VG-256)
Benchmarks: MS-COCO 2014 Pascal VOC 2007 VG-256
Metrics: mAP CP CR CF1 OP OR OF1 AP
Baselines: ResNet-101 ML-GCN KSSNet MS-CMA MCAR CPSD IDA ASL Q2L-TResL ML-Decoder SSGRL ADD-GCN
Builds on: total direct effect asymmetric loss ML-Decoder Query2Label
Closest papers by meaning
- In Pursuit of Causal Label Correlations for Multi-label Image Recognition (NeurIPS 2024 · similarity 0.91)
- Prototype-based Causal Intervention for Multi-Label Image Classification (CVPR 2026 · similarity 0.90)
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning (CVPR 2025 · similarity 0.86)
- Addressing Multi-Label Learning with Partial Labels: From Sample Selection to Label Selection (AAAI 2025 · similarity 0.85)
- Multi-Label Supervised Contrastive Learning (AAAI 2024 · similarity 0.84)
- Rethinking Visual Counterfactual Explanations Through Region Constraint (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.