Research map / Label-efficient and robust learning / Recognition and long-tail
In Pursuit of Causal Label Correlations for Multi-label Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a causal-intervention label-correlation module for multi-label image recognition that keeps causal label correlations while suppressing spurious ones from confounders modeled by K-means clustering of training spatial features.
Problem
Label correlations in multi-label image recognition can mislead classifiers under contextual bias in training data, and prior debiasing methods discard label correlation modeling altogether.
Approach
Decouple label-specific features with a Transformer decoder, model confounders as K-means cluster centers of spatial features from all training images, and use a cross-attention module with backdoor adjustment (NWGM approximation) to quantify causal label correlations, then average these with decoupled predictions.
Key results
- On COCO-Stuff common setting, the method reaches 60.6% All mAP versus 55.0% for the ResNet-50 baseline, and +1.5% Exclusive and +3.6% Co-occur over feature-split.
- In cross-dataset MS-COCO to NUS-WIDE, the method reaches 83.2 mAP vs 82.3 for IDA and 81.1 for the ResNet-101 baseline.
- On MS-COCO intra-dataset, the method reaches 84.9 mAP, tied with Q2L and above CCD (84.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply causal correlation modeling and decoupled label-specific features | ResNet-50 multi-label recognition · COCO-Stuff All | mAP: 55.0 -> 60.6 | Helped |
| apply the proposed method instead of feature-split | multi-label recognition · COCO-Stuff Exclusive | mAP: 28.8 -> 29.7 | Helped |
| apply the proposed method instead of IDA | ResNet-101 multi-label recognition · MS-COCO → NUS-WIDE | mAP: 82.3 -> 83.2 | Helped |
| remove the Decouple module (ablation) | full proposed method · COCO-Stuff All | mAP: 60.6 -> 56.8 | Hurt |
| add the Causal module to Decouple (ablation) | Decouple-only model · COCO-Stuff All | mAP: 56.8 -> 60.6 | Helped |
| reduce the number of clustering centers to 20 (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 59.6 | Hurt |
| increase clustering centers from 80 to 100 (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 60.5 | No clear effect |
| use ResNet-101 rather than ResNet-50 for confounder clustering (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 60.5 | No clear effect |
| replace K-means confounders with random vectors (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 56.1 | Hurt |
| replace K-means confounders with early-epoch features (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 60.1 | Hurt |
| replace K-means confounders with label-specific features (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 60.3 | Hurt |
| replace cross-attention with linear modeling and average merge (ablation) | proposed method · COCO-Stuff All | mAP: 60.6 -> 60.0 | Hurt |
Surprising findings
- Label correlation modeling via graph structures (ML-GCN, SSGRL) built from the training set fails to beat baseline and can degrade performance under contextual bias.
- Confounder backbone choice (ResNet-50, ResNet-101, BEiT3-Large) has negligible effect on final performance, suggesting a weak backbone suffices for confounder modeling.
Limitations
- Confounders are obtained by clustering features from a pre-trained CNN, which is dataset-dependent and gives limited interpretability of the confounder semantics.
- Only backdoor-adjustment causal intervention is used; higher-level causality such as counterfactual reasoning is not explored.
- Improvement over baseline in the cross-dataset setting is smaller than in the common setting, attributed to distribution shift beyond contextual bias.
Experimental setup
Datasets: COCO-Stuff DeepFashion MS-COCO NUS-WIDE ImageNet
Benchmarks: COCO-Stuff DeepFashion MS-COCO NUS-WIDE
Metrics: mAP top-3 recall
Baselines: Q2L ADD-GCN ML-GCN SSGRL C-Tran CCD TDRG IDA CAM-Based feature-split
Builds on: Q2L VC R-CNN CCD feature-split
Closest papers by meaning
- Prototype-based Causal Intervention for Multi-Label Image Classification (CVPR 2026 · similarity 0.92)
- Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training (ICML 2024 · similarity 0.91)
- Collaboration Based Multi-Label Learning (AAAI 2019 · similarity 0.89)
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning (CVPR 2025 · similarity 0.87)
- Multi-Label Supervised Contrastive Learning (AAAI 2024 · similarity 0.86)
- Correlation-Induced Label Prior for Semi-Supervised Multi-Label Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.