Research map / Vision-language and multimodal / Human and embodied modeling
Causal Inference over Visual-Semantic-Aligned Graph for Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSCNet, a tag-guided image classifier that locates semantic regions, refines tag predictions with a pre-learned visual-semantic exemplar hierarchy, and applies mask-based causal intervention on a heterogeneous graph.
Problem
Cross-modal enhanced image classification suffers from heterogeneity between visual and semantic (tag) modalities, so inaccurate region-to-tag mappings propagate errors into class predictions and spurious statistical correlations persist.
Approach
Combines adaptive region locating (CSNet with affine grid sampling), tag refinement via pre-learned visual prototypes and tag hierarchy, a heterogeneous region-tag-prototype graph, and learnable node/edge masks that split causal and non-causal subgraphs with a backdoor-adjustment-style loss.
Key results
- On Ingredient-101, VSCNet with ViT backbone reaches 0.886 Top-1 and 0.973 Top-5 accuracy, versus 0.871 / 0.975 for MGCC (ViT).
- VSCNet with ResNet50 reaches 0.865 Top-1 on Ingredient-101 vs 0.820 for ResNet50 baseline.
- On NUS-WIDE, VSCNet with ViT reaches P@1 0.826 and R@1 0.474, the best among compared methods in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add predicted tags to the base model (ablation) | VSCNet with ViT backbone · Ingredient-101 | accuracy (Acc@1): 0.854 -> 0.846 | Hurt |
| add predicted tags to the base model (ablation) | VSCNet with ViT backbone · NUS-WIDE | precision (P@1): 0.796 -> 0.777 | Hurt |
| add a heterogeneous graph to predicted-tag features (ablation) | VSCNet with ViT backbone · Ingredient-101 | accuracy (Acc@1): 0.846 -> 0.858 | Helped |
| add pre-learned visual-semantic exemplars to predicted-tag features (ablation) | VSCNet with ViT backbone · Ingredient-101 | accuracy (Acc@1): 0.846 -> 0.862 | Helped |
| add pre-learned visual-semantic exemplars and a heterogeneous graph to predicted-tag features (ablation) | VSCNet with ViT backbone · NUS-WIDE | precision (P@1): 0.777 -> 0.822 | Helped |
| add causal inference to the tag, exemplar, and graph configuration (ablation) | VSCNet with ViT backbone · Ingredient-101 | accuracy (Acc@1): 0.869 -> 0.871 | Helped |
| add causal inference to the tag, exemplar, and graph configuration (ablation) | VSCNet with ViT backbone · NUS-WIDE | recall (R@1): 0.469 -> 0.472 | Helped |
| add multi-view fusion to the causal-inference configuration (ablation) | VSCNet with ViT backbone · Ingredient-101 | accuracy (Acc@1): 0.871 -> 0.886 | Helped |
| add multi-view fusion to the causal-inference configuration (ablation) | VSCNet with ViT backbone · NUS-WIDE | precision (P@1): 0.824 -> 0.827 | Helped |
Surprising findings
- Using predicted tags alone (+L) can lower performance versus the base model on NUS-WIDE and Ingredient-101 Top-5 accuracy, attributed to noise in visual-semantic mapping.
- VSCNet with ResNet18 is below ViT and VanillaNet baselines on Ingredient-101 Top-1 (0.829 vs 0.854 for ViT).
Limitations
- Cost of constructing the visual-semantic exemplar hierarchy grows with dataset scale.
- Small visual targets and poor-quality images can cause wrong tag refinement and predictions, per error analysis.
- Evaluated only on two datasets with a fairly specific cross-modal setup using tags as privileged information.
Experimental setup
Datasets: Ingredient-101 NUS-WIDE
Benchmarks: Ingredient-101 NUS-WIDE
Metrics: Top-1 accuracy Top-5 accuracy precision@1 precision@5 recall@1 recall@5
Baselines: ResNet-18 ResNet-50 ViT RepVGG RepMLPNet VanillaNet CMFL IRRA MSMVFA SSAN Disentangle-VAE C2KD
Builds on: privileged information backdoor adjustment spatial transformer networks
Closest papers by meaning
- Visual Representation Learning through Causal Intervention for Controllable Image Editing (CVPR 2025 · similarity 0.88)
- From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.85)
- Multimodal Event Causality Reasoning with Scene Graph Enhanced Interaction Network (AAAI 2024 · similarity 0.85)
- A Recipe for Causal Graph Regression: Confounding Effects Revisited (ICML 2025 · similarity 0.84)
- In Pursuit of Causal Label Correlations for Multi-label Image Recognition (NeurIPS 2024 · similarity 0.84)
- Sparse Causal Discovery with Generative Intervention for Unsupervised Graph Domain Adaptation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.