Research map / Vision-language and multimodal / Visual question answering

Cross-Modal Feature Distribution Calibration for Few-Shot Visual Question Answering

AAAI 2024 · Vision-language and multimodal › Visual question answering · approach: Universal VQA models · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CDCIN, a few-shot VQA network that calibrates multimodal feature distributions by aligning pre- and post-inference "visual information entropy," with an inference-enhancement pre-training stage on base classes.

Problem

Few-shot visual question answering struggles with cross-modal inference and with dispersed multimodal feature distributions, since existing methods mostly extend few-shot classification methods to cross-modal tasks.

Approach

Defines visual information entropy (entropy of question-guided visual attention weights) and aligns its value before and after cross-modal reasoning via an information consistency loss; adds a co-attention inference module and a VQA-classification pre-training stage on base classes before few-shot fine-tuning.

Key results

• On Toronto COCO-QA, CDCIN-Swin reaches 79.08% (5-way 1-shot) and 89.48% (5-way 5-shot), improving over HGAT by 15.95% and 13.77%.
• On FS VQA, CDCIN-Swin achieves 80.23% (5-way 1-shot) and 90.14% (5-way 5-shot), versus 65.80% and 80.51% for ProtoNet-Res12.
• Ablation on Toronto COCO-QA shows CAIM, VIECM and inference-enhancement pre-training each add accuracy, with the full model reaching 72.26% at 5-way 1-shot for ResNet-12.

Measured effects reported in the paper

Surprising findings

• Performance of all models drops on the cleaned balanced datasets (e.g., CDCIN-Swin falls by 4.78% on COCO-QA 5-way 1-shot), which the authors attribute to removing frequently sampled classes.

Limitations

• Few-shot VQA evaluated only on answer-classification style datasets with restricted answer sets.
• Balanced datasets were constructed by the authors, making comparisons with prior work on original splits non-uniform.
• Visual-backbone comparisons confounded by different input resolutions (224 vs 84).

Experimental setup

Datasets: Toronto COCO-QA Visual Genome-QA VQA v2 FS COCO-QA FS VG-QA FS VQA

Benchmarks: Toronto COCO-QA Visual Genome-QA FS COCO-QA FS VG-QA FS VQA

Metrics: accuracy

Baselines: FPAIT HGAT Relation Net EGNN GCN FEAT ProtoNet MatchingNet

Builds on: FPAIT HGAT ProtoNet Swin Transformer

Closest papers by meaning

• FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.89)
• Counterfactual Samples Synthesizing for Robust Visual Question Answering (CVPR 2020 · similarity 0.88)
• When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.87)
• Towards Robust Visual Question Answering via Prompt-Driven Geometric Harmonization (AAAI 2025 · similarity 0.87)
• Core-to-Global Reasoning for Compositional Visual Question Answering (AAAI 2025 · similarity 0.87)
• VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.87)