Research map / Vision-language and multimodal / Multimodal LLMs
Counterfactual Debiasing for Physical Audiovisual Commonsense Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CF-PACR, a model-agnostic counterfactual debiasing framework that subtracts the natural direct effect of visual-only predictions from audiovisual total effects to reduce visual bias in physical audiovisual commonsense reasoning.
Problem
Physical audiovisual commonsense reasoning (PACR) models over-rely on visual features, learning spurious correlations from imbalanced object-material combinations in the PACS training data, which hurts generalization on rare combinations.
Approach
Use causal-graph counterfactual inference: estimate the natural direct effect of the visual pathway with a counterfactual model (learnable dummy vector for the no-treatment condition, trained with KL alignment) and subtract it from the total effect to obtain the total indirect effect used for prediction.
Key results
- CF-PACR with AudioCLIP improves PACS-QA accuracy from 60.0% to 63.6% and PACS-Material from 66.2% to 69.4%.
- MER-Res (L) + CF-PACR reaches 72.8% on PACS-QA and 76.5% on PACS-Material, up from 70.1% and 74.2%.
- Average gains across baselines are about 2.82 points on PACS-QA and 2.85 points on PACS-Material.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CF-PACR to AudioCLIP | AudioCLIP · PACS-QA | accuracy: 60.0±0.9 -> 63.6±0.8 | Helped |
| Apply CF-PACR to AudioCLIP | AudioCLIP · PACS-Material | accuracy: 66.2±1.0 -> 69.4±0.9 | Helped |
| Apply CF-PACR to MER-Res (L) | MER-Res (L) · PACS-QA | accuracy: 70.1±0.7 -> 72.8±1.1 | Helped |
| Apply CF-PACR to MER-Res (L) | MER-Res (L) · PACS-Material | accuracy: 74.2±0.9 -> 76.5±1.0 | Helped |
| Apply CF-PACR to MER-Res (B) | MER-Res (B) · PACS-QA | accuracy: 66.5±1.4 -> 68.7±0.8 | Helped |
| Apply CF-PACR to MER-Res (B) | MER-Res (B) · PACS-Material | accuracy: 72.2±1.2 -> 74.7±0.9 | Helped |
| Add counterfactual debiasing with SUM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-QA | accuracy: 60.7±0.8 -> 61.9±0.8 | Helped |
| Add counterfactual debiasing with SUM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-Material | accuracy: 66.6±1.1 -> 67.5±0.9 | Helped |
| Add counterfactual debiasing with HM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-QA | accuracy: 60.9±0.7 -> 62.8±0.9 | Helped |
| Add counterfactual debiasing with HM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-Material | accuracy: 66.8±0.9 -> 68.7±1.1 | Helped |
| Add counterfactual debiasing with LogSUM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-QA | accuracy: 61.4±1.1 -> 63.6±0.8 | Helped |
| Add counterfactual debiasing with LogSUM fusion (ablation) | CF-PACR with CLIP and AudioCLIP · PACS-Material | accuracy: 67.1±1.0 -> 69.4±0.9 | Helped |
Surprising findings
- Rebalancing visual object combinations in the training set (Late Fusion ‡) improves accuracy by 4.3 points on PACS-QA and 4.5 on PACS-Material, indicating substantial visual bias in PACS.
- Question-free object-only baseline (I+V+A) reaches near-chance accuracy on PACS-QA (51.9%), and single-modality plus question models do not beat chance much.
Limitations
- Some visually dominant object combinations show slight accuracy declines after debiasing.
- Gains are reported on a single benchmark (PACS) and hyperparameters (tau, fusion function) are tuned on the validation set.
Experimental setup
Datasets: PACS
Benchmarks: PACS-QA PACS-Material
Metrics: accuracy
Baselines: CLIP UNITER (L) MERLOT AudioCLIP MERLOT Reserve (B) MERLOT Reserve (L) Late Fusion
Builds on: AudioCLIP MERLOT Reserve CLIP
Closest papers by meaning
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.86)
- When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? (AAAI 2026 · similarity 0.85)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.85)
- Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (ICML 2025 · similarity 0.85)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.85)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.