Research map / Label-efficient and robust learning / Recognition and long-tail
Discovering Bias in Latent Space: An Unsupervised Debiasing Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SteerFair, an unsupervised inference-time method that finds option-position bias directions in attention-head activations via PCA over rule-mimicking demonstrations and steers activations away from them to reduce order sensitivity in VLM QA.
Problem
Foundation model QA performance is highly sensitive to superficial prompt variations such as option ordering, and existing debiasing methods either need labels, multiple inferences, or only operate in output space.
Approach
Enumerate simple bias association rules (e.g., always choose the j-th option), build demonstrations mimicking each rule from unlabeled questions, run PCA on attention-head activations to get bias directions, combine them via QR orthonormalization, and subtract them during inference with norm-preserving rescaling.
Key results
- SteerFair reduces order-bias std on VGR for LLaVA from 0.126 to 0.054 while raising Avg from 71.04% to 71.46%.
- SteerFair outperforms supervised ITI with 100 labels by an average of 10.86 accuracy points and 12.95 score points, and matches ITI with 500 labels.
- Without any hyperparameter tuning (all heads, alpha=1), SteerFair still improves over vanilla in most settings, e.g., LLaVA ScienceQA std 0.024 to 0.017.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply unsupervised SteerFair steering instead of vanilla inference | LLaVA (13B) · ScienceQA | average accuracy: 64.28% -> 65.46% | Helped |
| Apply unsupervised SteerFair steering instead of vanilla inference | LLaVA (13B) · ScienceQA | Std: 0.024 -> 0.017 | Helped |
| Apply unsupervised SteerFair steering instead of vanilla inference | LLaVA (13B) · MME | Std: 65.42 -> 2.72 | Helped |
| Apply unsupervised SteerFair steering instead of vanilla inference | LLaVA (13B) · VGR | Std: 0.126 -> 0.054 | Helped |
| Transfer bias directions from another dataset of the same task (ablation) | LLaVA (13B) · VGR | average accuracy: 71.04% -> 73.19% | Helped |
| Transfer bias directions from another dataset of the same task (ablation) | LLaVA (13B) · MME | score: 1333.43 -> 1305.57 | Hurt |
| Use transferred rather than same-dataset bias directions (ablation) | LLaVA (13B) · MME | Std: 2.72 -> 56.54 | Hurt |
| Set alpha = 1 and intervene all attention heads without tuning (ablation) | SteerFair · VGR | Std: 0.126 -> 0.023 | Helped |
| Set alpha = 1 and intervene all attention heads without tuning (ablation) | SteerFair · VGR | average accuracy: 71.04% -> 67.03% | Hurt |
| Set alpha and K both large (alpha >= 5, K >= 1000) (ablation) | SteerFair · VGR | accuracy | Hurt |
| Use fewer than 100 unlabeled samples to identify bias directions (ablation) | SteerFair · Not specified | accuracy: ± 2% | No clear effect |
| Use transferred rather than same-dataset bias directions (ablation) | IDEFICS (9B) · MME | score: 1044.70 -> 1060.94 | Helped |
Surprising findings
- Larger numbers of unlabeled samples (N>1000) gave noisier bias directions and no better debiasing; the lowest std occurred for N between 300 and 1000.
- Steering the original-dataset direction sometimes failed to help (e.g., InstructBLIP on MME), while transferred directions from another same-task dataset sometimes worked better.
Limitations
- Relies on bias being enumerable and well separated from useful knowledge in activation space; conditions under which this holds are unclear.
- No theoretical characterization of hyperparameters or required number of unlabeled samples.
- Simple averaging of bias directions limits effectiveness as the number of options grows (e.g., 5-option ScienceQA).
Experimental setup
Datasets: ScienceQA MME Visual Genome Relation (VGR)
Benchmarks: ScienceQA MME Visual Genome Relation (VGR)
Metrics: average accuracy across option orders (Avg%) standard deviation across option orders (Std)
Baselines: Vanilla inference ITI (100 labels) ITI (500 labels)
Builds on: Inference-Time Intervention (ITI) Calibrate Before Use Flamingo transformer mechanistic residual stream view
Closest papers by meaning
- Language‑Bias‑Resilient Visual Question Answering via Adaptive Multi‑Margin Collaborative Debiasing (NeurIPS 2025 · similarity 0.88)
- BayesVQA: Energy-Guided Bayesian Debiasing for Language-Bias-Robust Visual Question Answering (AAAI 2026 · similarity 0.88)
- Explicit Bias Discovery in Visual Question Answering Models (CVPR 2019 · similarity 0.87)
- Introspective Distillation for Robust Question Answering (NeurIPS 2021 · similarity 0.86)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.86)
- Debiasing Algorithm through Model Adaptation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.