Research map / Vision-language and multimodal / Human and embodied modeling
Evaluating Model Bias Requires Characterizing its Mistakes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkewSize, a metric that measures model bias from the distribution of prediction errors across subgroups by computing per-class Cramér's V effect sizes between a bias variable and predictions and aggregating them via Fisher-Pearson skewness.
Problem
Standard bias metrics (accuracy, worst-group accuracy, accuracy gap, and fairness metrics like DP and EO) ignore systematic differences in how a model mis-predicts across subgroups, and no real datasets exist for measuring such bias in open-vocabulary VLMs.
Approach
Treat bias as association between a spurious subgroup variable and the model's predicted labels within each ground-truth class, measure it per class with Cramér's V from a chi-squared contingency table, and aggregate across classes with the Fisher-Pearson skewness so that a few strongly affected classes yield a distinctive scalar.
Key results
- On dSprites, per-class effect size was ~0.70 only for the spurious class (Class 1) while accuracy-based metrics and DP/EO spread the bias across confused classes.
- On the BLIP-2 family, SkewSize for occupation was 0.233 (2.7B), -0.045 (6.7B), and 0.599 (FlanT5), suggesting scale amplifies and instruction tuning mitigates gender bias.
- On DomainNet, ResNet-50 reached 59.95% average accuracy with SkewSize 0.509; effect size correlated only mildly with accuracy (r=-0.291).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase spurious-correlation strength by removing more Green Class 1 training examples | ResNet18 on dSprites · dSprites, Class 1 | effect size: 0.020 -> 0.253 -> 0.492 -> 0.703 | Helped |
| Evaluate subgroup error distributions using SkewSize rather than accuracy-based metrics alone | ImageNet classifiers · ImageNet with synthetic backgrounds | bias characterization | Helped |
| Increase unsupervised language-model size from 2.7B to 6.7B | BLIP-2 · Occupation prediction | SkewSize: 0.233 -> -0.045 | Hurt |
| Use instruction-tuned FlanT5 instead of the unsupervised-trained language model | BLIP-2 · Occupation prediction | SkewSize: -0.045 -> 0.599 | Helped |
| Use instruction-tuned FlanT5 instead of the unsupervised-trained language model | BLIP-2 · Sports prediction | SkewSize: 0.360 -> 1.255 | Helped |
| Post-process raw model outputs (ablation) | BLIP2-2.7B, BLIP2-6.7B, and BLIP2-FlanT5 · Occupation prediction | SkewSize: 0.233 -> -0.005; -0.045 -> -0.130; 0.599 -> 0.124 | Mixed |
| Vary the minimum expected value threshold from MEV=6 to MEV=2 (ablation) | BLIP-2 family · Occupation prediction | SkewSize: BLIP-2.7: 0.235 -> 0.19; BLIP-6.7: -0.031 -> -0.102; BLIP-FlanT5: 0.625 -> 0.507 | No clear effect |
| Replace Cramer's V with the Phi coefficient to estimate effect size (ablation) | SkewSize on dSprites · dSprites | per-class effect size: Class 0: 0.670 -> 0.948; Class 1: 0.683 -> 0.966; Class 2: 0.696 -> 0.985 | No clear effect |
Surprising findings
- Accuracy-based metrics such as worst-group accuracy and accuracy gap can stay unchanged or even improve for a model whose error distribution is clearly biased.
- Per the paper, SkewSize values for BLIP-2 are higher (less biased) for the instruction-tuned FlanT5 model, while larger unsupervised-LM models show lower skewness.
Limitations
- SkewSize captures association, not causation, between the bias attribute and predictions.
- Binary gender and synthetic data used in VLM experiments are oversimplified and may inherit generator stereotypes.
- Low-count predictions create noise; filtering by minimum expected value is needed and manual post-processing of VLM outputs was required.
Experimental setup
Datasets: dSprites ImageNet TinyImageNet DomainNet VisoGender
Benchmarks: dSprites ImageNet (synthetic background variant) DomainNet VisoGender-style synthetic VLM gender-occupation/sport evaluation
Metrics: accuracy worst group accuracy accuracy gap demographic parity equalized odds effect size (Cramér's V) SkewSize
Baselines: accuracy worst group accuracy accuracy gap demographic parity equalized odds
Builds on: Cramér's V Pearson chi-squared test equalized odds demographic parity
Closest papers by meaning
- Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision (NeurIPS 2026 · similarity 0.87)
- TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- An Effective Theory of Bias Amplification (ICLR 2025 · similarity 0.86)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.85)
- Vision Language Models are Biased (ICLR 2026 · similarity 0.85)
- From Hope to Safety: Unlearning Biases of Deep Models via Gradient Penalization in Latent Space (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.