Research map / Vision-language and multimodal / Human and embodied modeling
TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRoVe, an automated method that discovers error-inducing static feature biases in temporal VLMs by clustering frame embeddings and scoring clusters with an error contribution score plus a static bias score, evaluated on 101 synthetic trained VLMs and real pretrained models.
Problem
Temporal vision-language models often rely on static feature biases (background, object, attribute) instead of dynamic visual changes, causing systematic errors; existing systematic-error discovery methods are designed for non-temporal settings and lack quantitative evaluation frameworks for tempora
Approach
Cluster image-level embeddings computed from static sequences (each frame repeated), then score each cluster by combining its effect on classification errors (ECS) with how confidently the model predicts the error label from static single-image sequences (SBS), enabling discovery without image-level feature annotations.
Key results
- On the synthetic 101-VLM framework, TRoVe achieves a 28.6% improvement over the closest baseline (Confidence) in discovery precision.
- On Kinetics400 with the top-20 TRoVe-identified static features, CoOp-based mitigation raises VideoCLIP-XL Label-y accuracy@5 from 51.7 to 94.4 and overall from 82.2 to 86.7.
- TRoVe identifies between 36 and 116 static feature biases per pretrained activity recognition VLM and 4 for BioViL-T on pneumonia progression.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TRoVe rather than the strongest baseline for identifying background-feature biases | Synthetic temporal VLM evaluation · Background features | P@10: 62.9 -> 100.0 | Helped |
| Use TRoVe rather than the strongest baseline for identifying object-feature biases | Synthetic temporal VLM evaluation · Object features | P@10: 97.8 -> 97.8 | No clear effect |
| Use TRoVE rather than the strongest baseline for identifying attribute-feature biases | Synthetic temporal VLM evaluation · Attribute features | P@10: 69.2 -> 100.0 | Helped |
| Use both Error Contribution Score and Static Bias Score instead of Static Bias Score alone (ablation) | TRoVe · Synthetic evaluation | P@10: 76.2 -> 99.1 | Helped |
| Use learned prompts on identified-feature sequences instead of default prompts | VideoCLIP-XL · Kinetics400 top-20 bias features, Label ỹ | Accuracy@5: 51.7 -> 94.4 | Helped |
| Use learned prompts on identified-feature sequences instead of default prompts | ViCLIP-B · Kinetics400 top-20 bias features, Label ỹ | Accuracy@5: 45.3 -> 95.8 | Helped |
| Use learned prompts on identified-feature sequences instead of default prompts | ViCLIP-L · Kinetics400 top-20 bias features, Label ỹ | Accuracy@5: 71.4 -> 96.9 | Helped |
| Apply TRoVe for static-bias discovery rather than Confidence | Synthetic temporal VLM evaluation · Three static-feature categories | Precision@K: 28.6% improvement over the closest baseline | Helped |
Surprising findings
- Non-temporal CLIP-style models exhibit more discovered static biases on average (134.5 vs 84.5) than temporal VLMs.
- The Confidence baseline is competitive on object features (P@10 97.8) but degrades on attribute and background features in the synthetic setting.
Limitations
- Evaluation relies on synthetic moving-circle data and small trained projection heads, which may not transfer fully to real-world biases.
- Real-world validation uses pseudolabels from open-vocabulary detectors (OwlV2) and entity extractors (RadGraph-XL) rather than ground truth.
Experimental setup
Datasets: Kinetics400 MS-CXR-T synthetic moving-circle dataset
Benchmarks: TRoVe synthetic evaluation framework (101 VLMs) Kinetics400 MS-CXR-T
Metrics: Precision@10 Precision@25 Precision@100 R-Precision Accuracy@5
Baselines: Random Domino George Distilling Failures Confidence
Builds on: Domino George Distilling Failures CoOp
Closest papers by meaning
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.87)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.87)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.87)
- RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.