Research map / Vision-language and multimodal / Human and embodied modeling
Adaptive Testing of Computer Vision Models
Irena Gao, Gabriel Ilharco, Scott Lundberg, Marco Tulio Ribeiro
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive test suggestions (hill-climb on previous failures) instead of topic-name-only retrieval, user study (ablation) | AdaVision · User study (classification) | failures found: d=0.588, p < 0.05 | Improved |
| use adaptive test suggestions instead of non-adaptive retrieval (ablation) | AdaVision · User study (object detection) | failures found: d=0.882, p < 0.005 | Improved |
| use adaptive test suggestions instead of non-adaptive retrieval (ablation) | AdaVision · User study (image captioning) | failures found: d=0.967, p < 0.05 | Improved |
| use AdaVision instead of NonAdaptive for perceived difficulty (ablation) | AdaVision · User study | perceived difficulty (1-5, lower easier): 4.10 ± 0.91 -> 3.05 ± 1.07 | Improved |
| use human-in-the-loop AdaVision topics instead of Domino automatic slices | ViT-H/14 · ImageNet categories, new retrievals | average failure rate: 8.6 -> 28.47 | Improved |
| transfer AdaVision topics found on ViT-H/14 to ResNet-50 | ResNet-50 · ImageNet categories, new retrievals | average failure rate: 25.45 -> 56.93 | Improved |
| finetune on AdaVision-found tests instead of generic retrieved images (ablation) | ViT-H/14 · AdaVision treatment topics (LAION-5B held-out) | accuracy: 82.5 -> 91.2 | Improved |
| finetune on AdaVision tests | ViT-H/14 · ImageNet val | accuracy: 88.4 -> 88.4 | No clear effect |
| finetune on AdaVision tests | ViT-H/14 · Five ImageNet OOD sets, treatment classes | accuracy: 78.0 -> 84.0 | Improved |
| finetune on AdaVision tests | ViT-H/14 · Control topics | accuracy: 91.3 -> 91.9 | No clear effect |
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.86)
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.86)
- Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models (ICML 2024 · similarity 0.86)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
