Research map / Label-efficient and robust learning / Recognition and long-tail
Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes double-correct predictions, curating a GPT-4-generated structured rationale dataset for ImageNet categories and a rationale-informed optimization (disentanglement plus reconstruction constraints solved via Lagrangian) that fine-tunes CLIP ViT-B/32 to improve both prediction accuracy and rationale localization.
Problem
Foundation models such as CLIP are evaluated mainly on prediction accuracy, ignoring whether correct predictions rest on valid visual evidence, which is needed for safe deployment.
Approach
Decompose ViT image embeddings per token and layer (weighted by mean-ablation importance) to get rationale heatmaps, then train CLIP with self-supervised constraints that make heatmaps of different rationales disentangled while their aggregate reconstructs the category text embedding, requiring no pixel annotations.
Key results
- Zero-shot classification average accuracy 73.1% vs 70.5% for the best baseline CLIP (+2.6%); linear probe average 87.5%.
- Rationale localization on CUB-Part mIoU 21.9 vs 14.4 for CLIP; PartImageNet 11.2 vs 5.2.
- Rationale disentanglement average 0.785 vs 0.420 for CLIP; GPT-based evaluation rates 964 of 1,000 categories' rationales at or above 4.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply rationale-informed optimization | CLIP ViT-B/32 · Nine classification datasets (AVG), zero-shot | accuracy (%): 2.6% | Helped |
| Apply rationale-informed optimization | CLIP ViT-B/32 · Nine classification datasets (AVG), linear probe | accuracy (%): 2.0% | Helped |
| Apply rationale-informed optimization | CLIP ViT-B/32 · CUB-Part | rationale localizability (mIoU): 7.5% | Helped |
| Apply rationale-informed optimization | CLIP ViT-B/32 · PartImageNet | rationale localizability (mIoU): 6.0% | Helped |
| Apply rationale-informed optimization | CLIP ViT-B/32 · Nine classification datasets | rationale disentanglability: 36.5% | Helped |
| Add rationale disentanglement constraint (ablation) | Rationale-informed optimization · CUB-Part | rationale localizability (mIoU): 10.4% | Helped |
| Add reconstruction constraint (ablation) | Rationale-informed optimization · CUB-Part | rationale localizability (mIoU): 13.3% | Helped |
| Add reconstruction constraint (ablation) | Rationale-informed optimization · CUB | accuracy (%): 30.2% | Helped |
| Use an external concept set (ablation) | Rationale-informed optimization · ImageNet, rationale-based prediction | accuracy (%) | Helped |
| Replace rationale concepts with random strings (ablation) | Rationale-informed optimization · ImageNet, rationale-based prediction | accuracy (%): -2.2 | Hurt |
Surprising findings
- Ablating the reconstruction constraint drastically lowered accuracy (56.0% to 25.8% on CUB), showing disentanglement alone collapses to trivial solutions.
- Random-string bottlenecks reduced accuracy of the proposed model (70.5% to 68.3%), while they helped CLIP and CLIP-ft.
Limitations
- Depends heavily on the quality of the structured rationale ontologies generated by GPT-4.
- Lack of explicit ground-truth rationale localization for large-scale datasets; relies on self-supervision.
- Computationally intensive, which may limit use in resource-constrained settings.
Experimental setup
Datasets: ImageNet CUB-200-2011 CUB-Part PartImageNet CIFAR-10 CIFAR-100 Caltech-101 Oxford-IIIT Pets Food-101 SUN397 Stanford Cars DTD
Benchmarks: CIFAR-10 CIFAR-100 CUB-200-2011 Caltech-101 Oxford-IIIT Pets Food-101 SUN397 Stanford Cars DTD Flickr30K MSCOCO CUB-Part
Metrics: accuracy mIoU disentanglability (1 - |cosine|) Likert score
Baselines: CLIP DeCLIP NegCLIP FILIP PyramidCLIP CLIP-ft CLIP-ft-vision
Builds on: CLIP DCLIP WaffleCLIP InfoNCE
Closest papers by meaning
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.88)
- Beyond Accuracy: On the Effects of Fine-Tuning Towards Vision-Language Model’s Prediction Rationality (AAAI 2025 · similarity 0.85)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.85)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.85)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.85)
- DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.