Research map / Vision-language and multimodal / CLIP and zero-shot
Reproducibility study of “LICO: Explainable Models with Language-Image Consistency"
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reproduces LICO (language-image consistency regularisation with manifold-matching and optimal-transport losses) for interpretable CNNs, finding that the claimed gains in saliency-map interpretability and classification accuracy mostly do not replicate.
Problem
Whether LICO, which uses CLIP-style text-encoder supervision via learnable prompts to regularise image features, consistently improves post-hoc saliency-map interpretability and classification accuracy as originally claimed.
Approach
A reproducibility study that reimplements the missing LICO training loop, manifold-matching loss and Insertion/Deletion metrics, then tests the paper's claims with ablations (OT-only, MM-only), a PartImageNet IoU check, and a learnable vs fixed temperature comparison.
Key results
- LICO reduced Insertion/Deletion-based interpretability for nearly every saliency method and dataset, e.g. Imagenette Grad-CAM Overall 54.1 baseline vs 49.0 with LICO.
- Test accuracy with LICO was slightly lower on all datasets, e.g. CIFAR-100 78.1±0.66 baseline vs 77.3±0.90 with LICO, and 28.5 vs 18.6 with 2,500 examples.
- On PartImageNet, mean Grad-CAM IoU fell from 0.185 (baseline) to 0.141 (LICO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LICO losses | ResNet-50 · ImageNet | accuracy: 49.75 -> 48.18 | Hurt |
| Add LICO losses | Wide ResNet-18 · Imagenette | accuracy: 87.3 ± 0.74 -> 86.8 ± 1.58 | No clear effect |
| Add LICO losses | Wide ResNet-18 · CIFAR-10 | accuracy: 93.9 ± 0.13 -> 93.6 ± 0.29 | Hurt |
| Add LICO losses | Wide ResNet-18 · CIFAR-100 | accuracy: 78.1 ± 0.66 -> 77.3 ± 0.90 | Hurt |
| Add LICO losses | Wide ResNet-18 · CIFAR-100 (2500 examples) | accuracy: 28.5 ± 0.39 -> 18.6 ± 1.64 | Hurt |
| Add LICO losses | ResNet-18 · PartImageNet | IoU: 0.185 ± 0.005 -> 0.141 ± 0.008 | Hurt |
| Add LICO losses | LICO saliency maps · CIFAR-10, CIFAR-100, Imagenette, ImageNet (20%) | Insertion/Deletion | Mixed |
| Add the MM loss alone (ablation) | LICO classifier · Imagenette | accuracy: 87.3 ± 0.74 -> 88.0 ± 0.26 | Helped |
| Add the MM loss alone (ablation) | LICO classifier · CIFAR-100 | accuracy: 77.7 ± 0.08 -> 78.6 ± 0.33 | Helped |
| Add the MM loss alone (ablation) | LICO classifier · CIFAR-100 (2500 examples) | accuracy: 28.5 ± 0.39 -> 26.1 ± 1.10 | Hurt |
| Add the OT loss alone (ablation) | LICO classifier · Imagenette | accuracy: 87.3 ± 0.74 -> 83.8 ± 1.58 | Hurt |
| Use both OT and MM losses instead of MM alone (ablation) | LICO classifier · CIFAR-100 | accuracy: 78.6 ± 0.33 -> 77.0 ± 1.08 | Hurt |
Surprising findings
- The OT loss implementation in the authors' public code reduces text features to a single vector per class via an EOT token, contradicting the paper's described fine-grained alignment.
- Using both OT and MM losses was consistently worse than MM loss alone, contrary to the original paper's ablation.
Limitations
- The authors' original code was incomplete (missing MM loss and training loop), so the reimplementation may differ substantially from the original.
- Authors did not respond to correspondence, so missing setup details could not be clarified.
- Some experiments were run only once (ImageNet) or with a single seed for hyperparameter search.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Imagenette PartImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Imagenette PartImageNet
Metrics: test accuracy Insertion score Deletion score Overall score (insertion - deletion) Intersection over Union
Baselines: Baseline ResNet/Wide ResNet without LICO MM-only loss OT-only loss
Builds on: LICO Grad-CAM RISE CLIP Sinkhorn algorithm
Closest papers by meaning
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.85)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.83)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.83)
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.83)
- JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.83)
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.