Research map / Vision-language and multimodal / Human and embodied modeling
ViTree: Single-Path Neural Tree for Step-Wise Interpretable Fine-Grained Visual Categorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViTree, a single-path neural decision tree over a Swin Transformer backbone that selects hard patches at each node and a single leaf path, achieving interpretable fine-grained classification on CUB-200-2011 and Stanford Cars.
Problem
Interpretability of fine-grained visual categorization models: existing neural trees and prototype methods use soft distributions, ensembles, or latent prototypes, which limit intrinsic step-wise interpretability while trailing transformer accuracy.
Approach
Combine a vision transformer with a binary neural tree where each node picks a hard patch from the transformer features via attention-based patch selection and refines the representation parent-to-child, and a learned path selector chooses a single leaf (hard path) for prediction.
Key results
- ViTree reaches 92.0% top-1 on CUB-200-2011, versus 91.2% for the SwinT-B backbone and 91.8% for the best transformer-only method (SIM-Trans, IELT).
- ViTree reaches 95.1% top-1 on Stanford Cars, above ViT-NeT (95.0%) and the SwinT-B backbone (94.5%).
- In a human survey, 70.04% of 671 responses reported understanding the decision path, and 82.06% of 513 responses correctly detected model errors given the path.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViTree with hard patches and a single learned tree path | SwinT-B · CUB-200-2011 | Top-1 accuracy: 91.8 -> 92.0 | Helped |
| Use ViTree with hard patches and a single learned tree path | SwinT-B · Stanford Cars | Top-1 accuracy: 95.0 -> 95.1 | Helped |
| Use only the vision-transformer loss ℓ_vit instead of the combined loss (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy: 92.03 -> 91.66 | Hurt |
| Use only the tree loss ℓ_leaf instead of the combined loss (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy: 92.03 -> 91.61 | Hurt |
| Select one learned highest-weight leaf instead of averaging all leaves (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy: 91.57 -> 92.03 | Helped |
| Select one leaf using path probabilities instead of softly aggregating leaves (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy: 91.52 -> 91.73 | Helped |
| Learn leaf weights instead of deriving them from path probabilities (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy: 91.52 -> 91.40; 91.73 -> 92.03 | Mixed |
| Use a 5-layer or 6-layer tree rather than a shallow or excessively deep tree (ablation) | ViTree · CUB-200-2011 | Top-1 accuracy | Helped |
Surprising findings
- Hard single-path selection outperforms soft path-probability and mean-over-leaves aggregation in the leaf-strategy ablation (92.03 vs 91.52 and 91.57 on CUB).
Limitations
- Gains over the transformer backbone are small (0.8% on CUB, 0.6% on Cars, with inconsistent figures in the text)
- Interpretability evaluated through small human surveys and qualitative case studies rather than formal metrics
- Hyperparameters chosen by heuristics and grid search; tree depth above 5-6 reportedly hard to train
Experimental setup
Datasets: CUB-200-2011 Stanford Cars
Benchmarks: CUB-200-2011 Stanford Cars
Metrics: Top-1 accuracy
Baselines: SwinT-B TransFG FFVT AFTrans SIM-Trans IELT ProtoPNet ProtoPool ProtoPFormer ACNet ProtoTree ViT-NeT
Builds on: Swin Transformer ProtoTree ViT-NeT ProtoPNet
Closest papers by meaning
- ViT-NeT: Interpretable Vision Transformers with Neural Tree Decoder (ICML 2022 · similarity 0.92)
- Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization (CVPR 2020 · similarity 0.89)
- Neural Prototype Trees for Interpretable Fine-Grained Image Recognition (CVPR 2021 · similarity 0.87)
- COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition (NeurIPS 2026 · similarity 0.85)
- PICNN: A Pathway towards Interpretable Convolutional Neural Networks (AAAI 2024 · similarity 0.84)
- Novel Class Discovery for Ultra-Fine-Grained Visual Categorization (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.