Research map / Label-efficient and robust learning / OOD and anomaly detection
LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LCA-on-the-Line, using in-distribution WordNet lowest-common-ancestor (taxonomic) distance to predict OOD accuracy across 75 vision and vision-language models, and shows taxonomy-aligned soft labels and prompts improve generalization.
Problem
Predicting models' out-of-distribution accuracy from in-distribution measurements fails when models differ in supervision and training data (vision models on ImageNet vs. VLMs on LAION), because ID accuracy no longer tracks OOD accuracy across families.
Approach
Use the in-distribution LCA distance over a class taxonomy (WordNet, or latent K-means hierarchies) as a mistake-severity measure; it shows a strong linear correlation with OOD top-1 accuracy across VMs and VLMs, unlike ID top-1 accuracy.
Key results
- ID LCA distance correlates strongly with OOD top-1 accuracy on ImageNet-S/R/A and ObjectNet (R^2 0.70-0.92, Pearson 0.84-0.96) across 75 models, while ID top-1 correlates poorly on those datasets (R^2 0.01-0.27).
- ID LCA gives lowest MAE for error prediction on ImageNet-S (0.093), ImageNet-A (0.103), ObjectNet (0.048) versus ID Top1 and agreement baselines.
- Adding LCA soft loss with linear interpolation improves OOD accuracy for linear probes, e.g., ResNet-50 ImageNet-R 36.5 to 42.5 and ImageNet-A 10.3 to 16.2, with ID accuracy preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ImageNet ID LCA instead of ID Top1 to predict OOD Top1 | 75 vision and vision-language models · ImageNet-Sketch | R²: 0.075 -> 0.816 | Helped |
| Use ImageNet ID LCA instead of ID Top1 to predict OOD Top1 | 75 vision and vision-language models · ImageNet-Rendition | R²: 0.020 -> 0.779 | Helped |
| Use ImageNet ID LCA instead of ID Top1 to predict OOD Top1 | 75 vision and vision-language models · ImageNet-Adversarial | R²: 0.009 -> 0.704 | Helped |
| Use ImageNet ID LCA instead of ID Top1 to predict OOD Top1 | 75 vision and vision-language models · ObjectNet | R²: 0.273 -> 0.915 | Helped |
| Use ImageNet ID LCA instead of ID Top1 to predict OOD Top1 | 75 vision and vision-language models · ImageNet-v2 | R²: 0.962 -> 0.339 | Hurt |
| Use WordNet LCA soft loss with cross-entropy and weight interpolation | ResNet-18 linear probe · ImageNet-Sketch | Top1 accuracy: 19.7 -> 20.7 (+1.0) | Helped |
| Use WordNet LCA soft loss with cross-entropy and weight interpolation | ResNet-18 linear probe · ImageNet-Rendition | Top1 accuracy: 31.9 -> 33.8 (+1.8) | Helped |
| Use WordNet LCA soft loss with cross-entropy and weight interpolation | ResNet-18 linear probe · ImageNet-Adversarial | Top1 accuracy: 1.1 -> 1.2 (+0.1) | Helped |
| Use WordNet LCA soft loss with cross-entropy and weight interpolation | ResNet-18 linear probe · ObjectNet | Top1 accuracy: 27.0 -> 28.0 (+1.0) | Helped |
| Replace WordNet soft-label hierarchy with a K-means latent hierarchy | ResNet-18 linear probe · ImageNet-Rendition | Top1 accuracy: 31.9 -> 35.1 (+3.2) | Helped |
| Add the correct taxonomy path and hierarchical relationship to zero-shot prompts | CLIP-ViT32 · ImageNet OOD datasets | Top1 accuracy | Helped |
| Provide the correct taxonomy path without explaining class-name relationships (Stack Parent) (ablation) | CLIP-ViT32 zero-shot prompting · ImageNet OOD datasets | Top1 accuracy | Mixed |
Surprising findings
- ID top-1 accuracy tracks OOD accuracy on ImageNet-v2 but not on the other shifted sets, and LCA's advantage vanishes on ImageNet-v2, which the authors attribute to its similarity to ImageNet.
- LCA distance and top-1 accuracy do not align consistently when VLMs are included, challenging the belief that they share the same ranking.
Limitations
- Not effective for OOD datasets visually similar to ImageNet (e.g., ImageNet-v2), where ID top-1 remains a strong indicator.
- Weaker model discrimination on datasets with few classes such as CIFAR.
- Correlational rather than causal; the paper acknowledges adversarial models can have low LCA with zero top-1 accuracy.
Experimental setup
Datasets: ImageNet ImageNet-v2 ImageNet-Sketch ImageNet-Rendition ImageNet-Adversarial ObjectNet WordNet
Benchmarks: ImageNet ImageNet-v2 ImageNet-Sketch ImageNet-Rendition ImageNet-Adversarial ObjectNet
Metrics: LCA distance top-1 accuracy top-5 accuracy R^2 Pearson correlation Kendall tau Spearman correlation MAE
Baselines: ID Top1 (accuracy-on-the-line) Average Confidence Agreement-on-the-line (Aline-D) Agreement-on-the-line (Aline-S)
Builds on: Accuracy-on-the-line Agreement-on-the-line effective robustness making better mistakes (hierarchical error)
Closest papers by meaning
- Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.87)
- OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable? (ICML 2025 · similarity 0.86)
- In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.86)
- AHA: Human-Assisted Out-of-Distribution Generalization and Detection (NeurIPS 2024 · similarity 0.86)
- Unraveling the Key Components of OOD Generalization via Diversification (ICLR 2024 · similarity 0.85)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.