Research map / Label-efficient and robust learning / Recognition and long-tail
I-Con: A Unifying Framework for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes I-Con, a single information-theoretic loss (integrated KL divergence between supervisory and learned neighborhood distributions) that unifies over 23 representation learning methods, and uses it to derive a debiased, KNN-propagated InfoNCE clustering method for unsupervised ImageNet-1K classification.
Problem
Representation learning has a proliferation of loss functions, making it hard to see how methods relate and which objective suits a task; prior unifications link only two or three methods at a time.
Approach
Many losses (clustering, spectral, contrastive, dimensionality reduction, supervised classification) are instances of minimizing the average KL divergence between a supervisory neighborhood distribution p(j|i) and a learned neighborhood distribution q(j|i); debiasing is applied by mixing a uniform distribution into the supervisory p, and neighbors are expanded via KNN graph walks.
Key results
- Debiased InfoNCE Clustering reaches 67.52% Hungarian accuracy on ImageNet-1K with DINO ViT-L/14, versus 59.84-60.15% for prior clustering baselines and 58.62% for TEMI on ViT-B/14, reporting +4.5% and +7.8% gains over TEMI for ViT-B/14 and ViT-L/14.
- Debiasing with alpha=0.4 improves t-SimCLR linear probing on CIFAR-10 from 90.97 to 92.70 (Student-t kernel).
- Ablation shows KNN propagation with 1-walks reaches 58.09 on DINO ViT-S/14 vs 55.51 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Debiased InfoNCE Clustering instead of TEMI | ImageNet-1K clustering with DiNO ViT-S/14 · ImageNet-1K | Hungarian accuracy: 56.84 -> 57.8 ± 0.26 | Helped |
| Use Debiased InfoNCE Clustering instead of TEMI | ImageNet-1K clustering with DiNO ViT-B/14 · ImageNet-1K | Hungarian accuracy: 58.62 -> 64.75 ± 0.18 | Helped |
| Add debiasing (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 57.27± 0.07; 63.03 -> 63.72 ± 0.09; 65.70 -> 66.87 ± 0.07 | Helped |
| Add KNN propagation (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 58.45 ± 0.23; 63.03 -> 64.87 ± 0.19; 65.70 -> 67.25 ± 0.21 | Helped |
| Add EMA (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 57.8 ± 0.26; 63.03 -> 64.75 ± 0.18; 65.70 -> 67.52 ± 0.28 | Helped |
| Add KNNs (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 56.43; 63.03 -> 64.26; 65.72 -> 65.70 | Mixed |
| Add 1-step walks on the KNN graph (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 58.09; 63.03 -> 64.29; 65.72 -> 65.97 | Helped |
| Add 2-step walks on the KNN graph (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 57.84; 63.03 -> 64.27; 65.72 -> 67.26 | Helped |
| Add 3-step walks on the KNN graph (ablation) | ImageNet-1K clustering · ImageNet-1K (DiNO ViT-S/14, ViT-B/14, ViT-L/14) | Hungarian accuracy: 55.51 -> 57.82; 63.03 -> 64.15; 65.72 -> 67.02 | Helped |
| Apply debiasing to contrastive feature learning (ablation) | Gaussian-kernel feature learner · CIFAR-100 (out of distribution), linear probing | accuracy: 32.44 -> 32.53; 32.44 -> 30.67 | Mixed |
| Use Student's t-distribution instead of Gaussian qφ (ablation) | Debiased feature learner, α = 0.4 · CIFAR-100 (out of distribution), linear probing | accuracy: 32.53 -> 41.98 | Helped |
| Apply debiasing to contrastive feature learning (ablation) | Student's t-kernel feature learner · STL-10, linear probing | accuracy: 85.11 -> 85.94; 85.11 -> 86.13; 85.11 -> 87.18 | Helped |
Surprising findings
- The DCL baseline diverges when using a Student's t-distribution kernel, while the proposed supervisory-side debiasing remains stable.
Limitations
- Table of unified methods is not exhaustive and many proofs are sketched or approximate (e.g., some limits and bounds are loose)
- Gains reported on ImageNet clustering are relative to a limited set of baselines, and the ViT-L/14 TEMI comparison is missing
- Hyperparameter sensitivity to alpha and propagation steps varies across backbones
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 STL-10 Oxford-IIIT Pet
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 STL-10 Oxford-IIIT Pet
Metrics: Hungarian accuracy linear probing accuracy KNN accuracy logistic regression accuracy
Baselines: TEMI SCAN Contrastive Clustering k-Means SimCLR DCL t-SimCLR
Builds on: InfoNCE SNE t-SNE SimCLR TEMI
Closest papers by meaning
- Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning (CVPR 2022 · similarity 0.87)
- Contrastive Learning Based Hybrid Networks for Long-Tailed Image Classification (CVPR 2021 · similarity 0.87)
- A Simple Framework for Contrastive Learning of Visual Representations (ICML 2020 · similarity 0.87)
- SCoRe: Submodular Combinatorial Representation Learning (ICML 2024 · similarity 0.86)
- Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses (ICML 2024 · similarity 0.85)
- A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.