Research map / Label-efficient and robust learning / Recognition and long-tail
Neural Clustering based Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FEC, a fully clustering-based visual backbone that treats feature extraction as iterative representative selection, yielding ad-hoc interpretable cluster hierarchies while remaining competitive on ImageNet, ADE20K and COCO.
Problem
Existing visual feature extractors (ConvNets, ViTs, MLPs) represent images as rigid grids of rectangular regions, which ignores the underlying pixel data distribution and yields black-box features that are hard to interpret.
Approach
Reformulates feature extraction as hierarchical neural clustering: each stage alternates clustering-based pooling (selecting cosine-similarity representatives via adaptive average pooling initialization) and clustering-based encoding (feature dispatching back to pixels), so cluster assignments across layers form a transparent segment pyramid.
Key results
- FEC-Small reaches 72.7% top-1 on ImageNet-1K with 5.5M parameters, outperforming ResNet18 (69.8%) with less than half its parameters.
- FEC-Base achieves 37.7 mIoU on ADE20K with Semantic FPN, vs 32.9 for ResNet18 and 36.7 for ResNet50.
- FEC-Small obtains 35.6 APbox and 33.6 APmask on COCO with Mask R-CNN, above ResNet18 (34.0 / 31.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FEC-Small instead of ResNet18 | Image classification backbone · ImageNet-1K val | Top-1 accuracy: 69.8 -> 72.7±0.06 | Helped |
| Use FEC-Base instead of ResNet18 | Semantic segmentation backbone · ADE20K val | mIoU: 32.9 -> 37.7±0.15 | Helped |
| Use FEC-Base instead of PVT-Tiny | Mask R-CNN backbone · COCO val2017 | APbox: 36.7 -> 37.9±0.06 | Helped |
| Use FEC-Base instead of PVT-Tiny | Mask R-CNN backbone · COCO val2017 | APmask: 35.1 -> 35.5±0.06 | Helped |
| Use dot product instead of Euclidean distance for similarity measurement (ablation) | FEC-Small · ImageNet-1K val | Top-1 accuracy: 72.2 -> 72.7 | Helped |
| Use cosine similarity instead of Euclidean distance for similarity measurement (ablation) | FEC-Small · ImageNet-1K val | Top-1 accuracy: 72.2 -> 72.7 | Helped |
| Add learnable similarity scaling and shifting parameters α and β (ablation) | FEC-Small feature dispatching · ImageNet-1K val | Top-1 accuracy: 72.1 -> 72.7 | Helped |
| Add similarity scaling parameter α (ablation) | FEC-Small feature dispatching · ImageNet-1K val | Top-1 accuracy: 72.4 -> 72.7 | Helped |
| Add similarity shifting parameter β (ablation) | FEC-Small feature dispatching · ImageNet-1K val | Top-1 accuracy: 72.2 -> 72.7 | Helped |
| Increase encoding channels from (48, 48, 96, 96) to (120, 120, 240, 240) (ablation) | FEC-Small · ADE20K val | mIoU: 34.9 -> 35.5 | Helped |
| Increase encoding channels from (48, 48, 96, 96) to (120, 120, 240, 240) (ablation) | FEC-Small · COCO val2017 | APbox: 34.9 -> 35.5 | Helped |
| Increase encoding channels from (48, 48, 96, 96) to (120, 120, 240, 240) (ablation) | FEC-Small · COCO val2017 | APmask: 33.4 -> 33.8 | Helped |
Surprising findings
- Cluster assignments emerge as semantically coherent object or part segments despite training only on classification with no cluster supervision.
Limitations
- Lower throughput than ResNet and not state-of-the-art; the authors state the goal is not to chase SOTA.
- Set-prediction detection heads (e.g., DETR) not explored; dense tasks rely on residual grid-style connections.
- Interpretability analysis is qualitative, relying on visualizations and K-Means reduction of clusters.
Experimental setup
Datasets: ImageNet-1K ADE20K MS COCO 2017
Benchmarks: ImageNet-1K ADE20K MS COCO 2017
Metrics: Top-1 accuracy mIoU AP AP50 AP75 FLOPs FPS
Baselines: ResNet18 ResNet50 PVT-Tiny PVT-Small CoC ViT-B/16 Swin-Tiny ResMLP MLP-Mixer ConvMixer
Builds on: Vision Transformer Context Clusters (CoC) Perceiver Mask R-CNN
Closest papers by meaning
- Deep Ensemble Clustering for Visual Representation Learning (ICML 2026 · similarity 0.87)
- Image as Set of Points (ICLR 2023 · similarity 0.86)
- Control-oriented Clustering of Visual Latent Representation (ICLR 2025 · similarity 0.85)
- Revisiting the Integration of Convolution and Attention for Vision Backbone (NeurIPS 2024 · similarity 0.84)
- Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations (CVPR 2024 · similarity 0.84)
- Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.