Research map / Graph learning / GNN architectures
Hypergraph Vision Transformers: Images are More than Nodes, More than Edges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HgVT, a vision transformer that builds a hierarchical bipartite hypergraph over image patches and virtual vertices/hyperedges, using dynamic cosine-based adjacency with diversity and population regularization and expert edge pooling, reporting ImageNet-1k classification and image retrieval results.
Problem
Vision Transformers model patch relations only weakly via full self-attention, while vision GNNs (ViG, ViHGNN) rely on costly clustering (KNN, Fuzzy C-Means) for edge/hyperedge generation, trading off efficiency against adaptability.
Approach
Represent the image as a bipartite hypergraph between patch vertices and hyperedges, build the adjacency dynamically by sharpened cosine similarity instead of clustering, regularize with diversity and population losses, and pool virtual hyperedges with an expert gate, so the hypergraph is an output usable for retrieval.
Key results
- HgVT-Ti reaches 76.2% ImageNet-1k Top-1, 1.9 points above ViHGNN-Ti (74.3%), with 7.7M params and 1.8B FLOPs.
- HgVT-S matches DeiT-B on ImageNet ReaL (86.7) with about a quarter of the parameters (22.9M vs 86.4M).
- On ImageNet-1k retrieval, HgVT-Ti with Im&Ex pooling and PS reaches 73.23 mAP@10, versus 64.94 for MRL-128 at 70.52 Top-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HgVT-Ti instead of ViHGNN-Ti | ImageNet isotropic classification models · ImageNet-1k | Top-1 accuracy: 74.3 -> 76.2 | Helped |
| use HgVT-S instead of DeiT-B | ImageNet classification models · ImageNet-1k ReaL | Top-1 accuracy: 86.7 -> 86.7 | No clear effect |
| use HgVT-Ti with Im&Ex pooling and PS instead of MRL-256 | ImageNet retrieval · ImageNet-1k retrieval | mAP@10 (%): 65.04 -> 73.23 | Helped |
| use HgVT-S instead of DINOv1-S/16 trained on GLDv2 | image retrieval · ROxford-M | mAP (%): 51.5 -> 28.0 | Hurt |
| add stochastic path dropout decay (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 81.89 -> 84.36 | Helped |
| add class dropout (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 84.36 -> 82.23 | Hurt |
| add diversity regularization (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 80.79 -> 84.36 | Helped |
| add population regularization (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 81.79 -> 84.36 | Helped |
| untie vertex and edge FFN weights (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 84.36 -> 82.89 | Hurt |
| tie adjacency and embedding features (Xadj = X) (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 84.36 -> 78.62 | Hurt |
| replace expert edge pooling with average edge pooling (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 82.06 vs. 82.52 | Hurt |
| use image pooling instead of expert edge pooling (ablation) | HgVT-Lt · ImageNet-100 | Top-1 accuracy: 82.52 -> 84.08 | Helped |
Surprising findings
- Image pooling gives the highest ablation accuracy (84.08) but degrades graph structure quality, showing a weak anti-correlation between graph quality metrics and accuracy.
- Removing population regularization yields near-zero sparsity, effectively reducing HgVT to a ViT with added complexity.
Limitations
- Isotropic-only evaluation; pyramidal variants not explored due to difficulty downsampling virtual tokens.
- HgVT-S was depth-reduced and only matches ViHGNN-S; the method is described as challenging to tune.
- Retrieval lags on ROxford-M compared with multi-scale ConvNets and self-supervised features.
Experimental setup
Datasets: ImageNet-1k ImageNet-100 CIFAR-100 Oxford Revisited Paris Revisited
Benchmarks: ImageNet-1k ImageNet ReaL ImageNet V2 ImageNet-100 ROxford RParis
Metrics: Top-1 accuracy mAP@10 1-NN-hit@10 mAP hyperedge entropy intra-cluster similarity inter-cluster distance silhouette score
Baselines: DeiT ViG ViHGNN ResMLP ConvMixer DINOv1 MRL ALEX+GeM RN101+R-MAC
Builds on: ViG ViHGNN DeiT MRL
Closest papers by meaning
- DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision Recognition (CVPR 2025 · similarity 0.89)
- GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs (CVPR 2024 · similarity 0.87)
- Vision Graph Prompting via Semantic Low-Rank Decomposition (ICML 2025 · similarity 0.87)
- Deep Hypergraph Neural Networks with Tight Framelets (AAAI 2025 · similarity 0.86)
- When Hypergraph Meets Heterophily: New Benchmark Datasets and Baseline (AAAI 2025 · similarity 0.86)
- Residual Hyperbolic Graph Convolution Networks (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.