Research map / Architectures and efficiency / Attention and transformers
Building Vision Models upon Heat Conduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Heat Conduction Operator (HCO), which models patch correlations as heat diffusion implemented via DCT/IDCT with learned frequency-value-embedding-predicted diffusivity, and builds the vHeat vision backbone family.
Problem
Attention-based vision backbones have O(N^2) cost when pursuing global receptive fields, making high-resolution inference expensive.
Approach
Treats image patches as heat sources and models their correlations via adaptive thermal diffusion; under a Neumann boundary assumption the heat equation solution is computed with DCT/IDCT, giving O(N^1.5) global mixing with learnable frequency value embeddings predicting thermal diffusivity.
Key results
- vHeat-B reaches 84.0% top-1 on ImageNet-1K with 68M params and 11.2G FLOPs, versus Swin-B's 83.5%, at 661 vs 456 img/s throughput.
- At 768x768 input, vHeat-B gives about 3x Swin-B throughput, with 80% less GPU memory and 35% fewer FLOPs.
- vHeat-B achieves 47.7 box AP on COCO Mask R-CNN 1x and 49.6 mIoU on ADE20K UperNet, both above Swin-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use vHeat-T instead of ConvNeXt-T | ImageNet-1K image classification · ImageNet-1K | top-1 accuracy (%): 82.1 -> 82.2 | Helped |
| use vHeat-S instead of ConvNeXt-S | ImageNet-1K image classification · ImageNet-1K | top-1 accuracy (%): 83.1 -> 83.6 | Helped |
| use vHeat-B instead of ConvNeXt-B | ImageNet-1K image classification · ImageNet-1K | top-1 accuracy (%): 83.8 -> 84.0 | Helped |
| use vHeat backbones instead of ConvNeXt backbones | COCO Mask R-CNN 1× object detection · COCO, Mask R-CNN 1× | box AP: 44.2 -> 45.1; 45.4 -> 46.8; 47.0 -> 47.7 | Helped |
| use vHeat backbones instead of ConvNeXt backbones | COCO Mask R-CNN 1× instance segmentation · COCO, Mask R-CNN 1× | mask AP: 40.1 -> 41.2; 41.8 -> 42.3; 42.7 -> 43.0 | Helped |
| use vHeat backbones instead of ConvNeXt backbones | ADE20K semantic segmentation with UperNet · ADE20K, UperNet | mIoU: 46.0 -> 46.9; 48.7 -> 49.1; 49.1 -> 49.6 | Helped |
| use vHeat-B instead of ConvNeXt-B | robust image classification · ObjectNet | top-1 accuracy (%): 26.1 -> 26.7 | Helped |
| use vHeat-B instead of ConvNeXt-B | robust image classification · ImageNet-A | top-1 accuracy (%): 36.5 -> 36.8 | Helped |
| replace SwinIR self-attention modules with HCO | vHeatIR low-level vision tasks · Set12 image denoising, σ = 15 | average PSNR: 33.33 -> 33.37 | Helped |
| replace SwinIR self-attention modules with HCO | vHeatIR low-level vision tasks · McMaster image denoising, σ = 15 | average PSNR: 35.55 -> 35.60 | Helped |
| replace SwinIR self-attention modules with HCO | vHeatIR low-level vision tasks · LIVE1 JPEG artifact reduction, q = 40 | average PSNR: 34.61 -> 34.64 | Helped |
| use adaptive thermal diffusivity predicted by shared FVEs rather than fixed k = 0.0 (ablation) | vHeat-T · ImageNet-1K | top-1 accuracy (%): 81.0 -> 82.2 | Helped |
Surprising findings
- Fixing the thermal diffusivity to k=0 degrades accuracy only modestly (81.0 vs 82.2), indicating the HCO structure contributes even without adaptive diffusivity.
Limitations
- Efficiency gains are most pronounced at high resolutions; at 224 resolution vHeat's advantage is smaller.
- Downstream tasks rely on interpolating FVEs and diffusivity to new input sizes, which is an additional design choice.
Experimental setup
Datasets: ImageNet-1K MS COCO 2017 ADE20K Set12 McMaster LIVE1 ObjectNet ImageNet-A
Benchmarks: ImageNet-1K COCO ADE20K ObjectNet ImageNet-A Set12 McMaster LIVE1
Metrics: top-1 accuracy box AP mask AP mIoU PSNR throughput FLOPs GPU memory
Baselines: Swin-T Swin-S Swin-B ConvNeXt-T ConvNeXt-S ConvNeXt-B Vim DCFormer GFNet SwinIR NAT ViL
Builds on: Swin Transformer Mamba / VMamba GFNet SwinIR
Closest papers by meaning
- CViT: Continuous Vision Transformer for Operator Learning (ICLR 2025 · similarity 0.85)
- Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset (CVPR 2025 · similarity 0.83)
- HeatFormer: A Neural Optimizer for Multiview Human Mesh Recovery (CVPR 2025 · similarity 0.83)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.82)
- Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.82)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.