Research map / Architectures and efficiency / Attention and transformers
Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies massive activations in Diffusion Transformers (DiTs) tied to AdaLN-zero residual scaling and proposes training-free DiTF, which uses AdaLN channel-wise modulation plus channel discard to extract features for visual correspondence.
Problem
Pre-trained DiT features perform poorly for dense visual correspondence because a few fixed dimensions carry massive activations that make spatial features uninformative and hard to discriminate by cosine similarity.
Approach
Massive activations in DiTs concentrate in fixed channels that coincide with AdaLN-zero residual scaling factors; applying the block's built-in AdaLN modulation (with timestep and text conditioning) to the pre-AdaLN feature suppresses them and yields semantically discriminative features, and zeroing residual weakly-massive channels further helps.
Key results
- DiTF-flux achieves 67.1% PCK@0.10 on SPair-71k, a 9.4% gain over DIFT (SD2-1).
- DiTF-sd3-5 with DINOv2 integration reaches 72.2% on SPair-71k, above GeoAware-SC's 69.6%.
- On ADE20K semantic segmentation, DiTF-flux improves mIoU from 44.8 (DINOv2) to 54.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiTF Flux features instead of DIFT features | Visual correspondence · SPair-71k | PCK@0.10: 52.9 -> 64.0; +9.4% | Helped |
| Use DiTF Flux features instead of DINOv2+NN features | Visual correspondence · AP-10K-I.S. | PCK@0.10: 60.9 -> 62.2; 1.3% ↑ | Helped |
| Integrate DINOv2 features with DiTF SD3-5 | DiTF SD3-5 · SPair-71k | PCK@0.10: 64.6% to 72.2% | Helped |
| Integrate DINOv2 features with DiTF Flux | DiTF Flux · SPair-71k | PCK@0.10: 67.1% to 67.6%; 0.5% gain | Helped |
| Use DiTF SD3-5 + DINOv2 instead of GeoAware-SC | Cross-category semantic correspondence · AP-10K-C.S. | PCK@0.10: 64.6% to 69.4%; nearly 4.8% | Helped |
| Add AdaLN channel modulation to DiT features (ablation) | DiT feature extraction · Semantic correspondence across datasets | PCK: more than 20% | Helped |
| Discard weakly massive channels after AdaLN modulation (ablation) | DiTF · Correspondence matching across datasets | matching accuracy | Helped |
| Condition AdaLN modulation on timestep t rather than only text embedding c (ablation) | DiT feature modulation · Semantic correspondence | PCK | Helped |
| Apply AdaLN channel modulation to Flux features (ablation) | Semantic segmentation · ADE20K | mIoU: 44.8 to 54.8 | Helped |
| Use DiTF Flux instead of DIFT | Temporal correspondence · DAVIS-2017 | J&F mean: 70.0 -> 72.2; 2.2% | Helped |
| Use DiTF Flux instead of DIFT | Geometric correspondence · HPatches, All, ε=1 | homography estimation accuracy: 45.6 -> 41.9 | Hurt |
Surprising findings
- Integrating DINOv2 features gives a large gain for SD3.5 (64.6% to 72.2%) but only a small gain for FLUX (67.1% to 67.6%), which the authors attribute to FLUX's close alignment with DINOv2.
- Timestep conditioning is essential for modulating massive activations, while modulation conditioned only on text embedding is ineffective.
Limitations
- Role of massive activations in the generative process of DiTs is unexplored.
- FLUX gains little from DINOv2 integration, suggesting feature redundancy.
- Geometric correspondence on HPatches is only comparable to DIFT rather than clearly superior.
Experimental setup
Datasets: SPair-71k PF-Pascal AP-10K HPatches DAVIS-2017 ADE20K
Benchmarks: SPair-71k PF-Pascal AP-10K HPatches DAVIS-2017 ADE20K
Metrics: PCK homography accuracy J&F mIoU
Baselines: DIFT DINOv2+NN SD+DINO GeoAware-SC DHF CATs++ SCorrSAN PMNC SCOT ASIC SIFT SuperPoint
Builds on: DIFT DINOv2 Massive activations in LLMs
Closest papers by meaning
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.89)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.89)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.89)
- Rethinking Cross-Layer Information Routing in Diffusion Transformer (NeurIPS 2026 · similarity 0.89)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.88)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.