Research map / Architectures and efficiency / Attention and transformers

FFT-Based Dynamic Token Mixer for Vision

AAAI 2024 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 64 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dynamic Filter, an FFT-based token mixer whose global frequency filters are linearly combined per channel from a learned basis with weights predicted by an MLP, and builds DFFormer and CDFFormer MetaFormer-style vision models.

Problem

Global FFT-based token mixers (global filters such as GFNet) are cheaper than self-attention at high resolution but have static filters and have not been examined within the MetaFormer architecture, lagging state-of-the-art models in accuracy.

Approach

Replace the static global filter with a dynamic filter: for each channel, input-dependent softmax weights (predicted by a small MLP from globally pooled features) linearly combine a basis of N learnable complex frequency filters, and the result is applied inside a MetaFormer block; hybrids mix in separable convolution blocks (CDFFormer).

Key results

• DFFormer-B36 (115M params) reaches 84.8% top-1 on ImageNet-1K at 224^2, and CDFFormer-B36 reaches 85.0%, the best among FFT-based models in the comparison table.
• CDFFormer-M36 reaches 48.6 mIoU on ADE20K with Semantic FPN, versus 42.4 for PoolFormer-M36.
• At 1024^2 resolution on V100, DFFormer-S18 throughput is 30.54 img/s versus 15.61 for CAFormer-S18, and peak memory is 1192 MB versus 4131 MB.

Measured effects reported in the paper

Surprising findings

• Despite lower theoretical complexity, FFT-based models have lower throughput than CAFormer at 224^2 resolution, which the authors attribute to FFT implementation and hardware.
• CAFormer-S18 acts as a high-pass filter in stage 3, while the dynamic filter model attenuates high frequencies, and FFT-based and MHSA models do not learn similar representations despite both being global.

Limitations

• At 224^2 resolution the proposed models have lower throughput than the attention-based CAFormer, contrary to the theoretical complexity argument.
• The default dynamic filter does not support arbitrary input resolutions; the global filter basis must be interpolated for detection and segmentation.
• Accuracy gains over the static global filter (GFFormer) are modest, about 0.3% top-1.

Experimental setup

Datasets: ImageNet-1K ADE20K COCO

Benchmarks: ImageNet-1K ADE20K COCO

Metrics: top-1 accuracy mIoU box AP throughput peak memory FLOPs linear CKA

Baselines: ConvNeXt ConvFormer CAFormer GFNet AMixer DynaMixer RMT MaxViT GCViT PoolFormer PVT ResNet

Builds on: GFNet MetaFormer AFNO ConvFormer StarReLU

Closest papers by meaning

• Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.82)
• Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.81)
• Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks (CVPR 2025 · similarity 0.81)
• Frequency Dynamic Convolution for Dense Image Prediction (CVPR 2025 · similarity 0.81)
• TransNeXt: Robust Foveal Visual Perception for Vision Transformers (CVPR 2024 · similarity 0.81)
• Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers (ICML 2025 · similarity 0.81)