Research map / Architectures and efficiency / Attention and transformers
Can We Get Rid of Handcrafted Feature Extractors? SparseViT: Nonsemantics-Centered, Parameter-Efficient Image Manipulation Localization Through Spare-Coding Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseViT, a Uniformer-based vision transformer whose dense self-attention is replaced by block-sparse attention with multi-scale learnable fusion, enabling image manipulation localization without handcrafted non-semantic feature extractors.
Problem
Image manipulation localization (IML) relies on handcrafted non-semantic feature extractors (Bayar, SRM, DCT, Sobel, Noiseprint), which hurt generalization to unseen or complex manipulation scenarios.
Approach
Reformulate ViT's global self-attention into a sparse, block-wise form with exponentially decaying sparsity rates across layers, so the model adaptively learns non-semantic (manipulation-sensitive) features while suppressing semantic content; pair it with a learnable feature fusion (LFF) prediction head.
Key results
- SparseViT reaches average pixel-level F1 of 0.671 across COVERAGE, Columbia, CASIAv1 and NIST16, above TruFor (0.627) and CAT-Net (0.601).
- SparseViT uses 50.3M parameters and 46.2G FLOPs versus TruFor's 68.7M and 236.5G, a reduction of over 80% in FLOPs relative to TruFor.
- In ablation, sparse attention alone raises Uniformer's CASIAv1 F1 from 0.789 to 0.813 and reduces FLOPs to 0.83x of the backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace global attention with sparse attention (ablation) | Uniformer · COVERAGE | pixel-level F1: 0.378 -> 0.485 | Helped |
| add sparse attention and LFF (ablation) | Uniformer · COVERAGE | pixel-level F1: 0.378 -> 0.513 | Helped |
| add LFF prediction head (ablation) | sparse-attention Uniformer · COVERAGE | pixel-level F1: 0.485 -> 0.513 | Helped |
| replace LFF with MLP prediction head (ablation) | SparseViT · COVERAGE | pixel-level F1: 0.513 -> 0.492 | Hurt |
| use single-scale features instead of multi-scale fusion (ablation) | SparseViT · COVERAGE | pixel-level F1: 0.485 -> 0.513 | Hurt |
| add Sobel feature extractor and LFF (ablation) | Uniformer · CASIAv1 | pixel-level F1: 0.789 -> 0.768 | Hurt |
| add Bayar feature extractor and LFF (ablation) | Uniformer · CASIAv1 | pixel-level F1: 0.789 -> 0.795 | Helped |
| add DCT feature extractor and LFF (ablation) | Uniformer · CASIAv1 | pixel-level F1: 0.789 -> 0.805 | Helped |
| add SRM feature extractor and LFF (ablation) | Uniformer · CASIAv1 | pixel-level F1: 0.789 -> 0.793 | Helped |
| use one sparsity rate rather than exponentially decreasing rates across layers (ablation) | SparseViT · COVERAGE | pixel-level F1: 0.461, 0.479, 0.484 -> 0.513 | Hurt |
| apply sparse attention (ablation) | ViT · AVG | pixel-level F1: 0.578 -> 0.598 | Helped |
| apply sparse attention (ablation) | VOLO · AVG | pixel-level F1: 0.510 -> 0.522 | Helped |
Surprising findings
- Several handcrafted extractors (Sobel, Bayar) combined with LFF did not improve over LFF alone and sometimes degraded performance, e.g. Uniformer (Sobel+LFF) F1 on COVERAGE fell to 0.312 from 0.473.
- CASIAv1 F1 was nearly identical across single-scale, MLP and LFF heads, so CASIAv1 was judged insufficient to reflect generalization.
Limitations
- Handcrafted-free claim is supported only by comparisons with a limited set of extractors and a single backbone family.
- Evaluation trains on a single CAT-Net dataset and tests cross-dataset; CASIAv1 results are not discriminative of generalization according to the authors.
- Sparsity-rate schedule is chosen heuristically; single fixed sparsity rates favor different datasets.
Experimental setup
Datasets: CAT-Net dataset ImageNet-1k DEFACTO MS-COCO
Benchmarks: CASIAv1 NIST16 COVERAGE Columbia DEF-12k
Metrics: pixel-level F1 pixel-level AUC pixel-level IoU parameters FLOPs
Baselines: ManTraNet PSCC-Net MVSS CAT-Net TruFor SegFormer MLP head ASPP AFF
Builds on: Uniformer CAT-Net SegFormer
Closest papers by meaning
- Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization (NeurIPS 2026 · similarity 0.86)
- Training-Free Image Manipulation Localization Using Diffusion Models (AAAI 2025 · similarity 0.85)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.85)
- Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation (CVPR 2025 · similarity 0.85)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.84)
- Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.