Research map / Detection and segmentation / CNN features
Mesoscopic Insights: Orchestrating Multi-Scale & Hybrid Architecture for Image Manipulation Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mesorch, a parallel CNN-Transformer hybrid for image manipulation localization that fuses multi-scale predictions with learned per-pixel scale weights and prunes low-weight scales, yielding two baseline models.
Problem
Image manipulation localization has relied on low-level (microscopic) traces, yet most tampering alters object-level semantics; existing hybrids combine CNNs and Transformers sequentially and weight decoder scales equally, limiting performance and efficiency.
Approach
Runs a CNN (local, high-frequency DCT-enhanced input) and a Transformer (global, low-frequency DCT-enhanced input) in parallel, decodes multi-scale predictions, fuses them with a learned per-pixel adaptive scale-weighting module, and prunes scales whose mean weight falls below a threshold.
Key results
- Mesorch achieves average F1 of 0.6771 across four datasets versus 0.6269 for TruFor, and permute-F1 average 0.7259.
- Mesorch-P (pruned) reaches 0.6762 average F1 with 62.235M parameters and 64.821G FLOPs, versus 68.697M and 236.544G for TruFor.
- Under GaussianBlur perturbation Mesorch-P gets the best average F1 of 0.6032 on CASIAv1 robustness tests.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Mesorch hybrid architecture | Image manipulation localization · Four-dataset average | F1: 0.6269 -> 0.6771 | Helped |
| Add scale pruning to Mesorch (ablation) | Mesorch · Four-dataset average | F1: 0.6771 -> 0.6762 | No clear effect |
| Add scale pruning to Mesorch (ablation) | Mesorch · 512x512, batch size 1 | FLOPs (G): 124.928 -> 64.821 | Helped |
| Add multi-scale processing (ablation) | CNN (ConvNeXt) · Four-dataset average | F1: 0.5610 -> 0.6031 | Helped |
| Add multi-scale processing (ablation) | Transformer (Segformer) · Four-dataset average | F1: 0.5723 -> 0.6096 | Helped |
| Add multi-scale processing (ablation) | CNN+Transformer hybrid · Four-dataset average | F1: 0.6422 -> 0.6501 | Helped |
| Add DCT feature extraction (ablation) | CNN+Transformer hybrid with multi-scale · Four-dataset average | F1: 0.6501 -> 0.6612 | Helped |
| Add adaptive weighting (ablation) | CNN+Transformer hybrid with multi-scale · Four-dataset average | F1: 0.6501 -> 0.6653 | Helped |
| Replace Segformer with MAE (ablation) | Mesorch with ConvNeXt · Four-dataset average | F1: 0.6771 -> 0.6215 | Hurt |
| Replace Segformer with PVT (ablation) | Mesorch with ConvNeXt · Four-dataset average | F1: 0.6771 -> 0.6704 | Hurt |
| Replace Segformer with Swin (ablation) | Mesorch with ConvNeXt · Four-dataset average | F1: 0.6771 -> 0.6089 | Hurt |
| Replace ConvNeXt with ResNet (ablation) | Mesorch with Segformer · Four-dataset average | F1: 0.6771 -> 0.6615 | Hurt |
Surprising findings
- Sequential or single-architecture models do not necessarily outperform, and the paper argues one branch can dominate in sequential designs.
- Pruning low-weight scales reduces FLOPs substantially with only a minor drop in average F1 (0.6771 to 0.6762).
Limitations
- Pruning threshold epsilon is predefined rather than learned, and the paper gives no sensitivity analysis.
- Gains over CAT-Net and TruFor are modest on some datasets (e.g., Columbia, where Mesorch-P is best in F1 but Mesorch is below CAT-Net on F1 in ablation scale).
Experimental setup
Datasets: CAT-Net protocol dataset
Benchmarks: CASIAv1 Coverage NIST16 Columbia
Metrics: F1 permute F1 AUC IOU FLOPs parameters
Baselines: MVSS-Net PSCC-Net CAT-Net TruFor ObjectFormer
Builds on: TruFor ObjectFormer MVSS-Net SegFormer
Closest papers by meaning
- Training-Free Image Manipulation Localization Using Diffusion Models (AAAI 2025 · similarity 0.87)
- Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization (NeurIPS 2026 · similarity 0.86)
- Towards Modern Image Manipulation Localization: A Large-Scale Dataset and Novel Methods (CVPR 2024 · similarity 0.84)
- GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization (AAAI 2025 · similarity 0.84)
- UnionFormer: Unified-Learning Transformer with Multi-View Representation for Image Manipulation Detection and Localization (CVPR 2024 · similarity 0.83)
- DiffForensics: Leveraging Diffusion Prior to Image Forgery Detection and Localization (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.