Research map / Detection and segmentation / CNN features
EFormer: Enhanced Transformer towards Semantic-Contour Features of Foreground for Portraits Matting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EFormer, a transformer decoder with cross-attention between high- and low-resolution features plus semantic/contour branches, for trimap-free portrait matting, outperforming prior methods on VideoMatte240K and AIM.
Problem
Self-attention in transformers captures low-frequency semantics but neglects high-frequency contour details, biasing portrait matte boundaries.
Approach
Uses cross-attention between high-resolution (1/8) and low-resolution (1/16) features to locate and emphasize high-frequency contour regions, cascaded with self-attention in a Semantic and Contour Detector, followed by separate MLP contour-edge and semantic extraction branches whose outputs are fused for the matte head.
Key results
- On VideoMatte240K-JPEG-SD at 512x288, EFormer reaches MAD 2.3097 vs 4.1858 for BGMv2 and 5.99 for RVM.
- On AIM at 512x512, EFormer reaches MAD 7.47 vs 14.84 for RVM and 21.66 for MODNet.
- Ablation: using 1/4 scale features as high-resolution input and 1/16 as low-resolution gives best MAD 3.9730 at 224x224 vs 4.1357 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EFormer instead of RVM | Portrait matting · VideoMatte240K-JPEG-SD test (512×288) | MSE: 1.1700 -> 0.6637 | Helped |
| Use EFormer instead of RVM | Portrait matting · AIM test | MAD: 14.84 -> 7.47 | Helped |
| Use EFormer instead of RVM | Portrait matting · VideoMatte240K-JPEG-SD test (224×224) | MAD: 6.1357 -> 4.1357 | Helped |
| Train for 25 rather than 20 epochs (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.1357 -> 4.1811 | Hurt |
| Train for 5 rather than 20 epochs (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.1357 -> 4.3953 | Hurt |
| Add self-attention after cross-attention (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.2762 -> 4.1357 | Helped |
| Add cross-attention before self-attention (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.1710 -> 4.1357 | Helped |
| Use Fenc 1/4 for FHR and Fenc 1/8 for FLR instead of Fenc 1/8 and Fenc 1/16 (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.1357 -> 3.9970 | Helped |
| Use Fenc 1/4 for FHR and Fenc 1/16 for FLR instead of Fenc 1/8 and Fenc 1/16 (ablation) | EFormer · VideoMatte240K-JPEG-SD test (224×224) | MAD: 4.1357 -> 3.9730 | Helped |
Surprising findings
- Removing the cross-attention layer only slightly degrades metrics, while removing self-attention causes a larger increase in error, suggesting semantic information dominates the matting metrics.
Limitations
- Reliance on composited training data rather than real-world portrait images with ground-truth mattes.
- Evaluation limited to VideoMatte240K and AIM; no comparison on additional benchmarks.
- Training uses a single RTX 3090 and only 25 epochs for ablations, limiting scale analysis.
Experimental setup
Datasets: VideoMatte240K (JPEG SD Format) Adobe Image Matting (AIM) BG10K
Benchmarks: VideoMatte240K-JPEG-SD test set AIM test set
Metrics: MAD MSE Grad Conn
Baselines: DeepLabV3 MODNet BGMv2 RVM
Builds on: DETR Inception Transformer (IFormer) ResNet BGMv2
Closest papers by meaning
- Attention-Guided Hierarchical Structure Aggregation for Image Matting (CVPR 2020 · similarity 0.87)
- Semantic Image Matting (CVPR 2021 · similarity 0.86)
- Unifying Automatic and Interactive Matting with Pretrained ViTs (CVPR 2024 · similarity 0.85)
- In-Context Matting (CVPR 2024 · similarity 0.84)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.84)
- DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.