Research map / Architectures and efficiency / Attention and transformers

Memory Efficient Matting with Adaptive Token Routing

AAAI 2025 · Architectures and efficiency › Attention and transformers · approach: Efficient image transformers · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MEMatte, a memory-efficient ViT-based image matting framework that routes informative tokens to global attention and others to a lightweight refinement branch, plus the UHR-395 ultra high-resolution matting dataset.

Problem

Global self-attention in transformer-based image matting has quadratic memory and compute cost, making full-resolution high-resolution matting infeasible on memory-limited GPUs; fixed-ratio token pruning/merging degrades accuracy.

Approach

A local-global router before each global attention block predicts per-token routing probabilities; a Batch-constrained Adaptive Token Routing mechanism reselects branches for all tokens at every block with a batch-level compression constraint, while a Lightweight Token Refinement Module (depth-wise conv + efficient channel attention), distilled from the ViTMatte teacher, handles the rest of the to

Key results

• On Composition-1K, MEMatte-S reduces memory by 88.5% (6.20 to 0.71 GB) and latency by 54.3% versus ViTMatte-S, with SAD 21.90 vs 21.46.
• MEMatte-B reduces memory from 12.53 to 1.49 GB (88.1%) and latency by 47.4% versus ViTMatte-B.
• On PPT-344, MEMatte-S achieves SAD 117.05 vs 141.98 for the best ViTMatte-S patch-based setting, and MEMatte-B achieves SAD 114.95 vs 151.86 for ViTMatte-B patch-based.

Measured effects reported in the paper

Surprising findings

• Token-compression baselines (DynamicViT, EViT, ToMe) substantially degrade matting accuracy (e.g., EViT SAD 42.66 vs 21.46) despite lower memory.
• Using distillation or LTRM alone gives roughly no improvement over neither, while combining both gives a significant gain.

Limitations

• Performance on Composition-1K is slightly lower than ViTMatte (about 5% degradation in SAD for MEMatte-S/B)
• Maximum-token cap k and target compression rho are manually chosen hyperparameters
• Real-world generalization evaluated only on AIM500 with training solely on synthetic data

Experimental setup

Datasets: DIM Composition-1K Distinctions-646 COCO VOC2012 UHR-395 PPM-100 Transparent-460 AIM-500

Benchmarks: Composition-1K Distinctions-646 UHR-395 PPT-344 AIM-500

Metrics: SAD MSE Grad Conn memory (GB) latency (ms) GFLOPs

Baselines: ViTMatte DIM IndexNet GCA HDMatt MGMatting SIM FBA TransMatting MatteFormer RMat SparseMat

Builds on: ViTMatte DynamicViT EViT ToMe Skip-At ECA

Closest papers by meaning

• MatAnyone: Stable Video Matting with Consistent Memory Propagation (CVPR 2025 · similarity 0.88)
• Path-Adaptive Matting for Efficient Inference Under Various Computational Cost Constraints (AAAI 2025 · similarity 0.87)
• Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.87)
• ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.85)
• MaGGIe: Masked Guided Gradual Human Instance Matting (CVPR 2024 · similarity 0.85)
• MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation (ICLR 2025 · similarity 0.85)