Research map / Vision-language and multimodal / Human and embodied modeling
Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Synergy Embedding Models (SEM), a Transformer-based multi-task dense prediction network with an intra-task hierarchy-adaptive deformable-attention module and an inter-task EM-based interaction module, reporting state-of-the-art results on NYUD-v2 and PASCAL-Context.
Problem
Multi-task dense scene understanding struggles to learn task-independent (intra-task) specialization across encoder stages and task-consensual (inter-task) dependencies efficiently, while all-pair Transformer attention across tasks is costly and noisy.
Approach
Attends over deformable points sampled from multiple encoder stages with learned per-point stage weights (hierarchy-adaptive keys/values) for each task, and models cross-task interaction through a compact set of shared bases learned by an EM / GMM-style iteration with EMA updates, replacing all-pair task attention.
Key results
- On NYUD-v2, SEM reaches 56.82 mIoU semantic segmentation and 0.4937 depth RMSE, beating TaskPrompter (55.30 mIoU, 0.5152 RMSE) with the same ViT-L encoder.
- On PASCAL-Context, SEM improves over TaskPrompter on semseg (81.66 vs 80.89 mIoU), parsing (69.90 vs 68.89 mIoU), saliency (84.95 vs 84.83 maxF) and boundary (73.80 vs 73.50 odsF), and lowers normal mErr (13.39 vs 13.72).
- Ablation on NYUD-v2 shows the intra-task module alone gives delta-m 1.95 and the inter-task module alone 1.75, with both combined 3.06 over the single-task baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the intra-task hierarchy-adaptive module to the baseline (ablation) | SEM baseline with ViT-L · NYUD-v2 | mIoU: 53.40 -> 56.01 | Helped |
| Add the inter-task EM-interactive module to the baseline (ablation) | SEM baseline with ViT-L · NYUD-v2 | mIoU: 53.40 -> 55.83 | Helped |
| Add both the intra-task and inter-task modules to the baseline (ablation) | SEM baseline with ViT-L · NYUD-v2 | mIoU: 53.40 -> 56.82 | Helped |
| Increase the number of encoder stages used by the intra-task module (ablation) | SEM · NYUD-v2 | accuracy | Helped |
| Add the inter-task module across more decoder stages (ablation) | SEM · NYUD-v2 | accuracy | Helped |
| Increase the EM iteration count to R = 5 (ablation) | SEM · NYUD-v2 | accuracy | Mixed |
Limitations
- Evaluated only on two indoor/natural-scene benchmarks with a small set of tasks.
- Gains over strongest baseline are modest on some PASCAL-Context tasks (e.g., saliency +0.12 maxF).
Experimental setup
Datasets: NYUD-v2 PASCAL-Context
Benchmarks: NYUD-v2 PASCAL-Context
Metrics: mIoU RMSE mErr odsF maxF delta-m
Baselines: Cross-Stitch PAD-Net MTI-Net ATRC MQTransformer DeMT InvPT TaskPrompter
Builds on: InvPT TaskPrompter EMANet Deformable attention DeMT
Closest papers by meaning
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.87)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.87)
- Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.86)
- LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · similarity 0.85)
- Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships (CVPR 2024 · similarity 0.85)
- Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.