Research map / 3D vision / Depth and stereo
Multi-task Learning with 3D-Aware Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a 3D-aware regularizer for multi-task dense prediction that projects shared encoder features into a tri-plane 3D representation, renders task predictions via differentiable volume rendering, and discards the regularizer at inference.
Problem
Cross-task correlations learned in unstructured high-dimensional shared feature spaces of multi-task dense prediction networks are noisy and prone to overfitting, hurting performance.
Approach
Regularize shared encoder features by mapping them to a tri-plane 3D neural representation, decoding per-task neural fields, and rendering predictions via differentiable volume rendering, so all tasks share a 3D-consistent structure; the regularizer is removed at inference and can be plugged into existing MTL backbones.
Key results
- On NYUv2 with InvPT (ViT-L) backbone, the 3D-aware regularizer improves segmentation mIoU from 53.56 to 54.87 and depth RMSE from 0.5183 to 0.5006.
- On PASCAL-Context with MTI-Net, the regularizer improves segmentation mIoU from 64.42 to 66.71 (+2.29).
- Adding the regularizer adds no inference cost since it is discarded at test time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the 3D-aware regularizer to MTI-Net | MTI-Net (HRNet-48) · NYUv2 | mIoU: 45.97 -> 46.67 | Helped |
| add the 3D-aware regularizer to InvPT | InvPT (ViT-L) · NYUv2 | mIoU: 53.56 -> 54.87 | Helped |
| add the 3D-aware regularizer to MTI-Net* | MTI-Net* (HRNet-48) · PASCAL-Context | mIoU: 64.42 -> 66.71 | Helped |
| add the 3D-aware regularizer to InvPT | InvPT (ViT-L) · PASCAL-Context | mIoU: 79.03 -> 79.53 | Helped |
| add cross-view consistency loss (ablation) | InvPT + 3D-aware regularizer · NYUv2 + video frames | RMSE: 0.4879 -> 0.4850 | Helped |
| replace the 3D-aware regularizer with auxiliary task heads (ablation) | InvPT · NYUv2 | mIoU: 53.56 -> 52.45 | Hurt |
| use regularizer predictions instead of task-specific heads (ablation) | InvPT + 3D-aware regularizer · NYUv2 | mIoU: 54.87 -> 51.79 | Hurt |
| average task-head and regularizer predictions (ablation) | InvPT + 3D-aware regularizer · NYUv2 | odsF: 78.30 -> 78.50 | Helped |
| render depth only instead of all tasks (ablation) | InvPT + 3D-aware regularizer · NYUv2 | mIoU: 54.87 -> 54.62 | No clear effect |
| render segmentation only instead of all tasks (ablation) | InvPT + 3D-aware regularizer · NYUv2 | mIoU: 54.87 -> 55.04 | Helped |
| render all tasks instead of one isolated task (ablation) | InvPT + 3D-aware regularizer · NYUv2 | RMSE: 0.5041 -> 0.5006 | Helped |
| add the 3D-aware regularizer | InvPT (ViT-L), 50% training data · NYUv2, 50% training data | mIoU: 49.24 -> 49.30 | Helped |
Surprising findings
- Adding auxiliary heads to InvPT lowers segmentation mIoU (53.56 to 52.45), indicating gains are not from extra heads alone.
- The regularizer's own rendered predictions are worse than the task-specific heads (e.g., Seg mIoU 51.79 vs 54.87), attributed to small rendering resolution.
Limitations
- Representing higher-resolution 3D volumes is expensive in memory and compute.
- Loss balancing uses fixed cross-validated hyperparameters rather than adaptive strategies.
- Regularizer-alone predictions are worse than task-specific heads due to small rendering resolution.
Experimental setup
Datasets: NYUv2 NYUv2 video frames PASCAL-Context
Benchmarks: NYUv2 PASCAL-Context
Metrics: mIoU RMSE mErr odsF maxF
Baselines: Cross-Stitch PAP PSD PAD-Net ATRC MTI-Net InvPT ASTMT
Builds on: MTI-Net InvPT EG3D tri-plane NeRF PixelNeRF
Closest papers by meaning
- 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding (CVPR 2026 · similarity 0.89)
- Joint-Task Regularization for Partially Labeled Multi-Task Learning (CVPR 2024 · similarity 0.86)
- A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.84)
- RRT-MVS: Recurrent Regularization Transformer for Multi-View Stereo (AAAI 2025 · similarity 0.84)
- DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions (CVPR 2024 · similarity 0.83)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.