Research map / Label-efficient and robust learning / Self-supervised and contrastive
MULTIMODALITY AS SUPERVISION: SELF-SUPERVISED SPECIALIZATION TO THE TEST ENVIRONMENT VIA MULTIMODALITY
Kunal Pratap Singh, Ali Garjani, Rishubh Singh, Muhammad Uzair Khattak, Jason Toskov, Efe Tarhan, Andrei Atanov, Oğuzhan Fatih Kar, Amir Zamir
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on test-space multimodal data with TST-MM instead of 4M-21 generalist | 4M-21 · ProcTHOR semantic segmentation | mIoU: 53.24 -> 60.85 | Improved |
| fine-tune 4M-21 on test-space data (adapted) | TST-MM · Scannet++ object detection | mAP: 31.54 -> 35.83 | Improved |
| adapt pretrained 4M-21 to test space | 4M-21 · Scannet++ semantic segmentation | mIoU: 27.59 -> 36.44 | Improved |
| add semantic segmentation modality (ablation) | TST-MM · ProcTHOR semantic segmentation | mIoU: 59.43 -> 60.85 | Improved |
| add object detection modality (ablation) | TST-MM · ProcTHOR semantic segmentation | mIoU: 59.38 -> 60.85 | Improved |
| add semantic segmentation, detection and SAM edge modalities (ablation) | TST-MM · ProcTHOR semantic segmentation | mIoU: 55.39 -> 60.85 | Improved |
| add semantic segmentation, detection and SAM edge modalities (ablation) | TST-MM · ProcTHOR object detection | mAP: 45.97 -> 49.38 | Improved |
| add object detection modality (ablation) | TST-MM · ProcTHOR object detection | mAP: 49.34 -> 49.38 | No clear effect |
| add SAM edges modality to RGB alone (ablation) | TST-MM · Scannet++ semantic segmentation | mIoU: +7.8% | Improved |
| add SAM edges modality to the seven others (ablation) | TST-MM · Scannet++ semantic segmentation | mIoU: 1.5% | Improved |
| use untrained pseudolabel networks as modalities (ablation) | TST-MM · test-space segmentation | mIoU | Hurt |
| mix transfer-set RGB frames into test-space pre-training (ablation) | TST-MM (ViT-S) · test-space segmentation | mIoU | Improved |
Closest papers by meaning
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.87)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
- On the Out-Of-Distribution Generalization of Large Multimodal Models (CVPR 2025 · similarity 0.86)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.86)
- ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (ICML 2026 · similarity 0.86)
- Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
