Research map / Vision-language and multimodal / Human and embodied modeling
Argus: A Compact and Versatile Foundation Model for Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Argus, a ~100M-parameter vision foundation model that pretrains a frozen DINOv2 ViT-B with a lightweight adapter on 5 core tasks, then adapts to 12 vision tasks by training only task-specific decoders.
Problem
Existing vision and multimodal foundation models are data- and compute-hungry to train and adapt, and perform inconsistently across vision tasks, while lacking support for many perception tasks such as depth and pose estimation.
Approach
A two-stage recipe: multitask pretraining with a frozen DINOv2 ViT plus a trainable ViT-Adapter variant (with GN instead of BN, and token-norm scaling), followed by extensible adaptation where new task decoders are trained on a frozen backbone.
Key results
- Argus (100M params) reaches 56.5% mIoU on ADE20K semantic segmentation, above Florence-2-B (54.9%, 232M params).
- Argus achieves top results on 10 of 12 tasks, with 58.6 COCO APb and 86.3% ImageNet Top-1 accuracy.
- Argus-Large (327M) improves over Argus-Base across tasks, e.g., COCO APb 60.2 vs 58.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Argus instead of DINOv2 (ViT-B) with matching decoders | Argus · ADE20K semantic segmentation | mIoU: 52.5 -> 56.5 | Helped |
| Use Argus instead of DINOv2 (ViT-B) with matching decoders | Argus · COCO object detection | APb: 55.1 -> 58.6; Florence 62.4 | Mixed |
| Use Argus instead of DINOv2 (ViT-B) with matching decoders | Argus · COCO instance segmentation | APm: 47.8 -> 51.8 | Helped |
| Use Argus instead of DINOv2 (ViT-B) with matching decoders | Argus · COCO pose estimation | APk: 57.2 -> 77.0 | Helped |
| Freeze the pretrained ViT during multitask pretraining (ablation) | Argus multitask pretraining · COCO object detection | APb: 55.3 -> 57.7 | Helped |
| Replace BN with GN in the adapter (ablation) | Argus multitask pretraining · ADE20K semantic segmentation | mIoU: 55.0 -> 55.6 | Helped |
| Scale ViT features before interaction (ablation) | Argus multitask pretraining · COCO pose estimation | APk: 74.7 -> 75.2 | Helped |
| Add EMA to the row 4 training recipe (ablation) | Argus multitask pretraining · ADE20K semantic segmentation | mIoU: 55.6 -> 56.9 | Helped |
| Increase multitask pretraining from 200k to 300k iterations (ablation) | Argus multitask pretraining · COCO object detection | APb: 57.9 -> 58.6 | Helped |
| Use empirical task weighting instead of FAMO or GradNorm (ablation) | Argus multitask pretraining · Five core tasks | task performance: FAMO 57.4; GradNorm 57.2; empirical 57.7 | Mixed |
| Remove object detection and instance segmentation from core-task pretraining (ablation) | Argus · COCO object detection | APb: 57.7 -> 54.2 | Hurt |
| Remove pose estimation from core-task pretraining (ablation) | Argus · COCO pose estimation | APk: 75.2 -> 74.4 | Hurt |
Surprising findings
- Freezing the pretrained ViT was crucial: training the full model did not converge well in multitask pretraining.
- Replacing batch normalization with group normalization in the adapter improved performance on all tasks, as shared BN statistics conflict across tasks.
Limitations
- Overfitting when expanding core pretraining tasks to 8 or 11 due to limited data for added tasks.
- Naive empirical task weighting used; standard MTL algorithms (FAMO, GradNorm) did not outperform it, and systematic MTL study left for future work.
Experimental setup
Datasets: COCO ADE20K ImageNet NYUv2 PASCAL-Context MVTecAD
Benchmarks: COCO ADE20K NYUv2 ImageNet PASCAL-Context MVTecAD
Metrics: AP mIoU PQ Top-1 accuracy RMSE odsF mErr maxF I-AUROC AR
Baselines: Florence Florence-2 Uni-Perceiver v2 Unified-IO XL Unified-IO 2 4M MAE GLID DINOv2 (with same decoders) InterImage
Builds on: ViT-Adapter DINOv2 Mask DINO UperNet DPT
Closest papers by meaning
- Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought (CVPR 2025 · similarity 0.87)
- AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models (CVPR 2026 · similarity 0.87)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.87)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.86)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.86)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.