Research map / Vision-language and multimodal / CLIP and zero-shot
TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransAgent, which distills knowledge from 11 frozen heterogeneous expert agents (vision, language, multi-modal) into CLIP via learnable prompts and mixture-of-agents gating, leaving no extra cost at inference.
Problem
Single CLIP-like vision-language models generalize poorly to downstream domains with large shifts (e.g., EuroSAT), especially in low-shot settings, while many expert vision/language/multi-modal models with heterogeneous structures remain unused.
Approach
Extract homogeneous features (intermediate vision features, LLM text features of class descriptions, and score vectors from T2I/I2T multi-modal agents) from frozen experts, fuse them with a mixture-of-agents gate, and distill them into learnable CLIP prompts via feature/KL losses; the agents are discarded at inference so the deployed model is the same as the original CLIP.
Key results
- On base-to-novel generalization averaged over 11 datasets, TransAgent reaches 81.27 HM (85.29 base, 77.62 novel), above PromptSRC at 79.97 HM.
- On EuroSAT, TransAgent reaches 89.89 HM versus 82.32 for PromptSRC; the abstract reports ~20% gains over CoOp in low-shot EuroSAT.
- Inference time on ImageNet is 2min35s versus 16min14s for CaFo, with accuracy 73.50 vs 72.46 for CaFo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| collaborate heterogeneous agents using TransAgent rather than prompt-learning baseline | CLIP ViT-B/16 · Base-to-novel average | HM: 79.97 -> 81.27 | Helped |
| collaborate heterogeneous agents using TransAgent rather than prompt-learning baseline | CLIP ViT-B/16 · EuroSAT base-to-novel | HM: 82.32 -> 89.89 | Helped |
| add layer-wise rather than last-layer visual feature distillation (ablation) | VAC · Base-to-novel average | HM: 79.97 -> 81.27 | Helped |
| replace average fusion with MoA gating (ablation) | VAC · Base-to-novel average | HM: 79.22 -> 81.27 | Helped |
| replace additive per-agent distillation with MoA gating (ablation) | VAC · Base-to-novel average | HM: 79.52 -> 81.27 | Helped |
| use [EOS] rather than [SOS] token for language feature distillation (ablation) | LAC · Base-to-novel average | HM: 79.47 -> 81.27 | Helped |
| replace average fusion with MoA gating (ablation) | LAC · Base-to-novel average | HM: 79.89 -> 81.27 | Helped |
| replace additive per-agent distillation with MoA gating (ablation) | LAC · Base-to-novel average | HM: 78.48 -> 81.27 | Helped |
| align learned scores instead of prompted logits with multi-modal agent scores (ablation) | MAC · Base-to-novel average | HM: 79.54 -> 81.27 | Helped |
| replace average fusion with MoA gating (ablation) | MAC · Base-to-novel average | HM: 79.41 -> 81.27 | Helped |
| replace additive per-agent distillation with MoA gating (ablation) | MAC · Base-to-novel average | HM: 79.61 -> 81.27 | Helped |
| add visual agent collaboration (ablation) | IVLP baseline · Base-to-novel average | HM: +1.53 | Helped |
Surprising findings
- The DINO vision agent lowered EuroSAT accuracy by 5.60% when used alone as a teacher, though it was kept in the agent set.
- L1 and MSE losses for the multi-modal score distillation underperformed KL by a large margin, since channel-wise losses do not suit probability score vectors.
Limitations
- Distilling external knowledge may harm CLIP's original representations by introducing irrelevant information from diversified agents.
- Large-scale pre-trained agents are data-hungry, so few-shot regimes may not fully exploit them.
- Training requires at least 48GB memory for extracting language and multi-modal agent knowledge, and the method depends on labeled domain data.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe RPO PromptSRC CaFo
Builds on: CoOp IVLP MaPLe PromptSRC CaFo
Closest papers by meaning
- Boosting Vision-Language Models with Transduction (NeurIPS 2024 · similarity 0.87)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.87)
- MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning (CVPR 2024 · similarity 0.86)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.85)
- Iterated Learning Improves Compositionality in Large Vision-Language Models (CVPR 2024 · similarity 0.85)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.