Research map / Vision-language and multimodal / CLIP and zero-shot
FETA: Towards Specializing Foundational Models for Expert Task Applications
Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Peter W. J. Staar, Kate Saenko, Rogerio Feris and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP using MIL-NCE bags instead of single paired labels | CLIP on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 27.7% -> 32.6% | Improved |
| fine-tune CLIP using MIL-NCE bags instead of single paired labels | CLIP on IKEA · All-Data | Image-to-Text Rec@1: 22.9% -> 26.8% | Improved |
| use MIL-CLIP rather than out-of-the-box CLIP | CLIP on Car-Manuals · Zero-Shot | Text-to-Image Rec@1: 10.1% -> 11.7% | Improved |
| freeze the image encoder during MIL fine-tuning (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 32.6% -> 34.5% | Mixed |
| freeze the text encoder during MIL fine-tuning (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 32.6% -> 30.6% | Hurt |
| freeze both encoders except the text projection layer (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 32.6% -> 30.1% | Hurt |
| add LoRA adapters with rank 32 (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 32.6% -> 35.6% | Improved |
| vary LoRA rank from 4 to 32 (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 33.8% -> 35.6% | Improved |
| vary LoRA rank from 32 to 256 (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 35.6% -> 35.5% | No clear effect |
| use MIL-NCE rather than MIL-SoftMax or MIL-Max (ablation) | CLIP-MIL on Car-Manuals · Few-Shot | Image-to-Text Rec@10: 47.5% / 47.8% -> 48.9% | Improved |
| use MIL-NCE rather than MIL-SoftMax or MIL-Max (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Text-to-Image Rec@10: 71.3% -> 72.3% | Improved |
| initialize MIL fine-tuning from pretrained CLIP400M instead of training from scratch (ablation) | CLIP-MIL on Car-Manuals · Many-Shot | Image-to-Text Rec@1: 29.1% -> 32.6% | Improved |
Closest papers by meaning
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.84)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.83)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.83)
- HEMM: Holistic Evaluation of Multimodal Foundation Models (NeurIPS 2024 · similarity 0.83)
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.83)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
