AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

FETA: Towards Specializing Foundational Models for Expert Task Applications

Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Peter W. J. Staar, Kate Saenko, Rogerio Feris and 1 more

NeurIPS 2022 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune CLIP using MIL-NCE bags instead of single paired labelsCLIP on Car-Manuals · Many-ShotImage-to-Text Rec@1: 27.7% -> 32.6%Improved
fine-tune CLIP using MIL-NCE bags instead of single paired labelsCLIP on IKEA · All-DataImage-to-Text Rec@1: 22.9% -> 26.8%Improved
use MIL-CLIP rather than out-of-the-box CLIPCLIP on Car-Manuals · Zero-ShotText-to-Image Rec@1: 10.1% -> 11.7%Improved
freeze the image encoder during MIL fine-tuning (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 32.6% -> 34.5%Mixed
freeze the text encoder during MIL fine-tuning (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 32.6% -> 30.6%Hurt
freeze both encoders except the text projection layer (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 32.6% -> 30.1%Hurt
add LoRA adapters with rank 32 (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 32.6% -> 35.6%Improved
vary LoRA rank from 4 to 32 (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 33.8% -> 35.6%Improved
vary LoRA rank from 32 to 256 (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 35.6% -> 35.5%No clear effect
use MIL-NCE rather than MIL-SoftMax or MIL-Max (ablation)CLIP-MIL on Car-Manuals · Few-ShotImage-to-Text Rec@10: 47.5% / 47.8% -> 48.9%Improved
use MIL-NCE rather than MIL-SoftMax or MIL-Max (ablation)CLIP-MIL on Car-Manuals · Many-ShotText-to-Image Rec@10: 71.3% -> 72.3%Improved
initialize MIL fine-tuning from pretrained CLIP400M instead of training from scratch (ablation)CLIP-MIL on Car-Manuals · Many-ShotImage-to-Text Rec@1: 29.1% -> 32.6%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy