Research map / Vision-language and multimodal / Cross-modal learning
Interfacing Foundation Models' Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIND, a lightweight transformer interface that aligns frozen foundation model embeddings (X-Decoder vision, LLaMA language) into a shared interleaved space for segmentation, grounding, and retrieval, and introduces the FIND-Bench interleaved COCO benchmark.
Problem
Foundation models are specialized in output type (language, images, masks), and there is no benchmark or unified interface for interleaved vision-language understanding spanning pixel-level grounding/segmentation and image-level retrieval.
Approach
A lightweight transformer interface with learnable queries that attend to frozen vision and language embeddings via content and conditional attention, projected into semantic and pixel spaces, so that one set of weights handles multimodal and interleaved tasks; trained on COCO with GPT-4 and SEEM-generated interleaved annotations (FIND-Bench).
Key results
- On FIND-Bench interleave grounding, FIND (L) reaches 76.3 cIoU on COCO-Entity versus SEEM (B) 69.4, roughly +8 points over the SEEM interleaved baseline.
- FIND (L) reaches 56.7 PQ on COCO generic segmentation, close to SEEM (L) 57.5 and X-Decoder, and 50.8 mAP, above Mask2Former (L) 48.6 mAP.
- FIND (L) reaches 53.4 IR@5 and 66.7 IR@10 on interleave retrieval with COCO-Entity, above FROMAGe and BLIP-2 baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train FIND on interleave tasks | FIND (L) · COCO-Entity interleave retrieval | IR@5: 26.8 -> 53.4 | Helped |
| Jointly train FIND on interleave tasks | FIND (L) · COCO-Paragraph interleave grounding | cIoU: 67.7 -> 74.7 | Helped |
| Train FIND jointly on image-text retrieval (ablation) | FIND interface · COCO-Entity image-text retrieval | IR@1: 44.3 -> 50.8 | Helped |
| Train FIND jointly on grounding (ablation) | FIND interface · COCO-Entity grounding | cIoU: 40.9 -> 73.0 | Helped |
| Train FIND jointly on interactive segmentation (ablation) | FIND interface · COCO-Entity grounding | cIoU: 36.5 -> 40.9 | Helped |
| Train FIND jointly on interleave tasks (ablation) | FIND interface · COCO class segmentation | mIoU: 61.4 -> 61.0 | No clear effect |
| Use LLaMA layer -12 embeddings instead of layer -1 (ablation) | FIND with X-Decoder (T) + UniCL · COCO-Karpathy image-text retrieval | IR@1: 38.9 -> 40.4 | Helped |
| Use LLaMA layer -12 embeddings instead of layer -6 (ablation) | FIND with X-Decoder (T) + UniCL · COCO-Karpathy image-text retrieval | IR@1: 38.1 -> 40.4 | Helped |
| Use LLaMA layer -12 embeddings instead of layer -18 (ablation) | FIND with X-Decoder (T) + UniCL · COCO-Karpathy image-text retrieval | IR@1: 40.1 -> 40.4 | Helped |
| Use LLaMA layer -12 embeddings instead of layer -24 (ablation) | FIND with X-Decoder (T) + UniCL · COCO-Karpathy image-text retrieval | IR@1: 40.2 -> 40.4 | Helped |
| Use LLaMA layer -12 embeddings instead of layer -30 (ablation) | FIND with X-Decoder (T) + UniCL · COCO-Karpathy image-text retrieval | IR@1: 37.9 -> 40.4 | Helped |
| Use LLaMA instead of UniCL as language encoder (ablation) | X-Decoder (T) interface · COCO-Karpathy image-text retrieval | IR@1: 40.4 -> 40.2 | No clear effect |
Surprising findings
- Image-text retrieval results are sub-optimal relative to specialized baselines, which the authors attribute to fine-tuning batch size and resolution choices.
- Using intermediate LLaMA layer (-12) embeddings works best, while output and input layers perform worse on retrieval.
Limitations
- Trained and evaluated only on COCO, so may not generalize to in-the-wild settings.
- Image-text retrieval performance trails strong baselines.
- FIND-Bench annotations are GPT-4 and SEEM generated, which may carry noise.
Experimental setup
Datasets: COCO RefCOCO-g COCO-Karpathy Pascal VOC COCO-Entity COCO-Paragraph
Benchmarks: COCO RefCOCO-g COCO-Entity COCO-Paragraph Pascal VOC COCO-Karpathy FIND-Bench
Metrics: PQ mAP mIoU cIoU IR@1 TR@1 IR@5 IR@10 AP50
Baselines: Mask2Former Grounding-SAM SEEM X-Decoder ImageBind FROMAGe BLIP-2 SAM
Builds on: X-Decoder SEEM Mask2Former SAM
Closest papers by meaning
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.84)
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.84)
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models (NeurIPS 2025 · similarity 0.83)
- GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model (CVPR 2025 · similarity 0.83)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.83)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.