Research map / Architectures and efficiency / RNNs and normalization

Your Embedding Model Is SMARTer Than You Think

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Learned embeddings for retrieval · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMART, which turns single-vector multimodal embedders into multi-vector retrievers by adding MaxSim late interaction over final-layer hidden states to the pooled score, used training-free or with lightweight adapter/LoRA post-training.

Problem

Single-vector multimodal retrievers compress token sequences into one pooled embedding, losing fine-grained local evidence; existing multi-vector retrievers require costly full training.

Approach

Contrastive training on the pooled eot token shapes preceding non-pooling hidden states via gradient flow, so their cosine geometry supports MaxSim late interaction; SMART adds this late-interaction score to the original single-vector score (hybrid scoring), either at inference only or as a training objective.

Key results

• Training-free SMART improves Qwen3-VL-Embedding-8B's MMEB-V2 average from 78.83 to 79.34 and 2B from 74.87 to 75.77.
• Qwen3-VL-Embedding-2B with a trained SMART adapter (about 1h50m on 8 A6000s) reaches 81.25 visdoc average, 0.34 above jina-embeddings-v4 (80.91).
• LamRA-Single-Convert (9.5 h) reaches 77.68 visdoc average vs. 78.31 for LamRA-Multi trained from scratch (12 h), saving about 20% training time.

Surprising findings

• On the adversarial toy benchmark, the hybrid score (42.6%) underperforms late interaction alone (56.8%) and falls below chance, since the pooled score is misleading there.
• Late interaction using hidden states from Layer 20 with the Layer 28 pooled anchor performs about as well as Layer 28 (80.16 vs 80.10).

Limitations

• Gains are not observed for global tasks like classification and VQA; SMART is not beneficial there.
• LamRA-Ret models were trained only on the visdoc subset due to limited compute.
• Video retrieval gains are limited for frame-based models like GME, and video moment retrieval is excluded.

Experimental setup

Datasets: ColPali training set MMEB-V2 Controlled local-evidence toy benchmark (generated)

Benchmarks: MMEB-V2 ViDoRe VisRAG Jina-VDR

Metrics: Recall@1 NDCG@5 pairwise accuracy

Baselines: VLM2Vec-V2.0 GME-2B GME-7B Qwen3-VL-Embedding-2B Qwen3-VL-Embedding-8B ColPali jina-embeddings-v4 LamRA-Single LamRA-Multi

Builds on: ColBERT LamRA InfoNCE LoRA

Closest papers by meaning

• MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.92)
• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.89)
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
• RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.89)
• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.89)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.88)