Research map / Large language models / Instruction and fine-tuning
Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking
Xin Zhang, Ziqi Dai, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with SFT yes/no classification instead of contrastive InfoNCE (ablation) | MLLM reranker · MRB | average score: 56.40 -> 58.09 | Improved |
| use SFT weight instead of CL weight (direction fixed) (ablation) | MLLM reranker · MRB | average score: 56.99 -> 58.09 (D_SFT); 56.40 -> 57.88 (D_CL) | Improved |
| use SFT direction instead of CL direction (weight fixed) (ablation) | MLLM reranker · MRB | average score: 57.88 -> 58.09 (W_SFT); 56.40 -> 56.99 (W_CL) | Improved |
| use constant weight 1 instead of instance-specific weights (ablation) | SFT reranker · MRB | average score: 58.19 -> 49.47 | Hurt |
| add tau mask that stops learning on well-learned pairs (ablation) | constant-weight baseline · MRB | average score: 49.47 -> 56.57 | Improved |
| replace SFT direction with random-initialized learnable projection (ablation) | SFT reranker · MRB | average score: 58.09 -> 56.75 | Hurt |
| use random-initialized learnable projection as CL direction (ablation) | CL reranker · MRB | average score: 56.40 -> 57.72 | Improved |
| expand SFT target tokens from yes/no to top-16 tokens (ablation) | SFT reranker · MRB | average score | No clear effect |
| compute CL loss in FP32 instead of FP16 (ablation) | CL reranker · MRB | average score: 56.09 -> 56.40 | Improved |
| freeze LM head (ablation) | SFT reranker · MRB | average score | No clear effect |
| increase number of negatives per sample up to 16 (ablation) | SFT reranker · MRB | average score | Improved |
| rerank with GMR-7B instead of GME-2B retrieval only | GME-2B top-100 · MRB | average score: 52.54 -> 63.85 | Improved |
Closest papers by meaning
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.91)
- Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.88)
- SRA-CL: Semantic Retrieval Augmented Contrastive Learning for Sequential Recommendation (NeurIPS 2025 · similarity 0.88)
- Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.87)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.87)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
