Research map / Vision-language and multimodal / Multimodal LLMs

F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines Fine-grained Fragment Retrieval over long multimodal dialogues, builds the MLDR dataset and a WeChat real-domain test set, and trains F²RVLM (SFT then GRPO with F1, format, and order-consistency rewards plus difficulty-aware curriculum) to outperform popular VLMs.

Problem

Locating query-relevant, semantically coherent fragments (utterances and images) scattered across long multimodal dialogues; existing VLMs often retrieve incoherent utterance-image sets and existing dialogue retrieval only selects single elements from short contexts.

Approach

Formulate fragment retrieval as generative ID prediction by a VLM, trained with SFT followed by GRPO using multi-objective rewards (F1 with length penalty, format, and CLIP-based cross-modal fragment order consistency), plus a difficulty-aware curriculum ranking training samples by F1 and prediction entropy.

Key results

• F²RVLM-Qwen2-VL-7B reaches 87.25% joint F1 on the MLDR validation set, the best among compared models.
• On the real-domain WeChat test set, F²RVLM-Qwen2-VL-7B achieves 62.07% F1 and 55.46% MCC, versus 48.89% F1 for GPT-4o and 53.21% for Gemini-2.5-Flash.
• Qwen2.5-VL-7B improves from 12.58% zero-shot F1 to 51.71% F1 on WeChat after MLDR fine-tuning.

Surprising findings

• Zero-shot Qwen2.5-VL-7B scores only 12.58% F1 on WeChat, while the same model reaches 51.71% after MLDR fine-tuning.
• A 3B F²RVLM variant outperforms larger models such as MiMo-7B-RL and GPT-4o on the in-domain set.

Limitations

• WeChat test set is small (12 volunteers, 580 samples) and skewed toward Work & Tech topics.
• MLDR is synthetic, generated by Qwen3 from triplets of short dialogues, with a fixed ~25-turn structure.
• Long dialogues require sliding-window inference for some baselines due to context limits.

Experimental setup

Datasets: MLDR DialogCC

Benchmarks: MLDR validation set WeChat real-domain test set

Metrics: Precision Recall F1 MCC Fragment Order Consistency Query-Fragment Similarity

Baselines: CLIP BLIP-2 E5-V GME GPT-4o Gemini-2.5-Flash Claude-Sonnet-4 Doubao-Seed-1.6 Qwen2.5-VL-72B Qwen2-VL-7B

Builds on: GRPO DAPO LoRA DialogCC ms-swift

Closest papers by meaning

• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.90)
• Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.90)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.88)
• Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
• V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts (CVPR 2025 · similarity 0.88)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)