Research map / Vision-language and multimodal / Cross-modal learning
Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train captioner with CIM reward (GRC + QIR) via GRPO | Qwen2.5-VL-7B · COCO-LN500 | Relations QA: 23.76 -> 44.15 | Improved |
| train captioner with CIM reward | Qwen2.5-VL-7B · COCO-LN500 | Attributes F1: 46.25 -> 58.51 | Improved |
| train captioner with CIM reward | Qwen2.5-VL-7B · COCO-LN500 | Objects precision: 82.35 -> 80.31 | Hurt |
| train captioner with CIM reward | LLaVA1.5-7B · COCO-LN500 | CAPTURE: 44.75 -> 48.62 | Improved |
| add QIR-only reward (vs base) (ablation) | Qwen2.5-VL-7B · COCO-LN500 | CAPTURE/Objects F1/Attributes F1/Relations QA: +4.0, +11.0, +10.8, +17.4 | Improved |
| add GRC-only reward (vs base) (ablation) | Qwen2.5-VL-7B · COCO-LN500 | CAPTURE/Objects F1/Attributes F1/Relations QA: +2.9, +3.6, +5.1, +0.9 | Improved |
| combine QIR and GRC rewards (ablation) | CIM · COCO-LN500 | CAPTURE and sub-metrics | Improved |
| CIM on top of SFT model vs SC-Captioner | Qwen2-VL-7B SFT · COCO-LN500 | CAPTURE: 47.51 -> 48.61 | Improved |
| CIM on top of SFT model vs SC-Captioner | LLaVA1.5-7B SFT · COCO-LN500 and DOCCI500 | CAPTURE: +0.4 avg | No clear effect |
| scale retrieval corpus (RefinedCaps -> +DenseFusion-1M) (ablation) | CIM · COCO-LN500 | Objects F1 / Attributes F1 / Relations QA: +1.1/+0.6/+1.3 then +0.4/+0.3/+1.7 | Improved |
| swap retrieval image/text encoders (ablation) | CIM · COCO-LN500 | CAPTURE: 48.17 - 48.99 | No clear effect |
| CIM reward vs CLIP-RL and Self Retrieval baselines | Qwen2.5-VL-7B · COCO-LN500 | CAPTURE: 46.71 -> 48.93 | Improved |
Closest papers by meaning
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models (NeurIPS 2025 · similarity 0.87)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
