Research map / Large language models / Instruction and fine-tuning
The Visual Prism: Refracting Images into Parallel Multilingual Descriptions with Structured Visual Guidance
Chengpeng Fu, Xiaocheng Feng, Yichong Huang, Wenshuai Huo, Baohang Li, Yang Xiang, Ting Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on PRISMS pseudo-parallel corpus instead of zero-shot | LLaMA3-8B · EN-FR Flores200 | BLEU: 31.19 -> 37.06 | Improved |
| fine-tune on PRISMS pseudo-parallel corpus instead of human parallel corpus | LLaMA3-8B · EN-FR average over 4 test sets | BLEU: 31.56 -> 30.28 | Hurt |
| fine-tune on PRISMS pseudo-parallel corpus instead of human parallel corpus | LLaMA3-8B · FR-EN Flores200 | BLEU: 41.32 -> 41.41 | No clear effect |
| use vanilla MLLM captions generated independently per language (ORI) as parallel corpus (ablation) | LLaMA3-8B · EN-FR Flores200 | BLEU: 31.19 -> 22.97 | Hurt |
| add structured visual guidance prompt to a vanilla MLLM without adherence training (ablation) | caption generation · EN-FR Flores200 | BLEU: 22.97 -> 2.34 | Hurt |
| add QE filtering to SVG-only descriptions (ablation) | caption generation · EN-FR Flores200 | BLEU: 2.34 -> 24.24 | Improved |
| add SVG adherence training (pseudo-data SFT) on top of SVG prompting (ablation) | caption generation · EN-FR Flores200 | BLEU: 24.24 -> 37.06 | Improved |
| replace pseudo-data SFT with Iter-DPO for SVG adherence (ablation) | caption generation · EN-ZH Flores200 | BLEU: 41.80 -> 41.71 | No clear effect |
| replace pseudo-data SFT with Iter-DPO for SVG adherence (ablation) | caption generation · ZH-EN Flores200 | BLEU: 23.42 -> 23.90 | No clear effect |
| add post-processing filter that drops descriptions with inconsistent triplet counts (ablation) | PRISMS corpus · EN-FR Flores200 | BLEU: 35.01 -> 37.06 | Improved |
| use PRISMS pseudo-parallel data to initialize unsupervised MT | XLM UNMT · UNMT Flickr2017 | BLEU: 38.57 -> 41.58 | Improved |
| fine-tune on PRISMS corpus | LLaMA3-8B · FR-EN Flores200 | BLEU: 37.39 -> 41.41 | Improved |
Closest papers by meaning
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.90)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.89)
- Parrot: Multilingual Visual Instruction Tuning (ICML 2025 · similarity 0.88)
- MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost (ICML 2025 · similarity 0.88)
- MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation (CVPR 2026 · similarity 0.88)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
