Research map / Vision-language and multimodal / Multimodal LLMs

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMCoIR, a multimodal multilingual code retrieval benchmark across five visual domains, and CodeMMR, a Qwen2-VL-2B-based embedding model trained with instruction-based contrastive alignment that outperforms prior embedding models and improves code RAG.

Problem

Existing code retrieval models are text-centric and ignore visual artifacts (web UIs, charts, SVGs, diagrams, UML), so there is no benchmark or model for retrieving code and images jointly.

Approach

A unified instruction-conditioned embedding model that maps natural language, code, and images into one shared space, trained with InfoNCE on a new multimodal code retrieval benchmark, and used as a retriever for RAG-based image-to-code generation.

Key results

• CodeMMR (2B) reaches 65.4 average Hit@1 and 68.0 average nDCG@10 on MMCoIR, versus 53.3 Hit@1 for VLM2Vec-v2 (2B) and 49.5 for GME (7B).
• In RAG on ChartMimic Direct, CodeMMR improves Execution Rate by an average of +10.0 over the No-RAG baseline.
• On WebCode2M-Mid, CodeMMR raises Visual Accuracy by +9.4 (Qwen2VL-2B from 25.2 to 35.9).

Surprising findings

• Adding conventional multimodal retrieval training data (VLM2Vec mix) gives no benefit and sometimes slightly hurts on MMCoIR (CodeMMR-Mix).
• SVG retrieval remains very hard, with image-to-code nDCG@10 around 10-20 even for CodeMMR.

Limitations

• SVG and long-context code retrieval remain poor, limited by 256-512 token truncation.
• Sketch2Code and ChartEdit compositional edit instructions show limited gains.
• Relies on a fixed 256-token input budget that truncates long structured code.

Experimental setup

Datasets: MMCoIR WebSight Web2Code Chart2Code ChartGen SVGStack MMSVG DATIKZ_v3 PlantUML VLM2Vec training mix

Benchmarks: MMCoIR ChartMimic Direct WebCode2M-Mid Sketch2Code ChartEdit DiagramGenBenchmark

Metrics: Hit@1 nDCG@10 Execution Rate High-Level Score Visual Accuracy CLIP Similarity

Baselines: UniIR (CLIP_SF) UniIR (BLIP_SF) VLM2Vec (2B) VLM2Vec (7B) VLM2Vec-v2 (2B) LamRA (7B) GME (2B) GME (7B) UniIR-FT

Builds on: VLM2Vec Qwen2-VL UniIR CoIR

Closest papers by meaning

• Towards A Generalist Code Embedding Model Based On Massive Data Synthesis (NeurIPS 2025 · similarity 0.90)
• JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.89)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.88)
• CONSIDER: Commonalities and Specialties Driven Multilingual Code Retrieval Framework (AAAI 2024 · similarity 0.87)
• PinNet: Pinpoint Instructive Information for Retrieval Augmented Code-to-Text Generation (ICML 2024 · similarity 0.86)
• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.86)