Research map / Vision-language and multimodal / Multimodal LLMs

SO-Bench: A Structural Output Evaluation of Multimodal LLM

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SO-Bench, a benchmark of ~1.8K image-schema-instruction samples for evaluating MLLMs' ability to produce schema-compliant structured JSON from visual inputs, and shows SFT and RLVR improve a 3B model.

Problem

MLLMs are deployed in agentic settings needing outputs that conform to customized JSON schemas grounded in visual inputs, but no benchmark systematically measures schema-grounded visual information extraction.

Approach

A multi-stage, human-in-the-loop auto-labeling pipeline (CLIP-based image-schema association, multi-image schema generation, persona-based user intents, critic-refined responses) builds diverse visual structured-output data, scored with schema validation, field matching and full structure matching using exact/fuzzy/ignore labels.

Key results

• Gemini-2.5-Pro achieves 97.74% schema validation and 18.91% full match (fuzzy), the best among evaluated models; no model exceeds 20% full match fuzzy.
• Small models show large gaps: Qwen3-VL (2B) reaches 16.28% schema validation, and Intern3.5-VL (4B) drops about 40% in schema validation at depth above six.
• On the 3B internal model, SFT on 50K examples raises schema validation from 58.7 to 85.8 and fuzzy field match from 45.6 to 56.5; RLVR alone gives 72.0 schema validation.

Surprising findings

• Structured output APIs raise schema validity for GPT-4o models but lower fuzzy field match accuracy relative to instruction-following prompts.
• RLVR on the 3B baseline gives small field-match gains and far less than SFT on the same 14K data, which the authors attribute to the baseline's capability ceiling.

Limitations

• Benchmark labels are generated by frontier proprietary models, so label errors may cap performance.
• Metrics capture recall and may not credit semantically correct but differently phrased values; multiple valid ground truths are not represented.
• FMA and FSMA reward recall and do not penalize extra predicted fields.

Experimental setup

Datasets: HierText AriaUI COYO JsonSchemaBench RICO WebUI ScreenSpot Pro OmniDocBench DocVQA InfographicVQA ChartQA-Pro ChartMuseum

Benchmarks: SO-Bench BFCL LiveBench MMMU MIABench IFEval OCRBenchV2 CC-OCR DocVQA RefCOCO MathVista

Metrics: schema validation accuracy field matching accuracy (exact/fuzzy) full structure matching accuracy (exact/fuzzy) Pearson correlation

Baselines: Qwen3-VL Qwen2.5-VL Gemma-3 InternVL3.5 Phi-4-Vision Pixtral Llama-4-Scout Claude-4.5-Haiku Claude-4.5-Sonnet GPT-4o GPT-4o-mini GPT-5

Builds on: BFCL JSONSchemaBench MDPO RLOO

Closest papers by meaning

• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.90)
• HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.89)