Research map / Large language models / Language, safety and interpretability

Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains Qwen2.5-VL-7B and Qwen2.5-Omni-7B on a 1K-sample text-only reasoning judge dataset (distilled from JudgeLRM-7B) to create zero-shot multimodal judges for image, video, audio, and molecule evaluation.

Problem

Multimodal LLM-as-a-Judge models typically need large modality-specific annotated data and generalize poorly across evaluation formats and modalities; commercial judge APIs are costly and unreliable.

Approach

Fine-tune multimodal LLMs only on a small curated set of text-only reasoning judgments; structured textual reasoning transfers zero-shot to image, video, and audio judgment tasks, with format-diverse post-processed outputs and inference-time scaling (majority voting, budget forcing).

Key results

• Flex-VL-7B reaches 48.60 overall on VL-RewardBench, above LLaVA-Critic-7B (43.7) and GPT-4o (65.8 is higher, so comparable only to open judges); Flex-Omni-7B attains 0.306 score-based correlation on MLLM-as-a-Judge, outperforming open-source judges in several subtasks.
• On audio MOS prediction, Flex-Omni-7B reaches 0.545 LCC on NISQA utterance-level, far above Gemini-2.0-Flash (0.408) and Qwen2.5-Omni-7B (0.210).
• On the molecule permeability task, Flex-Mol-LLaMA-based DPO reaches up to 80.10% accuracy and best-of-N selection reaches 77.49% at N=16.

Measured effects reported in the paper

Surprising findings

• Training on text-only reasoning data transfers zero-shot to image, video, and audio judgment without any modality-specific data.
• Training on a larger number of samples degraded multimodal benchmark performance due to catastrophic forgetting of visual/audio capabilities.

Limitations

• Performance on some subtasks (e.g., Pair w.o. Tie on some categories, safety checks for Flex-Omni) lags behind commercial models and some baselines.
• Flex-VL-7B cannot evaluate audio tasks; audio and video evaluation depends on the base models' capabilities.
• Some benchmarks (VL-RewardBench, MJ-Bench) were modified after experiments were completed, affecting reproducibility.

Experimental setup

Datasets: JudgeLM-100K RLHF-V Mol-LLaMA instruction tuning corpus

Benchmarks: MLLM-as-a-Judge VL-RewardBench MJ-Bench GenAI-Bench NISQA BVCC SOMOS VoxSim Multimodal RewardBench JudgeAnything

Metrics: Pearson correlation accuracy normalized Levenshtein distance LCC SRCC

Baselines: GPT-4o Gemini-2.5-Pro GPT-4V LLaVA-Critic-7B Prometheus-Vision-13B PickScore-v1 HPS-v2.1 ImageReward Qwen2.5-VL-7B Qwen2.5-Omni-7B

Builds on: JudgeLRM Qwen2.5-VL Qwen2.5-Omni Mol-LLaMA DPO

Closest papers by meaning

• Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following (CVPR 2026 · similarity 0.91)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.90)
• ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning (CVPR 2026 · similarity 0.89)
• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.89)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)