Research map / Vision-language and multimodal / Multimodal LLMs

Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that text+image coding agents in sandboxed terminals match or beat native omni-modal models on audio-video benchmarks, then proposes Code-X (SFT+GSPO RL on Qwen3.5-9B/Qwen3.6-27B with OmniCoding data) and TerminalBench-O.

Problem

Whether native omni-modal models are necessary for video and audio tasks, or whether text+image coding agents in a sandboxed terminal can solve them using tools like ffmpeg, ASR and self-written scripts.

Approach

Treat raw video/audio as workspace state rather than model context; coding agents stage media, extract transcripts/frames/metadata via tools and code, and reason over compact evidence. Adds failure taxonomy, skill injection (human, self-iteration, trace distillation), an RL recipe with process-aware verifiable rewards, and a process-level benchmark for omnimodal processing.

Key results

• GPT-5.4 xHigh in Codex reaches 75.0% average on OmniGAIA vs 66.1% for Gemini 3.1 Pro.
• Skill injection from execution-trace distillation raises GPT-5.4 high OmniGAIA average from 61.4% to 76.7%.
• Code-X on Qwen3.6-27B (SFT) reaches 43.3% on OmniGAIA and 60.0% on LVOmniBench, versus 38.2% and 51.0% for the direct MLLM baseline.

Surprising findings

• A text-only GPT-5.4 high coding agent matches or exceeds its vision-enabled version on OmniGAIA and SocialOmni.
• Coding agents consume fewer tokens than native omni models while achieving competitive accuracy.

Limitations

• Code-X validated only at 9B and 27B scales.
• Several closed-source agents (e.g., Claude Opus 4.6 on VideoZeroBench) were not fully evaluated due to API credit limits.
• Scope limited to offline tasks; results do not transfer directly to streaming or interactive settings.

Experimental setup

Datasets: OmniGAIA-SFT-2K OmniVideoBench AVUTBenchmark Video-MME-v2

Benchmarks: OmniGAIA SocialOmni LVOmniBench VideoZeroBench TerminalBench-O

Metrics: accuracy pass@1 pass rate avg score token usage

Baselines: Gemini 3.1 Pro Gemini 3 Flash Qwen 3.5 Omni Plus MiMo-V2-Omni OmniAgent Agent-Omni Claude Opus 4.6 Kimi K-2.5 MiniMax-M2.7 Direct MLLM

Builds on: GSPO DAPO Terminus-KIRA Meta-Harness AgentProcessBench

Closest papers by meaning

• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs (NeurIPS 2026 · similarity 0.88)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.87)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.86)
• VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.86)
• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.86)
• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.86)