Research map / Vision-language and multimodal / Human and embodied modeling
Symbolic Representation for Any-to-Any Generative Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A-LANGUAGE, a symbolic task description language (functions, parameters, topology) and a training-free GPT-4o-driven inference engine that turns natural-language any-to-any generation tasks into executable ComfyUI workflows.
Problem
Any-to-any multimodal generation is typically handled by implicitly trained unified neural models (limited extensibility, non-interruptible, costly) or by agentic multi-agent systems that lack a unified formal task representation.
Approach
An explicit symbolic representation of generative tasks as triples of functions, parameters, and topology (data-flow edges), inferred from natural language by a pre-trained LM in a two-stage (component then topology) process with iterative error-driven refinement, with no task-specific training.
Key results
- On ComfyBench the method reaches a 41.0% overall resolve rate versus 32.5% for ComfyAgent and 15.5% for LWM.
- On the 120-task suite, execution Pass@1 is 0.87 versus 0.63 for GenAgent; compilation Pass@1 is 0.98 versus 0.84.
- User study win rates reach 94% against Show-o and 98% against LVM on text-to-image-style comparisons, and 67% against Gen-3 on image-to-video alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use symbolic inference instead of GenAgent | 120-task suite · Compilation | Pass@1: 0.84 -> 0.98 | Helped |
| Use symbolic inference instead of GenAgent | 120-task suite · Execution | Pass@1: 0.63 -> 0.87 | Helped |
| Use the proposed method instead of ComfyAgent | ComfyBench · Total | resolve rate (%): 32.50 -> 41.00 | Helped |
| Add two-stage inference to the refinement-only system (ablation) | Ours on ComfyBench · Total | resolve rate (%): 28.50 -> 41.00 | Helped |
| Add iterative refinement to the two-stage-inference-only system (ablation) | Ours on ComfyBench · Total | resolve rate (%): 22.00 -> 41.00 | Helped |
| Use symbolic inference instead of neural unified models | Text to image · Average user-study rank | average rank: 2.0 -> 1.5 | Helped |
Surprising findings
- Complex agent designs did not necessarily outperform simpler ones; the simpler symbolic approach beat GenAgent on straightforward tasks.
- Higher-readability pseudo-natural syntax produced more format errors, while structurally rigid declarative/dataflow syntax produced more topological and connection errors.
Limitations
- Depends on a closed-source LM (GPT-4o) and on ComfyUI node availability.
- Agentic-style multi-step planning can be beneficial for more intricate workflows, which the symbolic method does not fully address.
- Evaluation relies on a small user study with five annotators and a self-constructed task suite.
Experimental setup
Datasets: ComfyBench 120-task generative task suite with 16 reference programs for retrieval
Benchmarks: ComfyBench 120-task generative task suite
Metrics: Pass@1 resolve rate win rate average rank
Baselines: GenAgent Show-o SEED-X LVM Unified-IO 2 Gen-3 ComfyAgent HuggingGPT LWM
Builds on: GenAgent VISPROG HuggingGPT ComfyUI
Closest papers by meaning
- GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models (CVPR 2026 · similarity 0.86)
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning (ICLR 2025 · similarity 0.86)
- GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules (ICLR 2024 · similarity 0.85)
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.85)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.85)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.