Research map / Vision-language and multimodal / Multimodal LLMs

ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ASCIIEval, a multiple-choice benchmark (3.5K human-drawn ASCII art samples, 359 concepts) plus the ASCIITune training set, evaluating LLMs and MLLMs on recognizing concepts from ASCII art in text, image, and combined modalities, with rationale-assisted fine-tuning and low-resolution prompting as enhancements.

Problem

Whether LLMs and MLLMs can perceive visual semantics embedded in consecutive text characters (ASCII art), which is under-studied and modality-agnostic, and how text and image inputs of the same art interact.

Approach

Frames ASCII art recognition as a multiple-choice task over human-drawn (not tone-converted) ASCII art, rendered as text and images, and measures modality-agnostic visual perception; it then tests rationale distillation from GPT-5 for LLMs and low-resolution prompting / LoRA on the vision encoder for MLLMs.

Key results

• GPT-5 reaches 87.81% macro-accuracy on image-only ASCIIEval, 20.01% above the best open-source MLLM (CogVLM2-Llama3-19B, 67.80%).
• Rationale-assisted fine-tuning of Qwen3-8B raises text-only accuracy from 28.28% to 35.66%, a 26.10% relative gain.
• Low-resolution prompting lifts Qwen2.5-VL-8B image-only accuracy by 17.49% to 52.32%; LoRA on the vision backbone reaches 75.48%.

Surprising findings

• Newer open-source MLLMs underperform their predecessors (e.g., Qwen-VL 52.32% vs. Qwen2.5-VL-7B 34.83% image-only), suggesting a trade-off between OCR-focused training and holistic visual perception.
• Adding text alongside the image consistently lowers accuracy compared with image-only input for all models.

Limitations

• Fine-tuning gains are post-hoc and do not fundamentally fix tokenization's destruction of 2D structure, per the authors.
• ASCIITune training set is noisy (human accuracy about 70% on a sample), and some test items remain ambiguous (under 1.67%).

Experimental setup

Datasets: ASCIIEval ASCIITune TableEval SGP-Bench OCRBench TextVQA

Benchmarks: ASCIIEval TableEval SGP-Bench OCRBench TextVQA

Metrics: macro-accuracy micro-accuracy

Baselines: GPT-4o Gemini-1.5-pro Gemini-2.5-pro Claude-opus-4 DeepSeek-V3 Qwen2.5-VL-72B CogVLM2-Llama3-19B InternVL3-78B

Builds on: TableEval SGP-Bench

Closest papers by meaning

• Textural or Textual: How Vision-Language Models Read Text in Images (ICML 2025 · similarity 0.89)
• Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.88)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.88)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.88)
• From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.88)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.88)