AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?

Michael Aerni, Joshua Swanson, Kristina Nikolić, Florian Tramèr

ICLR 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
compare text description with image generation of the same movie postersChatGPT-5 · Movie posters from memoryrubric error rate: text 45% vs image 6% errorsHurt
describe the learned faces in text instead of generating themfine-tuned Janus-Pro and Harmon · Synthetic facesattribute accuracy: description near random guessing (around 20%)Hurt
test on unseen concept combinations in image generation (ablation)fine-tuned Janus-Pro and Harmon · Abstract visual concepts (held-out test)image generation accuracyNo clear effect
ask the model to visualize the poster before describing itChatGPT-5 · Movie posters from memorydescription error rateNo clear effect
fine-tune only the LLM backbone, freezing other parametersJanus-Pro and Harmon · Controlled synthetic experimentsmodal aphasia emergenceNo clear effect
text-only refusal training for feet promptsJanus-Pro · Safety case studyrefusal rate on rare expression: 89% (real word) -> 24% (rare expression)Hurt
text-only refusal trainingJanus-Pro · Safety case studyimage generation capability for feetNo clear effect
different concept types (shape vs position)Janus-Pro · Abstract visual conceptsverbal accuracy: shapes around 23% vs random 14%; positions below random 25%Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy