AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images

Zhuowan Li, Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, Quan Hung Tran, Benjamin Van Durme, Alan Yuille

ICCV 2021 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add operation calibration (weighted merge) (ablation)NSCL baseline · GQA balanced testdev (gold programs)accuracy: 47.01 -> 51.30Improved
remove normalization of concept embeddings (keep magnitudes) (ablation)NSCL + operation calibration · GQA balanced testdev (gold programs)accuracy: 51.30 -> 54.65Improved
add concept calibration on top of unnormalized embeddings (ablation)NSCL + operation calibration · GQA balanced testdev (gold programs)accuracy: 54.65 -> 56.13Improved
add concept and operation calibration together (ablation)NSCL · GQA balanced testdev (gold programs)accuracy: 47.01 -> 56.13Improved
use symbolic scene representation instead of dense visual features (ablation)CCO · GQA balanced testdev (gold programs)accuracy: 53.00 -> 56.13Improved
replace MMN parser with MISO parserCCO · GQA testdev_balancedaccuracy: 54.01 -> 54.11No clear effect
replace MMN parser with MISO parserprogram parsing · GQA testdev_balancedexact match: 85.13 -> 91.18Improved
add operation calibration (ablation)CCO · GQA perturbed testdevaccuracy: 55.49 -> 56.13Improved
use gold scene graph instead of detected symbolic sceneCCO · GQA balanced testdev (gold programs)accuracy: 56.13 -> 89.61Improved
remove low-weight operations from questions (all removable)CCO · GQA perturbed testdevaccuracy: 56.13 -> -2.59Hurt
remove low-weight operations from questions (all removable)CCO · GQA visual-hard splitaccuracy: 37.42 -> -4.72Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy