AI Paper Atlas

Research map / Vision-language and multimodal / Visual question answering

Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning

Xu Yang, Hanwang Zhang, Chongyang Gao, Jianfei Cai

IJCV 2022 · Vision-language and multimodal › Visual question answering · approach: Vision-language model VQA evaluation · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fuse all four modules with equal weights instead of a single RELATION module (ablation)Module/R · MS-COCO Karpathy testCIDEr-D: 123.8 -> 125.6Improved
use soft learnable fusion weights instead of equal weights (ablation)Col/1 · MS-COCO Karpathy testCIDEr-D: 125.6 -> 127.3Improved
use soft fusion instead of hard Gumbel-Softmax selection (ablation)Col/H · MS-COCO Karpathy testCIDEr-D: 126.1 -> 127.3Improved
replace LSTM controller with multi-head attention controller (ablation)LSTM · MS-COCO Karpathy testCIDEr-D: 125.7 -> 126.7Improved
feed previous module layout Z into the MH-ATT controller (ablation)MH-ATT · MS-COCO Karpathy testCIDEr-D: 126.7 -> 127.3Improved
add part-of-speech syntax loss (ablation)MH-ATT+Z · MS-COCO Karpathy testCIDEr-D: 127.3 -> 128.4Improved
add part-of-speech syntax loss (ablation)MH-ATT+Z · MS-COCO Karpathy testmodule layout accuracy: 68.6 -> 77.6Improved
add Reason module (ablation)MH-ATT+Z · MS-COCO Karpathy testCIDEr-D: 127.3 -> 128.6Improved
add part-of-speech syntax loss (ablation)MH-ATT+Z · MS-COCO Karpathy testCHAIRi: 7.8 -> 7.4Improved
add Reason module on top of syntax loss (full CVLNM) (ablation)MH-ATT+Z+Ls · MS-COCO Karpathy testCIDEr-D: 128.4 -> 129.5Improved
change syntax-loss weight lambda to 5 (ablation)CVLNM · MS-COCO Karpathy testCIDEr-D: 129.5 -> 129.2No clear effect
pre-train CNN with 1600/400 objects/attributes instead of 305/103 (ablation)CVLNM · MS-COCO Karpathy testCIDEr-D: 129.5 -> 130.1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy