AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Human Attention in Image Captioning: Dataset and Analysis

Sen He, Hamed R. Tavakoli, Ali Borji, Nicolas Pugeault

ICCV 2019 · Vision-language and multimodal › Human and embodied modeling · approach: Human perception modeling · 49 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add free-viewing saliency guidance (foveated fixation locations) to soft-attention captioningsoft-attention captioning, ResNet-18 · Flickr30k testCIDEr: 0.352 -> 0.403Improved
add task saliency (trained on capgaze2 captioning fixations) to soft-attention captioningsoft-attention captioning, ResNet-18 · Flickr30k testCIDEr: 0.352 -> 0.416 (+18.2%)Improved
add task saliency (trained on capgaze2 captioning fixations) to soft-attention captioningsoft-attention captioning, ResNet-18 · MSCOCO testCIDEr: 0.81 -> 0.907 (+12%)Improved
add free-viewing saliency guidance to soft-attention captioningsoft-attention captioning, ResNet-18 · MSCOCO testCIDEr: 0.81 -> 0.889Improved
train the saliency model on captioning-task fixations (capgaze2) rather than free-viewing Salicon (ablation)saliency-guided captioning · Flickr30k testCIDEr: 0.403 -> 0.416Improved
train the saliency model on captioning-task fixations (capgaze2) rather than free-viewing Salicon (ablation)saliency-guided captioning · MSCOCO testCIDEr: 0.889 -> 0.907Improved
use a bottom-up saliency model (SalGAN) instead of top-down soft attention to predict captioning fixationsattention prediction · captioning fixations (NSS)NSS: 1.128 -> 1.618Improved
compare attention under captioning task versus free-viewing for CNN-encoder agreementVGG-16 conv5-3 · human-attended regions covered by CNNpercentage: 72.5% -> 78.1%Improved
use full-duration captioning fixations instead of first 3 seconds for CNN agreementVGG-16 conv5-3 · human-attended regions covered by CNNpercentage: 78.1% -> 77.9%No clear effect
correlate machine-human spatial attention congruency with caption qualitysoft-attention captioning · per-image CIDEr vs NSSSpearman correlation: 0.01No clear effect
correlate machine-human temporal attention distance with caption qualitysoft-attention captioning · per-image CIDEr vs DTW distanceSpearman correlation: -0.05No clear effect
describe the image while viewing (captioning) instead of free-viewing, over full durationhuman attention · attention ratio on described objectsattention ratio: 0.66 -> 0.63Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy