Research map / Vision-language and multimodal / Human and embodied modeling
UniAR: A Unified model for predicting human Attention and Responses on visual content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniAR, a unified T5-based multimodal encoder-decoder that, given an image and a text prompt, predicts saliency/importance heatmaps, scanpaths, and aesthetic/quality ratings across natural images, webpages, graphic designs and mobile UIs.
Problem
Human behavior modeling is fragmented: implicit perceptual behavior (attention, heatmaps, scanpaths) and explicit responses (ratings, aesthetics) are studied in isolation and with dedicated per-domain models, lacking a single model across diverse visual content.
Approach
A single ViT-B16 + T5-base encoder with a text prompt specifying input domain, output behavior type, and task query, plus three predictors (conv heatmap head, text-token autoregressive scanpath decoder, conv rating head), trained jointly on 11 public datasets.
Key results
- On KonIQ-10k, UniAR reaches PLCC 0.925 (best) and SRCC 0.905, and on Web Aesthetics PLCC 0.839 vs 0.820 for the best prior method.
- On WS-Scanpath, UniAR gets Sequence Score 0.267 and MultiMatch 0.887, outperforming the previous best (AGD-S w. layout, 0.224 and 0.755).
- Across heatmap benchmarks UniAR is best on 17 of 27 listed metrics and second-best on 6, with large gains on Mobile UI and Imp1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniAR instead of AGD-F (W/o-L) | WS-Saliency heatmap prediction · WS-Saliency | CC: 0.815 -> 0.827 | Helped |
| Use UniAR instead of AGD-F | FiWI heatmap prediction · FiWI | CC: 0.735 -> 0.734 | Hurt |
| Use UniAR instead of SAM-S2017 | Mobile UI heatmap prediction · Mobile UI | CC: 0.834 -> 0.879 | Helped |
| Use UniAR instead of DeepGaze IIE | CAT2000 heatmap prediction · CAT2000 | CC: 0.819 -> 0.870 | Helped |
| Use UniAR instead of SimpleNet w. PNASNet-5 | Salicon heatmap prediction · Salicon | CC: 0.907 -> 0.900 | Hurt |
| Use UniAR instead of [cite]'23 | OSIE heatmap prediction · OSIE | CC: 0.761 -> 0.742 | Hurt |
| Use UniAR instead of UMSI | Imp1k importance-heatmap prediction · Imp1k | CC: 0.875 -> 0.904 | Helped |
| Use UniAR instead of Gazeformer | COCO-Search18 scanpath prediction · COCO-Search18 | SemSS: 0.496 -> 0.521 | Helped |
| Use UniAR instead of AGD-S (w. layout) | WS-Scanpath prediction · WS-Scanpath | Sequence Score: 0.224 -> 0.267 | Helped |
| Use UniAR instead of BIQA (25 crops) | KonIQ-10k rating prediction · KonIQ-10k | SRCC: 0.906 -> 0.905 | Hurt |
| Use UniAR instead of Comparison-based Calista | Web Aesthetics rating prediction · Web Aesthetics | PLCC: 0.820 -> 0.839 | Helped |
| Train with COCO-Freeview scanpath plus WS-Saliency, in addition to COCO-Freeview saliency | UniAR transfer to WS-Scanpath · WS-Scanpath | Sequence Score: 0.190 -> 0.231 | Helped |
Surprising findings
- Transferring from COCO-FreeView (natural images) scanpath plus saliency data to webpage scanpaths achieves 0.231-0.261 Sequence Score without seeing webpage scanpaths in one setting, close to the in-domain previous SOTA.
Limitations
- Trained only on datasets from young, mostly university-affiliated participants, limiting demographic coverage.
- Using the model as a reward model may lead to reward hacking; preferences are not personalized and converge to a uniform notion of preference.
- Several benchmarks were reproduced with a randomly re-partitioned Mobile UI split and results are not uniformly comparable across papers.
Experimental setup
Datasets: Salicon OSIE CAT2000 WS-Saliency Mobile UI Imp1k WS-Scanpath FiWI COCO-Search18 Koniq-10k Web Aesthetics WebLI
Benchmarks: Salicon OSIE CAT2000 WS-Saliency Mobile UI Imp1k FiWI WS-Scanpath COCO-Search18 Koniq-10k Web Aesthetics
Metrics: CC KLD AUC-Judd sAUC SIM NSS RMSE R-squared SemSS SemFED
Baselines: SAM-ResNet EML-NET UMSI AGD-F DeepGaze IIE Gazeformer MUSIQ-single BIQA Calista
Builds on: T5 Vision Transformer GazeFormer UMSI
Closest papers by meaning
- Visual Persuasion: What Influences Decisions of Vision-Language Models? (ICML 2026 · similarity 0.86)
- Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms (NeurIPS 2024 · similarity 0.86)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.85)
- MVReward: Better Aligning and Evaluating Multi-View Diffusion Models with Human Preferences (AAAI 2025 · similarity 0.85)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.85)
- Learning Multi-Dimensional Human Preference for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.