AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Synergy of Sight and Semantics: Visual Intention Understanding with CLIP

Qu Yang, Mang Ye, Dacheng Tao

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add Hierarchical Class Integration (ablation)IntCLIP · Intentonomy testMicro F1: 47.05 -> 50.54Improved
add Sight-assisted Aggregation (ablation)IntCLIP · Intentonomy testMicro F1: 47.46 -> 50.54Improved
train all CLIP image layers instead of only semantic layers (ablation)IntCLIP dual branch · Intentonomy testMicro F1: 50.54 -> 22.64Hurt
add frozen sight branch alongside semantic branch (dual vs single, semantic layers trained) (ablation)IntCLIP image encoder · Intentonomy testMicro F1: 46.83 -> 50.54Improved
use LLM-integrated sentence labels instead of fine-grained labels only (ablation)IntCLIP · Intentonomy testMicro F1: 46.46 -> 50.54Improved
increase integrated label length from 11-15 to 16-20 words (ablation)HCI · Intentonomy testMacro F1: 38.40 -> 37.76Mixed
use Sight-assisted Aggregation instead of element-wise addition (ablation)IntCLIP · Intentonomy testMicro F1: 47.46 -> 50.54Improved
use Sight-assisted Aggregation instead of Concat + MLP (ablation)IntCLIP · Intentonomy testMicro F1: 46.94 -> 50.54Improved
add HCI labels to HLEGHLEG · Intentonomy validationMicro F1: 46.34 -> 48.20Improved
generate HCI labels from fine-grained labels only (HCI-FG) (ablation)HCI · Intentonomy validationMicro F1: 52.61 -> 52.14No clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy