Research map / Vision-language and multimodal / CLIP and zero-shot
Synergy of Sight and Semantics: Visual Intention Understanding with CLIP
Qu Yang, Mang Ye, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Hierarchical Class Integration (ablation) | IntCLIP · Intentonomy test | Micro F1: 47.05 -> 50.54 | Improved |
| add Sight-assisted Aggregation (ablation) | IntCLIP · Intentonomy test | Micro F1: 47.46 -> 50.54 | Improved |
| train all CLIP image layers instead of only semantic layers (ablation) | IntCLIP dual branch · Intentonomy test | Micro F1: 50.54 -> 22.64 | Hurt |
| add frozen sight branch alongside semantic branch (dual vs single, semantic layers trained) (ablation) | IntCLIP image encoder · Intentonomy test | Micro F1: 46.83 -> 50.54 | Improved |
| use LLM-integrated sentence labels instead of fine-grained labels only (ablation) | IntCLIP · Intentonomy test | Micro F1: 46.46 -> 50.54 | Improved |
| increase integrated label length from 11-15 to 16-20 words (ablation) | HCI · Intentonomy test | Macro F1: 38.40 -> 37.76 | Mixed |
| use Sight-assisted Aggregation instead of element-wise addition (ablation) | IntCLIP · Intentonomy test | Micro F1: 47.46 -> 50.54 | Improved |
| use Sight-assisted Aggregation instead of Concat + MLP (ablation) | IntCLIP · Intentonomy test | Micro F1: 46.94 -> 50.54 | Improved |
| add HCI labels to HLEG | HLEG · Intentonomy validation | Micro F1: 46.34 -> 48.20 | Improved |
| generate HCI labels from fine-grained labels only (HCI-FG) (ablation) | HCI · Intentonomy validation | Micro F1: 52.61 -> 52.14 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DualCoOp — DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations (NeurIPS 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding (AAAI 2026 · similarity 0.86)
- Uncertain Multimodal Intention and Emotion Understanding in the Wild (CVPR 2025 · similarity 0.85)
- VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridging (CVPR 2026 · similarity 0.84)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.84)
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.84)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
