Research map / Vision-language and multimodal / Human and embodied modeling
Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hierarchical Prompt Tuning (HPT), which uses ChatGPT-generated descriptions and entity/attribute/relationship graphs to learn low-, high-, and global-level text prompts in a CLIP dual-path asymmetric framework.
Problem
Prompt tuning of vision-language models with LLM-generated category descriptions ignores structured knowledge (entities, attributes, and their relations) within those descriptions, limiting few-shot and generalization performance.
Approach
Use ChatGPT to produce descriptions plus structured entity-attribute-relation graphs, and model them with a relationship-guided attention module (learned scalar biases added to self-attention) together with cross-level self-attention over low-level, high-level, and global-level prompts.
Key results
- Base-to-new generalization: HPT average new-class accuracy 76.86% vs. MaPLe 75.14%, harmonic mean 80.23% vs. 78.55%.
- Cross-dataset evaluation: HPT average 67.74% vs. MaPLe 66.30%.
- Domain generalization: HPT average 60.71% vs. MaPLe 60.26%, with the best results on ImageNet, ImageNetV2, ImageNet-S and ImageNet-R (not ImageNet-A).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add high-level prompts to global-level prompts (ablation) | HPT · Base-to-new average | harmonic mean: 78.99 -> 79.33 | Helped |
| add low-level prompts to global-level prompts (ablation) | HPT · Base-to-new average | harmonic mean: 78.99 -> 79.59 | Helped |
| add both high-level and low-level prompts to global-level prompts (ablation) | HPT · Base-to-new average | harmonic mean: 78.99 -> 80.23 | Helped |
| use entities only instead of no structured inputs (ablation) | HPT relationship-guided attention · Base-to-new average | harmonic mean: 79.33 -> 79.49 | Helped |
| use attributes only instead of no structured inputs (ablation) | HPT relationship-guided attention · Base-to-new average | harmonic mean: 79.33 -> 79.56 | Helped |
| use entities and relationships, without attributes (ablation) | HPT relationship-guided attention · Base-to-new average | harmonic mean: 79.33 -> 79.71 | Helped |
| use entities and attributes, without relationships (ablation) | HPT relationship-guided attention · Base-to-new average | harmonic mean: 79.33 -> 79.85 | Helped |
| use entities, attributes, and their relationships (ablation) | HPT relationship-guided attention · Base-to-new average | harmonic mean: 79.85 -> 80.23 | Helped |
| increase the number of category descriptions (ablation) | HPT · Base-to-new generalization | recognition accuracy | Helped |
| replace MaPLe with HPT | CLIP ViT-B/16 · Base-to-new average, new classes | accuracy: 75.14 -> 76.86 | Helped |
| replace MaPLe with HPT | CLIP ViT-B/16 · Cross-dataset average | accuracy: 66.30 -> 67.74 | Helped |
| replace MaPLe with HPT | CLIP ViT-B/16 · Domain generalization average | accuracy: 60.26 -> 60.71 | Helped |
Surprising findings
- On ImageNet-A, HPT is slightly below MaPLe (50.85 vs. 50.90), and on OxfordPets and Food101 base-to-new, HPT is slightly lower than MaPLe on some metrics.
Limitations
- Relies on ChatGPT-generated descriptions and graphs, whose quality and bias are not analyzed beyond an ethical statement.
- Only one randomly picked description per category is used for relationship-guided attention during training to save memory.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe
Builds on: CoOp CoCoOp MaPLe CLIP dual-path asymmetric network
Closest papers by meaning
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.89)
- Task-Aware Clustering for Prompting Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.