Research map / Vision-language and multimodal / CLIP and zero-shot
Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Multi-Path paradigm for compositional zero-shot learning with CLIP that trains separate state, object, and composition branches, implemented as Troika with branch-specific prompts, adapters, and a Cross-Modal Traction module.
Problem
Compositional zero-shot learning with pre-trained vision-language models: existing prompt-tuning methods model only composed state-object pairs, limiting use of pre-trained knowledge and generalization to unseen compositions.
Approach
Establish three independent recognition branches (state, object, composition) over a shared CLIP space, each with its own learnable prompt prefix and visual features (disentangled by MLPs), combine their probabilities at inference, and add a cross-attention Cross-Modal Traction module that shifts prompt representations toward image patch features.
Key results
- Closed-world: Troika improves HM by +2.0%, +7.4%, +2.3% and AUC by +1.5%, +5.7%, +1.9% over prior SOTA on MIT-States, UT-Zappos, C-GQA.
- Open-world: Troika outperforms best CLIP-based method by +0.8%, +3.8%, +0.5% HM and +0.4%, +2.7%, +0.3% AUC.
- Ablation: removing the Cross-Modal Traction module lowers UT-Zappos HM from 54.6 to 51.9 and AUC from 41.7 to 37.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the state and object branches alongside the composition branch during training and inference (ablation) | Troika · UT-Zappos closed-world | HM: 49.9 -> 54.6 | Helped |
| add the state and object branches alongside the composition branch during training and inference (ablation) | Troika · C-GQA closed-world | HM: 28.9 -> 29.4 | Helped |
| add independent branch-specific prefixes while sharing the primitive vocabulary (ablation) | Troika · UT-Zappos closed-world | HM: 54.0 -> 54.6 | Helped |
| add independent branch-specific prefixes while sharing the primitive vocabulary (ablation) | Troika · C-GQA closed-world | HM: 28.7 -> 29.4 | Helped |
| share the primitive vocabulary across branches instead of using an individual vocabulary for each branch (ablation) | Troika · C-GQA closed-world | HM: 28.9 -> 29.4 | Helped |
| add Cross-Modal Traction (ablation) | Troika · UT-Zappos closed-world | HM: 51.9 -> 54.6 | Helped |
| add Cross-Modal Traction (ablation) | Troika · C-GQA closed-world | AUC: 11.0 -> 12.4 | Helped |
| use Adapter instead of freezing the image encoder (ablation) | Troika · MIT-States closed-world | HM: 37.5 -> 39.3 | Helped |
| use Adapter instead of freezing the image encoder (ablation) | Troika · UT-Zappos closed-world | HM: 50.3 -> 54.6 | Helped |
| use Adapter instead of freezing the image encoder (ablation) | Troika · C-GQA closed-world | HM: 24.2 -> 29.4 | Helped |
| fully fine-tune the image encoder instead of using Adapter (ablation) | Troika · MIT-States and UT-Zappos closed-world | HM: 39.3 -> 28.7; 54.6 -> 34.4 | Hurt |
| fully fine-tune the image encoder instead of using Adapter (ablation) | Troika · C-GQA closed-world | HM: 29.4 -> 31.8 | Helped |
Surprising findings
- Fully fine-tuning the image encoder hurts performance on MIT-States and UT-Zappos, though it is best on C-GQA, suggesting overfitting.
- Building an individual primitive vocabulary per branch causes a significant performance drop, so a shared vocabulary is preferred.
Limitations
- Gains on open-world C-GQA and some metrics are small, and seen accuracy on UT-Zappos is slightly lower than DFSP(t2i).
- Full fine-tuning of the image encoder overfits on smaller datasets, so the adapter choice is dataset-dependent.
Experimental setup
Datasets: MIT-States UT-Zappos C-GQA
Benchmarks: MIT-States UT-Zappos C-GQA
Metrics: S U HM AUC
Baselines: CLIP CoOp CSP PromptCompVL DFSP
Builds on: CLIP CoOp CSP DFSP Adapter
Closest papers by meaning
- A Dynamic Learning Method towards Realistic Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.89)
- ProCC: Progressive Cross-Primitive Compatibility for Open-World Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.89)
- TOMCAT: Test-time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning (NeurIPS 2025 · similarity 0.88)
- Retrieval-Augmented Primitive Representations for Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.88)
- Beyond Seen Primitive Concepts and Attribute-Object Compositional Learning (CVPR 2024 · similarity 0.88)
- Revealing the Proximate Long-Tail Distribution in Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.