Research map / Label-efficient and robust learning / Recognition and long-tail
Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TASK2BOX, which maps task/dataset representations (CLIP, Task2Vec FIM, attributes) to axis-aligned box embeddings whose volumetric overlap encodes asymmetric task relationships like containment and transferability.
Problem
Representing and visualizing asymmetric relationships between tasks or datasets (containment, transferability) is not supported by symmetric Euclidean embeddings like t-SNE, and prior asymmetric measures such as KL divergence are weak.
Approach
Learn a mapping from dataset base representations to box embeddings and define the relation as the normalized intersection volume of boxes, so containment and partial transfer appear as asymmetric overlap; new datasets are embedded by a forward pass without retraining.
Key results
- On iNaturalist+CUB existing datasets with CLIP mean features, TASK2BOX (5D) reaches 84.67% F1 vs 45.25% for an MLP classifier.
- On ImageNet instruments existing datasets, TASK2BOX (5D) with CLIP mean features reaches 90.58% F1.
- On Taskonomy existing datasets, TASK2BOX (3D) achieves Spearman 0.93 vs 0.88 for MLP using attribute-based embeddings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5D rather than 2D boxes (ablation) | TASK2BOX on iNaturalist + CUB · Existing datasets, μCLIP | F1 score: 69.23% -> 84.67% | Helped |
| Use 5D rather than 2D boxes (ablation) | TASK2BOX on ImageNet · Existing datasets, μCLIP | F1 score: 62.72% -> 90.58% | Helped |
| Use 3D rather than 2D boxes (ablation) | TASK2BOX on Taskonomy · Novel datasets | Spearman’s ρ: 0.12 ± 0.21 -> 0.48 ± 0.24 | Helped |
| Use 5D rather than 3D boxes (ablation) | TASK2BOX on Taskonomy · Novel datasets | Spearman’s ρ: 0.48 ± 0.24 -> 0.39 ± 0.22 | Hurt |
| Use 5D rather than 3D boxes (ablation) | TASK2BOX on Taskonomy · Existing datasets | Spearman’s ρ: 0.93 ± 0.02 -> 0.94 ± 0.03 | No clear effect |
| Use box embeddings instead of directly predicting relationships with an MLP | iNaturalist + CUB and ImageNet hierarchical prediction · Existing and novel datasets | F1 score | Helped |
| Use CLIP features rather than FIM features | TASK2BOX · Hierarchical relationships on iNaturalist + CUB and ImageNet | F1 score | Mixed |
Surprising findings
- FIM features generalize poorly to novel datasets (e.g., 10.06% F1 for 2D on iNaturalist+CUB), far below CLIP-based features.
Limitations
- Novel-task performance on Taskonomy is weaker and more uncertain given only 25 tasks.
- Does not incorporate training procedures or model architecture information.
- 2D embeddings may restrict representation of complex relationships.
Experimental setup
Datasets: iNaturalist CUB ImageNet Taskonomy Hugging Face image classification datasets
Benchmarks: iNaturalist+CUB hierarchy ImageNet instruments hierarchy Taskonomy task affinity
Metrics: F1 score Spearman correlation
Baselines: MLP classifier linear classifier KL divergence asymmetric cosine similarity asymmetric euclidean similarity t-SNE
Builds on: box embeddings order embeddings Task2Vec
Closest papers by meaning
- Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.85)
- Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models (CVPR 2024 · similarity 0.85)
- Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings (NeurIPS 2025 · similarity 0.84)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.84)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.83)
- Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.