Research map / Large language models / Language, safety and interpretability
Deciphering Compatibility Relationships with Textual Descriptions via Extraction and Explanation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the PFE dataset of 6,407 fashion-pair compatibility explanations and a two-stage pipeline (feature extraction via rationale extraction or cross-attention, then fine-tuned LLM generation) that outputs explanations grounded in extracted features.
Problem
Generating accurate, diverse, and informative natural-language explanations of compatibility between pairs of fashion items, where existing outfit recommenders give rudimentary or repetitive explanations and general LLMs produce plausible but inauthentic ones.
Approach
A two-stage pipeline: a feature extractor (rationale extraction or cross-attention) trained on pair match labels selects important item features, and a generator LLM fine-tuned on the newly curated PFE explanation dataset turns the extracted features and categories into explanations.
Key results
- Fine-tuned Flan-T5-xl-F achieves the best BLEURT (0.0736), BLEU-4 (14.11) and ROUGE-1 (0.4691) on the PFE test set, while fine-tuned Flan-T5-large-F reaches the lowest FID (2.256).
- Rationale Extraction reaches 0.8531 accuracy on the Stage I matching task versus 0.8505 for Cross-Attn.
- In human rating (Amazon MTurk, 50 pairs), RE + Flan-T5-xl-F scores 6.43 versus 6.50 for ChatGPT and 5.48 for PEPLER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Flan-T5-xl on PFE instead of using the untuned model | Flan-T5-xl · PFE test set | BLEURT: -0.5783 -> 0.0736 | Helped |
| Fine-tune Flan-T5-xl on PFE instead of using the untuned model | Flan-T5-xl · PFE test set | BLEU-1: 23.64 -> 46.12 | Helped |
| Fine-tune Flan-T5-xl on PFE instead of using the untuned model | Flan-T5-xl · PFE test set | FID: 9.634 -> 3.046 | Helped |
| Use Rationale Extraction instead of Cross-Attn as the Stage I extractor (ablation) | Stage I extraction model · Combined test set | R@5: 0.1468 -> 0.3060 | Helped |
| Use Rationale Extraction instead of Cross-Attn as the Stage I extractor (ablation) | Stage I extraction model · Combined test set | accuracy: 0.8505 -> 0.8531 | Helped |
| Add Cross-Attn extraction instead of using no extractor (ablation) | GPT2-F · PFE test set | FID: 7.980 -> 6.718 | Helped |
| Add Cross-Attn extraction instead of using no extractor (ablation) | Flan-T5-large-F · PFE test set | FID: 11.141 -> 8.073 | Helped |
| Add Cross-Attn extraction instead of using no extractor (ablation) | Flan-T5-xl-F · PFE test set | FID: 16.647 -> 10.518 | Helped |
| Add Rationale Extraction instead of using no extractor (ablation) | GPT2-F · PFE test set | FID: 7.980 -> 7.689 | Helped |
| Add Rationale Extraction instead of using no extractor (ablation) | Flan-T5-large-F · PFE test set | FID: 11.141 -> 8.091 | Helped |
| Add Rationale Extraction instead of using no extractor (ablation) | Flan-T5-xl-F · PFE test set | FID: 16.647 -> 11.657 | Helped |
| Increase the PFE fine-tuning sample count from 5,000 to 6,000 (ablation) | Flan-T5-xl · PFE test set | BLEURT | No clear effect |
Surprising findings
- Fine-tuned GPT-2 attains the lowest CLIP-embedding FID in the extraction-impact analysis, which the authors note may reflect mimicking the training corpus rather than identifying compatibility relationships.
- Naive repetition of input features obtains a strong BLEU-1 (27.35) and FID that is competitive with several trained models.
Limitations
- The pipeline may struggle to reveal multi-object compatibility relationships.
- Human evaluation is small (50 pairs, three MTurk raters per item) and the automatic metrics do not capture accuracy of compatibility claims.
- The PFE dataset is relatively small (about 6.4k examples), and the authors observe that more samples may not significantly help.
Experimental setup
Datasets: PFE dataset Amazon Reviews (Clothing, Shoes and Jewelry) FashionVC Combined
Benchmarks: PFE test set Combined dataset (Stage I)
Metrics: BLEURT BLEU-1 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L FID Accuracy Recall@5 Recall@10
Baselines: PETER PEPLER-F GPT-2 Flan-T5-large Flan-T5-xl ChatGPT Naive repetition
Builds on: rationale extraction PEPLER PETER LoRA
Closest papers by meaning
- Learning to Match on Graph for Fashion Compatibility Modeling (AAAI 2020 · similarity 0.89)
- Context-Aware Visual Compatibility Prediction (CVPR 2019 · similarity 0.88)
- StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback (AAAI 2026 · similarity 0.82)
- Easy Regional Contrastive Learning of Expressive Fashion Representations (NeurIPS 2024 · similarity 0.81)
- Text-Driven Fashion Image Editing with Compositional Concept Learning and Counterfactual Abduction (CVPR 2025 · similarity 0.80)
- AIpparel: A Multimodal Foundation Model for Digital Garments (CVPR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.