Research map / Vision-language and multimodal / Cross-modal learning
Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization
chenyang zhu, Hongxiang Li, Xiu Li, Long Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| update knowledge in a frozen LLM encoder after concept learning instead of finetuning the encoder (Enc-FT) | MoKus · KnowCusBench generation | CLIP-I-Seg: 0.562 -> 0.718 | Improved |
| update knowledge in a frozen LLM encoder instead of finetuning the encoder (Enc-FT) | MoKus · KnowCusBench reconstruction | CLIP-I-Seg: 0.553 -> 0.764 | Improved |
| closed-form knowledge update instead of per-knowledge DreamBooth retraining (Naive-DB) | MoKus · KnowCusBench generation | CLIP-T: 0.291 -> 0.305 | Improved |
| closed-form knowledge update instead of Naive-DB | MoKus · KnowCusBench training | time (min): ~27min -> ~6min | Improved |
| MoKus versus Naive-DB on CLIP-I | MoKus · KnowCusBench reconstruction | CLIP-I: 0.874 -> 0.867 | Hurt |
| MoKus versus Naive-DB on generation CLIP-I | MoKus · KnowCusBench generation | CLIP-I: 0.789 -> 0.761 | Hurt |
| scaling factor eta 1e-4 instead of 1e-6 (ablation) | MoKus · KnowCusBench generation | Pick Score: 21.30 -> 18.40 | Hurt |
| scaling factor eta 1e-5 instead of 1e-6 (ablation) | MoKus · KnowCusBench generation | CLIP-I-Seg: 0.718 -> 0.717 | No clear effect |
| increase number of knowledge items from 1 to 5 (ablation) | MoKus · KnowCusBench generation | CLIP-T: 0.304 -> 0.305 | No clear effect |
| increase number of knowledge items from 1 to 5 (ablation) | MoKus · KnowCusBench training | time (s): 331.3s -> 360.0s | Hurt |
| apply knowledge update with world knowledge to Qwen-Image | MoKus · WISE subset | WiScore: 0.81 -> 1.33 | Improved |
| apply knowledge update with world knowledge to Qwen-Image | MoKus · WISE subset | Realism: 0.70 -> 1.42 | Improved |
Closest papers by meaning
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.86)
- Generative Multi-Modal Knowledge Retrieval with Large Language Models (AAAI 2024 · similarity 0.86)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.86)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.85)
- UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens (NeurIPS 2025 · similarity 0.85)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
