Research map / Vision-language and multimodal / CLIP and zero-shot
Harnessing the Power of Foundation Models for Accurate Material Classification
Qingran Lin, Fengwei Yang, Chaolun Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse DINOv2 vision features with CLIP-encoded GPT-4v descriptions (ablation) | material classifier trained on synthetic data · Google-test | mAcc: 0.89 -> 0.92 | Improved |
| add language prior stream to the DINOv2 vision stream (ablation) | material classifier · FMD | mAcc: 0.88 -> 0.89 | Improved |
| add vision stream to the language stream (ablation) | material classifier · FMD | mAcc: 0.85 -> 0.89 | Improved |
| add language stream on DMS-test (ablation) | material classifier · DMS-test | mAcc: 0.60 -> 0.64 | Improved |
| train on synthetic dataset instead of DMS | DINOv2 vision stream · Google-test | mAcc: 0.71 -> 0.89 | Improved |
| train on synthetic dataset instead of DMS | DINOv2 vision stream · FMD | mAcc: 0.79 -> 0.88 | Improved |
| train on synthetic dataset instead of DMS | DINOv2 vision stream · DMS-test | mAcc: 0.67 -> 0.60 | Hurt |
| fine-tune all DINOv2 parameters instead of the head only (ablation) | vision stream · FMD | mAcc: 0.88 -> 0.38 | Hurt |
| use ResNet-101 or ViT-L/16 instead of DINOv2 with head-only training (ablation) | vision stream · FMD | mAcc: 0.88 -> 0.64 | Hurt |
| scale synthetic data from 1x to 2x and beyond (ablation) | vision stream · FMD | mAcc | No clear effect |
| increase dataset scale from 0.2x to 1x (ablation) | vision stream · FMD | mIoU and mAcc | Improved |
| use auto-labeled semantic masks (Grounded SAM) instead of max-pooling all patches (ablation) | vision stream · FMD | mAcc: 0.85 -> 0.88 | Improved |
Closest papers by meaning
- Hierarchical Material Recognition from Local Appearance (ICCV 2025 · similarity 0.89)
- Foundation Model for Material Science (AAAI 2023 · similarity 0.88)
- Infusing Synthetic Data with Real-World Patterns for Zero-Shot Material State Segmentation (NeurIPS 2024 · similarity 0.88)
- MaRI: Material Retrieval Integration across Domains (CVPR 2025 · similarity 0.85)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.84)
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
