Research map / Label-efficient and robust learning / Recognition and long-tail
MaRI: Material Retrieval Integration across Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaRI, a dual-encoder DINOv2 model trained with InfoNCE on synthetic and ZeST-transferred real-world material data to retrieve materials from real images.
Problem
Material retrieval from real-world images is poorly served by generic image search models because material spaces differ from image spaces and there is a lack of material-aligned datasets with diverse shapes and lighting.
Approach
Learns a shared embedding space for image and material-sphere renderings by contrastively fine-tuning only the last Transformer block of two DINOv2 encoders, trained on a combined synthetic (Blender, Objaverse + AmbientCG + HDRI) and real-world (ZeST-transferred) dataset.
Key results
- MaRI reaches 26.0% Top-1 and 90.0% Top-5 instance accuracy on the Trained gallery, versus 8.5% and 16.0% for Make-it-Real.
- On the Unseen gallery MaRI achieves 54.0% Top-1 and 89.0% Top-5 instance accuracy, versus 42.5% and 75.0% for Make-it-Real.
- MaPa attains higher Top-3 IoU (0.80 vs 0.77 for MaRI) on the Trained set, while MaRI has the best Top-1 class accuracy (81.5%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the synthetic training data from 25% to 100% (ablation) | MaRI · Trained | T5I: 55.5% -> 90.0% | Helped |
| Increase the synthetic training data from 25% to 100% (ablation) | MaRI · Unseen | T1I: 44.5% -> 54.0% | Helped |
| Increase synthetic data from 50% to 100% (ablation) | MaRI · Trained | T1C: 82.0% -> 81.5% | Mixed |
| Add real-world training data (ablation) | MaRI dual encoder · Trained | T1I: 20.5% -> 26.0% | Helped |
| Add synthetic training data (ablation) | MaRI dual encoder · Trained | T1I: 9.0% -> 26.0% | Helped |
| Train with both real-world and synthetic data (ablation) | MaRI dual encoder · Unseen | T1I: 35.0% / 44.0% -> 54.0% | Helped |
| Replace the single encoder with dual encoders (ablation) | MaRI with real and synthetic data · Unseen | T1I: 49.5% -> 54.0% | Helped |
| Fine-tune only the last Transformer block instead of all parameters (ablation) | MaRI with InfoNCE · Trained | T1I: 13.0% -> 26.0% | Helped |
| Use InfoNCE instead of Triplet loss (ablation) | MaRI, last-block-only fine-tuning · Trained | T1I: 5.5% -> 26.0% | Helped |
| Use InfoNCE instead of Triplet loss (ablation) | MaRI, last-block-only fine-tuning · Unseen | T1I: 38.5% -> 54.0% | Helped |
| Use MaRI instead of Make-it-Real | Material retrieval · Trained | T1I: 8.5% -> 26.0% | Helped |
| Use MaRI instead of MaPa | Material retrieval · Trained | T3IoU: 0.80 -> 0.77 | Hurt |
Surprising findings
- Fine-tuning only the last Transformer block outperforms fine-tuning more layers, and InfoNCE outperforms triplet loss.
- Training on real-world data alone performs poorly on the trained gallery (T1I 9.0%), while synthetic-only training is much stronger on it.
Limitations
- No directly comparable prior method exists, so baselines are adapted general image search models and GPT-4V-based pipelines.
- Top-3 IoU is lower than MaPa's on the trained gallery, and the approach can return category mismatches.
- Real-world data covers mainly 8 material categories.
Experimental setup
Datasets: Objaverse AmbientCG HDRI Haven Grounded-SAM segmented real-world images Textures.com (unseen gallery)
Benchmarks: Trained gallery (AmbientCG-derived, ~200 materials) Unseen gallery (Textures.com, ~200 materials)
Metrics: Top-1 instance accuracy Top-5 instance accuracy Top-1 class accuracy Top-3 IoU
Baselines: ViT DINOv2 CLIP Make-it-Real MaPa
Builds on: CLIP DINOv2 ZeST InfoNCE
Closest papers by meaning
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.86)
- MAGE : Single Image to Material-Aware 3D via the Multi-View G-Buffer Estimation Model (CVPR 2025 · similarity 0.85)
- MARBLE: Material Recomposition and Blending in CLIP-Space (CVPR 2025 · similarity 0.85)
- MatExpert: Decomposing Materials Discovery By Mimicking Human Experts (ICLR 2025 · similarity 0.84)
- MatSynth: A Modern PBR Materials Dataset (CVPR 2024 · similarity 0.84)
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.