Research map / Point clouds / Point cloud classification
Physical Property Understanding from Language-Embedded Feature Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeRF2Physics, a training-free method that fuses CLIP features into a NeRF-derived point cloud and uses LLM-proposed materials with zero-shot CLIP kernel regression to densely predict physical properties such as mass, friction, and hardness.
Problem
Estimating dense, object-level physical properties (mass density, mass, friction, hardness) from a collection of images without labeled ground-truth data, in the open world.
Approach
Combine a language-embedded NeRF point cloud (patch-level CLIP features averaged per source point) with LLM-proposed material dictionaries containing property values, and predict per-point properties via zero-shot CLIP-similarity kernel regression, with LLM-estimated thickness for volumetric mass integration.
Key results
- On ABO-500 test mass estimation, NeRF2Physics achieves ADE 8.730 kg and MnRE 0.552, outperforming Image2mass (ADE 12.496, MnRE 0.341), 2D CNN, and LLaVA.
- For friction on the in-house dataset, the method reaches ADE 0.155 and PRA 0.800 versus GPT-4V's 0.209 and 0.467.
- For Shore hardness, it achieves the highest PRA (0.710) versus GPT-4V (0.609) and CLIP (0.647), though with similar ADE/MnRE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeRF2Physics instead of the strongest baseline for mass estimation | NeRF2Physics · ABO-500 test | MnRE: 0.362 -> 0.552 | Helped |
| use NeRF2Physics instead of CLIP global features | NeRF2Physics · In-house hardness | PRA: 0.647 -> 0.710 | Helped |
| use NeRF2Physics instead of GPT-4V | NeRF2Physics · In-house kinetic friction | PRA: 0.533 -> 0.800 | Helped |
| add LLM-estimated thickness rather than integrate occupied voxels from depth-based carving (ablation) | NeRF2Physics · ABO-500 val | ADE: 18.587 -> 9.786 | Helped |
| use kernel regression instead of retrieving only the most likely material (ablation) | NeRF2Physics · ABO-500 val | ADE: 12.266 -> 9.786 | Helped |
| use fused patch CLIP embeddings instead of one global CLIP embedding (ablation) | NeRF2Physics · ABO-500 val | ADE: 10.396 -> 9.786 | Helped |
Surprising findings
- The paper reports that GPT-4V performs similarly to the uniform CLIP baseline on friction, suggesting difficulty distinguishing surfaces within a scene.
- Retrieval-only (temperature to zero) performs worse than kernel regression on mass estimation due to inherent material uncertainty.
Limitations
- Boundaries between parts are not localized precisely and similar materials (e.g., stainless steel and aluminum) get mixed.
- Internal object parts that are not visible cannot be reasoned about from surface features; mass relies on LLM thickness estimates.
- Evaluation of dense properties uses a small in-house dataset (6 friction points, 31 hardness points).
Experimental setup
Datasets: ABO-500 (subset of Amazon Berkeley Objects) in-house real-world friction and hardness dataset
Benchmarks: ABO-500 in-house friction and hardness dataset
Metrics: ADE ALDE APE MnRE PRA
Baselines: Image2mass 2D CNN LLaVA GPT-4V CLIP (global embedding)
Builds on: Nerfacto CLIP BLIP-2 Image2mass
Closest papers by meaning
- Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties (NeurIPS 2023 · similarity 0.88)
- Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.84)
- A General Protocol to Probe Large Vision Models for 3D Physical Understanding (NeurIPS 2024 · similarity 0.83)
- Physically Compatible 3D Object Modeling from a Single Image (NeurIPS 2024 · similarity 0.83)
- GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation (NeurIPS 2024 · similarity 0.83)
- ContPhy: Continuum Physical Concept Learning and Reasoning from Videos (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.