Research map / Image generation / Text-to-image diffusion

PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Layout-to-image control

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PhysTacGen, which uses GTPO (GRPO-style RL with format, semantic and vocabulary rewards) to refine a VLM for material descriptions, plus DINOv2 pair curation and depth-conditioned SDXL ControlNet to synthesize optical tactile images from RGB.

Problem

Visual-to-tactile image synthesis is hard because visual appearance does not reflect contact compliance and real visual-tactile pairs are spatially misaligned, while tactile data is scarce and expensive.

Approach

Combine RL-refined (GTPO) physical material descriptions as text conditions with DINOv2-filtered pairs and monocular relative-depth as a 4-channel ControlNet condition on SDXL, with contact-aware loss reweighting.

Key results

• PhysTacGen reaches SSIM 0.964 and LPIPS 0.559 on the curated SSVTP split, versus 0.715 SSIM for the adapted VisGel baseline.
• In a blinded 22-participant forced choice, GTPO descriptions were preferred 57% of the time versus 27% for SFT and 16% for the classifier.
• On the heuristic force-coefficient proxy, RGB plus generated tactile reduced test MSE from 0.1455 to 0.1330 (real-tactile oracle 0.1310).

Surprising findings

• The 7-label classifier prompts achieve slightly lower LPIPS (0.557) than GTPO prompts (0.559) in the matched comparison.
• Mix-Fusion obtains the lowest legacy nFD (0.241), better than the full PhysTacGen (0.278).

Limitations

• Results limited to one curated SSVTP dataset with no cross-sensor generalization.
• Force-coefficient labels are heuristic proxies, so no measured-force or real robot transfer is shown.
• Uses legacy nFD rather than standard FID; DINO filtering before split shapes the test distribution.

Experimental setup

Datasets: SSVTP

Benchmarks: SSVTP curated split 452-pair description benchmark

Metrics: SSIM LPIPS legacy nFD SES BLEU ROUGE PPL physics plausibility score human preference test MSE

Baselines: Pix2Pix Pix2Pix-Turbo VisGel SingleS2R 7-label classifier SFT Qwen

Builds on: GRPO SDXL ControlNet DINOv2 Depth-Anything-V2

Closest papers by meaning

• Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation (NeurIPS 2024 · similarity 0.89)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation (NeurIPS 2025 · similarity 0.89)
• Tactile-Augmented Radiance Fields (CVPR 2024 · similarity 0.88)
• AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.88)
• EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.88)
• Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.88)