Research map / Vision-language and multimodal / Robot action and navigation

HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Memory and subgoal-based manipulation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HandEdit, a large-scale egocentric dataset (200M editing instances from five hand-object datasets) and two-track URDF-conditioned benchmark for editing human hands/arms into dexterous robot embodiments, evaluating 11 image editors with generic, VLM-based and embodiment-aware metrics.

Problem

Human egocentric hand videos are abundant but differ in appearance, articulation and viewpoint from dexterous robot hands, and there is no dedicated dataset, protocol, or benchmark for URDF-conditioned human-to-robot dexterous hand image editing.

Approach

Builds pseudo-ground-truth edited targets via hand segmentation (SAM3), background inpainting (ProPainter), retargeting to URDF joints, virtual-base arm placement, rendering/compositing and harmonization, and pairs this with a metric suite combining generic similarity, GPT-4o VLM judgment, and embodiment-aware metrics (skin-based removal, DINOv2 structure, CLIP/CIELAB identity, object-region LPIPS

Key results

• GPT-Image-2 achieves the best LPIPS(ROI) (0.482) and FID(ROI) (99.71) on Hand-only and highest structural fidelity (0.780) and interaction (0.703).
• GPT-Image-1.5 obtains the highest VLM score (0.765 Hand-only; 0.856 Hand-Arm) but lower interaction (0.435) than GPT-Image-2.
• FireRed-Image-Edit-1.1 has the highest PQ (0.754) yet a low VLM score (0.292) on Hand-only, showing perceptual quality alone does not ensure embodiment correctness.

Surprising findings

• Visually natural edits (high PQ, FireRed) can still have low semantic consistency and embodiment fidelity.
• VLM judgment and embodiment-aware metrics disagree: the top VLM scorer (GPT-Image-1.5) does not lead on structural fidelity or interaction preservation.

Limitations

• Targets are pseudo-GT composites rather than real paired human-robot images, leaving a gap in appearance and contact dynamics.
• Retargeting and IK filtering reject a large share of frames (about 33% non-kept on ARCTIC run) and arm base placement relies on a human sequence-level selection.
• Benchmark uses a single GPT-4o judge and a 1K-image test set per track.

Experimental setup

Datasets: EgoDex ARCTIC OakInk2 HOI4D HO-Cap

Benchmarks: HandEdit Hand-only track HandEdit Hand-Arm track

Metrics: PSNR SSIM LPIPS FID VLM score (SC/PQ) Human-hand removal Structural fidelity ID fidelity Interaction consistency

Baselines: GPT-Image-2 GPT-Image-1.5 Nano-Banana Nano-Banana-2 Seedream-4.5 FLUX-2-Pro FLUX-Kontext-Max Hunyuan-Image-3.0 Qwen-Image-Edit-2511 FireRed-Image-Edit-1.1 OmniGen2

Builds on: SAM 3 ProPainter Harmonizer DINOv2 CLIP

Closest papers by meaning

• ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.89)
• MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.89)
• StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video (NeurIPS 2026 · similarity 0.89)
• VinT-6D: A Large-Scale Object-in-hand Dataset from Vision, Touch and Proprioception (ICML 2024 · similarity 0.88)
• EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.88)
• FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.88)