Research map / Vision-language and multimodal / CLIP and zero-shot
SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemTra, a two-level (task and skill) hierarchical framework that uses a pretrained language model and multi-modal skill encoders to translate interleaved multi-modal task prompts into semantic skill sequences and parametric skills for zero-shot cross-domain policy adaptation.
Problem
Zero-shot policy adaptation to new domains for long-horizon tasks specified by interleaved multi-modal task prompts (video, sensor data, text), where the target domain differs in subtask order or skill-level contexts such as wind and speed.
Approach
Decomposes cross-domain adaptation into task-level seq-to-seq translation of extracted semantic skills by a PLM (GPT-2) and skill-level parametric instantiation (domain factor and magnitude) conditioned on domain contexts, with a behavior decoder conditioned on a single executable skill plus an online context encoder.
Key results
- On Franka Kitchen, SemTra reaches 66.2% N-rate cross-domain, 36.34% higher than the best baseline VCP.
- On Meta-World, SemTra reaches 79.3% N-rate cross-domain, 66.24% higher than the best baseline TP-GPT.
- Using only 50 annotated videos per semantic skill (about 0.23% of the dataset) gives N-rate of 71.1% versus 72.7% with every-timestep annotations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SemTra instead of the strongest baseline VCP | Franka Kitchen cross-domain adaptation · Franka Kitchen cross-domain | N-rate: 29.8 -> 66.2 | Helped |
| use SemTra instead of TP-GPT | Meta-World cross-domain adaptation · Meta-World cross-domain | N-rate: 13.1 -> 79.3 | Helped |
| translate task instructions into semantic skill sequences with PLM-based task adaptation | SemTra · Franka Kitchen task-level reverse | N-rate: 50.9 -> 71.3 | Helped |
| translate task instructions into semantic skill sequences with PLM-based task adaptation | SemTra · Meta-World task-level reverse | N-rate: 8.3 -> 76.3 | Helped |
| condition behavior decoding on language-space rather than video-space skill embeddings (ablation) | SemTra behavior decoder · Franka Kitchen behavior decoding | N-rate: 19.4 -> 40.2 | Helped |
| condition behavior decoding on language-space rather than video-space skill embeddings (ablation) | SemTra behavior decoder · Meta-World behavior decoding | N-rate: 8.1 -> 11.8 | Helped |
| use retrieved video semantics rather than conditioning only on the multimodal task prompt (ablation) | SemTra behavior decoder · Franka Kitchen behavior decoding | N-rate: 4.1 -> 19.4 | Helped |
| use retrieved video semantics rather than conditioning only on the multimodal task prompt (ablation) | SemTra behavior decoder · Meta-World behavior decoding | N-rate: 2.7 -> 8.1 | Helped |
| generate a semantic skill sequence rather than conditioning on the unsegmented entire executable sequence (ablation) | SemTra behavior decoder · Franka Kitchen behavior decoding | N-rate: 42.8 -> 72.2 | Helped |
| generate a semantic skill sequence rather than conditioning on the unsegmented entire executable sequence (ablation) | SemTra behavior decoder · Meta-World behavior decoding | N-rate: 34.3 -> 76.1 | Helped |
| use 50 annotated videos per semantic skill instead of annotations at every timestep (ablation) | SemTra · Cross-domain adaptation | N-rate: 72.7 -> 71.1 | No clear effect |
Surprising findings
- GPT-3 and GPT-3.5 struggle to reverse skill sequences in zero-shot use, while GPT-4 and PaLM perform better.
- In the RLBench use case, GPT-3.5 scored lower than GPT-3 on the summarized-sentence command (53.3 vs 76.6).
Limitations
- Evaluation relies on simulated domains with synthetically constructed domain contexts (wind, speed) and rule-based expert policies.
- Skill sequence generation is not dynamically adjusted to real-time observations, as acknowledged as future work.
Experimental setup
Datasets: D4RL Franka Kitchen offline dataset Meta-World multi-stage expert trajectories RLBench semantically annotated trajectories (460)
Benchmarks: Franka Kitchen Meta-World RLBench CARLA
Metrics: K-rate N-rate normalized return
Baselines: VIMA TP-GPT VCP TP-BCZ SayCan
Builds on: V-CLIP Decision Transformer VIMA SayCan BCZ
Closest papers by meaning
- In-Context Policy Adaptation via Cross-Domain Skill Diffusion (AAAI 2025 · similarity 0.89)
- LLM-based Skill Diffusion for Zero-shot Policy Adaptation (NeurIPS 2024 · similarity 0.89)
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.88)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.87)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.87)
- Hypothesis, Verification, and Induction: Grounding Large Language Models with Self-Driven Skill Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.