Research map / Probabilistic and scientific modeling / Molecules and proteins
QueST: Self-Supervised Skill Abstractions for Learning Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QueST, a causal FSQ-quantized skill autoencoder with a GPT-style skill prior that learns discrete temporal action abstractions for multitask and few-shot robot imitation.
Problem
Generalization of robot policies to new tasks in low-data regimes; existing discrete latent skill models (VQ-BeT, PRISE) have architectures that don't capture transferable, temporally structured action abstractions.
Approach
An encoder with causal strided convolutions and masked self-attention maps an action sequence to multiple FSQ-quantized skill tokens, and a decoder cross-attends fixed sinusoidal queries to those tokens, so the latent sequence is causal yet decoder sees all codes; a transformer prior autoregressively predicts skill tokens from observations and task embedding.
Key results
- On LIBERO-90 multitask, QueST reaches 88.6% success, outperforming VQ-BeT and Diffusion Policy by 8 and 13 points.
- In 5-shot LIBERO-LONG transfer, QueST surpasses all baselines by an absolute margin of about 14% (68.8% with decoder finetuning).
- Ablations show FSQ over VQ, decoder cross-attention, and causal encoder each improve performance (e.g., few-shot 68.8 vs 62.5 for VQ).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace FSQ with VQ (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 88.6 \pm 0.4 -> 81.2 \pm 0.6; 68.8 \pm 1.7 -> 62.5 \pm 2.0 | Hurt |
| condition the decoder on observations (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 88.6 \pm 0.4 -> 81.9 \pm 1.1; 68.8 \pm 1.7 -> 61.3 \pm 2.2 | Hurt |
| replace cross-attention decoder with a mirrored decoder (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 88.6 \pm 0.4 -> 86.3 \pm 0.9; 68.8 \pm 1.7 -> 45.4 \pm 2.0 | Hurt |
| add causal masking to the encoder (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 82.0 \pm 1.6 -> 88.6 \pm 0.4; 58.8 \pm 3.0 -> 68.8 \pm 1.7 | Helped |
| add causal masking to the decoder (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 85.1 \pm 1.8 -> 88.6 \pm 0.4; 61.6 \pm 2.5 -> 68.8 \pm 1.7 | Helped |
| add causal masking to both encoder and decoder (ablation) | QueST · LIBERO-90 and LIBERO few-shot | success rate: 78.5 \pm 0.5 -> 88.6 \pm 0.4; 56.1 \pm 1.8 -> 68.8 \pm 1.7 | Helped |
| finetune the decoder during few-shot learning (ablation) | QueST · LIBERO few-shot | success rate: 66.0 \pm 3.6 -> 68.8 \pm 1.7 | Helped |
| increase decoder loss scale from 10 to 100 (ablation) | QueST · LIBERO few-shot | success rate: 68.8 \pm 1.7 -> 66.0 \pm 1.0 | Hurt |
| use QueST rather than the next-best baseline | multitask imitation learning · LIBERO-90 | success rate: +8% | Helped |
| use QueST rather than the next-best baseline | few-shot imitation learning · LIBERO-LONG | success rate: 14% | Helped |
| finetune the decoder during few-shot learning | QueST · LIBERO-LONG | success rate: 1.5% | Hurt |
| use QueST rather than other methods | multitask imitation learning · MetaWorld ML45 | success rate | No clear effect |
Surprising findings
- Conditioning the action decoder on observations hurt performance (LIBERO-90 81.9 vs 88.6).
- Freezing the decoder in few-shot transfer reduced performance only marginally (68.8 to 66.0).
Limitations
- Held-out tasks are structurally similar to training tasks, so few-shot transfer may fail for more diverse tasks.
- Only causality is used as an inductive bias.
- Evaluated only in simulation benchmarks (LIBERO, MetaWorld).
Experimental setup
Datasets: LIBERO-90 LIBERO-LONG MetaWorld ML45
Benchmarks: LIBERO-90 LIBERO-LONG MetaWorld ML45
Metrics: success rate
Baselines: ResNet-T Diffusion Policy ACT VQ-BeT PRISE
Builds on: Finite Scalar Quantization VQ-BeT PRISE Behavior Transformer
Closest papers by meaning
- Learning transferable motor skills with hierarchical latent mixture policies (ICLR 2022 · similarity 0.90)
- STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization (ICML 2025 · similarity 0.89)
- PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control (ICML 2024 · similarity 0.89)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.88)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.88)
- CCIL: Continuity-Based Data Augmentation for Corrective Imitation Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.