Research map / Large language models / Alignment and preferences
ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference
Sanjana Srivastava, Kangrui Wang, Yung-Chieh Chan, Tianyuan Dai, Manling Li, Ruohan Zhang, Mengdi Xu, Jiajun Wu, Li Fei-Fei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Preference Grounding module (vs passing raw preference to Staging) (ablation) | ROSETTA · context-dependent preferences | alignment (satisfaction) | Improved |
| add Staging module (vs grounded preference directly to Coding) (ablation) | ROSETTA · all preference types | alignment | Improved |
| add domain-knowledge verification follow-up questions (ablation) | ROSETTA · goal-related, behavioral, preferential preferences | alignment | Improved |
| add follow-up verification questions (ablation) | ROSETTA · corrective preferences | alignment | No clear effect |
| add Preference Grounding module (ablation) | ROSETTA · optimizability (success rate) | success rate >50% | Hurt |
| use open-source Qwen models instead of gpt-4o/o1-mini | ROSETTA · mean best reward | mean best reward: 0.623 -> 0.741 | Improved |
| paraphrase prompts with gpt-4o | ROSETTA · mean best reward | mean best reward: 0.623 -> 0.658 | Mixed |
| paraphrase prompts with Gemini 2.5 | ROSETTA · mean best reward | mean best reward: 0.623 -> 0.584 | Hurt |
| add iterative error correction to generated reward code | ROSETTA · generated reward functions that run | % running: 79.35% -> 95.38% | Improved |
| increase preference iteration from 1 to 4 | ROSETTA · human preference sequences | success rate: 94.6 -> 78.1 | Mixed |
Closest papers by meaning
- Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data (NeurIPS 2025 · similarity 0.90)
- REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.90)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.89)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.89)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Self-Rewarding Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
