Research map / Large language models / Alignment and preferences
Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GazeReward, which feeds eye-tracking features predicted by frozen ET generator models into a QLoRA-tuned reward model alongside text, improving RM accuracy on OASST1 and HelpSteer2.
Problem
Reward models for LLM human alignment learn human preferences from text alone and miss implicit behavioral signals; real eye-tracking data is costly and unavailable at inference time.
Approach
Augment the reward model with eye-tracking features predicted by pretrained, frozen gaze-prediction models from the prompt-response text, integrated either by concatenating special-token-delimited gaze embeddings (GazeConcat) or by elementwise addition (GazeAdd).
Key results
- On OASST1, GazeConcat with fcomb_2.2 on Llama-3-8B reaches 71.2% vs 65.5% baseline (about 8.8% relative gain), and Mistral-7B reaches 71.0% vs 66.3%.
- On HelpSteer2, GazeAdd fcomb_1 raises Llama-3-8B accuracy to 62.4% vs 53.3% baseline (17% relative gain).
- On RewardBench (OASST1-trained), Mistral-7B accuracy rises from 41.2% to 60.5% with GazeConcat fcomb_2.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add eye-tracking features to the reward model using GazeConcat or GazeAdd | Llama-3-8B-Instruct reward model · OASST1 | accuracy: 65.9 $\pm$ 0.5${}$ -> 70.2 $\pm$ 0.3${**}$ | Helped |
| Add eye-tracking features to the reward model using GazeConcat or GazeAdd | Llama-3-8B reward model · OASST1 | accuracy: 65.5 $\pm$ 2.1${}$ -> 71.5 $\pm$ 0.5${}$ | Helped |
| Add eye-tracking features to the reward model using GazeConcat | Mistral-7B reward model · OASST1 | accuracy: 66.3 $\pm$ 0.1${}$ -> 71.0 $\pm$ 1.0${}$ | Helped |
| Add eye-tracking features to the reward model using GazeAdd | Llama-3-8B-Instruct reward model · HelpSteer2 | accuracy: 54.7 $\pm$ 0.7${}$ -> 62.3 $\pm$ 0.6${**}$ | Helped |
| Add eye-tracking features to the reward model using GazeAdd | Llama-3-8B reward model · HelpSteer2 | accuracy: 53.3 $\pm$ 0.8${}$ -> 62.4 $\pm$ 1.0${**}$ | Helped |
| Add eye-tracking features to the reward model using GazeConcat | Mistral-7B reward model · HelpSteer2 | accuracy: 54.1 $\pm$ 0.3${}$ -> 58.7 $\pm$ 2.4${}$ | Helped |
| Add eye-tracking features using GazeConcat | Llama-3-8B-Instruct reward model trained on OASST1 · RewardBench | accuracy: 46.9 -> 58.4 | Helped |
| Add eye-tracking features using GazeConcat | Llama-3-8B reward model trained on OASST1 · RewardBench | accuracy: 50.9 -> 58.5 | Helped |
| Add eye-tracking features using GazeConcat | Mistral-7B reward model trained on OASST1 · RewardBench | accuracy: 41.2 -> 60.5 | Helped |
| Use TRT alone from eye-tracking predictor 1 instead of five features or TRT+FFD (ablation) | GazeReward · OASST1 and HelpSteer2 | accuracy | Mixed |
| Use five features (nFix, FFD, GPT, TRT, fixProp) instead of TRT+FFD (ablation) | GazeReward with GazeConcat · OASST1 and HelpSteer2 | accuracy | Mixed |
| Use GazeAdd instead of GazeConcat (ablation) | GazeReward · OASST1 and HelpSteer2 | accuracy | Mixed |
Surprising findings
- Some gaze feature combinations (fcomb_2.5) performed worse than the text-only baseline on certain settings.
- The ET-feature gains on RewardBench were large (over 40% relative for Mistral-7B), though the authors attribute part of the baseline weakness to small base models and datasets.
Limitations
- ET prediction models were trained on small English-only corpora not tailored to LLM responses.
- Only two integration methods were explored and GazeAdd could not be applied to Mistral due to tokenizer mapping issues.
- Experiments use PEFT on 7-8B models and small datasets; scaling to larger models is untested.
Experimental setup
Datasets: OASST1 HelpSteer2 ZuCo 1 ZuCo 2 PROVO Dundee GECO
Benchmarks: OASST1 test HelpSteer2 test RewardBench
Metrics: preference accuracy (%)
Baselines: text-only reward model (same backbone, no ET features)
Builds on: Bradley-Terry reward modeling QLoRA RLAIF DPO
Closest papers by meaning
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.90)
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.89)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.89)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.89)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.88)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.