Research map / Large language models / AI and society
Position: Challenges and Future Directions of Data-Centric AI Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing for data-centric AI alignment, using a qualitative re-annotation of 160 Anthropic-HH samples to identify six sources of human-feedback unreliability and outlining future directions for data collection, cleaning, and AI-feedback verification.
Problem
Alignment methods assume preference data reflects true human preferences, but human feedback is unreliable, non-representative, temporally drifting and context-dependent, and AI-generated feedback inherits model limitations and biases.
Approach
Shifts emphasis from algorithms and loss functions to the quality and representativeness of alignment feedback data, categorizing six sources of human-feedback unreliability through qualitative re-labeling and proposing data collection, cleaning, and verification directions.
Key results
- Re-annotation of 160 Anthropic-HH samples (80 helpful, 80 harmless) by three annotators yielded Fleiss's kappa of 0.46 (moderate agreement).
- About 25% of re-annotated samples were labeled 'rejected is better' (contradicting original labels) and another 25% 'both are bad'.
- Six sources of unreliability were identified; low-IAA samples were dominated by different thresholds of criteria (37%) and different preference criteria (29%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add “both are good” and “both are bad” options to preference labeling | Human preference data collection · Preference labeling tasks | reliability of human feedback | Helped |
| Clean feedback data with a committee of reward models and filter low-agreement data | Models trained on human feedback · Alignment training | model performance | Helped |
Surprising findings
- About 25% of re-annotated Anthropic-HH samples contradicted the original labels, with another 25% judged as 'both are bad'.
- Harmful suggestions and misinformation in both responses account for most of the 'both are bad' cases.
Limitations
- Qualitative analysis covers only 160 samples from one dataset (Anthropic-HH) with three lab-member annotators.
- Proposed future directions are not empirically validated.
- Position-style paper with mostly literature-based arguments for AI-feedback challenges.
Experimental setup
Datasets: Anthropic-HH
Metrics: Fleiss's kappa proportion of unreliability sources
Builds on: RLHF DPO Anthropic-HH
Closest papers by meaning
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.90)
- Measuring Error Alignment for Decision-Making Systems (AAAI 2025 · similarity 0.88)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.88)
- MAP: Multi-Human-Value Alignment Palette (ICLR 2025 · similarity 0.87)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.87)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.