AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

GneissWeb: Preparing High Quality Data for LLMs at Scale

Hajar Emami Gohari, Swanand Ravindra Kadhe, Yousaf Shah, Constantin M Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai and 20 more

ICLR 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train on GneissWeb instead of FineWeb-V1.1.01.4B model, 350B tokens · High-Signal avgaccuracy: 56.26 -> 58.40Improved
train on GneissWeb instead of FineWeb-V1.1.07B model, 350B tokens · High-Signal avgaccuracy: 64.61 -> 67.34Improved
train on GneissWeb instead of FineWeb-V1.1.03B model, 350B tokens · Extended avgaccuracy: 50.15 -> 52.1Improved
train on GneissWeb instead of DCLM-Baseline1.4B model, 350B tokens · High-Signal avgaccuracy: 61.36 -> 58.40Hurt
train on GneissWeb instead of FineWeb-Edu-Score-21.4B model, 350B tokens · World Knowledgeaccuracy: 42.24 -> 41.09Hurt
add exact substring deduplication to FineWeb (ablation)1.4B model · High-Signal avg, 350B tokensaccuracy: 55.99 -> 57.39Improved
add category-aware readability score filter (ablation)FineWeb-V1.1.0 · High-Signal avg, 35B tokensaccuracy: 51.94 -> 53.20Improved
add category-aware extreme-tokenized filter (ablation)FineWeb-V1.1.0 · High-Signal avg, 35B tokensaccuracy: 51.94 -> 52.78Improved
extend lenient thresholds to news, business, personal finance categories (ablation)readability/extreme-token filters · High-Signal avgaccuracyHurt
use GneissWeb ensemble rule instead of rule 4 (AND of fastText) (ablation)ensemble quality filter · High-Signal avg, 1.4B 35Baccuracy: 51.29 -> 51.66Improved
use GneissWeb ensemble rule instead of rule 1 (ablation)ensemble quality filter · High-Signal avg, 7B 100Baccuracy: 62.65 -> 63.09Improved
use ensemble rule 3 instead of FineWeb baseline (ablation)ensemble quality filter · High-Signal avg, 1.4B 35Baccuracy: 50.74 -> 51.06No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy