AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika and 1 more

ICML 2023 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train on the near-deduplicated Pile instead of the original Pile (ablation)Pythia · Language modeling benchmarksevaluation performanceNo clear effect
Use parallel attention and feedforward sublayers at all model scalesPythia · NLP benchmarksperformance relative to OPTNo clear effect
Use batch sizes 4× to 8× standard for models under 1 billion parametersPythia · Training runs for models under 1 billion parametersconvergenceNo clear effect
Use larger batch sizes for smaller modelsPythia · Training time for smaller modelswall-clock speed-up: 10$\times$Improved
Replace masculine pronouns with feminine counterparts in the final 7% or 21% of trainingPythia-deduped · WinoBiasstereotypical accuracyImproved
Replace masculine pronouns with feminine counterparts in the final 7% or 21% of trainingPythia-deduped · CrowS-Pairs gender% StereotypeImproved
Replace masculine pronouns with feminine counterparts in the final 7% or 21% of trainingPythia-deduped · LAMBADAperplexityImproved
Place sequences at different positions in trainingPythia · Training corpus memorizationmemorized-sequence occurrence rateNo clear effect
Increase model size to 2.8 billion parameters or more and continue training past 65,000 stepsPythia-deduped · Arithmetic and TriviaQAcorrelation of accuracy with relevant term frequency: 65,000 training stepsImproved
Train on a second epoch of the deduplicated PilePythia-deduped · Language-modeling benchmarksevaluation scoresNo clear effect

Later papers that compared with this method

2 later accepted papers report Pythia-12B as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy