ICML 2024 accepted papers on Large language models

310 of the 2,610 papers accepted at ICML 2024 (11.9%) are on Large language models, compared with 2.4% at ICML 2023. They fall into 7 topics.

See ICML 2024 in the interactive map

Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.

Topics and papers

Language, safety and interpretability · 100 papers

Improving Factuality and Reasoning in Language Models through Multiagent Debate (94 citations)
Position: Will we run out of data? Limits of LLM scaling based on human-generated data (83 citations)
Position: On the Possibilities of AI-Generated Text Detection (53 citations)
A Tale of Tails: Model Collapse as a Change of Scaling Laws (39 citations)
Magicoder: Empowering Code Generation with OSS-Instruct (26 citations)
Copyright Traps for Large Language Models (24 citations)
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (16 citations)
In-Context Unlearning: Language Models as Few-Shot Unlearners (13 citations)
Large Language Models are Geographically Biased (13 citations)
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (13 citations)
Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains (13 citations)
InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (11 citations)

and 88 more in the interactive map.

Agents, code and math · 46 papers

Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? (21 citations)
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (17 citations)
Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (14 citations)
TravelPlanner: A Benchmark for Real-World Planning with Language Agents (14 citations)
Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks (11 citations)
BetterV: Controlled Verilog Generation with Discriminative Guidance (9 citations)
LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery (9 citations)
Structured Chemistry Reasoning with Large Language Models (8 citations)
AST-T5: Structure-Aware Pretraining for Code Generation and Understanding (7 citations)
DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning (7 citations)
OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models (7 citations)
CHEMREASONER: Heuristic Search over a Large Language Model’s Knowledge Space using Quantum-Chemical Feedback (6 citations)

and 34 more in the interactive map.

AI and society · 42 papers

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (70 citations)
Position: TrustLLM: Trustworthiness in Large Language Models (54 citations)
Position: Levels of AGI for Operationalizing Progress on the Path to AGI (53 citations)
Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? (21 citations)
Assessing Large Language Models on Climate Information (18 citations)
Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning (16 citations)
The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning (13 citations)
Position: A Safe Harbor for AI Evaluation and Red Teaming (11 citations)
Position: AI-Powered Autonomous Weapons Risk Geopolitical Instability and Threaten AI Research (5 citations)
Position: Explain to Question not to Justify (5 citations)
Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind (4 citations)
AI Control: Improving Safety Despite Intentional Subversion (3 citations)

and 30 more in the interactive map.

Instruction and fine-tuning · 36 papers

Cell2Sentence: Teaching Large Language Models the Language of Biology (62 citations)
Accurate LoRA-Finetuning Quantization of LLMs via Information Retention (42 citations)
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (19 citations)
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (15 citations)
LESS: Selecting Influential Data for Targeted Instruction Tuning (14 citations)
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (13 citations)
LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (13 citations)
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (11 citations)
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (7 citations)
CogBench: a large language model walks into a psychology lab (5 citations)
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (5 citations)
Privacy-Preserving Instructions for Aligning Large Language Models (5 citations)

and 24 more in the interactive map.

Reasoning and chain of thought · 33 papers

Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution (26 citations)
Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models (13 citations)
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator (12 citations)
Better & Faster Large Language Models via Multi-token Prediction (9 citations)
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (6 citations)
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution (5 citations)
Stay on Topic with Classifier-Free Guidance (5 citations)
Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (4 citations)
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (4 citations)
Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Sampling (4 citations)
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (3 citations)
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (3 citations)

and 21 more in the interactive map.

Alignment and preferences · 32 papers

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (103 citations)
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (69 citations)
Fundamental Limitations of Alignment in Large Language Models (45 citations)
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (25 citations)
ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (22 citations)
Self-Rewarding Language Models (10 citations)
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (8 citations)
Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback (8 citations)
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (7 citations)
AI Alignment with Changing and Influenceable Reward Functions (6 citations)
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (4 citations)
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation (4 citations)

and 20 more in the interactive map.

Retrieval and knowledge · 21 papers

How Language Model Hallucinations Can Snowball (71 citations)
Debating with More Persuasive LLMs Leads to More Truthful Answers (11 citations)
A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules? (10 citations)
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (7 citations)
Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (6 citations)
InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (6 citations)
Language Models as Science Tutors (6 citations)
MEMORYLLM: Towards Self-Updatable Large Language Models (4 citations)
C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models (3 citations)
Repoformer: Selective Retrieval for Repository-Level Code Completion (3 citations)
Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models (2 citations)
Larimar: Large Language Models with Episodic Memory Control (2 citations)

and 9 more in the interactive map.