ICLR 2024 accepted papers on Large language models

Language, safety and interpretability · 121 papers

• A Simple and Effective Pruning Approach for Large Language Models (53 citations)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (52 citations)
• Language Models Represent Space and Time (42 citations)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (37 citations)
• Efficient Streaming Language Models with Attention Sinks (36 citations)
• Large Language Models Cannot Self-Correct Reasoning Yet (36 citations)
• Can LLM-Generated Misinformation Be Detected? (34 citations)
• The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (31 citations)
• BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (30 citations)
• Language Modeling Is Compression (28 citations)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (28 citations)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (27 citations)

and 109 more in the interactive map.

Agents, code and math · 65 papers

• Large Language Models as Optimizers (93 citations)
• Teaching Large Language Models to Self-Debug (77 citations)
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (57 citations)
• DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models (39 citations)
• LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models (32 citations)
• Let's Verify Step by Step (31 citations)
• Learning Performance-Improving Code Edits (23 citations)
• Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (22 citations)
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (21 citations)
• Large Language Models as Tool Makers (20 citations)
• Domain-Agnostic Molecular Generation with Chemical Feedback (17 citations)
• Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification (17 citations)

and 53 more in the interactive map.

Instruction and fine-tuning · 53 papers

• WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (105 citations)
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct (81 citations)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (40 citations)
• Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models (40 citations)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (39 citations)
• Fine-Tuned Language Models Generate Stable Inorganic Materials as Text (35 citations)
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (29 citations)
• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (28 citations)
• OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (27 citations)
• UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (25 citations)
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (21 citations)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (20 citations)

and 41 more in the interactive map.

Retrieval and knowledge · 38 papers

• Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation (47 citations)
• Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (40 citations)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models (33 citations)
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (28 citations)
• KoLA: Carefully Benchmarking World Knowledge of Large Language Models (24 citations)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (22 citations)
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context (21 citations)
• Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources (18 citations)
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning (15 citations)
• Retrieval meets Long Context Large Language Models (15 citations)
• Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (12 citations)
• RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (12 citations)

and 26 more in the interactive map.

Alignment and preferences · 24 papers

• Eureka: Human-Level Reward Design via Coding Large Language Models (48 citations)
• Chain of Hindsight aligns Language Models with Feedback (27 citations)
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (22 citations)
• GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (14 citations)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets (9 citations)
• Human Feedback is not Gold Standard (7 citations)
• RAIN: Your Language Models Can Align Themselves without Finetuning (7 citations)
• RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment (4 citations)
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (3 citations)
• Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (2 citations)
• Language Model Self-improvement by Reinforcement Learning Contemplation (2 citations)
• Tool-Augmented Reward Modeling (2 citations)

and 12 more in the interactive map.

Reasoning and chain of thought · 15 papers

• Large Language Models as Analogical Reasoners (13 citations)
• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning (12 citations)
• BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models (8 citations)
• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (7 citations)
• ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (7 citations)
• Amortizing intractable inference in large language models (4 citations)
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation (4 citations)
• Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning (3 citations)
• CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (2 citations)
• Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (2 citations)
• Think before you speak: Training Language Models With Pause Tokens (2 citations)
• BatchPrompt: Accomplish more with less (1 citation)

and 3 more in the interactive map.

AI and society · 12 papers

• The Alignment Problem from a Deep Learning Perspective (63 citations)
• Listen, Think, and Understand (18 citations)
• The Generative AI Paradox: “What It Can Create, It May Not Understand” (11 citations)
• GAIA: a benchmark for General AI Assistants (10 citations)
• BatteryML: An Open-source Platform for Machine Learning on Battery Degradation (8 citations)
• Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (6 citations)
• Quality-Diversity through AI Feedback (6 citations)
• SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (6 citations)
• OMNI: Open-endedness via Models of human Notions of Interestingness (4 citations)
• Context is Environment
• CircuitNet 2.0: An Advanced Dataset for Promoting Machine Learning Innovations in Realistic Chip Design Environment
• On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs