Research map / Large language models / Reasoning and chain of thought
ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsistentEE, an RL-based early exiting method where each instance needs to be correctly predicted by only one internal classifier, with a hardness-aware reward based on a "memorized layer" concept.
Problem
Early exiting training objectives require all internal classifiers to predict all instances correctly, while inference only needs one correct exit, creating a train-inference mismatch that hurts shallow classifiers and efficiency-accuracy trade-offs.
Approach
Formulates early exiting as a policy-gradient RL problem where a policy network decides exit or continue per layer, so only the exiting classifier incurs loss; the reward's accuracy-acceleration trade-off is scaled per instance by a memorized-layer hardness measure (the layer from which an instance stays correctly classified until the final layer).
Key results
- On dev sets, ConsistentEE saves 34% of layers averaged across datasets with no accuracy loss (1.54x speedup) for BERT-Base.
- With ~1% accuracy tolerance, ConsistentEE saves 51% of layers on average, outperforming the best baseline BERxiT by 16%.
- On generation with LLaMA backbones, ConsistentEE outperforms CALM at speedups above 2x on Alpaca/Dolly and consistently on CNN/DM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| RL-based early exiting (ConsistentEE) replacing weighted-sum CE training | BERT-Base early exiting · GLUE dev, 8 datasets | averaged score at no accuracy loss: 86.9 vs 86.4 BERT-Base (with -34% layers) | Helped |
| Hardness-guided reward (memorized layer) vs vanilla reward (ablation) | ConsistentEE reward function · ablation under varied alpha (GLUE) | accuracy vs speedup trade-off | Helped |
| Hardness-guided reward uses memorized layer, instead of fixed trade-off coefficient | generation (LLaMA with CALM comparison) · Alpaca/Dolly, CNN/DM | Rouge-L / BERT-F vs speedup | No clear effect |
| ConsistentEE vs CALM for generation | LLaMA backbones · Alpaca/Dolly | Rouge-L/BERT-F at higher speedup | Mixed |
| Larger trade-off alpha (ablation) | ConsistentEE · GLUE dev hyperparameter sweep | speedup and accuracy: accuracy varies only 4% over alpha range | Mixed |
| ConsistentEE vs PABEE/BERxiT on ALBERT-Base/BERT-Large/ALBERT-Large | PLM backbones (RTE, MRPC, SST-2) · test set | accuracy/F1 with saved layers | Helped |
Surprising findings
- Under the traditional objective, the first three internal classifiers of BERT on RTE fall below the 50% random-guess baseline.
- Under ConsistentEE, accuracy at some shallow layers is not available because no training samples exit there on RTE.
Limitations
- Hardness-guided reward not used for LLM generation experiments due to cost of computing memorized layer over tokens.
- Gains over baselines are modest on some datasets; RTE accuracy gaps are small.
- Evaluation on LLMs limited to LLaMA-7b/13b with limited compute, LoRA-trained backbone initialization only.
Experimental setup
Datasets: RTE MRPC SST-2 QNLI QQP MNLI M-CID StackOverflow Alpaca Dolly CNN/DM
Benchmarks: RTE MRPC SST-2 QNLI QQP MNLI M-CID StackOverflow Dolly CNN/DM
Metrics: accuracy F1 saved layers saved runtime Rouge-L BERTScore F1
Baselines: DeeBERT PABEE BERxiT Right-Tool PCEE-BERT HashEE TR-BERT CALM
Builds on: DeeBERT PABEE BERxiT unforgettable examples CALM
Closest papers by meaning
- Optimizing Language Models for Inference Time Objectives using Reinforcement Learning (ICML 2025 · similarity 0.89)
- COSEE: Consistency-Oriented Signal-Based Early Exiting via Calibrated Sample Weighting Mechanism (AAAI 2025 · similarity 0.89)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.88)
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.87)
- Reinforcement Mid-Training (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.