Research map / Architectures and efficiency / RNNs and normalization
PALBERT: Teaching ALBERT to Ponder
Nikita Balagansky, Daniil Gavrilov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sampled exit-layer inference with deterministic Q-exit (ablation) | PonderNet adapted to ALBERT · SST-2, RTE, CoLA | task score: 91.1 ± 0.6 -> 92.2 ± 0.3; 73.5 ± 1.9 -> 77.3 ± 1.4; 50.8 ± 2.2 -> 55.7 ± 0.9 | Improved |
| Use closed-form expectation instead of sampling the exit layer (ablation) | PonderNet adapted to ALBERT · SST-2, RTE, CoLA | task score: 91.1 ± 0.6 -> 92.3 ± 0.4; 73.5 ± 1.9 -> 76.8 ± 3.0; 50.8 ± 2.2 -> 55.9 ± 2.2 | Improved |
| Fine-tune the Lambda layer with its own learning rate (ablation) | PALBERT with Q-exit · SST-2 | accuracy: 92.2 ± 0.3 -> 92.7 ± 0.4 | Improved |
| Fine-tune the Lambda layer with its own learning rate (ablation) | PALBERT with Q-exit · RTE | accuracy: 77.3 ± 1.4 -> 77.3 ± 1.4 | No clear effect |
| Fine-tune the Lambda layer with its own learning rate (ablation) | PALBERT with Q-exit · CoLA | Matthews correlation: 55.7 ± 0.9 -> 56.5 ± 1.2 | Improved |
| Use a 3-layer rather than 1-layer Lambda MLP (ablation) | PALBERT with Lambda LR · SST-2, RTE, CoLA | task score: 92.7 ± 0.4 -> 92.6 ± 0.3; 77.3 ± 1.4 -> 77.0 ± 1.4; 56.5 ± 1.2 -> 56.3 ± 2.4 | Mixed |
| Concatenate the current and previous hidden states for Lambda input (ablation) | PALBERT with Q-exit and Lambda LR · SST-2, RTE, CoLA | task score: 92.7 ± 0.4 -> 93.0 ± 0.3; 77.3 ± 1.4 -> 76.5 ± 1.6; 56.5 ± 1.2 -> 56.9 ± 1.9 | Mixed |
| Use Q-exit and the revised Lambda architecture instead of sampled PonderNet | ALBERT early-exit models · GLUE test | Macro: 77.2 -> 80.6 | Improved |
| Use PALBERT instead of ALBERT PABEE | ALBERT early-exit models · GLUE test | Macro: 79.3 -> 80.6 | Improved |
| Use PRoBERTa instead of RoBERTa PABEE | RoBERTa early-exit models · GLUE test | Macro: 81.3 -> 82.1 | Improved |
| Increase the Q-exit threshold above 0.5 (ablation) | PALBERT · GLUE dev tasks | task score | Hurt |
| Use a smaller Q-exit threshold (ablation) | PALBERT · GLUE tasks | inference speed | Improved |
Closest papers by meaning
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.86)
- Switchable Decision: Dynamic Neural Generation Networks (ICML 2024 · similarity 0.85)
- PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space (ICML 2026 · similarity 0.85)
- Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.85)
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.85)
- AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
