Research map / Architectures and efficiency / Quantization
Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CALR instead of AdaGrad | LSTM on SST2 · SST2 test | accuracy: 84.90 -> 86.03 | Helped |
| Use CALR instead of CLR (ablation) | LSTM on SST2 · SST2 test | accuracy: 84.71 -> 86.03 | Helped |
| Use CALR instead of AdaGrad | DAN-RAND on SST2 · SST2 test | accuracy: 83.20 -> 84.61 | Helped |
| Use CALR instead of CLR (ablation) | DAN-RAND on SST2 · SST2 test | accuracy: 83.84 -> 84.61 | Helped |
| Add post-SVD retraining of the embedding factors | LSTM, 90% reduction · SST2 test | accuracy: 82.54 -> 85.11 | Helped |
| Add post-SVD retraining of the embedding factors | DAN-RAND, 90% reduction · SST2 test | accuracy: 82.59 -> 83.11 | Helped |
| Add post-SVD retraining of the embedding factors | LSTM, 90% reduction · Books Intent test | accuracy: 89.83 -> 90.94 | Helped |
| Add post-SVD retraining of the embedding factors | DAN-RAND, 90% reduction · Books Intent test | accuracy: 83.72 -> 89.23 | Helped |
| Use online SVD compression and retraining instead of 16-bit quantization | LSTM, 50% reduction · SST2 test | accuracy: 85.08 -> 85.67 | Helped |
| Use online SVD compression and retraining instead of 8-bit quantization | LSTM, 75% reduction · SST2 test | accuracy: 85.01 -> 85.45 | Helped |
| Increase compression to 90% using online SVD and retraining (ablation) | LSTM · SST2 test | model size: 53.25 -> 6.94 | Helped |
| Increase compression to 90% using online SVD and retraining (ablation) | DAN-RAND · SST2 test | accuracy: 84.61 -> 83.11 | Hurt |
Closest papers by meaning
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.87)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.86)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.86)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.86)
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.86)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.