Research map / Large language models / Instruction and fine-tuning
TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching
Runjia Zeng, Qifan Wang, Qiang Guan, Ruixiang Tang, Lifu Huang, Zhenting Wang, XUELING ZHANG, Cheng Han, Dongfang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TokenSeek instead of random token selection (TokenTune) with QLoRA | Qwen2.5-0.5B QLoRA · Few-shot QA average | average score: 46.34 -> 48.45 | Improved |
| add TokenSeek instead of random token selection with QLoRA | Llama3.2-1B QLoRA · Few-shot QA average | average score: 49.22 -> 52.61 | Improved |
| add TokenSeek instead of random token selection with LoHa | Llama3.2-3B LoHa · Few-shot QA average | average score: 59.13 -> 61.02 | Improved |
| add TokenSeek (10% tokens) to full-parameter tuning | Qwen2.5-0.5B full tuning · Few-shot QA average | average score: 48.43 -> 41.94 | Hurt |
| add TokenSeek (10% tokens) to full-parameter tuning | Llama3.2-1B full tuning · Few-shot QA average | average score: 40.82 -> 41.13 | No clear effect |
| add TokenSeek to QLoRA, reducing memory | Llama2-7B QLoRA · Few-shot QA average | average score: 62.55 -> 62.14 | Hurt |
| add gradient information to context-only scoring (alpha=5, beta=5) (ablation) | TokenSeek on Qwen2.5-0.5B QLoRA · Few-shot QA average | average score: 48.45 -> 48.65 | Improved |
| add context information to gradient-only scoring (alpha=5, beta=5) (ablation) | TokenSeek on Qwen2.5-0.5B QLoRA · Few-shot QA average | average score: 46.39 -> 48.65 | Improved |
| raise tunable token ratio from 10% to 30% (ablation) | TokenSeek on Llama3.2-1B QLoRA · Few-shot QA average | average score: 52.61 -> 52.75 | No clear effect |
| lower tunable token ratio from 10% to 3% (ablation) | TokenSeek on Qwen2.5-0.5B QLoRA · Few-shot QA average | average score: 48.45 -> 43.79 | Hurt |
| raise tunable token ratio from 10% to 60% (ablation) | TokenSeek on Qwen2.5-0.5B QLoRA · Few-shot QA average | average score: 48.45 -> 48.43 | No clear effect |
| add 100K MiniPile examples to Open-Platypus training data (ablation) | TokenSeek QLoRA on Llama2-7B · Few-shot QA average | average score: 62.14 -> 63.26 | Improved |
Closest papers by meaning
- Getting the most out of your tokenizer for pre-training and domain adaptation (ICML 2024 · similarity 0.88)
- Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025 · similarity 0.87)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.87)
- D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Unlocking Full Efficiency of Token Filtering in Large Language Model Training (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
