Research map / Large language models / Language, safety and interpretability
Flora: Effortless Context Construction to Arbitrary Length and Scale
Tianxiang Chen, Zhentao Tan, Xiaofan Bo, Yue Wu, Tao Gong, Qi Chu, Jieping Ye
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on Flora-80k long-context data | Llama-3-8B-Instruct · LongBench v1 | average score: 38.93 -> 50.73 | Improved |
| fine-tune on Flora-128k long-context data | QwQ-32B · LongBench v1 | average score: 66.23 -> 70.07 | Improved |
| fine-tune on Flora-128k | QwQ-32B · LongBench v2 (CoT) | average accuracy: 45.6 -> 50.5 | Improved |
| fine-tune on LongAlpaca-12k instead of Flora | QwQ-32B · LongBench v2 (CoT) | average accuracy: 45.6 -> 42.5 | Hurt |
| add Fewshot QA strategy (1/4 of data) (ablation) | Mosaic-IT baseline · LongBench v1 | average score: 45.54 -> 47.40 | Improved |
| add Answer Before or After strategy (1/4 of data) (ablation) | Mosaic-IT baseline · LongBench v1 | average score: 45.54 -> 47.93 | Improved |
| add Answer No Answer strategy (1/4 of data) (ablation) | Mosaic-IT baseline · LongBench v1 | average score: 45.54 -> 47.97 | Improved |
| add Answer to ID strategy (1/4 of data) (ablation) | Mosaic-IT baseline · LongBench v1 | average score: 45.54 -> 48.38 | Improved |
| combine all four strategies with Mosaic (3/7 base + 4/7 all) (ablation) | Mosaic-IT baseline · LongBench v1 | average score: 45.54 -> 48.99 | Improved |
| replace augmented samples under 2k tokens with original SFT samples (ablation) | Flora 3/7+4/7 mix · Short-context tasks | average score: 43.02 -> 43.41 | Improved |
| use proposed exponential token-length distribution instead of normal (ablation) | Flora 8k-sample set · LongBench v2 | average accuracy: 30.5 -> 32.3 | Improved |
| use proposed distribution instead of normal (ablation) | Flora 8k-sample set · Short-context tasks | average score: 43.56 -> 44.82 | Improved |
Closest papers by meaning
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.90)
- LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.89)
- LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.89)
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
