Research map / Architectures and efficiency / Pruning and efficient tuning
FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tensorized Tucker factor tuning with rank searched up to 32 instead of NOAH | ViT-B/16 · VTAB-1K | average accuracy: 75.5 -> 75.6 | Helped |
| Use tensorized Tensor-Train factor tuning with rank searched up to 16 instead of NOAH | ViT-B/16 · VTAB-1K | average accuracy: 75.5 -> 75.3 | No clear effect |
| Use rank 4 Tensor-Train factor tuning instead of full fine-tuning | ViT-B/16 · VTAB-1K | average accuracy: 68.9 -> 73.5 | Helped |
| Use rank 4 Tensor-Train factor tuning instead of VPT-Deep | ViT-B/16 · VTAB-1K | average accuracy: 72.0 -> 73.5 | Helped |
| Use partitioned Tensor-Train factor tuning instead of VPT-Deep | Swin-B · VTAB-1K | average accuracy: 71.6 -> 77.4 | Helped |
| Use Tucker decomposition rather than Tensor-Train decomposition | FacT on ViT-B/16 · VTAB-1K | average accuracy: 75.3 -> 75.6 | No clear effect |
| Use non-normalized rather than normalized VTAB-1K inputs | PETL methods, including FacT · VTAB-1K | average accuracy: about 1-2% | Helped |
| Tensorize MHSA blocks instead of FFN blocks (ablation) | FacT-TT · VTAB-1K Natural, Specialized, and Structured groups | accuracy | Helped |
| Tensorize all MHSA and FFN blocks rather than only one block type (ablation) | FacT-TT · VTAB-1K Specialized and Structured groups | accuracy | Helped |
| Tensorize only MHSA rather than all blocks at larger ranks (ablation) | FacT-TT · VTAB-1K Natural group | accuracy | Helped |
| Increase Tensor-Train decomposition rank (ablation) | FacT-TT · VTAB-1K | average accuracy | Helped |
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.88)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.88)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.88)
- V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark (NeurIPS 2024 · similarity 0.87)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.86)
- Memory Efficient Transformer Adapter for Dense Predictions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.