Research map / Vision-language and multimodal / Multimodal LLMs
HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperET, a parameter-efficient method that adjusts vision encoder features in Poincaré-ball hyperbolic space via Möbius matrix multiplication to scale their hyperbolic radius (granularity), improving MLLM fine-tuning and pre-training with under 1% extra parameters.
Problem
MLLM training is extremely resource-intensive, which the authors attribute to CLIP/SAM/DINOv2 vision encoders being aligned with language at a single granularity level, creating a granularity mismatch during cross-modal alignment.
Approach
Learnable scaling matrices (diagonal, block-diagonal, banded, or full) applied via Möbius multiplication in the Poincaré ball to frozen pretrained weights, so that the hyperbolic radius of visual representations is continuously adjusted to match textual granularity, with Theorem-based claims that radius scales with the matrix norm ratio.
Key results
- On ScienceQA with MemVP + LLaMA-7B, HyperET (diagonal matrices) reaches 93.78 average accuracy, matching MemVP with LLaMA-13B; with LLaMA-13B and Mobius scaling the W^D_s variant reaches 94.72 vs 93.78 for MemVP.
- LLaVA-1.5 (Vicuna-7B) with HyperET improves VQAv2 from 78.5 to 80.3 and MME from 1510.7 to 1536.2, and LLaVA-1.5 (Vicuna-13B) improves VQAv2 from 80.0 to 82.3.
- Ablation shows Euclidean-space matrices with equal parameter counts give no gain (93.81 average) versus 94.72 for hyperbolic Mobius version, and removing Mobius multiplication drops accuracy by 0.81%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HyperET with full scaling matrices | LLaVA-1.5 with Vicuna-13B · TextVQA | top-1 accuracy: 61.3 -> 63.9 | Helped |
| add HyperET with full scaling matrices | LLaVA-1.5 with Vicuna-13B · POPE | top-1 accuracy: 85.9 -> 88.7 | Helped |
| add diagonal HyperET scaling matrices | MemVP with LLaMA-7B · ScienceQA test | average accuracy: 93.07 -> 93.78 | Helped |
| add diagonal HyperET scaling matrices | LaVIN with LLaMA-13B · ScienceQA test | average accuracy: 90.50 -> 91.46 | Helped |
| add diagonal HyperET scaling matrices | MemVP with LLaMA-13B · ScienceQA test | average accuracy: 93.78 -> 94.72 | Helped |
| use Euclidean scaling matrices instead of hyperbolic scaling (ablation) | MemVP with LLaMA-13B · ScienceQA test | average accuracy: 94.72 -> 93.81 | Hurt |
| replace Möbius multiplication with standard matrix multiplication (ablation) | MemVP+HyperET with LLaMA-13B · ScienceQA test | average accuracy: 94.72 -> 93.91 | Hurt |
| increase block-diagonal scaling flexibility to block size 4 (ablation) | MemVP+HyperET with LLaMA-13B · ScienceQA test | average accuracy: 94.72 -> 94.84 | Helped |
| increase block-diagonal scaling flexibility to block size 8 (ablation) | MemVP+HyperET with LLaMA-13B · ScienceQA test | average accuracy: 94.72 -> 94.82 | Helped |
| increase banded scaling bandwidth from 1 to 2 (ablation) | MemVP+HyperET with LLaMA-13B · ScienceQA test | average accuracy: 94.89 -> 94.82 | Hurt |
| increase banded scaling bandwidth from 1 to 4 (ablation) | MemVP+HyperET with LLaMA-13B · ScienceQA test | average accuracy: 94.89 -> 94.83 | Hurt |
| use HyperET rather than Euclidean parameter augmentation (ablation) | MemVP with LLaMA-13B · ScienceQA test | average accuracy: 93.81 -> 94.72 | Helped |
Surprising findings
- Euclidean matrix multiplication with the same number of added parameters yields little or no improvement, suggesting gains are not explained by parameter count alone.
- Increasing banded or block-diagonal flexibility yields only marginal gains and can overfit.
Limitations
- Pre-training variant with full matrices W_s uses about 50M parameters, which is not strictly lightweight and is close to full-adapter scale.
- Experiments limited to ScienceQA and LLaVA-family/MemVP/LaVIN backbones with CLIP-based encoders; generalization to other encoders is only partially tested.
- Hyperbolic radius adjustment rationale depends on the claim that radius encodes granularity, which is asserted by prior work rather than directly validated.
Experimental setup
Datasets: ScienceQA LLaVA-1.5 pre-training and instruction data
Benchmarks: ScienceQA VQAv2 GQA VizWiz TextVQA POPE MME MMBench MMBench-Chinese SEED-Bench LLaVA-Bench (In-the-Wild) MM-Vet
Metrics: accuracy average accuracy
Baselines: LaVIN MemVP LLaVA LLaVA-1.5 LLaVA-Next
Builds on: LoRA LLaVA-1.5 MemVP LaVIN Poincaré ball model
Closest papers by meaning
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.90)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.90)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.90)
- Towards Multimodal Large Language Models with Both Training and Inference Efficiency (ICML 2026 · similarity 0.90)
- Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.