Research map / Large language models / Language, safety and interpretability

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-eRank, a rank-based intrinsic metric that measures how much effective rank of LLM hidden representations drops from an untrained to a trained model, and an eRank-based modality alignment score for multimodal LLMs.

Problem

Existing LLM evaluation metrics (loss, accuracy) are prediction-based; there is no intrinsic, representation-based metric that captures how well a pretrained LLM removes redundant information (noise reduction), nor a way to assess modality alignment in multimodal LLMs.

Approach

Measure the effective rank (exponentiated matrix entropy) of the covariance of normalized token representations, and define Diff-eRank as the reduction of this eRank between an untrained and a trained model; for MLLMs, compare eRanks across vision encoder, connector and LLM stages to define image reduction ratio and image-text alignment scores.

Key results

• On OPT models (125M to 13B), Diff-eRank on dolly-15k rises from 54.35 to 89.81 and on openwebtext2 from 103.23 to 262.70, tracking reduced loss.
• On openbookqa, Diff-eRank increases from 1.410 (125M) to 3.032 (13B) alongside accuracy rising from 0.276 to 0.366 (the 6.7B point dips slightly).
• The paper reports that LLaVA-1.5 achieves higher Image-Text Alignment than MiniGPT-v2 and both have similar Image Reduction Ratio, with the metrics decreasing under image rotation.

Measured effects reported in the paper

Surprising findings

• Image rotation lowers Image Reduction Ratio and Image-Text Alignment, while image-related eRanks increase, suggesting the model perceives spatial-semantic changes.

Limitations

• Training dynamics over full pretraining and post-training were not studied due to compute limits.
• Results on the outlier-prone 6.7B and 13B points are not monotonic on some benchmarks.
• Multimodal evaluation uses only two MLLMs and the alignment metrics are not validated against downstream benchmarks beyond qualitative consistency.

Experimental setup

Datasets: Wikipedia openwebtext2 dolly-15k hh-rlhf openbookqa piqa winogrande detail_23k cc_sbu_align

Benchmarks: openbookqa piqa winogrande

Metrics: Diff-eRank reduced cross-entropy loss accuracy Image Reduction Ratio Image-Text Alignment

Baselines: cross-entropy loss (reduced loss) benchmark accuracy

Builds on: effective rank (Roy & Vetterli) matrix entropy compute-optimal scaling

Closest papers by meaning

• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.89)
• Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)
• DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models (NeurIPS 2024 · similarity 0.88)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
• Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.87)