Research map / Detection and segmentation / CNN features

Tensorized Attention for Understanding Multi-Object Relationships

AAAI 2025 · Detection and segmentation › CNN features · approach: Local features and context in CNNs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Tensorized Attention Model (TAM), which uses Tucker decomposition to compute attention over three object types (query, key/value memory, and semantics) and integrates it into Transformer encoders and as a LoRA adapter for Llama2-13B.

Problem

Standard attention only models relationships between one or two object types, so it cannot capture simultaneous multi-object relationships (e.g., comments, replies, and subjects) that matter in real-world data such as Reddit and X conversations.

Approach

Extends the Tensorized Transformer by using a Tucker-decomposed core tensor whose first axis matches the query token length, computing 3D attention among query, key, and semantic components, then collapsing the semantic axis (semantic fusion) to a 2D attention applied to values; also applied as a LoRA adapter.

Key results

• Trained from scratch, TAM reaches R10@1 of 45.61 on NFL, 52.12 on Politics, and 47.15 on TweetQA, outperforming BERT(SCR) (35.76, 37.09, 23.39) and ActBERT (43.89, 48.05, 29.74).
• As a LoRA adapter on Llama2-13B, TAM achieves Rouge-1 of 14.06 on Tea versus 13.39 for LoRA, and 76.12 versus 75.87 on SQuAD.
• Human and LLM judgments on 100 Tea assessments indicate TAM produced over 15% more informative outputs than LoRA while preserving naturalness.

Measured effects reported in the paper

Surprising findings

• Tensorized Transformer variants performed similar to or worse than BERT on most datasets, with TTrans(SCR) dropping on NFL (32.05 vs 35.76 for BERT(SCR)).

Limitations

• Experiments are limited to Reddit, TweetQA, and SQuAD with specific semantic-object designs chosen per dataset.
• Parameter count slightly exceeds BERT; Longformer-style layers suggested but not tested.
• LLM adapter experiments run for only one epoch and small dimension settings were chosen per dataset.

Experimental setup

Datasets: NFL (Reddit) Politics (Reddit) TweetQA Tea (Reddit) SQuAD 1.1

Benchmarks: NFL Politics TweetQA Tea SQuAD 1.1

Metrics: Recall10@1 Recall10@2 Recall10@5 ROUGE-1 ROUGE-2 ROUGE-L BLEU-1 BLEU-2

Baselines: BERT Tensorized Transformer ViLBERT ActBERT LoRA

Builds on: Tensorized Transformer LoRA BERT ViLBERT ActBERT

Closest papers by meaning

• Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.84)
• Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.84)
• SAS: Simulated Attention Score (NeurIPS 2025 · similarity 0.84)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.84)
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.84)
• MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.84)