Research map / Reinforcement learning / Visual and representation RL
The Adaptive Q-Network for Recommendation Tasks with Dynamic Item Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dynamic recommendation task with a shifting item pool and AdaQN, a value-based RL method that learns Q-values over frozen pre-trained item-characteristic embeddings using a lightweight Hadamard-product Q-network (CEQN).
Problem
RL-based recommenders assume a fixed item action space, so they generalize poorly when new items emerge and old ones are removed at test time.
Approach
Replace the state-item Q-function with a state-characteristic Q-function over a continuous item-feature space (frozen embedding layer from a pre-trained DeepFM), so new test items can be scored by their characteristics; the CEQN fuses state and characteristic via a Hadamard product instead of concatenation to reduce parameters.
Key results
- On the dynamic recommendation task on KuaiRec (FreeB), AdaQN reaches 13.11 cumulative reward vs. 9.47 for the best baseline PG(C).
- On MovieLens dynamic task, AdaQN achieves 32.56 under FreeB, comparable to PG(C) at 32.37 and above other baselines.
- Partial-overlap ablation: AdaQN's overlap item ratio (32.8%) is closest to Random (33.2%), with reward 14.11 vs. HAC(C) 10.82.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaQN instead of the strongest listed baseline | dynamic recommendation · KuaiRec, FreeB | cumulative reward: 9.4740 -> 13.1100 | Helped |
| Use AdaQN instead of the strongest listed baseline | dynamic recommendation · MovieLens, NX 10 | cumulative reward: 32.3740 -> 32.5607 | Helped |
| Use AdaQN instead of the strongest listed baseline | standard recommendation · KuaiRec, NX 10 | cumulative reward: 20.1416 -> 17.9038 | Hurt |
| Use AdaQN instead of the strongest listed baseline | standard recommendation · MovieLens, FreeB | cumulative reward: 34.0812 -> 25.4937 | Hurt |
| Use AdaQN instead of the strongest listed baseline | standard recommendation · MovieLens, NX 0 | cumulative reward: 23.9807 -> 27.1657 | Helped |
| Add a pre-trained, frozen embedding layer rather than using no pre-training (ablation) | AdaQN · KuaiRec dynamic recommendation, FreeB | cumulative reward: 8.0821 -> 13.1100 | Helped |
| Freeze the pre-trained embedding layer rather than unfreezing it (ablation) | AdaQN · KuaiRec dynamic recommendation, FreeB | cumulative reward: 7.1675 -> 13.1100 | Helped |
| Use CEQN Hadamard product rather than concatenating state and characteristics (ablation) | AdaQN · KuaiRec dynamic recommendation, FreeB | cumulative reward: 12.6659 -> 13.1100 | Helped |
| Use CEQN Hadamard product rather than concatenation (ablation) | AdaQN · KuaiRec dynamic recommendation | parameters: 87637 -> 19828 | Helped |
Surprising findings
- On the MovieLens dynamic task, AdaQN's reward under NX0 (13.09) is below Random-level variants of other baselines' scores on some modes, and PG(C) and Random are close to AdaQN on MovieLens FreeB, showing limited margin there.
Limitations
- Evaluation uses simulated environments built from offline public datasets rather than live user feedback.
- Item characteristic embedding is frozen after pre-training, so it cannot adapt to the RL signal; authors note unfrozen embeddings degrade performance.
- Gains over baselines on MovieLens are small in some evaluation modes.
Experimental setup
Datasets: MovieLens-1M KuaiRec
Benchmarks: MovieLens KuaiRec
Metrics: cumulative reward R under FreeB, NX0, NX10 evaluation modes overlap item ratio
Baselines: Random SL DQN PPO PG(C) HAC(C)
Builds on: DQN DeepFM PPO HAC
Closest papers by meaning
- On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation (ICML 2024 · similarity 0.87)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Contrastive Representation for Interactive Recommendation (AAAI 2025 · similarity 0.85)
- UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution (AAAI 2024 · similarity 0.84)
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.84)
- ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.