Research map / Large language models / Instruction and fine-tuning
Breaking the Gradient Barrier: Unveiling Large Language Models for Strategic Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLIM, a gradient-free strategic classification method where LLMs with in-context learning implicitly simulate the bi-level strategic manipulation and decision-rule optimization without fine-tuning.
Problem
Existing strategic classification methods rely on linear models or shallow networks and explicit gradient retraining, which do not scale to large, evolving financial and internet datasets.
Approach
Theoretically show that linear self-attention forward passes under ICL can reproduce the gradient-descent updates for both the agent-side feature manipulation and the decision-maker's rule update, so pre-trained LLMs can solve the Stackelberg bi-level SC problem without any parameter updates.
Key results
- On PhiUSIIL under the strategic setting, GPT-4o with GLIM reaches 86.50% accuracy versus 65.65% for the MLP baseline.
- On Adult, GLIM with GPT-4o improves accuracy by 8.36% from non-strategic to strategic setting (91.35% vs 80.23% reported in table for strategic vs non-strategic).
- ICL-based updates converge toward gradient-descent updates with cosine similarity near 0.95 and L2 distance near 0.1 in decision rule optimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o with GLIM instead of MLP | Strategic classification · PhiUSIIL | accuracy: 65.65$_{\pm1.14}$ -> 86.50$_{\pm0.91}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Strategic classification · CISFraud | accuracy: 65.04$_{\pm1.27}$ -> 86.89$_{\pm1.08}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Strategic classification · Synthetic | accuracy: 70.90$_{\pm2.49}$ -> 86.83$_{\pm2.35}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Strategic classification · Credit | accuracy: 77.06$_{\pm0.41}$ -> 89.64$_{\pm0.27}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Strategic classification · Adult | accuracy: 78.74$_{\pm1.83}$ -> 91.35$_{\pm1.29}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Strategic classification · Spam | accuracy: 91.05$_{\pm0.54}$ -> 95.97$_{\pm0.61}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · PhiUSIIL | accuracy: 59.25$_{\pm0.57}$ -> 79.14$_{\pm0.94}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · CISFraud | accuracy: 59.03$_{\pm1.06}$ -> 80.15$_{\pm1.10}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · Synthetic | accuracy: 65.39$_{\pm2.03}$ -> 81.19$_{\pm2.19}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · Credit | accuracy: 71.50$_{\pm0.33}$ -> 80.96$_{\pm0.44}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · Adult | accuracy: 73.57$_{\pm1.55}$ -> 80.23$_{\pm1.31}$ | Helped |
| Use GPT-4o with GLIM instead of MLP | Non-strategic classification · Spam | accuracy: 89.01$_{\pm0.69}$ -> 91.28$_{\pm0.65}$ | Helped |
Surprising findings
- The strategic setting yields higher accuracy than the non-strategic setting for GLIM on several datasets, which the authors attribute to robustness.
Limitations
- Theoretical analysis is restricted to linear classifiers and linear self-attention; non-linear results are empirical only.
- The homogeneous-label assumption used in the construction is a simplification; the heterogeneous case is handled with an approximation bound.
Experimental setup
Datasets: CISFraud PhiUSIIL PaySim-based Synthetic Adult Spam Credit
Benchmarks: CISFraud PhiUSIIL Synthetic Credit Adult Spam
Metrics: accuracy cosine similarity L2 distance KL divergence cross-entropy loss
Baselines: Linear Model MLP
Builds on: in-context learning as implicit gradient descent strategic classification performative prediction
Closest papers by meaning
- One-Shot Strategic Classification Under Unknown Costs (ICML 2024 · similarity 0.86)
- Strategic Classification with Non-Linear Classifiers (NeurIPS 2025 · similarity 0.86)
- Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.85)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.85)
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.84)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.