Research map / Large language models / AI and society
Context is Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-Context Risk Minimization (ICRM), which treats domain generalization as next-token prediction with unlabeled test-environment context, and shows gains over ERM, ARM and TENT on four image DG benchmarks.
Problem
Domain generalization methods fail to beat ERM across real-world benchmarks; existing invariance and marginal-transfer methods discard or coarsely summarize environment information at the example level.
Approach
Treat environments as context: train a transformer to predict each label from the input plus preceding unlabeled examples from the same environment, so the model adaptively "zooms in" on the test-environment risk minimizer and can exploit example-level signals; the extended input-context feature space also reveals invariances that ERM misses.
Key results
- On FEMNIST with 25 context samples, ICRM reaches 87.2% average OOD accuracy versus 79.3% for ERM and 83.9% for ARM.
- On WILDS Camelyon17 with no test context, ICRM achieves 92.0% average accuracy versus 68.6% for ERM.
- On Tiny ImageNet-C with 0 context, ICRM reaches 38.3% average accuracy versus 31.8% for ERM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 25 unlabeled same-environment test examples as context | ICRM · FEMNIST | average test accuracy: 78.7 -> 87.2 | Helped |
| Add 25 unlabeled same-environment test examples as context | ICRM · Rotated MNIST | average test accuracy: 93.6 -> 96.1 | Helped |
| Add 25 unlabeled same-environment test examples as context | ICRM · WILDS Camelyon17 | average test accuracy: 92.0 -> 90.7 | Hurt |
| Add 25 unlabeled same-environment test examples as context | ICRM · Tiny ImageNet-C | average test accuracy: 38.3 -> 39.2 | Helped |
| Use pooled iid samples rather than same-environment examples in training context sequences (ablation) | ICRM · FEMNIST | average test accuracy: 87.2 -> 81.1 | Hurt |
| Use pooled iid samples rather than same-environment examples in training context sequences (ablation) | ICRM · Rotated MNIST | average test accuracy: 96.1 -> 92.6 | Hurt |
| Use pooled iid samples rather than same-environment examples in training context sequences (ablation) | ICRM · WILDS Camelyon17 | average test accuracy: 90.7 -> 90.7 | No clear effect |
| Use pooled iid samples rather than same-environment examples in training context sequences (ablation) | ICRM · Tiny ImageNet-C | average test accuracy: 39.2 -> 39.3 | No clear effect |
| Use the ICRM Transformer architecture without context (ablation) | ERM · WILDS Camelyon17 | average test accuracy: 68.6 -> 50.1 | Hurt |
| Use the ICRM Transformer architecture without context (ablation) | ERM · FEMNIST | average test accuracy: 79.3 -> 77.4 | Hurt |
| Use the ICRM Transformer architecture for ARM (ablation) | ARM · Tiny ImageNet-C | average test accuracy: 30.8 -> 5.5 | Hurt |
| Use the ICRM Transformer architecture for ARM (ablation) | ARM · FEMNIST | average test accuracy: 49.5 -> 71.4 | Helped |
Surprising findings
- ICRM-Mix (training on samples pooled across environments, without environment labels) performs similarly to ICRM on WILDS Camelyon17 and Tiny ImageNet-C.
- Simply adding the transformer architecture without context (ERM+) hurts performance relative to ERM on all four datasets.
Limitations
- The model could zoom in on toxic spurious correlations that have high predictive power within certain environments.
- Theoretical guarantees rely on idealized assumptions such as Gaussian latent structure and ideal amortization functions.
- Gains vanish or reverse on some settings, e.g. TENT and ARM collapse at certain context sizes on Tiny ImageNet-C.
Experimental setup
Datasets: FEMNIST Rotated MNIST WILDS Camelyon17 Tiny ImageNet-C
Benchmarks: FEMNIST Rotated MNIST WILDS Camelyon17 Tiny ImageNet-C
Metrics: average test accuracy worst-case test accuracy
Baselines: ERM ARM TENT ICRM-Mix ERM+ ARM+
Builds on: empirical risk minimization adaptive risk minimization GPT-2 DomainBed
Closest papers by meaning
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Generalization to New Sequential Decision Making Tasks with In-Context Learning (ICML 2024 · similarity 0.88)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.88)
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.88)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.87)
- Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.