Research map / Adversarial robustness and security / Certified and empirical robustness

LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders

NeurIPS 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LORE, an unsupervised adversarial fine-tuning method for visual encoders (mainly CLIP ViT-B/32) that uses Lagrangian-dual constrained optimization to keep clean embeddings close to a frozen reference encoder while improving robustness.

Problem

Unsupervised adversarial fine-tuning of visual encoders such as CLIP is unstable early in training and gives a poor robustness-clean accuracy trade-off, especially at large perturbation budgets.

Approach

Replace heuristic loss weighting with a semi-infinite constrained formulation enforcing embedding-space proximity to a frozen reference encoder, solved via a primal-dual scheme where the Lagrange multiplier is an input-dependent Softplus-output MLP on reference embeddings, with a scale-invariant margin.

Key results

• On ImageNet-100 with ViT-B/32 LAION, LORE^4 reaches 68.4 clean vs 52.7 for FARE^4, and 9.06 vs 0.06 robust accuracy at eps=8 (ViT-B/16 LORE^4 vs FARE^4 at eps=8: 9.06 vs 2.22).
• Across 13 zero-shot datasets with ViT-B/32 CLIP, LORE improves average zero-shot clean accuracy over FARE by 0.5 to 5.5 points across training budgets eps=1,2,4.
• Extending the proximity term to supervised TeCoA raises clean accuracy at eps=8 from 30.22 to 67.23 while robust accuracy rises from 2.35 to 4.23.

Measured effects reported in the paper

Surprising findings

• Fixed proximity margins can be overly restrictive, with rho=0.6 sharply reducing eps=4 robustness versus the adaptive margin.
• Adversarially fine-tuned LORE-CLIP outperforms the original pretrained CLIP on some ImageNet-C corruptions (e.g., jpeg compression).

Limitations

• Performance ceiling is bounded by the frozen reference encoder's quality.
• Neural-network parameterization of Lagrange multipliers is heuristic and no convergence guarantees for the non-convex alternating optimization.
• No error bars or multiple seeds reported.

Experimental setup

Datasets: ImageNet ImageNet-100

Benchmarks: ImageNet ImageNet-100 CLIP-benchmark 13 zero-shot datasets ImageNet-C

Metrics: clean accuracy adversarial accuracy under APGD (CE and DLR) at epsilon 1,2, Pareto frontier average cosine similarity to text templates

Baselines: FARE TeCoA naive L2 regularization CLIP

Builds on: FARE TeCoA Lagrangian constrained learning for adversarial robustness PGD AutoAttack

Closest papers by meaning

• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.87)
• Revisiting Adversarial Training at Scale (CVPR 2024 · similarity 0.87)
• SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.87)
• Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.87)
• Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.86)
• Adversarially Robust Few-shot Learning via Parameter Co-distillation of Similarity and Class Concept Learners (CVPR 2024 · similarity 0.86)