Research map / Large language models / AI and society

Defining Operational Conditions for Safety-Critical AI-Based Systems from Data

NeurIPS 2026 · Large language models › AI and society · approach: Human-centered trust and agency · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a deterministic, permutation-stable kernel-based (RBF, noisy-OR superposition) method to construct Operational Design Domains from data, with OOD consistency adjustment and conformal threshold calibration, validated on synthetic benchmarks and a VCAS/HCAS aviation use case.

Problem

Defining the Operational Design Domain (ODD) of safety-critical AI-based systems is hard to do manually, and existing data-driven ODD approches lack a deterministic, order-independent representation and a formal similarity notion.

Approach

Assign an RBF kernel to each ID anchor point with covariance calibrated from nearest-neighbour gaps, superpose affinities via noisy-OR, enforce an upper affinity bound at observed OOD samples by shrinking the most-contributing kernel, and pick the membership threshold by split-conformal calibration; also formalizes ODD similarity as equality of generated output images up to Lebesgue-null sets.

Key results

• On 2D suite at N=1000, autoSAFE is within 0.01313 AUPR of the strongest baseline and outperforms the convex hull on non-convex domains (e.g., annulus IoU 0.864 vs 0.898 for hull, two-blob IoU 0.811 vs 0.488).
• On the VCAS excluded region, false-positive rate falls from 0.547 (convex hull) to 0.037 (autoSAFE) and 0.0115 with OOD constraints.
• Monte Carlo AUPR reaches 0.99902(11) at N=10000 for the linear ODD, while the 5D constrained ODD is sparse-data sensitive (AUPR 0.079 at s=3 vs 0.909 at s=1).

Surprising findings

• The 5D constrained ODD with default calibration (s=3) performs very poorly (AUPR 0.079), below several baselines, showing sensitivity to the bandwidth constant in sparse high-dimensional settings.
• Convex hull IoU slightly exceeds autoSAFE on the annulus (0.898 vs 0.864), though autoSAFE has much smaller false-positive boundary distance.

Limitations

• Affinity depends on kernel calibration constants; sparse high-dimensional case is sensitive to s.
• Derived ODD reflects data density, not causal or system-level safety; unsafe conditions present in data remain inside the ODD.
• No evaluation under distribution shift or mislabeled samples; validation draws from the same generating process as anchors.

Experimental setup

Datasets: VCAS state vectors (aviation) HCAS state vectors (aviation) synthetic analytic ODDs (half-space, annulus, two-blob, banana, 5D constrained)

Benchmarks: synthetic analytic ODDs (2D-12D Monte Carlo) VCAS aviation use case HCAS aviation use case

Metrics: AUPR precision recall IoU false-positive rate false-exclusion rate

Baselines: KDE GMM (BIC) k-NN distance OC-SVM Isolation Forest convex hull

Builds on: Development Assurance Levels split-conformal prediction kernel density estimation DBSCAN

Closest papers by meaning

• Position: Require Frontier AI Labs To Release Small "Analog" Models (NeurIPS 2025 · similarity 0.84)
• Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.83)
• SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior (ICML 2025 · similarity 0.83)
• Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? (NeurIPS 2024 · similarity 0.82)
• Towards Computational Foreseeability (AAAI 2025 · similarity 0.82)
• A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems (ICML 2025 · similarity 0.81)