Research map / Architectures and efficiency / Attention and transformers
Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CarFormer, a causal Transformer pretrained on vehicle diagnostic trouble code sequences with time and mileage context, and EPredictor, a cross-attention decoder that predicts which error patterns occur and when.
Problem
Predicting when and which error patterns (EPs) will occur in a car from multivariate, irregular diagnostic trouble code (DTC) event streams with high cardinality, class imbalance, and limited labels.
Approach
Treat DTC sequences as language: a causal Transformer encoder (CarFormer) with continuous time and mileage context embedded into queries and keys and trained with next-event, time, and random-event losses, feeding an autoregressive EPredictor decoder that outputs multi-label EP probabilities and time-to-occurrence via cross-attention.
Key results
- Best CarFormer (rot-ce) reaches 22.64% next-DTC accuracy versus 19.89% for the GPT baseline.
- Best EPredictor (rotcross-key-value-scaled-ce-2) reaches 84.38% micro F1 on the test set.
- With half the sequence observed (81 events), the model enters the confident predictive maintenance window; average time error is 58.4 ± 13.2 h.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add key-value cross-attention (ablation) | EPredictor · EP test set | Micro F1 (%): 80.74 -> 84.38 | Helped |
| Add cross-attention (ablation) | EPredictor · EP test set | Micro F1 (%): 80.74 -> 84.38 | Helped |
| Add context embedding to layer 2 rather than both layers (ablation) | EPredictor · EP test set | Micro F1 (%): 82.69 -> 84.38 | Helped |
| Add attention scaling (ablation) | EPredictor · EP test set | Micro F1 (%): 84.38 -> 84.38 | No clear effect |
| Add mileage through mixed feed-forward network (ablation) | EPredictor · EP test set | MAPE (%): 35.89 -> 33.37 | Helped |
| Add relative speed matrix (ablation) | EPredictor · EP test set | Micro F1 (%): 84.38 -> 83.53 | Hurt |
| Add absolute time embedding (ablation) | CarFormer · Next event prediction | ACC (%): 22.64 -> 21.48 | Hurt |
| Add mileage embedding to time embedding at the input (ablation) | CarFormer · Next event prediction | ACC (%): 21.48 -> 21.38 | Hurt |
| Use c2m and m2c attention terms with time embedding (ablation) | CarFormer · Next event prediction | ACC (%): 21.48 -> 21.58 | Helped |
| Use RoPE with context embedding (ablation) | CarFormer · Next event prediction | ACC (%): 21.48 -> 22.64 | Helped |
| Use context embedding with RoPE rather than time embedding only (ablation) | CarFormer · Next event prediction | MAPE (%): 2.9 -> 3.2 | Hurt |
| Set minimum context to 30 observations (ablation) | EPredictor · EP test set | F1 score: +2% | Helped |
Surprising findings
- Adding a learned speed (relative time-mileage) matrix to attention did not improve metrics, which the authors attribute to missing modalities during training.
- A context size of c=30 helped F1 by about 2%, but c=10, 20, 40 performed worse than no context, showing non-monotonic sensitivity.
Limitations
- Dataset is proprietary and anonymized, so results are hard to reproduce on public data.
- Error pattern labels are sparse and imbalanced, requiring up-sampling, down-sampling, and class dropping.
- Confident predictive maintenance window depends on an ad hoc threshold and noise adjustment.
Experimental setup
Datasets: anonymized vehicular DTC sequence dataset (1.7M sequences) repair and warranty error pattern labels
Benchmarks: in-house vehicular DTC test set
Metrics: accuracy micro F1 MAPE MAE RMSE CPMWAUC
Baselines: GPT
Builds on: GPT Transformer Hawkes Process RoPE DeBERTa disentangled attention
Closest papers by meaning
- Accident Anticipation via Temporal Occurrence Prediction (NeurIPS 2025 · similarity 0.86)
- From Faults to Features: Pretraining to Learn Robust Representations against Sensor Failures (NeurIPS 2025 · similarity 0.86)
- Progressive Risk Estimation for Accident Anticipation (NeurIPS 2026 · similarity 0.84)
- TraffiDent: A Dataset for Understanding the Interplay Between Traffic Dynamics and Incidents (NeurIPS 2025 · similarity 0.81)
- ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles (CVPR 2024 · similarity 0.81)
- LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.