Draft:Perception Error Model

In autonomous vehicle testing, a perception error model (PEM) is an approach to simulating the behaviour of sensing and perception systems by modeling the errors they produce rather than simulating the underlying sensor physics. This differs from high-fidelity sensor simulation techniques, where one wishes to generate synthetic sensor signals that the actual perception algorithms would process. Perception Error Models are sometimes referred to as surrogate models for perception systems, because they approximate the output of computationally expensive object detectors at a fraction of the computational cost.

The primary motivation for PEMs is to enable efficient virtual testing of autonomous vehicle driving policies. Since the failure modes of perception systems have significant impact on downstream planning and control, testing these systems requires capturing the dependencies between perception errors and vehicle behaviour. PEMs provide a computationally efficient alternative to running full physics-based sensor simulations with actual perception algorithms.

Piazzoni et al. (2020, 2023) proposed a formal definition of Perception Error Models as an approximation of the combined function of sensing subsystem S and perception subsystem P:^{[1] [2]}

${\displaystyle {\text{PEM}}(W)\approx S\&P(W)={\tilde {W}}=W+E}${\displaystyle {\text{PEM}}(W)\approx S\&P(W)={\tilde {W}}=W+E}

where W denotes the ground truth world state consisting of surrounding objects, ${\displaystyle {\tilde {W}}}${\displaystyle {\tilde {W}}} denotes the perceived world, and E represents the perception error. The PEM receives the ground truth world W and returns the perceived world ${\displaystyle {\tilde {W}}}${\displaystyle {\tilde {W}}}.

The driving policy DP then generates a response R by analyzing the perceived world:

${\displaystyle R=DP({\tilde {W}})=DP(W+E)}${\displaystyle R=DP({\tilde {W}})=DP(W+E)}

This definition does not depend on specific sensor types and provides a standard interface for integration in simulation pipelines.

Piazzoni et al. (2023) outlined four key factors that affect the manifestation of perception errors in autonomous vehicles, which should be considered when designing a PEM.^[2]

Positional aspects: The relative position of objects with respect to the ego-vehicle affects the quality of perception. Sensors have limited Field of View (FoV), including limited range and blind spots. Signal strength and resolution degrade with distance, and perception systems typically perform better in areas where multiple sensor fields of view overlap.

Parameter inter-dependencies: Object parameters such as class and size affect error manifestation. Larger objects are more likely to be detected, slow-moving vehicles are easier to track, and classification errors may affect size estimation. Parameters not directly relevant to driving policy may also be important, such as object colour or material affecting LiDAR or radar performance.

Occlusion: Objects in the scene may influence detection of other objects. Large vehicles such as trucks may occlude smaller objects such as cars or cyclists. Occlusion significantly impacts perception reliability, and cumulative statistics should be reported separately for different occlusion levels.

Temporal aspects: Objects move through the scene, causing previously occluded objects to become visible and vice versa. Algorithm uncertainties and filtering techniques depend on their previous state, so errors evolve over time and should be modeled as time series using dynamical models.

Piazzoni et al. (2023) proposed a generalized data-driven approach to PEM construction using the Apollo autonomous driving software and the nuScenes public dataset.^[2] Their approach models the Sensing and Perception system as a whole using Hidden Markov Models, partitioned based on spatial location around the ego-vehicle and occlusion levels. The key contribution was demonstrating PEM integration into a simulation pipeline, enabling analysis of how perception errors affect autonomous vehicle safety. Similarly, Sadeghi et al. (2021) tested neural networks as efficient surrogates in the CARLA simulator, demonstrating a reduction in computation time while maintaining similar downstream behaviour to actual detectors.^[3]

In addition, Piazzoni et al. (2022) extended PEMs to cooperative perception (CoPEM) scenarios, studying how vehicle-to-everything (V2X) communication can improve safety by compensating for perception limitations through information sharing between vehicles and infrastructure.^[4]

Innes and Ramamoorthy (2023) deployed PEMs in emergency braking scenarios using efficient importance sampling strategies to estimate rare collision probabilities.^[5] This enables likely and safety-critical perception errors to be identified.

In a similar fashion, Sadeghi et al. (2023) described a method to identify perception errors that score highly on standard quality metrics but cause planning failures, termed adversarial perception errors.^[6] Using a boundary-attack algorithm on black-box planners in the CARLA simulator, they showed that such errors can be systematically constructed, highlighting limitations of standard perception metrics for predicting downstream safety. Hanselmann et al. (2025) extended this adversarial approach with EMPERROR, a transformer-based generative PEM using the conditional variational autoencoder (CVAE) framework.^[7] Unlike earlier PEMs that model errors independently for each object, EMPERROR reasons over entire scene context, capturing correlated errors across groups of objects. Optimization in the CVAE latent space enables generation of adversarial errors constrained to high-likelihood regions.

Category:Self-driving_cars Category:Machine learning Category:Simulation

• Draft articles on computing
• Draft articles on technology
• AfC submissions on science, mathematics and engineering
• Pending AfC submissions
• AfC pending submissions by age/2 days ago
• AfC submissions by date/03 January 2026