Probability of Detection (POD)-based Metric for Evaluation of Classifiers Used in Driving Behavior Prediction



Published Sep 22, 2019
Daniel Adofo Ameyaw Qi Deng Dirk Söffker


Classifiers are functional tools/algorithms that implement classifications and are widely used in science and technology for state of health estimation, diagnosis systems, and situation/intention recognition of human operators. Certification of these classifiers plays a crucial role in their selection for a specific task. Current certification approaches utilize the Receiver Operator Curve (ROC) as a standard tool that provides graphically the performance of classifiers. Beside the ratio of Detection Rate and False Alarm Rate (combined as ROC), other properties related to process parameters are not considered. In this paper, a new evaluation method based on the Probability of Detection (POD) reliability measure is developed discussing the effect of further process parameters on the classification results. Probability of Detection (POD) serves as a performance measure for quantifying the reliability of conventional Nondestructive Testing (NDT) procedures and Structural Health Monitoring (SHM) systems. The approach considers statistical variability of sensor-based measurements. In this publication for the first time the signal-response and the binary (hit/miss) approaches are implemented in combination with a process parameter. As example in this publication, the prediction of driving behavior classification is used as process parameter. The signal response approach is applied to compare the driving behavior prediction capabilities of Fuzzy Logic-Hidden Markov Model (FL-HMM), Artificial Neural Network (ANN), and Support Vector Machine (SVM) with respect to the reliability of the prediction for driver behavior related to prediction time. The hit/miss approach is also applied on FL-HMM as example for predicting an upcoming driving maneuver. To account for data uncertainty and variability, confidence bounds are established. A typical and useful criteria for detection at a 90 % probability of detection level with 95 % confidence level is successfully implemented as a new reliability measure and certification standard for classifiers. In this article a new approach is established permitting a new evaluation approach to classifiers. The new approach introduces a POD-based measure for comparison of binary classifiers.

How to Cite

Ameyaw, D. A., Deng, Q., & Söffker, D. (2019). Probability of Detection (POD)-based Metric for Evaluation of Classifiers Used in Driving Behavior Prediction. Annual Conference of the PHM Society, 11(1).
Abstract 188 | PDF Downloads 166



Probability of Detection, Classifiers, Situation/Intension recognition, Driving behavior prediction, Artificial Neural Network, Support Vector Machine, Fuzzy-logic Hidden Markov Model, Reliability certification

Technical Papers