28. Estimation method for factors contributing to decision errors by train forward surveillance AI

Efforts are being made to apply AI to train-front images, aiming to automate and reduce the labor involved in diagnostic and decision-making tasks previously performed visually on railways.
Furthermore, as AI is expected to make safety decisions during automatic train operation without personnel at the front of the train, it is crucial to understand the causes of accidents that may result from AI decision errors.
However, there is currently no adequate method for this. Given this situation, we developed a method to estimate the factors behind AI decision errors, such as missing a detection target, for a forward monitoring system designed to detect obstructions in front of a train.

In the developed method, we estimate the factors by dividing the process into three stages: inspecting input images for errors caused by the photographing method, evaluating the AI's capabilities for errors due to differences in its recognition method, and reviewing the training data for biases (Figure 1).
In the inspection of input images, we modify factors such as blurring and brightness to identify conditions that the AI can detect, thereby estimating factors that may cause decision errors related to the image capture conditions (Figure 2).
In the inspection of the AI's capabilities, we estimate factors by comparing the performance of different types and mechanisms of AIs and identifying models they can detect. In the inspection of training data, we analyze the distribution of image features and perform similar image searches to determine the proportion of images in the training data that resemble situation images where decision errors occurred. This helps identify any biases in the training data. From 215 hours of train forward surveillance video, we created approximately 75,000 evaluation images for unsafe situations. Applying the developed method to the 2,668 images that the AI had missed, we were able to estimate all the factors contributing to these omissions.

When an accident or incident occurs due to an AI decision error, this factor estimation method can be used to narrow down the target of countermeasures to be considered. It can also be used to check performance when developing AI-based systems.