Avaliação de datasets e de algoritmos de detecção de mudança utilizando mapas de dificuldade

Abstract

Evaluating a change detection algorithm must show the superiority of its performance concerning state-of-the-art algorithms’ performance. The steps of evaluating an algorithm comprise executing it to segment a set of videos from a dataset and comparing the results with ground truth. In this work, we propose using additional information in evaluating change detection algorithms: the level of difficulty to classify each pixel of each frame of the videos from a dataset. For each video frame, we created a structured called difficulty map, which stores values representing the level of difficulty required by an algorithm to classify each pixel of that frame. Based on the difficulty maps, we developed two metrics. The first aims to evaluate the performance of algorithms about the difficulty map. The second metric aims to estimate the level of difficulty that each dataset video requires from the algorithms to classify its frames’ pixels. In this work, we also developed a method for selecting representative videos based on their difficulty level. The evaluation algorithms’ results showed that the algorithms that represent the state-of-the-art fail in the same regions of the frame. The metric that estimates video difficulty levels has been demonstrated that many videos from dataset CDNet 2014 have similar difficulty levels. This finding corroborates the method’s results to generate a representative subset since the selected subset has fewer videos and has the same evaluation potential as the original video set.