B-On-Track: evaluation metric for multiple object tracking with ground-truth complexity consideration

Multiple Object Tracking (MOT) is a critical task in computer vision with numerous applications, including surveillance, autonomous driving, and crowd analysis. Evaluating the performance of MOT algorithms is crucial for understanding their strengths and weaknesses and guiding the development of more robust and accurate tracking systems. However, existing evaluation metrics often fail to capture the complexity and challenges of real-world tracking scenarios, leading to an incomplete assessment of tracking performance. This thesis addresses the limitations of current evaluation metrics by proposing B-on-Track, a novel adaptive evaluation metric for multiple object tracking. B-on-Track incorporates multiple factors, such as tracking duration, identity preservation, track loss patterns, and overlapped rates, to provide a comprehensive assessment of tracking performance. The metric adapts to the complexity of tracking scenarios by considering the path difficulty of each tracked object, which is estimated using factors extracted from the PersonPath22 dataset, a large-scale pedestrian tracking dataset with diverse scenes and challenges. The development of B-on-Track begins with an in-depth analysis of the PersonPath22 dataset, where important factors such as variance of path trajectory, overlapped rate, and occlusion are extracted to quantify the complexity of tracking each pedestrian trajectory. This analysis provides valuable insights into the challenges posed by different tracking scenarios and informs the design of the adaptive evaluation metric. The B-on-Track metric is formulated using mathematical notations and includes adjustable parameters that control the emphasis on different aspects of tracking performance. The metric is validated through extensive experiments on the PersonPath22 dataset, where its performance is compared with existing evaluation metrics such as MOTA, IDF1, and HOTA. The results demonstrate that B-on-Track provides a more nuanced and informative assessment of tracking performance, capturing the strengths and weaknesses of different tracking algorithms in various scenarios. Furthermore, the impact of adjusting the parameters of the B-on-Track metric is analyzed, showcasing its adaptability to different tracking requirements and dataset characteristics. This analysis provides insights into how the metric behaves under different parameter settings and guides the selection of appropriate parameter values based on the specific needs of the application. The main contributions of this thesis include: (1) the development of a path difficulty score that quantifies the complexity of pedestrian tracking scenarios, (2) the proposal of the B-on-Track metric, a novel adaptive evaluation metric for multiple object tracking, (3) the validation of B-on-Track through extensive experiments and comparisons with existing metrics, and (4) the analysis of the impact of adjusting the metric's parameters. Future work includes extending B-on-Track to other tracking domains, integrating it with deep learning-based tracking algorithms, exploring real-time performance evaluation, incorporating additional tracking challenges, and developing a standardized benchmark dataset for evaluating multiple object tracking algorithms using the B-on-Track metric.