Computer vision methods for tattoo detection, location and classification

Abstract

Tattoos are still poorly explored as a biometric factor for human identification, especially in law enforcement, where they can play an important role in identifying criminals, victims or other persons of interest. Tattoos are classified as soft biometrics as they are not permanent and can change over time, unlike hard biometric traits (fingerprint, iris, DNA, etc.). In this way, the main objective of this work is to apply computer vision methods and transfer learning to the problems of tattoo detection, location and classification in images. Given the scarcity of datasets available in the literature for these problems, specific annotated datasets were created for each problem addressed here. For the tattoo detection problem, a deep learning model based on transfer learning was presented. Data augmentation technique was also applied to improve the diversity of the training sets to obtain a better classification accuracy, and comparative experiments were carried out to evaluate the diversity of images in the data sets and the accuracy of the proposed model. For the tattoo location problem, an approach was presented by retraining the Mask R-CNN network with a tattoo dataset, and a fine-tuning was performed on the network to find the set of parameters that presented the best results in training the network. For the tattoo classification problem, the proposed model was also based on using deep networks with transfer learning to classify a set of 40 tattoo categories, many of them with practical meaning for law enforcement. Data augmentation technique was also used to improve the diversity and robustness of the training data. In tattoo detection, the results were very promising, achieving an accuracy of 95.1% in the test dataset and an F1-score of 0.79 in an external dataset, which, in general, were satisfactory, given the complexity of the problem. In tattoos location, the results reached an average accuracy of 89.3%, showing that the Mask R-CNN network has great adaptability to the tattoo environment, in addition to performing a qualitative analysis that helped to understand how the characteristics of images and annotations influence the results. In tattoos classification, the results reached accuracy of 85.24% when using cross validation and data augmentation, showing that the transfer learning approach adopted has good capacity for this problem. Future work will include improving the quality and volume of the databases, conducting a more in-depth study on the fine-tuning of network parameters, and studies of open-world techniques for classifying tattoos, as well as developing models for other problems that compose the tattoo recognition roadmap.