Explorando abordagens para legendagem automática de imagens para a identificação do sujeito principal em fotografias de eventos de formaturas

Abstract

The selection of photographs from a graduation event is an essential task for companies that organize such events, as the sale of personalized albums is a significant part of their revenue. With that in mind, this work focuses on the application of pre-trained neural networks to assist in organizing graduation albums, exploring techniques for automatic image captioning and main subject identification. The study begins with a comparative analysis of three deep learning- based approaches to automatic image captioning in the context of graduations. The One For All (OFA) model, based on transformers, stands out as a promising choice. OFA is pre-trained on a wide variety of data and has been specialized for the specific task of captioning graduation images. Additionally, the work investigates the use of the implicit knowledge in automatic captioning models to identify the main subject in an image. This is essential for the effective organization of graduation albums, where it is crucial to highlight the main protagonists. The improvements, with average Intersection over Union measures of 0.47, compared to 0.17 without specialization. We also explore the use of the captions generated by the model to generate a word cloud, which can be useful for filtering photographs. The contributions of this work are directly relevant to the organization of graduation albums, including the generation of captions and bounding boxes for the main subject in photographs, as well as the creation of word clouds for efficient album organization. In summary, this study highlights the effectiveness of pre-trained neural networks in automatic image captioning and subject identification, providing significant benefits in automating the organization of graduation albums, a valuable task for companies and graduates.