Abstract

Returning lost and found items in public spaces is challenging with traditional methods, and while technological advancements have led to systematic approaches, they often rely on query-based searches or image classification. This research provides a solution that combines textual and visual data to improve the semantic matching of lost and found items to address these problems. Three deep learning models for image similarity, text similarity, and fusion are implemented in a progressive web application (PWA) to support user data input and matching alerts. A fusion model was created by combining the SBERT model, which was refined using a dataset of 2,600 lost and found description pairs both for English and Sinhala languages, and the Siamese network, which was trained on 848 bag images using MobileNetV2. This fusion model also incorporates location and time features to give priority to recent activities and places to enhance matching accuracy. A neural network was trained using the dataset for the fusion model, which included image similarity, text similarity, location similarity and time similarity features as well as a target column that represents the similarity level of the two given bags. The accuracy of the Siamese model was 0.75, whereas the SBERT model demonstrated an accuracy of 0.9526 and an F1 score of 0.9405. The fusion model, which combined text and image data, achieved an accuracy of 0.87 and an F1 score of 0.98. The developed web application offers a community-driven platform to assist users in locating misplaced items, showcasing the system's practical usefulness.

Statistics