Manuscript received April 23, 2024; revised June 11, 2024; accepted June 18, 2024; published September 6, 2024

Abstract—Recently people often purchase their daily needs at retail stores. Therefore, crowds might happen due to a manual queueing system. To overcome the problem, the smart system based on object detection has been conducted using several object detection methods. This study proposed YOLOv8 combined with transformer Real-Time Detection Transformer (RT-DETR) model to enhance the method performance in detecting the detail products. The intra-Class Variation method has been used to recognize the characteristics of the products such as size, color, and variant of the product. To validate the proposed model, three different datasets have been applied that is grocery dataset that displays products one by one in the training and validation process, the RPC-dataset that has many products in one image, and the D2S dataset with products that have varying lighting and stacked products. Results showed that the proposed model outperformed compared to other models, with a mean Average Precision (mAP) of 99.5% for the grocery dataset, 99.3% RPC-dataset, and 85.5% D2S dataset, respectively.

Keywords—intra-class variation, mean Average Precision (mAP), retail product, Real-Time Detection Transformer (RT-DETR), YOLOv8

Cite: Andi Wahyu Maulana, Suryo Adhi Wibowo, Unang Sunarya, Rissa Rahmania, and Asep Insani, "Performance Improvement of YOLOv8-RTDETR Method Based Retail Product Detection," Journal of Image and Graphics, Vol. 12, No. 3, pp. 292-301, 2024.

Copyright © 2024 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.