Multimodal Deep Learning and IoT Sensor Fusion for Real-Time Beef Freshness Detection

Abstract

Beef freshness quality is one of the important indicators in ensuring food safety and suitability. However, conventional methods such as manual visual inspection and laboratory testing cannot be widely applied in real-time and mass scale. To overcome these challenges, this study proposes a meat freshness detection system based on a multimodal approach that combines visual imagery and gas sensor data in a single IoT-based framework. This system is designed by utilizing the YOLOv11 architecture that has been optimized using the Adam optimizer. The dataset consisted of 540 original beef images, expanded into 1,296 images after augmentation. The model is trained on these augmented images and is able to achieve detection performance with a mAP@0.5 value of 99.4% and mAP@0.5:0.95 of 95.7%. As a further improvement, the cropped image features from the YOLOv11 model are processed through a combination of the ViT model and CNN to classify the level of meat freshness into three classes: Fresh, Medium, and Rotten with an accuracy of 99%. On the other hand, chemical data was obtained from the MQ136 and MQ137 gas sensors to detect H₂S and NH₃ levels which are indicators of meat spoilage. Data from visual and chemical data were then combined through a multimodal fusion method and classified using the Random Forest algorithm, producing a final prediction of Fit for Consumption, Need to Check, and Not Fit for Consumption. This multimodal model achieved a classification accuracy of 98% with a ROC-AUC score approaching 1.00 across all classes. While the proposed system achieved very high accuracy, further validation across diverse real-world environments is recommended to establish its generalizability.