Result of leftover level estimation (MAE (S.D)).

Monitoring the remaining food in patients’ trays is a routine activity in healthcare facilities as it provides valuable insights into the patients’ dietary intake. However, estimating food leftovers through visual observation is time-consuming and biased. To tackle this issue, we have devised an efficient deep learning-based approach that promises to revolutionize how we estimate food leftovers. Our first step was creating the LeFoodSet dataset, a pioneering large-scale open dataset explicitly designed for estimating food leftovers. This dataset is unique in its ability to estimate leftover rates and types of food. To the best of our knowledge, this is the first comprehensive dataset for this type of analysis. The dataset comprises 524 image pairs representing 34 Indonesian food categories, each with images captured before and after consumption. Our prediction models employed a combined visual feature extraction and late fusion approach utilizing soft parameter sharing. Here, we used multi-task (MT) models that simultaneously predict leftovers and food types in training. In the experiments, we tested the single task (ST) model, the ST Model with Ground Truth (ST-GT), the MT model, and the MT model with Inter-task Connection (MT-IC). Our AI-based models, particularly the MT and MT-IC models, have shown promising results, outperforming human observation in predicting leftover food. These findings show the best with the ResNet101 model, where the Mean Average Error (MAE) of leftover task and food classification accuracy task is 0.0801 and 90.44% in the MT Model and 0.0817 and 92.56% in the MT-IC Model, respectively. It is proved that the proposed solution has a bright future for AI-based approaches in medical and nursing applications.

监控患者餐盘中的剩余食物是医疗机构的一项常规工作，该操作可为患者的膳食摄入情况提供极具价值的参考依据。然而，通过人工目视观察估算食物剩余量不仅耗时耗力，还会引入主观偏差。为解决这一问题，我们研发了一种高效的深度学习方法，有望彻底革新食物剩余量的估算范式。我们的首要工作是构建LeFoodSet数据集（LeFoodSet），这是一款专为食物剩余量估算设计的开创性大规模开源数据集，其独特优势在于可同时估算食物剩余率与食物种类。据我们所知，这是首个针对该类分析任务的全面型数据集。该数据集包含524组图像对，涵盖34种印尼食品类别，每类均配有食用前与食用后的拍摄图像。我们的预测模型采用了融合视觉特征提取与软参数共享的后融合技术路径；训练阶段中，我们采用了多任务（Multi-task, MT）模型，可同时预测食物剩余量与食物类别。在实验环节中，我们分别测试了单任务（Single Task, ST）模型、带真值标签的单任务（ST with Ground Truth, ST-GT）模型、多任务（MT）模型以及带任务间连接的多任务（MT with Inter-task Connection, MT-IC）模型。我们的人工智能模型，尤其是MT与MT-IC模型，取得了颇具前景的实验结果，在食物剩余量预测任务上超越了人工目视观察的表现。该研究以残差网络101（ResNet101）取得了最优效果：在MT模型中，食物剩余量估算任务的平均绝对误差（Mean Absolute Error, MAE）为0.0801，食物分类准确率为90.44%；而在MT-IC模型中，两项指标分别为0.0817与92.56%。实验结果证实，所提出的解决方案在医疗与护理领域的人工智能应用中拥有广阔的发展前景。