Interactive Clustering of Cooking Recipe Instructions:Towards the Automatic Detection of Events Involving Kitchen Devices

Abstract: Cooking recipes are a rich source of semantic information. They contain instructions for food preparation tasks, specifying the actions that should be carried out which typically involve various ingredients and kitchen devices. In an IoT scenario, instructions in cooking recipes can form the basis for automatically controlling kitchen devices without any programming. However, as these instructions are written in natural language, they first need to be transformed or parsed into machine-interpretable commands. As a step towards this, we investigate methods for identifying the various types of actions (events) that kitchen devices are involved in. We cast this problem as a clustering task, whereby recipe instructions involving a given device of interest, are automatically grouped according to the type of event described. Each sentence in every instruction is represented by its embedding vector which is computed using a BERT-based model, specifically one pre-trained using a Roberta architecture. We cluster these sentence embeddings using our newly proposed interactive machine learning (IML)-based framework underpinned by the HDBScan clustering technique. We demonstrate that our IML framework can detect events in sentences with satisfactory accuracy, reaching almost the same level as human performance.摘要：烹饪食谱是语义信息的丰富来源。其包含食品制备任务的操作指南，明确了需执行的各类操作步骤，这类操作通常涉及多种食材与厨房设备。在物联网（Internet of Things, IoT）场景中，烹饪食谱中的操作指南可作为无需编程即可自动控制厨房设备的基础。但由于这类指南以自然语言撰写，因此首先需要将其转换或解析为机器可理解的指令。为此，本研究探索了用于识别厨房设备所参与的各类操作（事件，event）的方法。我们将该问题建模为聚类任务，即针对涉及指定目标设备的食谱操作指南，根据其描述的事件类型自动进行分组。每条操作指南中的句子均通过嵌入向量进行表征，该向量由基于BERT的模型计算得到，具体为采用RoBERTa架构预训练的模型。我们使用全新提出的基于交互式机器学习（Interactive Machine Learning, IML）的框架对这些句子嵌入向量进行聚类，该框架以HDBSCAN聚类技术为支撑。实验结果表明，所提出的IML框架能够以令人满意的准确率识别句子中的事件，其性能几乎可达到人类水平。