Interactive Clustering of Cooking Recipe Instructions:Towards the Automatic Detection of Events Involving Kitchen Devices

Abstract: Cooking recipes are a rich source of semantic information. They contain instructions for food preparation tasks, specifying the actions that should be carried out which typically involve various ingredients and kitchen devices. In an IoT scenario, instructions in cooking recipes can form the basis for automatically controlling kitchen devices without any programming. However, as these instructions are written in natural language, they first need to be transformed or parsed into machine-interpretable commands. As a step towards this, we investigate methods for identifying the various types of actions (events) that kitchen devices are involved in. We cast this problem as a clustering task, whereby recipe instructions involving a given device of interest, are automatically grouped according to the type of event described. Each sentence in every instruction is represented by its embedding vector which is computed using a BERT-based model, specifically one pre-trained using a Roberta architecture. We cluster these sentence embeddings using our newly proposed interactive machine learning (IML)-based framework underpinned by the HDBScan clustering technique. We demonstrate that our IML framework can detect events in sentences with satisfactory accuracy, reaching almost the same level as human performance.