COFFAIL: A Dataset of Successful and Anomalous Robot Skill Executions in the Context of Coffee Preparation

Dataset description This dataset includes both successful and anomalous execution episodes of the Jessie robot (https://www.h-brs.de/en/a2s/robots) performing seven different skills in the context of preparing coffee. All episodes were collected by demonstration (either by executing hand-coded scripts that produce the desired behaviour or using kinesthetic teaching). The included skills are: Picking up a cup from a countertop Moving a cup by pushing Pouring from one cup into another (note that, for safety reasons, liquid was not used in these trials) Placing a cup on a countertop Picking up a teaspoon Stirring Placing a spoon in a sink The dataset includes robot observations and actions from the execution episodes: Observations: RGB images (of size 266x200 pixels) from a head camera and wrist cameras attached to each arm (for skills performed by a single arm, wrist images of the stationary arm are not collected) Joint states (measured positions, velocities, and efforts) End effector poses (with respect to the arm's base) Actions in the form of end-effector delta motions (x, y, z position changes and roll, pitch, yaw orientation changes) Important note for correct action interpretation: The actions are given with respect to each manipulator's base frame, such that, considering the constraints imposed by the arm arrangement on Jessie, forward arm motion is actually along the negative x-axis of the manipulator frames (and not along the positive x, as would be expected); the sign of the y motions is also inverted accordingly. Data format The data is saved in a RLDS-like format, but in the form of MongoDB databases. In particular, the dataset includes 15 separate database dumps: Seven dumps corresponding to successful execution episodes (in total, 79 successful episodes over all skills) Seven dumps corresponding to episodes with execution anomalies (in total, 48 anomalous episodes over all skills) One dump of a database with anomaly annotations (anomalies are annotated using intervals, namely the annotations include entries for when each anomaly starts and when it ends, together with a comment on what the anomaly was) In the databases corresponding to execution episodes: each collection corresponds to a separate episode each document corresponds to an execution step Note that the recordings include idle states before and after each skill execution; these should be filtered out if the data is used for policy learning. Usage examples A Python package for working with the COFFAIL dataset is provided in the following repository: https://github.com/KEROL-project/coffail-utils/ Citing If you find COFFAIL useful, please cite the following paper: @inproceedings{coffail2026, author = {Mitrevski, Alex and Salunke, Ayush}, title = {{COFFAIL: A Dataset of Successful and Anomalous Robot Skill Executions in the Context of Coffee Preparation}}, booktitle = {2nd German Robotics Conference (GRC)}, year = {2026}, url = {https://arxiv.org/abs/2604.18236}}数据集描述 本数据集涵盖了杰西机器人（Jessie robot，https://www.h-brs.de/en/a2s/robots）在备咖啡场景下执行7种不同技能时的成功执行片段与异常执行片段。所有片段均通过演示方式采集：既可以通过手动编写的脚本生成预期行为，也可采用动觉示教（kinesthetic teaching）。 本次数据集覆盖的技能如下： 1. 从操作台拾取杯子 2. 推动杯子移动 3. 杯间倾倒（注：出于安全考量，本次试验未使用实际液体） 4. 将杯子放置在操作台上 5. 拾取茶匙 6. 搅拌操作 7. 将勺子放置在水槽中 本数据集包含执行片段中的机器人观测数据与动作数据： ### 观测数据 - RGB图像：尺寸为266×200像素，采集自头部相机与安装在每只机械臂上的腕部相机（针对单臂执行的技能，不采集静止机械臂的腕部图像） - 关节状态：包括实测位置、速度与受力 - 末端执行器位姿：相对于机械臂基座的位姿 ### 动作数据 动作以末端执行器增量运动的形式给出，包含x、y、z位置变化与滚转（roll）、俯仰（pitch）、偏航（yaw）姿态变化。 ### 动作解释注意事项 动作的参考坐标系为各机械臂自身的基座坐标系。结合杰西机器人的机械臂布局约束，正向机械臂运动实际沿机械臂坐标系的负x轴方向（而非常规预期的正x轴）；y轴运动的符号也相应反转。 ### 数据格式 本数据集采用类RLDS（RLDS-like）格式，但以MongoDB数据库的形式存储。具体而言，数据集包含15个独立的数据库备份文件： - 7个备份对应成功执行片段：所有技能总计79条成功执行片段 - 7个备份对应存在执行异常的片段：所有技能总计48条异常执行片段 - 1个备份为异常标注数据库：异常以时间区间形式标注，每条标注包含异常的起始与结束时间，以及对应异常的说明文本。 在执行片段对应的数据库中： - 每个集合（collection）对应一条独立的执行片段 - 每个文档（document）对应一个执行步骤 需注意：录制数据包含每项技能执行前后的空闲状态，若将数据用于策略学习，需将这些空闲状态过滤掉。 ### 使用示例 本数据集对应的Python工具包已在如下开源仓库提供：https://github.com/KEROL-project/coffail-utils/ ### 引用 若您使用本COFFAIL数据集，请引用如下论文： @inproceedings{coffail2026, author = {Mitrevski, Alex and Salunke, Ayush}, title = {{COFFAIL: A Dataset of Successful and Anomalous Robot Skill Executions in the Context of Coffee Preparation}}, booktitle = {2nd German Robotics Conference (GRC)}, year = {2026}, url = {https://arxiv.org/abs/2604.18236}}