Visuo-motor dataset recorded from a micro-farming robot

This is the accompanying dataset of the paper [1] describing algorithms for intrinsic motivation and episodic memory on the Sony LettuceThink microfarming robot. The LettuceThink microfarming robot developed by Sony Computer Science Laboratories consists of an aluminium frame with an X-Carve CNC machine mounted on it. The CNC machine is used to provide 3-axes movements to a depth camera (Sony DepthSense) mounted at the tip of the vertical z-axis (the end-effector camera). In the experiments presented in the paper, the end-effector camera is facing top-down and only two motors are used (x and y). A simulator of the LettuceThink robot has been developed to ease the testing of different configurations of the learning system. The simulator generates sensorimotor data from requested trajectories of the end-effector camera. Knowing the initial position of the CNC machine and the target position, the simulator linearly interpolates the trajectory and returns the intermediate positions of the camera together with the images captured from each specific position. The sensorimotor data returned by the simulator have been prerecorded by performing a full scan of the (x,y) plane of the CNC machine using a resolution of 5mm. This resulted in 24,964 images, each mapped to an (x,y) position of the CNC machine. The dataset published here contains these images. In particular, the dataset consists of a set of images, each named with the specific position of the 2 motors of the robot. A python script for generating visuo-motor trajectories (sequences of data consisting of [image, motor_x, motor_y]) from this dataset is available at the following github page: https://github.com/guidoschillaci/sonylettucethink_dataset Provided with the dataset is also a python script that allows to easily read the images and to generate trajectories (returning This work has been supported by the EU-H2020 ROMI Project and by the EU-H2020 Marie Sklodowska Curie project "Predictive Robots" (grant agreement no. 838861)References: [1] Schillaci, G., Villalpando, A. P., Hafner, V. V., Hanappe, P., Colliaux, D., & Wintz, T. (2020). Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces. arXiv preprint arXiv:2001.01982.

本数据集为论文[1]的配套数据集，该论文针对索尼LettuceThink微型农耕机器人的内在动机（intrinsic motivation）与情景记忆（episodic memory）算法进行阐述。由索尼计算机科学实验室（Sony Computer Science Laboratories）开发的LettuceThink微型农耕机器人，整体采用铝制机架并搭载X-Carve型计算机数控（CNC）机床。该CNC机床可带动安装于垂直Z轴末端（即末端执行器相机（end-effector camera））的深度相机（Sony DepthSense）实现三轴运动。在论文呈现的实验中，末端执行器相机采用俯拍视角，且仅启用X、Y轴两台电机。为简化学习系统不同配置的测试流程，研发了LettuceThink机器人仿真器。该仿真器可根据指定的末端执行器相机运动轨迹生成传感运动数据（sensorimotor data）。已知CNC机床初始位置与目标位置后，仿真器将对轨迹进行线性插值，并返回相机的中间位置，以及各特定位置下采集的图像。仿真器返回的传感运动数据，系通过以5mm分辨率对CNC机床的X-Y平面进行全扫描预先录制所得。此次扫描共得到24964张图像，每张图像均对应CNC机床的一组X-Y坐标位置。本次发布的数据集即包含上述图像。具体而言，本数据集包含一组图像，每张图像均以机器人两台电机的对应坐标位置命名。可通过以下GitHub页面获取用于从本数据集生成视觉-运动轨迹（visuo-motor trajectories，即由[图像、电机X轴坐标、电机Y轴坐标]组成的数据序列）的Python脚本：https://github.com/guidoschillaci/sonylettucethink_dataset。本数据集还附带一款Python脚本，可便捷读取图像并生成轨迹。本研究得到欧盟“地平线2020”（EU-H2020）ROMI项目以及欧盟“地平线2020”玛丽·斯克洛多夫斯卡-居里"Predictive Robots"项目（资助协议编号：838861）的支持。参考文献：[1] Schillaci, G., Villalpando, A. P., Hafner, V. V., Hanappe, P., Colliaux, D., & Wintz, T. (2020). 面向高维感官空间机器人探索的内在动机与情景记忆. arXiv预印本arXiv:2001.01982.