See & Grasp 2 Data Set

This is the second See & Grasp data set introduced from Yildirim & Jacobs (2013). It is a data set containing both the visual and haptic features for "Fribbles". The set of "Fribbles" is larger than the original See & Grasp dataset, which is available here. Fribbles are complex, 3-D objects with multiple parts and spatial relations among the parts. Moreover, Fribbles have a categorical structure—that is, each Fribble is an exemplar from a category formed by perturbing a category prototype. There are 4 Fribble families, each with 4 species. Each Fribble has 4 slots, and one of 3 possible parts are attached to each slot, leading to (4 × (34)) = 324 total Fribbles in each family. The unmodified 3-D object files for the whole set of Fribbles can be found on Mike Tarr's (Department of Psychology, Carnegie Mellon University) web pages (TarrLab webpage). The See & Grasp data set contains 891 items corresponding to 891 Fribbles (324 for the A and C families, and 243 for family B because the 3D model files for one of 4 species in family B seem to be corrupt). There are 3 entries associated with each item. One entry is the 3-D object model for a Fribble. The second entry is an image of a Fribble rendered from a canonical viewpoint so that the Fribble's parts and spatial relations among the parts are clearly visible. (Using the 3-D object model, users can easily generate new images of a Fribble from any desired viewpoint.) The third entry is a way of representing a Fribble's haptic features. It is a set of joint angles obtained from a grasp simulator known as "GraspIt!" (Miller & Allen, 2004). GraspIt! contains a simulator of a human hand. When forming the representation of a Fribble's haptic features, the input to GraspIt! was the 3-D object model for the Fribble. Its output was a set of 16 joint angles of the fingers of a simulated human hand obtained when the simulated hand "grasped" the Fribble. Grasps—or closings of the fingers around a Fribble—were performed using GraspIt!'s AutoGrasp function. Each Fribble was grasped 24 times, with the fribble rotated 8 times (by 45°) on each axis. To be sure that Fribbles fit inside GraspIt!'s hand, their sizes were reduced by 29%. Please see the figure for sample images from the dataset and an illustration of the grasping procedure. The file contains three folders and a CSV file. The folders “Fa”, “Fb”, and “Fc” contain the files for each family of Fribble we examined—A, B, and C. These files each contain the following folders: “renders” contains the images of each Fribble in the species. “obj” contains the MTL and OBJ files for each Fribble in the species. “parts” contains the original Fribble parts from Mike Tarr's web pages which are composited into the individual Fribbles. “vrml” contains the VRML file for each Fribble in the species. The files are named based on the parts which were combined to make that Fribble. E.g., 4_a1b3c2d3 was made by combining 4_body.obj, 4_a1.obj, 4_b3.obj, 4_c2.obj, and 4_d3.obj. Finally, “haptics.csv” contains the haptic features for each of the Fribbles. It is a CSV file with 892 rows (including one header) and 255 columns. The first column is the Fribble's species and name, the next 254 columns are the joint angles from Graspit!: we performed a grasp for each of 24 different angles (rotating 45° around each axis), and each grasp has 16 DOF values associated with it, representing the position of the hand's joints. The rest of the columns contain the data for the contact points between the Fribble and the hand in Graspit. The first number for each grasp (there are 24 grasps for each Fribble) is the number of contact points for that grasp. For each contact point, the first six numbers relate to the contact's wrench, the next 3 numbers specify the contact's location in body coordinates, and the last number specifies the scalar constraint error for that contact. Sample code for rendering objects visually with VTK and haptically with GraspIt can be seen in the folder SampleCode. Please read GraspItUpdate.pdf file to see how to update GraspIt’s TCP server code for haptic rendering. Sample code folder also contains aomr.xml, the world file (i.e., setup of the human hand) used for haptic rendering in GraspIt. Please cite the following paper in relation to the See & Grasp 2 data set. Yildirim, I. & Jacobs, R. A. (2013). Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies. Cognition, 126, 135-148. Citation for GraspIt! Miller, A., & Allen, P. K. (2004). Graspit!: A versatile simulator for robotic grasping. IEEE Robotics and Automation Magazine, 11, 110-122.本数据集为源自Yildirim与Jacobs（2013）研究的第二代See & Grasp数据集。其包含针对Fribble（Fribble）的视觉与触觉特征两类数据，且Fribble样本规模大于原版See & Grasp数据集，原版数据集可通过此处获取。Fribble为具备多部件及部件间空间关系的复杂三维物体，且具有类别结构——即每个Fribble均为通过扰动类别原型生成的类别样本。本次数据集共包含4个Fribble类别族，每个类别族下设4个物种。每个Fribble设有4个安装槽，每个槽位可装配3种可选部件中的一种，因此每个类别族中共包含(4 × 3⁴) = 324个Fribble样本。全套未修改的Fribble三维物体文件，可从卡内基梅隆大学心理学系Mike Tarr的个人网页（TarrLab网页）获取。本次See & Grasp数据集共包含891条数据，对应891个Fribble样本：其中A、C类别族各含324个样本，B类别族仅含243个样本，原因是B族中4个物种之一的三维模型文件存在损坏。每条数据均包含3项内容：第一项为Fribble的三维物体模型；第二项为从标准视角渲染得到的Fribble图像，可清晰展示其部件及部件间的空间关系（用户可基于该三维模型，轻松生成任意视角下的Fribble图像）；第三项为Fribble触觉特征的表征数据，该数据来自名为"GraspIt!"的抓取模拟器（Miller & Allen, 2004）的输出结果，具体为一组关节角度值。GraspIt!内置了人手仿真模拟器。在生成Fribble的触觉特征表征时，我们将Fribble的三维物体模型作为GraspIt!的输入，模拟器输出仿真人手在"抓取"该Fribble时的16个手指关节角度值。抓取动作（即仿真手指环绕Fribble闭合）通过GraspIt!的AutoGrasp功能完成。每个Fribble共被抓取24次：沿每个轴旋转8次（每次旋转45°）。为确保Fribble可适配GraspIt!的仿真手，我们将其尺寸缩小了29%。可参考数据集附带的示例图像及抓取流程示意图。本数据集文件包含3个文件夹与1个CSV文件。文件夹"Fa"、"Fb"、"Fc"分别对应本次研究涉及的A、B、C三个Fribble类别族，每个类别族文件夹下均包含以下子文件夹："renders"文件夹存储该类别下每个Fribble的渲染图像；"obj"文件夹存储该类别下每个Fribble的MTL与OBJ格式文件；"parts"文件夹存储源自Mike Tarr网页的原始Fribble部件，用于组合生成单个Fribble；"vrml"文件夹存储该类别下每个Fribble的VRML格式文件。文件命名基于组合生成该Fribble的部件名称，例如4_a1b3c2d3由4_body.obj、4_a1.obj、4_b3.obj、4_c2.obj及4_d3.obj组合而成。最后，"haptics.csv"文件存储了所有Fribble的触觉特征数据。该CSV文件包含892行（含1行表头）与255列：第一列为Fribble的类别族与样本名称；后续254列为GraspIt!输出的关节角度数据：我们针对24种不同的旋转姿态（沿每个轴旋转45°）各执行一次抓取，每次抓取对应16个自由度（Degree of Freedom, DOF）值，用于表征仿真手的关节位置；剩余列存储GraspIt!中Fribble与仿真手的接触点数据。每次抓取（每个Fribble共24次抓取）的第一个数值为该次抓取的接触点数量；每个接触点的前6个数值对应接触点的力旋量（wrench），接下来3个数值为接触点在物体坐标系下的位置，最后1个数值为该接触点的标量约束误差。使用VTK进行物体视觉渲染、使用GraspIt!进行触觉渲染的示例代码可在SampleCode文件夹中获取。请参考GraspItUpdate.pdf文件，了解如何更新用于触觉渲染的GraspIt! TCP服务器代码。示例代码文件夹中还包含aomr.xml文件，该文件为GraspIt!触觉渲染所用的世界场景文件（即仿真人手的配置文件）。如使用本See & Grasp 2数据集，请引用以下文献：Yildirim, I. & Jacobs, R. A. (2013). 跨视觉与触觉模态的物体类别知识迁移：实验与计算研究. *Cognition*, 126, 135-148. 如需引用GraspIt!模拟器，请参考：Miller, A. & Allen, P. K. (2004). GraspIt!: 面向机器人抓取的多功能模拟器. *IEEE Robotics and Automation Magazine*, 11, 110-122.