BitRobot/FrodoBots-2K

--- license: cc-by-sa-4.0 task_categories: - reinforcement-learning - robotics language: - en annotations_creators: - experts-generated tags: - self-driving - robotics navigation pretty_name: FrodoBots 2K Dataset --- ## Dataset Description - **Homepage:** https://www.frodobots.ai/ - **Hours of tele-operation:** ~2,000 Hrs - **Dataset Size:** 1TB - **Point of Contact:** [e-mail] # FrodoBots 2K Dataset The **FrodoBots 2K Dataset** is a diverse collection of camera footage, GPS, IMU, audio recordings & human control data collected from ~2,000 hours of tele-operated sidewalk robots driving in 10+ cities. This dataset is collected from [**Earth Rovers**](https://www.frodobots.ai/earth-rovers), a global scavenger hunt "Drive to Earn" game developed by [**FrodoBots Lab**](https://www.frodobots.ai). Please join our [**Discord**](https://discord.com/invite/AUegJCJwyb) for discussions with fellow researchers/makers! - If you're interested in contributing driving data, you can buy your own unit(s) from our [**online shop**](https://shop.frodobots.com/) (US$299 per unit) and start driving around your neighborhood (& earn in-game points in the process)! - If you're interested in testing out your AI models on our existing fleet of Earth Rovers in various cities or your own Earth Rover, feel free to DM [**Michael Cho on Twitter/X**](https://twitter.com/micoolcho) to gain access to our Remote Access SDK. - If you're interested in playing the game (ie. remotely driving an Earth Rover), you may join as a gamer at [**Earth Rovers School**](https://www.frodobots.ai/earth-rovers-school). [](https://youtu.be/yZVflvg4kBw?si=4lVWnrywLr1dOcjR) ## Dataset Summary There are 7 types of data that are associated with a typical Earth Rovers drive, as follows: 1. **Control data**: Gamer's control inputs captured at a frequency of 10Hz (Ideal) as well as the RPM (revolutions per minute) readings for each of the 4 wheels on the robot. 2. **GPS data**: Latitude, longitude, and timestamp info collected during the robot drives at a frequency of 1Hz. 3. **IMU (Inertial Measurement Unit) data**: 9-DOF sensor data, including acceleration (captured at 100Hz), gyroscope (captured at 1Hz), and magnetometer info (captured at 1Hz), along with timestamp data. 4. **Rear camera video**: Video footage captured by the robot's rear-facing camera at a typical frame rate of 20 FPS with a resolution of 540x360. 5. **Front camera video**: Video footage captured by the robot's front-facing camera at a typical frame rate of 20 FPS with a resolution of 1024x576. 6. **Microphone**: Audio recordings captured by the robot's microphone, with a sample rate of 16000Hz, channel 1. 7. **Speaker**: Audio recordings of the robot's speaker output (ie. gamer's microphone), also with a sample rate of 16000Hz, channel 1. Note: As of 12 May 2024, ~1,300 hrs are ready for download. The remaining ~700 hours are still undergoing data cleaning and will be available for download by end May or early June. **Video Walkthrough** Our cofounder, Michael Cho, walks through the core components of the dataset, as well as a discussion on latency issues surrounding the data collection. [](https://youtu.be/M4l67YXnhqk) In total, there were 9,000+ individual driving sessions recorded. The chart below shows the distribution of individual driving session duration. |  | |:--:| These drives were done with Earth Rovers in 10+ cities. The chart below shows the distribution of recorded driving duration in the various cities. | | |:--:| ## About FrodoBots [**FrodoBots**](https://www.frodobots.ai/) is a project aiming to crowdsource the world's largest real-world teleoperation datasets with robotic gaming. We have 3 core thesis: 1. **Robotic gaming can be a thing**: It is possible to create fun gaming experience where gamers control robots remotely to complete missions in real life. 2. **Affordable robots are just as useful in collecting data for Embodied AI research**: We design our robots to be like "toys", so that as many people as possible can afford to buy one and play with them. 3. **DePIN can scale this project**: We can create a global community of robot hardware owners/operators by incentivizing them with well-designed tokenomics, taking best practices from other DePIN (Decentralized Physical Infrastructure Network) projects. |  | |:--:| | *Testing in Madrid* | |  | |:--:| | *Testing in London* | |  | |:--:| | *Testing in Stockholm* | |  | |:--:| | *Testing in Wuhan* | |  | |:--:| | *Testing in Liuzhou* | |  | |:--:| | *Testing in Berlin* | |  | |:--:| | *Game Controller + Browser = Control FrodoBots Anywhere* | |  | |:--:| | *Chatting with locals via built-in microphone/speaker* | |  | |:--:| | *Zero turning radius = Easy maneuvering* | |  | |:--:| | *Night driving test in Palo Alto* | |  | |:--:| | *Driving through rain* | |  | |:--:| | *Road crossing in Palo Alto* | |  | |:--:| | *Earth Rover being tested in Stanford University campus* | ## Motivations for open-sourcing the dataset The team behind FrodoBots is focused on building an real-world video gaming experience using real-life robots (we call it "robotic gaming"). A by-product of gamers playing the game is the accompanying dataset that's generated. By sharing this dataset with the research community, we hope to see new innovations that can (1) take advantage of this dataset & (2) leverage our existing fleet of community-sourced robots (via our Remote Access SDK) as a platform for testing SOTA Embodied AI models in the real world. ## Help needed! We are a very small team with little experience in various downstream data pipeline and AI research skillsets. One thing we do have is lots of real-world data. Please reach out or join our [**Discord**](https://discord.com/invite/AUegJCJwyb) if you'd have any feedback or like to contribute to our efforts, especially on following: - **Data cleaning**: We have way more data than what we've open-sourced in this dataset, primarily because we struggle with variuos data cleaning tasks. - **Data analytics**: We have done a couple charts but that's about it. - **Data annotations**: We have open-sourced the raw files, but it'll be great to work with teams with data annotation know-how to further augment the current dataset. - **Data visualization**: A lot more can be done to visualize some of these raw inputs (eg. layering timestamped data on top of the video footage). - **Data anonymization**: We'd like to build in various data anonymization (eg. face blurring) in future releases. We attempted to do this but struggled with downstream data manipulation issues (eg. dropped frames, lower video resolution, etc) - **Data streaming & hosting**: If this project continues to scale, we'd have millions of hours of such data in the future. Will need help with storage/streaming. ## **Download** Download FrodoBots dataset using the link in this [csv file](https://frodobots-2k-dataset.s3.ap-southeast-1.amazonaws.com/complete-dataset.csv). ## **Helper code** We've provided a helpercode.ipynb file that will hopefully serve as a quick-start for researchers to play around with the dataset. ## **Contributions** The team at FrodoBots Lab created this dataset, including [Michael Cho](https://twitter.com/micoolcho), Sam Cho, Aaron Tung, Niresh Dravin & Santiago Pravisani.--- 许可证：CC BY-SA 4.0（知识共享署名-相同方式共享4.0协议） 任务类别： - 强化学习（reinforcement-learning） - 机器人学（robotics） 语言： - 英语（en） 注释生成者： - 专家生成（experts-generated） 标签： - 自动驾驶（self-driving） - 机器人导航（robotics navigation） 展示名称：FrodoBots 2K 数据集（FrodoBots 2K Dataset） --- ## 数据集说明 - **主页**：https://www.frodobots.ai/ - **远程操控总时长**：约2000小时 - **数据集规模**：1TB - **联系人**：[e-mail] # FrodoBots 2K 数据集（FrodoBots 2K Dataset） **FrodoBots 2K 数据集**是一个多样化的多模态数据集，涵盖摄像头影像素材、全球定位系统（Global Positioning System, GPS）数据、惯性测量单元（Inertial Measurement Unit, IMU）数据、音频录制内容与人类操控数据，采集自10余个城市中约2000小时的远程操控人行道机器人行驶过程。 本数据集采集自由FrodoBots实验室（FrodoBots Lab）开发的全球寻宝式“边驾驶边赚取奖励”游戏**Earth Rovers（Earth Rovers）**。 欢迎加入我们的[Discord社区](https://discord.com/invite/AUegJCJwyb)，与研究人员及创客交流探讨！ - 若您希望贡献驾驶数据，可通过我们的[官方线上商店](https://shop.frodobots.com/)购买机器人设备（单价299美元），即可在社区周边开展驾驶任务，并在此过程中赚取游戏积分！ - 若您希望在我们分布于全球多城市的现有Earth Rovers机器人集群或自有Earth Rovers设备上测试AI模型，可直接通过Twitter/X向[Michael Cho](https://twitter.com/micoolcho)发送私信，以获取远程访问软件开发工具包（Remote Access SDK, SDK）的使用权限。 - 若您希望体验该游戏（即远程操控Earth Rovers机器人），可作为玩家加入[Earth Rovers学院](https://www.frodobots.ai/earth-rovers-school)。 [](https://youtu.be/yZVflvg4kBw?si=4lVWnrywLr1dOcjR) ## 数据集摘要 典型的Earth Rovers驾驶任务对应7类数据，具体如下： 1. **操控数据**：以理想状态下10赫兹的频率采集的玩家操控输入数据，以及机器人4个车轮各自的每分钟转数（revolutions per minute, RPM）读数。 2. **GPS数据**：以1赫兹的频率采集的机器人行驶过程中的纬度、经度与时间戳信息。 3. **IMU数据**：9自由度传感器数据，包含以100赫兹采集的加速度数据、以1赫兹采集的陀螺仪数据与磁力计数据，以及配套的时间戳数据。 4. **后置摄像头影像**：由机器人后置摄像头采集的影像素材，典型帧率为20帧每秒（Frames Per Second, FPS），分辨率为540×360。 5. **前置摄像头影像**：由机器人前置摄像头采集的影像素材，典型帧率为20帧每秒（Frames Per Second, FPS），分辨率为1024×576。 6. **麦克风音频**：由机器人内置麦克风采集的音频录制内容，采样率为16000赫兹，单声道。 7. **扬声器音频**：机器人扬声器输出的音频录制内容（即玩家麦克风的声音），采样率同样为16000赫兹，单声道。 注：截至2024年5月12日，约1300小时的数据已可下载。剩余约700小时的数据仍在进行数据清洗工作，预计将于5月底或6月初开放下载。 **视频演示** 我们的联合创始人Michael Cho将为您讲解本数据集的核心组成部分，并探讨数据采集过程中涉及的延迟问题。 [](https://youtu.be/M4l67YXnhqk) 本次共记录了9000余段独立驾驶会话，下图展示了各驾驶会话的时长分布。 |  | |:--:| 这些驾驶任务均在10余个城市的Earth Rovers机器人上完成，下图展示了各城市的记录驾驶时长分布。 | | |:--:| ## 关于FrodoBots [**FrodoBots**](https://www.frodobots.ai/) 是一个旨在通过机器人游戏众包构建全球规模最大的真实世界远程操控数据集的项目。 我们秉持三大核心理念： 1. **机器人游戏具备可行性**：我们可以打造富有乐趣的游戏体验，让玩家远程操控真实世界中的机器人完成任务。 2. **平价机器人同样可用于具身人工智能（Embodied AI）研究的数据采集**：我们将机器人设计为类似“玩具”的形态，以便尽可能多的用户能够负担并体验它们。 3. **去中心化物理基础设施网络（Decentralized Physical Infrastructure Network, DePIN）可助力项目规模化**：我们可以借鉴其他DePIN项目的最佳实践，通过设计合理的代币经济学（tokenomics）机制激励机器人硬件所有者与操作者，从而构建全球性的社区生态。 |  | |:--:| | *在马德里进行测试* | |  | |:--:| | *在伦敦进行测试* | |  | |:--:| | *在斯德哥尔摩进行测试* | |  | |:--:| | *在武汉进行测试* | |  | |:--:| | *在柳州进行测试* | |  | |:--:| | *在柏林进行测试* | |  | |:--:| | *游戏控制器+浏览器：随时随地操控FrodoBots机器人* | |  | |:--:| | *通过内置麦克风与扬声器与当地人交流* | |  | |:--:| | *零转弯半径：灵活机动* | |  | |:--:| | *帕洛阿尔托夜间行驶测试* | |  | |:--:| | *雨天行驶* | |  | |:--:| | *帕洛阿尔托路口通行* | |  | |:--:| | *Earth Rovers在斯坦福大学校园进行测试* | ## 开源本数据集的初衷 FrodoBots团队致力于打造基于真实机器人的真实世界视频游戏体验（我们称之为“机器人游戏”），而玩家游玩该游戏所产生的附属数据则构成本数据集。 通过向研究社区共享本数据集，我们期望催生两类创新：一是充分利用本数据集开展研究，二是借助我们的社区众包机器人集群（通过远程访问软件开发工具包Remote Access SDK）作为测试平台，在真实世界中验证前沿的具身人工智能模型。 ## 寻求协助！ 我们团队规模较小，在各类下游数据管线与人工智能研究技能方面经验不足，但我们拥有大量真实世界的数据集。 若您有任何反馈或希望为我们的工作贡献力量，欢迎随时联系我们或加入[Discord社区](https://discord.com/invite/AUegJCJwyb)，尤其欢迎在以下领域提供协助： - **数据清洗**：我们拥有远超本次开源数据集的原始数据，主要原因是我们在各类数据清洗任务上存在瓶颈。 - **数据分析**：我们仅制作了少量图表，后续还有大量工作待开展。 - **数据标注**：我们已开源原始数据文件，若能与具备数据标注专业能力的团队合作进一步丰富本数据集，将不胜荣幸。 - **数据可视化**：针对各类原始输入数据（例如将带时间戳的数据叠加至影像素材中），我们还有更多可视化方案可供探索。 - **数据匿名化**：我们计划在未来的版本中加入各类数据匿名化处理功能（例如人脸模糊），但此前的尝试因下游数据处理问题（例如丢帧、视频分辨率降低等）未能成功。 - **数据流式传输与托管**：若本项目持续规模化发展，我们未来将面临数百万小时的此类数据存储与传输需求，亟需相关领域的协助。 ## **下载** 可通过此[CSV文件](https://frodobots-2k-dataset.s3.ap-southeast-1.amazonaws.com/complete-dataset.csv)中的链接下载FrodoBots 2K数据集。 ## **辅助代码** 我们提供了helpercode.ipynb文件，以期为研究人员快速上手本数据集提供指引。 ## **贡献人员** FrodoBots实验室团队打造了本数据集，成员包括[Michael Cho](https://twitter.com/micoolcho)、Sam Cho、Aaron Tung、Niresh Dravin与Santiago Pravisani。