dora-rs/dora-idefics2

# Getting Started

Create a new conda environment for robomaster

```bash
conda create -n robomaster python=3.8
pip install robomaster dora-rs==0.3.3
```

Create a new conda environment for idefics2. This requirements file suppose that your using cu122.

```bash
conda create -n idefics2 python=3.10
conda activate idefics2
pip install -r requirements.txt
```

## Robomaster Jailbreak

### Installation of the Robomaster S1 Hack

This guide is an updated version of the original [Robomaster S1 SDK Hack Guide](https://www.bug-br.org.br/s1_sdk_hack.zip) and is intended for use on a Windows 11 system.

#### Prerequisites

Before you get started, you'll need the following:

- Robomaster S1 (do not update it to the latest version, as it may block the hack).
- [Robomaster App](https://www.dji.com/fr/robomaster-s1/downloads).
- [Android SDK Platform-Tools](https://developer.android.com/tools/releases/platform-tools). Simply unzip it and keep the path handy.
- A micro USB cable. If this guide doesn't work, there might be an issue with the cable, and you may need to replace it with one that supports data transfer.

#### Instructions

1. Start the Robomaster App and connect the Robomaster S1 using one of the two options provided (via router or via Wi-Fi).
2. While connected, use a micro USB cable to connect the robot to the computer's USB port. You should hear a beep sound, similar to when you connect any device. (Please note that no other Android device should be connected via USB during this process).
3. In the Lab section of the app, create a new Python application and paste the following code:

```python
def root_me(module):
__import__ = rm_define.__dict__['__builtins__']['__import__']
return __import__(module, globals(), locals(), [], 0)

builtins = root_me('builtins')
subprocess = root_me('subprocess')
proc = subprocess.Popen('/system/bin/adb_en.sh', shell=True, executable='/system/bin/sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
```

4. Run the code; there should be no errors, and the console should display **Execution Complete**
5. Without closing the app, navigate to the folder containing the Android SDK Platform-Tools and open a terminal inside it.
6. Run the ADP command `.\adb.exe devices `. If everything is working correctly, you should see output similar to this: 
7. Execute the upload.sh script located in the folder `s1_SDK`.
8. Once everything has been executed, restart the S1 by turning it off and then back on. While it's booting up, you should hear two chimes instead of the usual single chime, indicating that the hack has been successful.

## Robomaster Connection

Make sure to be connected using the wifi hotspot of the robomaster which is the most stable one.

The default password for the hotpsot is: 12341234

You might need to have a second wifi card if you want to be able to run the demo with internet on.

## Post-Installation test

Please try running idefics2 with:

```bash
conda activate idefics2
python tests/test_idefics2.py
```

Please try running robomaster with:

```bash
conda activate robomaster
python tests/test_robomaster.py
```

## Running the demo

```bash
export HF_TOKEN=<TOKEN>
conda activate idefics2
# This requires dora==0.3.3, update with:
# cargo install dora-cli
dora up
dora start graphs/dataflow_robot_vlm.yml --attach --hot-reload
```

Current way to interact is by press up arrow key on laptop to record a message and send to the VLM

## Running the demo without robot

```bash
export HF_TOKEN=<TOKEN>
conda activate idefics2
# This requires dora==0.3.3, update with:
# cargo install dora-cli
dora up
dora start graphs/dataflow_vlm_basic.yml --attach --hot-reload
```

Current way to interact is by press up arrow key on laptop to record a message and send to the VLM

## Kill process in case of failure

Due to a Python GIL issue, we currently meed to kill processes manually. You can use the following command to do so:

```bash
pkill -f 'import dora;'
```

## LICENSE

While the source of this library is licensed under Apache-2.0, the usage of the Text to Speech(TTS) SystemEngine is licensed under Mozilla Public License 2.0 and GNU Lesser General Public License (LGPL) version 3.0.

Feel free to remove the TTS SystemEngine.

原始信息汇总

数据集概述

安装指南

• Robomaster Jailbreak:

  • 设备要求:
    • Robomaster S1（需避免更新至最新版本）
    • Robomaster App
    • Android SDK Platform-Tools
    • 支持数据传输的micro USB电缆
  • 操作步骤:
    1. 通过Robomaster App连接Robomaster S1。
    2. 使用micro USB电缆将机器人连接至计算机。
    3. 在App的Lab部分创建Python应用并执行特定代码。
    4. 执行完成后，通过Android SDK Platform-Tools进行进一步操作。
    5. 重启S1，成功后将听到两声启动音。

• Robomaster Connection:

  • 推荐使用Robomaster的Wi-Fi热点连接，密码为12341234。
  • 如需同时使用互联网，可能需要第二张Wi-Fi卡。

测试与运行

• Post-Installation Test:

  • 激活相应环境后，分别运行test_idefics2.py和test_robomaster.py进行测试。

• Running the demo:

  • 需要设置环境变量HF_TOKEN并激活idefics2环境。
  • 使用dora命令启动特定配置文件进行演示。

• Running the demo without robot:

  • 同上，但使用不同的配置文件进行无机器人演示。

• Kill process in case of failure:

  • 使用命令pkill -f import dora;手动终止进程。

许可证信息

• 本数据集的源代码遵循Apache-2.0许可证。
• 文本到语音系统引擎的使用遵循Mozilla Public License 2.0和LGPL 3.0。