tom-jerry-123/Physical-AI-AV-IT

--- license: other task_categories: - robotics tags: - autonomous-driving - waypoints - webdataset size_categories: - 1M<n<10M --- # PhysicalAI-AV-SFT Supervised fine-tuning (SFT) dataset for an autonomous-vehicle vision-language waypoint-prediction model. Contains **29,991 samples** from ~150 000 driving scenes (~18 seconds per scene, sampled at anchor times 2s..16s) recorded in the United States. ## Format [WebDataset](https://github.com/webdataset/webdataset) — 3 uncompressed `.tar` shards, each containing pairs of files per sample: | Entry | Description | |---|---| | `{key}.jpg` | Front-facing wide-angle camera frame (JPEG quality 95, 640 × 360 px) | | `{key}.json` | Metadata (see schema below) | **Key format:** `{scene_id}__{sample_idx:02d}` **Shard assignment:** `sha256(scene_id) % 3` — all frames of a scene land in the same shard, preventing scene leakage across train/eval splits. ## Metadata schema (`{key}.json`) ```json { "scene_id": "UUID string — identifies the driving scene", "chunk_name": "chunk_XXXX — source data chunk", "sample_idx": "int 2–16 — target second within the scene (15 anchors/scene)", "global_idx": "int — globally unique datum index", "target_t_rel_us": "int — timestamp relative to scene start (microseconds)", "target_frame_index": "int — video frame index", "egomotion": "list[list[float]] — full available past trajectory (incl. anchor), target-relative, 0.25s granularity", "waypoints": "list[list[float]] — full available future trajectory, target-relative, 0.25s granularity", "is_long_tail": "bool — long-tail driving scenario flag" } ``` Coordinate convention: all `x`/`y`/`yaw` in the **ego-vehicle frame** at target time, +x = forward, +y = left, yaw in radians CCW from forward. ## Loading ```python import webdataset as wds, json from PIL import Image import io # Local (after cloning the repo) ds = wds.WebDataset("shards/train-{00000..00002}-of-00003.tar").shuffle(1000) for sample in ds: img = Image.open(io.BytesIO(sample["jpg"])) meta = json.loads(sample["json"]) # meta["waypoints"] → full available future waypoints at 0.25s granularity ``` ## Shard index `index.parquet` — one row per sample, columns: `key`, `shard`, `scene_id`, `chunk_name`, `sample_idx`, `global_idx`, `target_t_rel_us`, `is_long_tail`. ```python import pandas as pd df = pd.read_parquet("index.parquet") lt = df[df["is_long_tail"]] # long-tail subset ```--- 许可证：其他 任务类别： - 机器人学 标签： - 自动驾驶 - 航点 - WebDataset（WebDataset） 规模类别： - 1M<n<10M --- # PhysicalAI-AV-SFT 本数据集为自动驾驶视觉语言航点预测模型的监督微调（Supervised Fine-Tuning，SFT）数据集。数据集包含来自约15万个驾驶场景的29991条样本，每个场景时长约18秒，在2秒至16秒的锚点时刻进行采样，数据采集于美国境内。 ## 数据格式 采用WebDataset（项目地址：https://github.com/webdataset/webdataset）格式，包含3个未压缩的`.tar`分片，每个样本对应一组文件： | 条目 | 描述 | |---|---| | `{key}.jpg` | 前置广角相机拍摄帧（JPEG 质量95，分辨率640×360像素） | | `{key}.json` | 元数据文件（详见下文的元数据 schema） | **键格式：** `{scene_id}__{sample_idx:02d}` **分片分配规则：** 通过`sha256(scene_id) % 3`计算分片索引，同一驾驶场景的所有帧将被分配至同一个分片，避免训练/评估划分出现场景泄露问题。 ## 元数据 schema（`{key}.json`文件） json { "scene_id": "UUID字符串，用于标识驾驶场景", "chunk_name": "格式为chunk_XXXX，代表源数据块", "sample_idx": "整数，取值范围2至16，对应场景内的目标时刻（每个场景共15个锚点样本）", "global_idx": "整数，全局唯一的数据索引", "target_t_rel_us": "整数，相对于场景起始时刻的时间戳，单位为微秒", "target_frame_index": "整数，视频帧索引", "egomotion": "二维浮点列表，包含完整可用的历史轨迹（含当前锚点），采用目标相对坐标系，粒度为0.25秒", "waypoints": "二维浮点列表，包含完整可用的未来轨迹，采用目标相对坐标系，粒度为0.25秒", "is_long_tail": "布尔值，用于标记是否为长尾驾驶场景" } **坐标系约定：** 所有`x`/`y`/`yaw`均采用目标时刻下的**自车坐标系**，其中+x轴指向车辆前方，+y轴指向车辆左侧，yaw为相对于前方的逆时针弧度值。 ## 数据加载 python import webdataset as wds, json from PIL import Image import io # 本地加载（克隆仓库后执行） ds = wds.WebDataset("shards/train-{00000..00002}-of-00003.tar").shuffle(1000) for sample in ds: img = Image.open(io.BytesIO(sample["jpg"])) meta = json.loads(sample["json"]) # meta["waypoints"] → 粒度为0.25秒的完整可用未来航点轨迹 ## 分片索引 `index.parquet`为分片索引文件，每条样本对应一行，包含以下列：`key`、`shard`、`scene_id`、`chunk_name`、`sample_idx`、`global_idx`、`target_t_rel_us`、`is_long_tail`。 python import pandas as pd df = pd.read_parquet("index.parquet") lt = df[df["is_long_tail"]] # 长尾场景子集