govtech/SynthSite
收藏资源简介：
--- license: other license_name: govtech-singapore license_link: LICENSE language: - en tags: - video - safety - construction - synthetic - benchmark - computer-vision --- # SynthSite **SynthSite** is a curated benchmark of **227 synthetic construction site safety videos** (115 unsafe, 112 safe) generated using four text-to-video models: `Sora 2 Pro`, `Veo 3.1`, `Wan 2.2-14B`, and `Wan 2.6`. Each video was independently labeled by **2–3 human reviewers** for the presence of a **Worker Under Suspended Load** hazard, producing a binary classification: **unsafe** (`True_Positive` — worker remains in the suspended-load fall zone) or **safe** (`False_Positive` — no worker in fall zone or load not truly suspended). All content is fully AI-generated — no real people or real incidents are depicted. The terms "lifted load" and "suspended load" are used interchangeably throughout this repo. <!-- Paper link: TODO — update once preprint is available --> --- ## Sample Frames Each video is a short clip (5–10 seconds, 1280×720, 16–30 FPS). Frames below are sampled from unsafe and safe videos. ### Unsafe — Worker Under Suspended Load <table> <tr> <td align="center"><img src="assets/tp_sora_site.png" width="220"/></td> <td align="center"><img src="assets/tp_sora_steel.png" width="220"/></td> <td align="center"><img src="assets/tp_veo_night.png" width="220"/></td> </tr> <tr> <td align="center"><img src="assets/tp_veo_rain.png" width="220"/></td> <td align="center"><img src="assets/tp_wan22_pipe.png" width="220"/></td> <td align="center"><img src="assets/tp_wan26_rebar.png" width="220"/></td> </tr> </table> ### Safe — No Hazard Present <table> <tr> <td align="center"><img src="assets/fp_sora_site.png" width="220"/></td> <td align="center"><img src="assets/fp_veo_day.png" width="220"/></td> <td align="center"><img src="assets/fp_veo_sunset.png" width="220"/></td> </tr> <tr> <td align="center"><img src="assets/fp_veo_night.png" width="220"/></td> <td align="center"><img src="assets/fp_wan22_quiet.png" width="220"/></td> <td align="center"><img src="assets/fp_wan26_tunnel.png" width="220"/></td> </tr> </table> --- ## Generation Videos were generated from four models: | Model | Provider | Model License | Output Rights | Videos | |-------|----------|---------------|---------------|--------| | `Sora 2 Pro` | OpenAI | Proprietary (API ToS) | Assigned to user | 6 | | `Veo 3.1` | Google | Proprietary (API ToS) | Not claimed by Google | 124 | | `Wan 2.2-14B` | Alibaba / Wan-AI | Apache 2.0 | Not claimed by creators | 77 | | `Wan 2.6` | Alibaba / Wan-AI | Apache 2.0 | Not claimed by creators | 20 | All four models either assign output rights to the user or explicitly disclaim ownership of generated content, permitting redistribution under this dataset's license. The terms of service for each model were reviewed and confirmed to be compatible with open dataset release for academic and research use. No real video footage is included. The generation process used text intermediaries that sufficiently abstract away details from any source material, preventing re-identification. Provenance metadata (C2PA for Sora, SynthID for Veo) has been preserved. From an initial pool of **487** candidates, each video was manually screened before hazard annotation. Videos were excluded if they contained synthesis failures that prevented reliable hazard judgment — including severe artifacts, missing or fused workers/loads/machinery, abrupt appearance or disappearance of key entities, or major temporal corruption. Minor imperfections that did not hinder hazard interpretation (e.g., mild geometric distortion, low resolution, or corrupted text overlays) were retained. **227 videos (47%)** passed screening and form the final benchmark. ### Gemini 2.5 Flash Evaluation Each video was assessed using a rubric-guided Gemini 2.5 Flash pipeline with schema-constrained structured outputs, characterizing diversity, complexity, and realism. The evaluation script, rubric, and per-video results are in `code/gemini_synthsite_eval.py` and `results/gemini_synthsite_eval/`. ### VBench Statistics Scores are normalized to [0, 1], with higher values indicating better temporal stability. | Metric | Mean | Median | Std | |---|---|---|---| | Subject consistency | 0.9770 | 0.9826 | 0.0191 | | Background consistency | 0.9701 | 0.9769 | 0.0191 | | Motion smoothness | 0.9952 | 0.9962 | 0.0024 | | Temporal flickering | 0.9903 | 0.9935 | 0.0127 | --- ## Inter-Annotator Agreement | Scope | Videos | Weighted Avg Cohen's Kappa | Krippendorff's Alpha | |-------|--------|---------------------------|----------------------| | Global | 227 | 0.20 | 0.42 | | Tier 1 | 150 | 1.00 | 1.00 | | Tier 2 | 77 | -0.24 | -0.33 | The low global Kappa (0.20) is expected — it is driven by the intentionally-included ambiguous Tier 2 videos. Tier 1 achieves perfect agreement (1.00), while Tier 2 is below chance, confirming these videos are genuinely ambiguous for human reviewers. Pairwise metrics available in `results/annotators_agreement/cohens_kappa.csv`. --- ## Tier System Videos are assigned to tiers based on inter-annotator agreement: | Tier | Condition | Count | Interpretation | |------|-----------|-------|----------------| | Tier 1 | All reviewers agree | **150** (66%) | High-confidence ground truth | | Tier 2 | Reviewers disagree | **77** (34%) | Ambiguous — genuine disagreement | Systems can be scored primarily on Tier 1 (reliable ground truth), with Tier 2 performance reported separately as the hard set. --- ## Dataset Structure ``` SynthSite/ ├── LICENSE ├── README.md ├── synthetic_video_labels.csv ├── assets/ — Sample frame images for README ├── videos/ │ └── *.mp4 (227 files) ├── results/ │ ├── annotators_agreement/ │ │ ├── cohens_kappa.csv │ │ ├── krippendorffs_alpha.csv │ │ └── summary.csv │ ├── detector_outcome/ │ │ ├── confusion_matrix_grounding_dino.csv │ │ ├── confusion_matrix_yolo_world.csv │ │ ├── results_grounding_dino.csv │ │ └── results_yolo_world.csv │ ├── gemini_synthsite_eval/ │ │ └── *.json (227 files — Gemini 2.5 Flash per-video assessments) │ └── vbench_eval/ │ ├── synthsite_eval_results.json │ └── synthsite_scores.csv ├── code/ │ ├── gemini_synthsite_eval.py — Gemini video evaluation pipeline │ ├── gemini_agg_analysis.py — Aggregate Gemini outputs into tables/figures │ └── hazard_detector_eval.py — YOLO World + Grounding DINO evaluation └── docs/ ├── detection_design.md — Technical reference on detection approach ├── detection_parameters.md — Filter cascade design and calibration ├── detector_scores.md — Per-tier, per-generator detection results └── vbench_scores.md — VBench quality metrics for synthetic videos ``` ### Fields (`synthetic_video_labels.csv`) | Field | Type | Description | |-------|------|-------------| | `filename` | string | Video filename (matches file in `videos/`) | | `num_labelers` | int | Number of reviewers (2-3) | | `tier` | int | Agreement tier: `1` (agreement) or `2` (disagreement) | | `resolved_label` | string | Majority-vote label: `True_Positive` or `False_Positive` | | `labeler_N_name` | string | Anonymized reviewer ID (`Reviewer_01` through `Reviewer_08`) | | `labeler_N_label` | string | `True_Positive` or `False_Positive` | Columns repeat for N = 1 to 3. A 3rd reviewer is present only when the first two disagreed (tiebreaker). Empty values indicate fewer reviewers. --- ## Intended Uses - Benchmarking computer vision systems for construction site hazard detection - Studying the effectiveness of synthetic data for safety-critical applications - Evaluating inter-annotator agreement on ambiguous safety scenarios - Comparing video generation models for domain-specific content creation ## Out-of-Scope Uses - Misrepresenting synthetic content as real incident footage - Training systems intended to cause harm or circumvent safety measures - Substituting for real-world safety assessments or compliance evaluations --- ## Ethics - **No real people** appear in any video — all content is AI-generated - **No real incidents** are depicted — scenarios are synthetic constructions - Source material was abstracted through text intermediaries, preventing re-identification - Generative model terms of service were reviewed and complied with - Provenance metadata (C2PA, SynthID) preserved for transparency - Reviewer identities have been anonymized --- ## License This dataset is released under a custom GovTech Singapore license. See [`LICENSE`](LICENSE) for full terms. --- ## Getting Started This repository uses **Git LFS** for large files (videos, images). To clone with all assets: ```bash git lfs install git clone <repo-url> ``` Without Git LFS, video and image files will be downloaded as small pointer files.
--- license: 其他 license_name: govtech-singapore license_link: LICENSE language: - en tags: - 视频 - 安全 - 建筑 - 合成 - 基准测试 - 计算机视觉 --- # SynthSite **SynthSite** 是一个经过精选的基准数据集，包含227段合成建筑工地安全视频（其中115段标注为不安全场景，112段为安全场景），由四款文本到视频（text-to-video）模型生成，分别为`Sora 2 Pro`、`Veo 3.1`、`Wan 2.2-14B`以及`Wan 2.6`。每段视频均由2至3名标注人员独立标注，以判断是否存在**悬吊载荷下的工人（Worker Under Suspended Load）** 安全隐患，最终分为二分类标签：**不安全（`True_Positive`——工人处于悬吊载荷坠落危险区域内）** 与**安全（`False_Positive`——坠落危险区域内无工人，或载荷未真正处于悬吊状态）**。 所有内容均由人工智能生成，未包含真实人物或真实事故场景。本仓库中“起吊载荷”与“悬吊载荷”可互换使用。 <!-- Paper link: TODO — update once preprint is available --> --- ## 示例帧 每段视频均为短时长片段（5至10秒，分辨率1280×720，帧率16至30 FPS）。以下示例帧分别取自不安全场景与安全场景的视频。 ### 不安全场景——悬吊载荷下的工人 <table> <tr> <td align="center"><img src="assets/tp_sora_site.png" width="220"/></td> <td align="center"><img src="assets/tp_sora_steel.png" width="220"/></td> <td align="center"><img src="assets/tp_veo_night.png" width="220"/></td> </tr> <tr> <td align="center"><img src="assets/tp_veo_rain.png" width="220"/></td> <td align="center"><img src="assets/tp_wan22_pipe.png" width="220"/></td> <td align="center"><img src="assets/tp_wan26_rebar.png" width="220"/></td> </tr> </table> ### 安全场景——无安全隐患 <table> <tr> <td align="center"><img src="assets/fp_sora_site.png" width="220"/></td> <td align="center"><img src="assets/fp_veo_day.png" width="220"/></td> <td align="center"><img src="assets/fp_veo_sunset.png" width="220"/></td> </tr> <tr> <td align="center"><img src="assets/fp_veo_night.png" width="220"/></td> <td align="center"><img src="assets/fp_wan22_quiet.png" width="220"/></td> <td align="center"><img src="assets/fp_wan26_tunnel.png" width="220"/></td> </tr> </table> --- ## 视频生成 视频由以下四款模型生成： | 模型 | 提供商 | 模型许可证 | 输出权利 | 视频数量 | |-------|----------|---------------|---------------|--------| | `Sora 2 Pro` | OpenAI | 专有（API服务条款） | 归属于用户 | 6 | | `Veo 3.1` | Google | 专有（API服务条款） | 谷歌未主张所有权 | 124 | | `Wan 2.2-14B` | Alibaba / Wan-AI | Apache 2.0 | 创作者未主张所有权 | 77 | | `Wan 2.6` | Alibaba / Wan-AI | Apache 2.0 | 创作者未主张所有权 | 20 | 四款模型均将生成内容的输出权利归属于用户，或明确声明不拥有生成内容的所有权，允许依据本数据集的许可证进行再分发。本团队已审阅所有模型的服务条款，确认其符合学术与研究用途的开源数据集发布要求。 本数据集未包含真实视频素材。生成过程采用文本中介层，充分抽象了原始素材的细节，避免了身份重识别风险。同时保留了溯源元数据（Sora使用C2PA，Veo使用SynthID）以保障透明度。 本次数据集的初始候选视频共487段，所有视频在标注安全隐患前均经过人工筛选。若视频存在影响隐患判断的合成缺陷（包括严重伪影、工人/载荷/机械设备缺失或融合、关键实体突然出现或消失、严重时序损坏等），则会被排除。仅轻微影响隐患判断的瑕疵（如轻微几何畸变、分辨率较低、文本叠加损坏等）可保留。最终共有227段视频（占比47%）通过筛选，构成本基准数据集的主体内容。 ### Gemini 2.5 Flash 评估 所有视频均通过遵循评分准则的Gemini 2.5 Flash流水线进行评估，采用模式约束的结构化输出，从多样性、复杂度与真实感三个维度进行量化分析。评估脚本、评分准则及单视频评估结果可在`code/gemini_synthsite_eval.py`与`results/gemini_synthsite_eval/`路径下获取。 ### VBench 统计指标 所有指标得分均归一化至[0, 1]区间，得分越高代表时序稳定性越好。 | 指标 | 均值 | 中位数 | 标准差 | |---|---|---|---| | 主体一致性 | 0.9770 | 0.9826 | 0.0191 | | 背景一致性 | 0.9701 | 0.9769 | 0.0191 | | 运动平滑度 | 0.9952 | 0.9962 | 0.0024 | | 时序闪烁 | 0.9903 | 0.9935 | 0.0127 | --- ## 标注员间一致性 | 范围 | 视频数量 | 加权平均科恩kappa系数 | Krippendorff阿尔法系数 | |-------|--------|---------------------------|----------------------| | 全局范围 | 227 | 0.20 | 0.42 | | Tier 1 | 150 | 1.00 | 1.00 | | Tier 2 | 77 | -0.24 | -0.33 | 全局范围的科恩kappa系数较低（0.20）属于预期结果，这是由于数据集故意引入了存在歧义的Tier 2视频所致。Tier 1的标注一致性达到完美（1.00），而Tier 2的系数低于随机水平，证实此类视频对人工标注员而言确实存在歧义。两两标注员一致性指标可在`results/annotators_agreement/cohens_kappa.csv`中查看。 --- ## 层级分类体系 视频根据标注员间的一致性分为不同层级： | 层级 | 判定条件 | 数量 | 解读 | |------|-----------|-------|----------------| | Tier 1 | 所有标注员意见一致 | **150**（占比66%） | 高置信度真实标签 | | Tier 2 | 标注员意见分歧 | **77**（占比34%） | 存在歧义——标注员真实意见分歧 | 模型可主要基于Tier 1数据集（可靠真实标签）进行性能评估，Tier 2数据集可作为困难样本集单独报告性能。 --- ## 数据集结构 SynthSite/ ├── LICENSE ├── README.md ├── synthetic_video_labels.csv ├── assets/ — 用于README的示例帧图片 ├── videos/ │ └── *.mp4（共227个文件） ├── results/ │ ├── annotators_agreement/ │ │ ├── cohens_kappa.csv │ │ ├── krippendorffs_alpha.csv │ │ └── summary.csv │ ├── detector_outcome/ │ │ ├── confusion_matrix_grounding_dino.csv │ │ ├── confusion_matrix_yolo_world.csv │ │ ├── results_grounding_dino.csv │ │ └── results_yolo_world.csv │ ├── gemini_synthsite_eval/ │ │ └── *.json（共227个文件——Gemini 2.5 Flash单视频评估结果） │ └── vbench_eval/ │ ├── synthsite_eval_results.json │ └── synthsite_scores.csv ├── code/ │ ├── gemini_synthsite_eval.py — Gemini视频评估流水线 │ ├── gemini_agg_analysis.py — 聚合Gemini输出以生成表格/图表 │ └── hazard_detector_eval.py — YOLO World + Grounding DINO评估脚本 └── docs/ ├── detection_design.md — 检测方法技术参考 ├── detection_parameters.md — 过滤级联设计与校准 ├── detector_scores.md — 各层级、各生成模型的检测结果 └── vbench_scores.md — 合成视频的VBench质量指标 ### 字段说明（`synthetic_video_labels.csv`） | 字段 | 类型 | 描述 | |-------|------|-------------| | `filename` | 字符串 | 视频文件名（与`videos/`路径下的文件一致） | | `num_labelers` | 整数 | 标注员数量（2-3名） | | `tier` | 整数 | 标注一致性层级：`1`代表意见一致，`2`代表意见分歧 | | `resolved_label` | 字符串 | 最终标注标签：`True_Positive`或`False_Positive` | | `labeler_N_name` | 字符串 | 匿名化标注员ID（格式为`Reviewer_01`至`Reviewer_08`） | | `labeler_N_label` | 字符串 | 标注员N的标注结果：`True_Positive`或`False_Positive` | 标注员字段按N=1至3循环。仅当前两位标注员意见分歧时，才会加入第三位标注员作为平局破译者。字段为空则代表该位置无对应标注员。 --- ## 预期用途 - 针对建筑工地安全隐患检测的计算机视觉系统基准测试 - 研究合成数据在安全关键型应用中的有效性 - 评估歧义安全场景下的人工标注一致性 - 对比不同视频生成模型在垂直领域内容创作中的表现 --- ## 禁止用途 - 将合成内容虚假表述为真实事故画面 - 训练用于造成危害或规避安全措施的系统 - 替代真实世界的安全评估或合规性审查 --- ## 伦理规范 - **无真实人物**：所有视频均未包含真实人物，内容均由人工智能生成 - **无真实事故**：所有场景均为合成构建，未还原真实事故 - 原始素材通过文本中介层进行抽象处理，避免身份重识别风险 - 已审阅并遵守所有生成式模型的服务条款 - 保留溯源元数据（C2PA、SynthID）以保障透明度 - 标注员身份已匿名化 --- ## 许可证 本数据集采用新加坡政府科技局（GovTech Singapore）自定义许可证发布。完整条款请参阅[`LICENSE`](LICENSE)。 --- ## 快速入门 本仓库使用Git LFS存储大文件（视频、图片）。如需克隆所有资源，请执行以下命令： bash git lfs install git clone <repo-url> 若未使用Git LFS，视频与图片文件将仅作为小型指针文件下载。
