hoffman-lab/SkyScenes

Name: hoffman-lab/SkyScenes
Creator: hoffman-lab
Published: 2024-03-04T19:15:44+08:00

Hugging Face2024-09-25 更新2024-03-04 收录

空中场景理解

数据集成

数据链接：

https://hf-mirror.com/datasets/hoffman-lab/SkyScenes 数据链接链接失效反馈

官方服务：

资源简介：

--- license: mit language: - en task_categories: - object-detection - depth-estimation - image-segmentation tags: - dataset - aerial - synthetic - domain adaptation - sim2real ---  # SkyScenes: A Synthetic Dataset for Aerial Scene Understanding [Sahil Khose](https://sahilkhose.github.io/)\*, [Anisha Pal](https://anipal.github.io/)\*, [Aayushi Agarwal](https://www.linkedin.com/in/aayushiag/)\*, [Deepanshi](https://www.linkedin.com/in/deepanshi-d/)\*, [Judy Hoffman](https://faculty.cc.gatech.edu/~judy/), [Prithvijit Chattopadhyay](https://prithv1.xyz/)  [![HuggingFace Dataset](https://img.shields.io/badge/🤗-HuggingFace%20Dataset-cyan.svg)](https://huggingface.co/datasets/hoffman-lab/SkyScenes)[![Project Page](https://img.shields.io/badge/Project-Website-orange)](https://hoffman-group.github.io/SkyScenes/)[![arXiv](https://img.shields.io/badge/arXiv-SkyScenes-b31b1b.svg)](https://arxiv.org/abs/2312.06719) <img src="./assets/teaser.jpeg" width="100%"/> ## Dataset Summary Real-world aerial scene understanding is limited by a lack of datasets that contain densely annotated images curated under a diverse set of conditions. Due to inherent challenges in obtaining such images in controlled real-world settings, we present SkyScenes, a synthetic dataset of densely annotated aerial images captured from Unmanned Aerial Vehicle (UAV) perspectives. **SkyScenes** images are carefully curated from **CARLA** to comprehensively capture diversity across layout (urban and rural maps), weather conditions, times of day, pitch angles and altitudes with corresponding semantic, instance and depth annotations. **SkyScenes** features **33,600** images in total, which are spread across 8 towns, 5 weather and daytime conditions and 12 height and pitch variations. ## 📣 Announcement SkyScenes has been accepted at [ECCV 2024](https://www.ecva.net/papers/eccv_2024/papers_ECCV/html/10113_ECCV_2024_paper.php) ! ## SkyScenes Details <details> <summary>Click to view the detailed list of all variations</summary> - **Layout Variations(Total 8):**: - Town01 - Town02 - Town03 - Town04 - Town05 - Town06 - Town07 - Town10HD _Town07 features Rural Scenes, whereas the rest of the towns feature Urban scenes_ - **Weather & Daytime Variations(Total 5):** - ClearNoon - ClearSunset - ClearNight - CloudyNoon - MidRainyNoon - **Height and Pitch Variations of UAV Flight(Total 12):** - Height = 15m, Pitch = 0° - Height = 15m, Pitch = 45° - Height = 15m, Pitch = 60° - Height = 15m, Pitch = 90° - Height = 35m, Pitch = 0° - Height = 35m, Pitch = 45° - Height = 35m, Pitch = 60° - Height = 35m, Pitch = 90° - Height = 60m, Pitch = 0° - Height = 60m, Pitch = 45° - Height = 60m, Pitch = 60° - Height = 60m, Pitch = 90° </details> <details> <summary>Click to view class definitions, color palette and class IDs for Semantic Segmentation</summary> **SkyScenes** semantic segmentation labels span 28 classes which can be further collapsed to 20 classes. | Class ID | Class ID (collapsed) | RGB Color Palette | Class Name | Definition | |----------|--------------------|-------------------|------------------|----------------------------------------------------------------------------------------------------| | 0 | -1 | (0, 0, 0) | unlabeled | Elements/objects in the scene that have not been categorized | | 1 | 2 | (70, 70, 70) | building | Includes houses, skyscrapers, and the elements attached to them | | 2 | 4 | (190, 153, 153) | fence | Wood or wire assemblies that enclose an area of ground | | 3 | -1 | (55, 90, 80) | other | Uncategorized elements | | 4 | 11 | (220, 20, 60) | pedestrian | Humans that walk | | 5 | 5 | (153, 153, 153) | pole | Vertically oriented pole and its horizontal components if any | | 6 | 16 | (157, 234, 50) | roadline | Markings on road | | 7 | 0 | (128, 64, 128) | road | Lanes, streets, paved areas on which cars drive | | 8 | 1 | (244, 35, 232) | sidewalk | Parts of ground designated for pedestrians or cyclists | | 9 | 8 | (107, 142, 35) | vegetation | Trees, hedges, all kinds of vertical vegetation (ground-level vegetation is not included here) | | 10 | 13 | (0, 0, 142) | cars | Cars in scene | | 11 | 3 | (102, 102, 156) | wall | Individual standing walls, not part of buildings | | 12 | 7 | (220, 220, 0) | traffic sign | Signs installed by the state/city authority, usually for traffic regulation | | 13 | 10 | (70, 130, 180) | sky | Open sky, including clouds and sun | | 14 | -1 | (81, 0, 81) | ground | Any horizontal ground-level structures that do not match any other category | | 15 | -1 | (150, 100, 100) | bridge | The structure of the bridge | | 16 | -1 | (230, 150, 140) | railtrack | Rail tracks that are non-drivable by cars | | 17 | -1 | (180, 165, 180) | guardrail | Guard rails / crash barriers | | 18 | 6 | (250, 170, 30) | traffic light | Traffic light boxes without their poles | | 19 | -1 | (110, 190, 160) | static | Elements in the scene and props that are immovable | | 20 | -1 | (170, 120, 50) | dynamic | Elements whose position is susceptible to change over time | | 21 | 19 | (45, 60, 150) | water | Horizontal water surfaces | | 22 | 9 | (152, 251, 152) | terrain | Grass, ground-level vegetation, soil, or sand | | 23 | 12 | (255, 0, 0) | rider | Humans that ride/drive any kind of vehicle or mobility system | | 24 | 18 | (119, 11, 32) | bicycle | Bicycles in scenes | | 25 | 17 | (0, 0, 230) | motorcycle | Motorcycles in scene | | 26 | 15 | (0, 60, 100) | bus | Buses in scenes | | 27 | 14 | (0, 0, 70) | truck | Trucks in scenes | | </details> ## Dataset Structure The dataset is organized in the following structure:  ``` ├── Images (RGB Images) │ ├── H_15_P_0 │ │ ├── ClearNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── ClearSunset │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── ClearNight │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── CloudyNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ └── MidRainyNoon │ │ ├── Town01 │ │ │ └── Town01.tar.gz │ │ ├── Town02 │ │ │ └── Town02.tar.gz │ │ ├── ... │ │ └── Town10HD │ │ └── Town10HD.tar.gz │ ├── H_15_P_45 │ │ └── ... │ ├── ... │ └── H_60_P_90 │ └── ... ├── Instance (Instance Segmentation Annotations) │ ├── H_35_P_45 │ │ └── ClearNoon │ │ ├── Town01 │ │ │ └── Town01.tar.gz │ │ ├── Town02 │ │ │ └── Town02.tar.gz │ │ ├── ... │ │ └── Town10HD │ │ └── Town10HD.tar.gz │ └── ... ├── Segment (Semantic Segmentation Annotations) │ ├── H_15_P_0 │ │ ├── ClearNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── H_15_P_45 │ │ │ └── ... │ │ ├── ... │ │ └── H_60_P_90 │ │ └── ... │ └── ... └── Depth (Depth Annotations) ├── H_35_P_45 │ └── ClearNoon │ ├── Town01 │ │ └── Town01.tar.gz │ ├── Town02 │ │ └── Town02.tar.gz │ ├── ... │ └── Town10HD │ └── Town10HD.tar.gz └── ... ``` **Note**: Since the same viewpoint is reproduced across each weather variation, hence ClearNoon annotations can be used for all images pertaining to the different weather variations. ## Dataset Download The dataset can be downloaded using wget. Since SkyScenes offers variations across different axes we enable different subsets for download that can aid in model sensitivity analysis across these axes. ### Download instructions: wget **Example script for downloading different subsets of data using wget** ```bash #!/bin/bash #Change here to download a specific Height and Pitch Variation, for example - H_15_P_0 # HP=('H_15_P_45' 'H_15_P_60' 'H_15_P_90') HP=('H_15_P_0' 'H_15_P_45' 'H_15_P_60' 'H_15_P_90' 'H_35_P_0' 'H_35_P_45' 'H_35_P_60' 'H_35_P_90' 'H_60_P_0' 'H_60_P_45' 'H_60_P_60' 'H_60_P_90') #Change here to download a specific weather subset, for example - ClearNoon #Note - For Segment, Instance and Depth annotations this field should only have ClearNoon variation # weather=('ClearNoon' 'ClearNight') weather=('ClearNoon' 'ClearNight' 'ClearSunset' 'CloudyNoon' 'MidRainyNoon') #Change here to download a specific Town subset, for example - Town07 layout=('Town01' 'Town02' 'Town03' 'Town04' 'Town05' 'Town06' 'Town07' 'Town10HD') #Change here for any specific annotation, for example - https://huggingface.co/datasets/hoffman-lab/SkyScenes/resolve/main/Segment base_url=('https://huggingface.co/datasets/hoffman-lab/SkyScenes/resolve/main/Images') #Change here for base download folder base_download_folder='SkyScenes' for hp in "${HP[@]}"; do for w in "${weather[@]}"; do for t in "${layout[@]}"; do folder=$(echo "$base_url" | awk -F '/' '{print $(NF)}') download_url="${base_url}/${hp}/${w}/${t}/${t}.tar.gz" download_folder="${base_download_folder}/${folder}/${hp}/${w}/${t}" mkdir -p "$download_folder" echo "Downloading: $download_url" wget -P "$download_folder" "$download_url" done done done ```  ## BibTex If you find this work useful please like ❤️ our dataset repo and cite 📄 our paper. Thanks for your support! ``` @misc{khose2023skyscenes, title={SkyScenes: A Synthetic Dataset for Aerial Scene Understanding}, author={Sahil Khose and Anisha Pal and Aayushi Agarwal and Deepanshi and Judy Hoffman and Prithvijit Chattopadhyay}, year={2023}, eprint={2312.06719}, archivePrefix={arXiv}, primaryClass={cs.CV} } ```

提供机构：

hoffman-lab

原始信息汇总

SkyScenes 数据集概述

数据集简介

SkyScenes 是一个合成数据集，包含从无人机视角捕捉的密集标注的航拍图像。该数据集旨在解决真实世界航拍场景理解中缺乏多样化条件下的密集标注图像的问题。SkyScenes 数据集包含 33,600 张图像，涵盖 8 个城镇、5 种天气和白天条件以及 12 种高度和俯仰角度的变化。

详细变化列表

布局变化（共 8 种）：
- Town01 至 Town10HD
- 其中 Town07 为乡村场景，其余为城市场景
天气与白天变化（共 5 种）：
- ClearNoon
- ClearSunset
- ClearNight
- CloudyNoon
- MidRainyNoon
无人机飞行高度和俯仰角变化（共 12 种）：
- 高度 = 15m, 俯仰角 = 0°, 45°, 60°, 90°
- 高度 = 35m, 俯仰角 = 0°, 45°, 60°, 90°
- 高度 = 60m, 俯仰角 = 0°, 45°, 60°, 90°

语义分割类别定义

SkyScenes 数据集的语义分割标签涵盖 28 个类别，可进一步合并为 20 个类别。

类别 ID	合并类别 ID	RGB 颜色	类别名称	定义
0	-1	(0, 0, 0)	未标注	场景中未分类的元素
1	2	(70, 70, 70)	建筑物	包括房屋、摩天大楼及其附属元素
2	4	(190, 153, 153)	栅栏	围栏或电线组成的围栏
3	-1	(55, 90, 80)	其他	未分类的元素
4	11	(220, 20, 60)	行人	步行的人
5	5	(153, 153, 153)	杆	垂直的杆及其水平组件
6	16	(157, 234, 50)	道路线	道路上的标记
7	0	(128, 64, 128)	道路	车道、街道、铺砌的行驶区域
8	1	(244, 35, 232)	人行道	行人或自行车专用的地面部分
9	8	(107, 142, 35)	植被	树木、灌木等垂直植被
10	13	(0, 0, 142)	汽车	场景中的汽车
11	3	(102, 102, 156)	墙	独立的墙，不包括建筑物的一部分
12	7	(220, 220, 0)	交通标志	由国家/城市当局安装的交通管制标志
13	10	(70, 130, 180)	天空	开放的天空，包括云和太阳
14	-1	(81, 0, 81)	地面	不符合其他类别的水平地面结构
15	-1	(150, 100, 100)	桥	桥的结构
16	-1	(230, 150, 140)	铁路轨道	非汽车可行驶的铁路轨道
17	-1	(180, 165, 180)	护栏	护栏/防撞栏
18	6	(250, 170, 30)	交通灯	交通灯箱，不包括杆
19	-1	(110, 190, 160)	静态	场景中的不可移动元素和道具
20	-1	(170, 120, 50)	动态	位置随时间变化的元素
21	19	(45, 60, 150)	水	水平水表面
22	9	(152, 251, 152)	地形	草地、地面植被、土壤或沙子
23	12	(255, 0, 0)	骑手	骑乘任何交通工具或移动系统的人
24	18	(119, 11, 32)	自行车	场景中的自行车
25	17	(0, 0, 230)	摩托车	场景中的摩托车
26	15	(0, 60, 100)	公交车	场景中的公交车
27	14	(0, 0, 70)	卡车	场景中的卡车

数据集结构

数据集按以下结构组织：

├── Images (RGB Images) │ ├── H_15_P_0 │ │ ├── ClearNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── ClearSunset │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── ClearNight │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── CloudyNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ └── MidRainyNoon │ │ ├── Town01 │ │ │ └── Town01.tar.gz │ │ ├── Town02 │ │ │ └── Town02.tar.gz │ │ ├── ... │ │ └── Town10HD │ │ └── Town10HD.tar.gz │ ├── H_15_P_45 │ │ └── ... │ ├── ... │ └── H_60_P_90 │ └── ... ├── Instance (Instance Segmentation Annotations) │ ├── H_35_P_45 │ │ └── ClearNoon │ │ ├── Town01 │ │ │ └── Town01.tar.gz │ │ ├── Town02 │ │ │ └── Town02.tar.gz │ │ ├── ... │ │ └── Town10HD │ │ └── Town10HD.tar.gz │ └── ... ├── Segment (Semantic Segmentation Annotations) │ ├── H_15_P_0 │ │ ├── ClearNoon │ │ │ ├── Town01 │ │ │ │ └── Town01.tar.gz │ │ │ ├── Town02 │ │ │ │ └── Town02.tar.gz │ │ │ ├── ... │ │ │ └── Town10HD │ │ │ └── Town10HD.tar.gz │ │ ├── H_15_P_45 │ │ │ └── ... │ │ ├── ... │ │ └── H_60_P_90 │ │ └── ... │ └── ... └── Depth (Depth Annotations) ├── H_35_P_45 │ └── ClearNoon │ ├── Town01 │ │ └── Town01.tar.gz │ ├── Town02 │ │ └── Town02.tar.gz │ ├── ... │ └── Town10HD │ └── Town10HD.tar.gz └── ...

数据集下载

数据集可通过 Hugging Face 的 datasets 库或 wget 下载。

使用 wget 下载

示例脚本： bash #!/bin/bash HP=(H_15_P_0 H_15_P_45 H_15_P_60 H_15_P_90 H_35_P_0 H_35_P_45 H_35_P_60 H_35_P_90 H_60_P_0 H_60_P_45 H_60_P_60 H_60_P_90) weather=(ClearNoon ClearNight ClearSunset CloudyNoon MidRainyNoon) layout=(Town01 Town02 Town03 Town04 Town05 Town06 Town07 Town10HD) base_url=(https://huggingface.co/datasets/hoffman-lab/SkyScenes/resolve/main/Images) base_download_folder=SkyScenes

for hp in "${HP[@]}"; do for w in "${weather[@]}"; do for t in "${layout[@]}"; do folder=$(echo "$base_url" | awk -F / {print $(NF)}) download_url="${base_url}/${hp}/${w}/${t}/${t}.tar.gz" download_folder="${base_download_folder}/${folder}/${hp}/${w}/${t}" mkdir -p "$download_folder" echo "Downloading: $download_url" wget -P "$download_folder" "$download_url" done done done

使用 datasets 下载

示例脚本： python from datasets import load_dataset dataset = load_dataset(hoffman-lab/SkyScenes, name="H_35_P_45 images")

注意事项

深度和实例分割地图仅适用于 H_35_P_45，其他变化将很快提供。
为避免使用 datasets 库加载数据集时出现问题，建议避免下载包含重叠目录的子集。如果存在重叠目录，请在继续下载新子集之前清除 .cache 目录中的重叠部分。

搜集汇总

数据集介绍

构建方式

在无人机视觉感知领域，真实世界数据的获取常受限于环境多样性与标注密度。SkyScenes数据集通过合成数据生成技术，系统性地构建了涵盖城市与乡村布局的空中场景。该数据集基于CARLA仿真平台，精心设计了8种城镇布局、5类天气与光照条件，并结合12种飞行高度与俯仰角组合，生成了总计33,600幅图像。每幅图像均配备了精确的语义分割、实例分割及深度标注，通过程序化控制实现了多维度条件组合，为仿真到真实场景的域适应研究提供了可控且丰富的实验环境。

特点

SkyScenes数据集的核心特征在于其系统化的多维可控性与密集标注体系。数据集覆盖了从15米至60米飞行高度、0°至90°俯仰角的无人机视角，同时囊括了晴朗正午、日落、夜晚、多云及雨中正午等多种光照与气象条件。语义标注体系包含28个精细类别，可进一步归纳为20个通用类别，涵盖建筑、道路、植被、车辆及行人等关键场景要素。实例分割与深度信息的同步提供，使得该数据集能够支持物体检测、场景解析与三维感知等多任务联合研究，其结构化变体设计便于进行模型对特定环境因素的敏感性分析。

使用方法

研究者可通过Hugging Face平台直接访问SkyScenes数据集，利用提供的wget脚本按需下载特定子集。数据集按飞行参数、天气条件和城镇布局分层组织，用户可根据实验需求选择下载特定高度俯仰组合下的图像及其对应标注。对于语义分割、实例分割及深度估计任务，可分别加载对应的标注文件。由于相同视点在不同天气条件下具有一致性，标注文件可跨天气变体重用，这简化了数据管理并支持跨域分析。该数据集适用于无人机视觉算法的训练、仿真到真实场景的域适应研究，以及模型在多样化环境条件下的鲁棒性评估。

背景与挑战

背景概述

在无人机自主导航与环境感知领域，真实世界数据的稀缺性与标注成本高昂构成了显著瓶颈。为应对这一挑战，由佐治亚理工学院Judy Hoffman教授团队主导，于2023年发布了SkyScenes合成数据集。该数据集基于CARLA仿真平台构建，旨在通过精心控制布局、天气、时间、飞行高度与俯仰角等多维度变量，生成涵盖33,600张图像的密集标注空中视角数据。其核心研究问题聚焦于提升模型在复杂空中场景下的语义分割、实例检测与深度估计能力，并为仿真到真实领域的自适应研究提供了关键基准，相关成果已被ECCV 2024收录。

当前挑战

SkyScenes数据集致力于解决空中场景理解中模型泛化能力不足的挑战，其核心在于通过合成数据缓解真实数据在多样环境条件下采集与标注的困难。构建过程中的主要挑战体现在多维变量系统的设计上：需在仿真环境中精确模拟八种城镇布局、五种天气与光照条件、十二种飞行高度与俯仰角组合，以确保数据分布的多样性与可控性；同时，生成像素级语义、实例及深度标注时，需保持跨模态标注的一致性，并处理仿真引擎固有的纹理简化与物理失真问题，以逼近真实世界的视觉复杂性。

常用场景

经典使用场景

在无人机视觉感知领域，SkyScenes数据集以其精心构建的合成数据特性，成为模型训练与评估的经典基准。该数据集通过CARLA仿真引擎生成了涵盖城市与乡村布局、多种天气与光照条件、不同飞行高度与俯仰角度的无人机视角图像，并提供了密集的语义分割、实例分割及深度标注。研究者通常利用其丰富的环境变体，系统性地探究模型在复杂空中场景下的鲁棒性与泛化能力，尤其在跨域适应研究中，该数据集常作为源域数据，用于模拟到真实场景的知识迁移。

解决学术问题

SkyScenes数据集有效应对了真实世界无人机数据采集成本高昂、标注稀疏且环境多样性不足的核心挑战。它为解决计算机视觉领域中的关键学术问题提供了重要支撑，包括在有限真实标注下如何利用合成数据进行域适应学习，如何提升模型对光照、天气及视角变化的不变性，以及如何推动密集预测任务（如语义分割、深度估计）在航空影像上的进展。该数据集通过提供大规模、高质量、多模态的标注数据，显著降低了相关研究的入门门槛，并促进了仿真到真实迁移学习范式的深化与发展。

衍生相关工作

围绕SkyScenes数据集，已催生了一系列聚焦于空中场景理解与域适应的经典研究工作。这些工作主要探索如何利用其丰富的合成数据提升模型在真实航空影像上的性能，包括开发新颖的域适应算法以弥合仿真与真实之间的鸿沟，设计针对空中视角的专用网络架构，以及利用其多任务标注进行联合学习以增强特征表示。该数据集作为ECCV 2024的收录成果，已成为该领域新兴的基准测试平台，激励着后续研究在仿真数据利用、跨域泛化及三维场景解析等方向持续深入。

以上内容由遇见数据集搜集并总结生成