Alvin16/SpaceSense-Bench

---
license: cc-by-nc-4.0
size_categories:
- 10GB<n<100GB
pretty_name: SpaceSense-Bench
task_categories:
- object-detection
- image-segmentation
- depth-estimation
- other
tags:
- space
- satellite
- multi-modal
- lidar
- pose-estimation
---

# SpaceSense-Bench: Multi-Modal Spacecraft Perception and Pose Estimation Dataset

[**Project Page**](https://wuaodi.github.io/SpaceSense-Bench/) | [**Paper**](https://huggingface.co/papers/2603.09320) | [**Toolkit & Code**](https://github.com/wuaodi/SpaceSense-Bench)

SpaceSense-Bench is a high-fidelity simulation-based multi-modal (RGB, Depth, LiDAR Point Cloud) dataset for spacecraft component-level semantic understanding, containing **136 satellite models** with synchronized multi-modal data.


## Dataset Overview

| Item | Detail |
|------|--------|
| Satellite Models | 136 (sourced from NASA/ESA 3D models) |
| Data Modalities | RGB, Depth, Semantic Segmentation, LiDAR Point Cloud, 6-DoF Pose |
| Image Resolution | 1024 x 1024 |
| Camera FOV | 50 degrees |
| Semantic Classes | 7 (main_body, solar_panel, dish_antenna, omni_antenna, payload, thruster, adapter_ring) |
| Simulation Platform | Unreal Engine 5.2.0 + AirSim 1.8.1 |

## Sample Usage

The [SpaceSense-Toolkit](https://github.com/wuaodi/SpaceSense-Bench/tree/main/SpaceSense-Toolkit) provides tools for converting raw data to standard formats and visualizing the results.

### Installation

```bash
pip install -r requirements.txt
```

### Conversion and Visualization

```bash
# Visualize the raw data
python SpaceSense-Toolkit/visualize/raw_data_web_visualizer.py --raw-data data_example

# Convert to Semantic-KITTI (3D segmentation)
python SpaceSense-Toolkit/convert/airsim_to_semantickitti.py --raw-data data_example --output output/semantickitti --satellite-json SpaceSense-Toolkit/configs/satellite_descriptions.json

# Convert to MMSegmentation (2D segmentation)
python SpaceSense-Toolkit/convert/airsim_to_mmseg.py --raw-data data_example --output output/mmseg

# Convert to YOLO (Object detection)
python SpaceSense-Toolkit/convert/airsim_to_yolo.py --raw-data data_example --output output/yolo
```

## Data Modalities

| Modality | Format | Unit / Range | Description |
|----------|--------|-------------|-------------|
| RGB | PNG (1024x1024) | 8-bit color | Scene rendering from `cam0` |
| Depth | NPZ (compressed) | int32, millimeters (`depth` key); background = 10,000 m | Per-pixel metric depth exported from AirSim `DepthPerspective` |
| Semantic Segmentation | PNG (1024x1024, RGB color mask) | color-coded part labels | Component-level segmentation mask; toolkit maps colors to 7 semantic classes |
| LiDAR Point Cloud | ASC (x,y,z per line) | meters, 3 decimal places | Sparse 3D point cloud |
| 6-DoF Pose | CSV | meters + Hamilton quaternion (w,x,y,z) | Target-spacecraft pose in camera frame, plus service/camera pose in world frame |

## Coordinate System & Units

| Item | Convention |
|------|-----------|
| Camera Frame | X-forward, Y-right, Z-down (FRD, right-handed) |
| World Frame | AirSim NED, target spacecraft fixed at origin |
| Quaternion | Hamilton convention: w + xi + yj + zk |
| Euler Angles | ZYX intrinsic (Yaw-Pitch-Roll) |
| Position | meters (m), 6 decimal places |
| Depth Map | millimeters (mm), stored as compressed `.npz`; deep-space background = 10,000 m |
| LiDAR | meters (m), `.asc` format (`x,y,z`), 3 decimal places |
| Timestamp | `YYYYMMDDHHMMSSmmm` |

## Sensor Configuration

### Camera (`cam0`)

- Resolution: 1024 x 1024
- FOV: 50 degrees
- Image types captured: RGB (`Scene`, type 0), Segmentation (type 5), Depth (`DepthPerspective`, type 2)
- TargetGamma: 1.0
- Mounting offset: 1 m in front of the service spacecraft body origin

### LiDAR

- Range: 300 m
- Channels: 256
- Vertical FOV: -20 to +20 degrees
- Horizontal FOV: -20 to +20 degrees
- Data frame: `SensorLocalFrame`

## Data Split (Zero-shot / OOD)

The training and validation sets contain **completely non-overlapping satellite models**, so validation performance reflects zero-shot generalization to unseen spacecraft.

| Split | Satellites | Rule |
|-------|----------:|------|
| Train | 117 | All satellites excluding val and excluded |
| Test | 14 | Every 10th by index: seq 00, 10, 20, ..., 130 |
| Validation | 5 | Seq 131-135, reserved for future testing |

## Data Organization

Each `.tar.gz` file in the `raw/` folder contains data for one satellite. The toolkit expects trajectory folders containing `image/`, `seg/`, `depth/`, `lidar/`, and `pose_ground_truth.csv`.

Typical layout:

```text
<satellite_name>/ or <timestamp>_<satellite_name>/
├── approach_front/
│ ├── image/ # RGB images (.png)
│ ├── depth/ # Depth maps (.npz, int32, mm, key = depth)
│ ├── seg/ # Semantic masks (.png, RGB color encoding)
│ ├── lidar/ # Point clouds (.asc)
│ └── pose_ground_truth.csv
├── approach_back/
├── orbit_xy/
└── ...
```

Some HuggingFace exports may contain one additional nesting level:

```text
<satellite_name>/
└── <satellite_name>/
├── approach_front/
├── approach_back/
├── orbit_xy/
└── ...
```

## Semantic Class Definitions

| Class ID | Name | Description |
|:--------:|------|-------------|
| 0 | background | Deep space background |
| 1 | main_body | Spacecraft main body / bus |
| 2 | solar_panel | Solar panels / solar arrays |
| 3 | dish_antenna | Dish / parabolic antennas |
| 4 | omni_antenna | Omnidirectional antennas / booms |
| 5 | payload | Scientific instruments / payloads |
| 6 | thruster | Thrusters / propulsion systems |
| 7 | adapter_ring | Launch adapter rings |

## License

This dataset is released under the [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. Non-commercial use only.

## Citation

```bibtex
@article{SpaceSense-Bench,
title={SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation},
author={Aodi Wu, Jianhong Zuo, Zeyuan Zhao, Xubo Luo, Ruisuo Wang, Xue Wan},
year={2026},
url={https://arxiv.org/abs/2603.09320}
}
```