FRESH (Fusion with RGB and Events for Spacecraft pose estimation under Harsh lighting) Dataset

FRESH (Fusion with RGB and Events for Spacecraft pose estimation under Harsh lighting) is a dataset to evaluate RGB and event sensors for satellite pose estimation under challenging real-world lighting. The RGB and event sensors used to capture the data have been optically aligned via a glass beamsplitter and temporally synchronized using a custom multithreaded software. In the dataset, three 3D-printed models of satellite objects have been captured in rotating sequences in an uncluttered environment. The challenging lighting conditions were engineered using a directional 170k lux (@1m distance) daylight temperature (5600K) LED lamp. This light creates glare, blooming, overexposure and lens flare artifacts which are observed in real orbital scenarios and occlude the structure of the target satellite. The primary purpose of the dataset is to provide optically and temporally aligned RGB and event data to lead research into event-RGB fusion for satellite pose estimation during harsh lighting conditions. Furthermore, the aligned dataset allows a like-to-like evaluation of both sensors. In particular, an event-based camera (Prophesee Gen 4 with Prophesee-Sony IMX636 sensor) and an RGB sensor (Basler a2A1920-160ucPRO with Sony IMX392 sensor) were used for capturing this dataset. Ground-truthing was performed by manual labelling of the sequences using Blender (see citation below for full details). Thus, the dataset consists of ground-truth poses of the satellite relative to the capture setup. The structure of the dataset is as follows: synthetic satty frames train.json test.json cassini frames train.json test.json soho frames train.json test.json real 24 sequences in the format: <object>-<trajectory_index>-<distance> events events.txt frames test.json timestamps.txt harsh-and-slow-frames.json models <object i.e satty/cassini/soho> dense.json: json file containing one key “dense_points” which is a Wx3 list of 3D points representing a dense point cloud used to evaluate the event certifier method of: M. Jawaid, R. Talak, Y. Latif, L. Carlone, T.-J. Chin, Test-time certifiable self-supervision to bridge the sim2real gap in event-based satellite pose estimation, in: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024. STL file(s) for the satellite models In each frames directory the frames are in the following formats: %05d_rgb.png (for the RGB frames) %05d_event.png (for the event frames) The events.txt contains the raw event data for each sequence in the following header-less csv format: timestamp, x, y, polarity Where the timestamp is in microseconds, x and y are the coordinates of the triggered event and the polarity is either 0 or 1 to indicate a positive or negative event. The train.json and test.json annotations files for both synthetic and real data are in the following format: “landmarks_3d”: Nx3 list of 3D landmarks marked on the satellite object “wireframe_points”: Mx3 list of 3D points of a simple wireframe for visualisation of the pose through reprojection. “wireframe_faces”: PxQ list of P faces of the simple wireframe which each Qth element of the face P is an index into the “wireframe_points” list above. “instrinsics”: 3x3 camera intrinsic matrix obtained after calibration and undistortion “annotations”: list of annotations for each frame in the following format: “filename_rgb”: filename in format %05d_rgb.png of the RGB frame this annotation corresponds to “filename_event”: filename in format %05d_event.png of the event frame this annotation corresponds to “pose”: 4x4 pose matrix with both rotation and translation components in OpenCV coordinate system. “keypoints”: Nx2 matrix of the “landmarks_3d” reprojected onto the frame using the “pose” “bbox”: x1,y1,x2,y2 format 2D bounding box of the satellite in the frame. The timestamps.txt files for each sequence in the real data are in the following header-less csv format: %05d_rgb, timestamp where %05d_rgb is the RGB frame name without the file extension and the corresponding timestamp is in microseconds with a zero padded width of 16 characters. Code relating to this dataset will be maintained at: https://github.com/mohsij/space-event-rgb-fusion For more details please refer to the following article. If you found this dataset useful, please consider citing our paper: @article{JAWAID2025111039,title = {Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting},journal = {Aerospace Science and Technology},pages = {111039},year = {2025},issn = {1270-9638},doi = {https://doi.org/10.1016/j.ast.2025.111039},url = {https://www.sciencedirect.com/science/article/pii/S1270963825011022},author = {Mohsi Jawaid and Marcus Märtens and Tat-Jun Chin},keywords = {event-based pose estimation, rendezvous, domain gap, sensor fusion, close proximity, harsh lighting}}FRESH（全称：Fusion with RGB and Events for Spacecraft pose estimation under Harsh lighting，即恶劣光照下航天器位姿估计的RGB与事件融合数据集）是一款用于在复杂真实光照环境下，评估RGB与事件传感器在卫星位姿估计任务中表现的数据集。用于采集本数据集的RGB与事件传感器通过玻璃分光镜实现光学对齐，并通过定制化多线程软件完成时间同步。本数据集包含3个3D打印的卫星实体模型，在无杂物的环境中以旋转序列的形式完成采集。复杂光照环境通过定向170k勒克斯（1米距离处，色温5600K）的LED日光灯管构建，该光源会产生眩光、泛光、过曝与镜头光晕等伪影——这些现象在真实轨道场景中均会出现，且会遮挡目标卫星的结构特征。 本数据集的核心目的是提供光学与时间均对齐的RGB与事件数据，以推动恶劣光照条件下卫星位姿估计的事件-RGB融合研究。此外，该对齐数据集还支持对两类传感器进行对等性能评估。本次采集所使用的设备包括：基于事件的相机（搭载Prophesee-Sony IMX636传感器的Prophesee Gen 4），以及RGB传感器（搭载Sony IMX392传感器的Basler a2A1920-160ucPRO）。 地面真值通过Blender对序列进行手动标注获取（完整细节请见下文引用文献）。因此，本数据集包含卫星相对于采集装置的地面真值位姿。 ### 数据集结构 本数据集分为合成数据（synthetic）与真实数据（real）两大模块： #### 合成数据模块 包含satty、cassini、soho三个子目录，每个子目录下均设有frames文件夹，内含train.json与test.json标注文件。 #### 真实数据模块 包含24个命名格式为<object>-<trajectory_index>-<distance>的序列，其下设有events、frames、models三个子目录： 1. events目录：包含events.txt原始事件数据文件，以及harsh-and-slow-frames.json文件。 2. frames目录：内含test.json标注文件与timestamps.txt时间戳文件。 3. models目录：包含卫星模型（即satty/cassini/soho），以及dense.json文件——该JSON文件包含一个名为"dense_points"的键，其值为W×3的三维点列表，代表用于评估M. Jawaid等人提出的事件验证方法的密集点云。相关引用文献为：M. Jawaid, R. Talak, Y. Latif, L. Carlone, T.-J. Chin, *Test-time certifiable self-supervision to bridge the sim2real gap in event-based satellite pose estimation*，发表于IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024。同时目录中还包含卫星模型的STL文件。 每个frames目录下的帧文件格式如下： - `%05d_rgb.png`：RGB帧文件 - `%05d_event.png`：事件帧文件 events.txt文件格式说明： 该文件采用无表头CSV格式，每一行包含四列数据，依次为：timestamp, x, y, polarity。其中timestamp单位为微秒，x与y为触发事件的像素坐标，polarity取值为0或1，分别代表负事件与正事件。 train.json与test.json标注文件格式（合成数据与真实数据通用）： 标注文件包含以下字段： - "landmarks_3d"：N×3的三维地标点列表，标注于卫星实体表面 - "wireframe_points"：M×3的三维点列表，构成用于可视化位姿的简单线框模型 - "wireframe_faces"：P×Q的列表，代表P个线框面，每个面的Q个元素均为上述"wireframe_points"列表的索引值 - "intrinsics"：经过校准与去畸变后的3×3相机内参矩阵 - "annotations"：每一帧的标注列表，单条标注的格式如下： - "filename_rgb"：对应RGB帧的文件名，格式为`%05d_rgb.png` - "filename_event"：对应事件帧的文件名，格式为`%05d_event.png` - "pose"：4×4位姿矩阵，包含旋转与平移分量，采用OpenCV坐标系 - "keypoints"：N×2的矩阵，为"landmarks_3d"通过"pose"重投影至图像平面后的关键点坐标 - "bbox"：卫星在帧中的2D边界框，格式为x1,y1,x2,y2 真实数据的timestamps.txt文件格式说明： 该文件采用无表头CSV格式，每一行格式为：`%05d_rgb, timestamp`。其中`%05d_rgb`为不带扩展名的RGB帧文件名，对应的timestamp单位为微秒，采用16位零填充格式。 本数据集相关代码将持续维护于：https://github.com/mohsij/space-event-rgb-fusion 如需获取更多细节，请参阅下述论文。若本数据集对你的研究有所帮助，请引用我们的文章： @article{JAWAID2025111039,title = {Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting},journal = {Aerospace Science and Technology},pages = {111039},year = {2025},issn = {1270-9638},doi = {https://doi.org/10.1016/j.ast.2025.111039},url = {https://www.sciencedirect.com/science/article/pii/S1270963825011022},author = {Mohsi Jawaid and Marcus Märtens and Tat-Jun Chin},keywords = {event-based pose estimation, rendezvous, domain gap, sensor fusion, close proximity, harsh lighting}}