OctoSense

# Dataset Card for OctoSense (FiftyOne Multimodal)  OctoSense is a time-synchronized, calibrated multi-sensor robot perception dataset captured by an open-source eight-sensor rig on three platforms: a car, a boat, and a Unitree Go2-W quadruped. This FiftyOne version packages eight representative episodes as MCAP files — one sample per episode — loadable directly in the FiftyOne App's multimodal viewer with synchronized playback across all sensor streams. ## Installation ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo import fiftyone.utils.huggingface as fouh dataset = fouh.load_from_hub("Voxel51/OctoSense") session = fo.launch_app(dataset) ``` ## Dataset Details ### Dataset Description OctoSense provides 59 hours of time-synchronized driving data from eight sensors: stereo RGB and event cameras, a thermal camera, an Ouster OS1-64 LiDAR, an IMU, RTK-corrected GPS, and vehicle CAN bus (car) or joint-angle proprioception (quadruped). Sensors are hardware PPS-locked to a single clock and post-processed with Kalman-smoothed affine synchronization. All sensors record at native rates without dropped data: the RGB pair at 100 Hz, event cameras at up to ~7 MEv/s, LiDAR at 10 Hz, IMU at 400 Hz (VectorNav) and 100 Hz (in-LiDAR), and GPS at 5 Hz. Car sequences additionally include ego-motion optical flow, dense LiDAR-projected depth, pseudo-label semantic segmentation, LiDAR-inertial odometry, and a fused GPS/LIO trajectory. The dataset spans urban, suburban, and rural environments at sunrise through nighttime, and includes deliberately degraded-sensor recordings. - **Curated by:** Harpreet Sahota (FiftyOne multimodal packaging) - **Original dataset by:** Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari (GRASP Laboratory, University of Pennsylvania; Brown University) - **Funded by:** [More Information Needed] - **Shared by:** Harpreet Sahota - **Language(s):** Not applicable (sensor data) - **License:** MIT ### Dataset Sources - **Repository:** https://huggingface.co/datasets/anthonytec2/OctoSense - **Paper:** https://arxiv.org/abs/2606.27317 - **Demo:** https://abisulco.com/octosense/ ## Uses ### Direct Use - Visual inspection and debugging of synchronized multi-sensor recordings in the FiftyOne multimodal viewer (Image, 3D, Map, Plot, and Logs tiles on a shared timeline). - Filtering episodes by platform, split, sensor coverage, geographic bounding box, speed, idle fraction, or ground-truth availability using the FiftyOne SDK and App sidebar. - Prototyping multi-modal perception pipelines against a known-good subset before scaling to the full 382-sequence release. - Evaluating sensor coverage and calibration quality across platforms before importing additional sequences. ### Out-of-Scope Use This FiftyOne package contains eight sequences (a curated subset). Training production models requires the full OctoSense release (`anthonytec2/OctoSense`). Radar range and bearing values use fitted scales (1/16 m per count, −1 bearing sign) derived from LiDAR cross-validation; they are not manufacturer-documented and should not be used as ground truth for radar calibration. ## Dataset Structure ### Overview **Media type:** `multimodal` — one `fo.Sample` per episode, `filepath` pointing to a `.mcap` file. All sensor streams are encoded inside the MCAP with time-aligned messages on a shared nanosecond clock. The FiftyOne App renders them in synchronized Image, 3D, Map, Plot, and Logs tiles. **Sample count:** 8 episodes (6 car, 1 boat, 1 Unitree quadruped) **Total recording time:** ~46 minutes across all 8 episodes ### Sample Fields | Field | FiftyOne type | Description | |---|---|---| | `filepath` | `StringField` | Absolute path to the episode's `.mcap` file | | `bag_id` | `StringField` | ROS 2 bag identifier (e.g. `rosbag2_2026_01_09-11_32_05`) | | `session` | `StringField` | Session label within the platform (e.g. `sess8`) | | `platform` | `StringField` | Recording platform: `car`, `boat`, or `unitree` | | `start_time` | `StringField` | Sequence wall-clock start time (ISO 8601) | | `split` | `StringField` | `train` or `test` for car episodes; null for boat and unitree | | `is_daytime` | `BooleanField` | Whether the sequence was recorded in daylight | | `degraded` | `BooleanField` | Whether sensor degradation was logged for this sequence | | `has_seg` | `BooleanField` | Whether pseudo-label semantic segmentation GT is available | | `duration_s` | `FloatField` | Episode duration in seconds (from MCAP message timestamps) | | `message_count` | `IntField` | Total MCAP messages across all channels | | `channel_count` | `IntField` | Number of distinct MCAP channels | | `topics` | `ListField(StringField)` | All MCAP topic strings in the episode | | `schemas` | `ListField(StringField)` | All MCAP schema names (use for capability filtering) | | `n_lidar_frames` | `IntField` | Number of LiDAR sweeps | | `n_rgb_frames` | `IntField` | Number of RGB camera frames | | `n_imu_samples` | `IntField` | Number of IMU samples | | `n_gps_fixes` | `IntField` | GPS fixes logged (includes zero-fix rows on unitree) | | `n_gps_valid` | `FloatField` | GPS fixes with valid lock | | `n_events_left` | `IntField` | Total events from left event camera | | `n_events_right` | `IntField` | Total events from right event camera | | `gps_quality` | `StringField` | GPS fix quality string (e.g. `single_m`, `rtk`) | | `gps_lat_min` | `FloatField` | Southern edge of episode geographic bounding box (degrees) | | `gps_lat_max` | `FloatField` | Northern edge of episode geographic bounding box (degrees) | | `gps_lon_min` | `FloatField` | Western edge of episode geographic bounding box (degrees) | | `gps_lon_max` | `FloatField` | Eastern edge of episode geographic bounding box (degrees) | | `mean_speed_mph` | `FloatField` | Mean driving speed in mph | | `idle_fraction` | `FloatField` | Fraction of recording time at near-zero speed | | `distance_m` | `FloatField` | Total distance driven in metres | | `rgb_cal_id` | `StringField` | Camera calibration file identifier | | `imu_cal_id` | `StringField` | IMU calibration file identifier | | `lidar_cal_id` | `StringField` | LiDAR calibration file identifier | | `sensor_dropout` | `StringField` | Dropout description if a sensor outage was logged; null otherwise | ### MCAP Stream Contents Each `.mcap` file is a multi-channel recording. Topics and their schemas vary by platform: | Topic | Schema | Tile | Platforms | |---|---|---|---| | `/camera/left/image_raw` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | all | | `/camera/right/image_raw` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | all | | `/camera/infrared/image_raw` | `foxglove.CompressedVideo` (H.264, ~50 fps) | Image | all | | `/camera/left_rect/depth` | `foxglove.RawImage` (32fc1 metres, 10 Hz) | Image | car | | `/camera/left_rect/semantic` | `foxglove.CompressedImage` (PNG, 10 Hz) | Image | car daytime | | `/camera/left_rect/flow` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | car | | `/lidar/points` | `foxglove.PointCloud` (10 Hz) | 3D | all | | `/gps` | `foxglove.LocationFix` | Map | car, boat | | `/odom` | `foxglove.PoseInFrame` | 3D | car | | `/imu/vectornav` | `imu_vectornav` (JSON) | Plot | all | | `/imu/ouster` | `imu_ouster` (JSON) | Plot | all | | `/car/*` | JSON scalars (speed, wheels, steer, CAN) | Plot | car | | `/robot/*` | JSON scalars (low_state, sport, controller) | Plot | unitree | | `/captions` | `foxglove.Log` | Logs | car | | `/tf_static` | `foxglove.FrameTransforms` | — | all | ### Ground Truth Coverage | Ground truth | Source | Coverage | Notes | |---|---|---|---| | Dense depth | LiDAR accumulation, `32fc1` metres | All 6 car episodes | ~12 % of pixels valid; 3–102 m range | | Semantic segmentation | EoMT-Cityscapes-DINOv2-L-1024 pseudo-labels | Daytime car only (all 5 daytime car episodes in this subset) | 19 Cityscapes classes; `255` = ignore | | Ego-motion optical flow | Derived at conversion from depth + poses | All 6 car episodes | Ego-motion only; moving objects are holes | | Radar tracks | Mazda CAN `track_{1..6}` | Car | Range 1/16 m per count (fitted); bearing sign inverted | ### Filtering by Capability Use the `schemas` list field to filter episodes rather than topic names (topic substrings false-positive against non-decodable packet channels): ```python import fiftyone as fo from fiftyone import ViewField as F dataset = fo.load_dataset("octosense") # Car episodes with semantic segmentation car_with_seg = dataset.match(F("has_seg") == True) # Episodes with GPS has_gps = dataset.match(F("schemas").contains("foxglove.LocationFix")) # Filter by platform cars = dataset.match(F("platform") == "car") ``` ## Dataset Creation ### Curation Rationale The eight episodes were selected to provide maximum coverage of platforms, modalities, and conditions within a manageable subset: - **Daytime car with all GT** (`car/sess8`, 53 s): shortest car sequence with depth, semantic segmentation, and all CAN channels present. - **Car sessions** (sess8 long, sess9, sess10 ×2, sess11): 163–597 s each, from both train and test splits, spanning all four car sessions; selected for near-zero sensor dropout, valid GPS lock, and minimal idle time. - **Boat** (`boat/sess1`, 54 s): the only platform where the LiDAR ships raw range images rather than XYZ, and where the boat IMU extrinsic differs. - **Unitree** (`unitree/sess1`, 112 s): the only platform with `/robot` joint state and no GPS or CAN data. ### Source Data #### Data Collection and Processing Data was collected from a single OctoSense rig mounted on each platform. All sensors are hardware PPS-locked for temporal alignment and post-processed with Kalman-smoothed affine synchronization. Source recordings are ROS 2 bags converted to HDF5 and H.265 video files; the raw release is hosted at `anthonytec2/OctoSense` on Hugging Face (~8.5 TB). This FiftyOne package re-encodes video streams from HEVC to H.264 (libx264, no B-frames, one access unit per MCAP message) for browser WebCodecs compatibility and packages all channels into MCAP files using the Foxglove schema registry. LiDAR points are stored at a 20-byte stride (x, y, z, intensity as float32; r, g, b, a as uint8), with colour sampled from the nearest left-camera frame. #### Who are the source data producers? The OctoSense hardware platform and dataset were developed at the General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania (Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Pratik Chaudhari) and Brown University (Randall Balestriero). Data was collected on Long Island and in Philadelphia, PA, USA. ### Annotations #### Annotation process All ground truth in OctoSense is machine-generated: - **Depth** is accumulated from LiDAR sweeps projected onto the rectified left camera frame. - **Semantic segmentation** is produced by EoMT-Cityscapes-DINOv2-L-1024 applied to CLAHE-enhanced rectified left frames; it is a model's output, not human annotation. - **Optical flow** is derived at MCAP-conversion time from depth and LiDAR-inertial poses (`flow_gap = 2` frames, i.e. 0.2 s lookahead); it captures ego-motion only — pixels covering dynamic objects (vehicles, pedestrians) were excluded from the depth accumulation and appear as invalid. - **Radar detection geometry** (range, bearing) uses scales fitted against LiDAR-projected vehicle pixels in bird's-eye view (1/16 m per `DIST_OBJ` count, negative `ANG_OBJ` sign). These are not manufacturer-documented values. #### Who are the annotators? No human annotators. All labels are generated programmatically from sensor data and pre-trained models as described above. #### Personal and Sensitive Information Recordings were made on public roads and a private waterway. No personally identifiable information was intentionally captured. Faces and licence plates visible in RGB frames are not blurred in the source data. ## Citation **BibTeX:** ```bibtex @article{bisulco2026octosense, title={OctoSense: Self-Supervised Learning for Multimodal Robot Perception}, author={Bisulco, Anthony and Wang, Jeremy and Daniilidis, Kostas and Balestriero, Randall and Chaudhari, Pratik}, journal={arXiv preprint arXiv:2606.27317}, year={2026} } ``` **APA:** Bisulco, A., Wang, J., Daniilidis, K., Balestriero, R., & Chaudhari, P. (2026). OctoSense: Self-Supervised Learning for Multimodal Robot Perception. *arXiv preprint arXiv:2606.27317*. ## More Information - Full dataset (8.5 TB, 382 sequences): https://huggingface.co/datasets/anthonytec2/OctoSense - Devkit and conversion scripts: https://github.com/anthonytec2/OctoSense - Project page and supplementary videos: https://abisulco.com/octosense/ - FiftyOne multimodal documentation: https://docs.voxel51.com/user_guide/app.html#multimodal-viewer ## Dataset Card Authors Harpreet Sahota ## Dataset Card Contact harpreetsahota