Tartan训练RGB顺序分片

# Tartan RGB training shards This private mirror contains the **exact RGB corpus selected by the existing training schedule**: 1,247 trajectories, 1,751,563 original PNG files, and 11,318 independent windows. It is a selected-camera repack of upstream datasets, not an independent dataset release or the complete collection of every modality/camera. | Source | Selected camera | Trajectories | RGB frames | | --- | --- | ---: | ---: | | TartanAir V1 | left | 369 | 306,637 | | TartanGround | left front (`lcam_front`) | 878 | 1,444,926 | Only original RGB and indexing/provenance are included. There are no model checkpoints, frozen features, generated supervision, KITTI images or labels, depth arrays, poses, or camera calibration labels. Historical opaque pose-member hashes in the unchanged schedule identify the original selection; they are not pose values or training inputs. ## Storage and fidelity - 622 ZIP shards, with a target size of 2 GiB each. Boundaries occur between complete logical training updates (eight original windows, six in the final update). - The source ZIP member compression streams are copied directly. Decompressed PNG bytes are unchanged: no image decoding, resizing, quantization, or re-encoding is applied when packing. - Compressed image payload totals 1,079,274,591,904 bytes (about 1.08 decimal TB), plus small ZIP/index metadata. Original PNG bytes total 1,221,737,915,354 bytes. - Each shard is independently readable without extraction. The server retains a bounded set of staging shards and removes only its own verified-upload staging files; original source archives are retained. - Every PNG has a SHA256, byte length, original archive SHA256/member name, and exact window/frame identity in `shards/<id>/index.jsonl.gz`. Every uploaded ZIP/index has a size and SHA256 receipt. ## Layout ```text schedule/windows.original.json # unchanged original schedule and order schedule/shard_plan.json # shard/update/window ranges sources.json # selected archives and source provenance shards/00000/rgb.zip # original PNG bytes, renamed for uniqueness shards/00000/index.jsonl.gz # per-PNG checksums and source mapping receipts/train-00000.json # uploaded-file size/SHA256 verification dataset_manifest.json # final coverage and all shard hashes download_shards.py rgb_shard_reader.py ``` `dataset_manifest.json` is marked `complete` only after all shards are uploaded and independently rechecked against repository file sizes/LFS SHA256. During upload, only shards with an `uploaded_verified` receipt are available for use. ## Small-disk use Authenticate with the ModelScope SDK in the normal credential cache, or set `MODELSCOPE_API_TOKEN` in your process environment. Do not put a token in a script, Git repository, or a command-line argument. ```bash python download_shards.py --root /data/tartan-rgb --from-update 1 --ahead 1 ``` This fetches the shard containing the requested update and one following shard, resumes partial transfers, and checks SHA256 before making each file usable. No extraction is required. Call `Downloader.fetch_for_update(next_update, ahead=1)` from a background CPU worker while training on the current shard. Remove a consumed shard **only after** saving a checkpoint whose exact sampling cursor has passed its `position_end`. The helper never deletes data automatically. `RGBShardReader.load(row)` returns the original native RGB window in the same order. Connect it only to data reading; retain the existing teacher/student preprocessing, 81-frame encoder packaging, native backend order, logical rank/RNG assignments, loss, optimizer, depth branch, and saved cursor. This repository does not modify a training run or migrate an existing checkpoint automatically. The original schedule is bound by SHA256: `fd0365321281f2f2d223ae81c032a53788a1ee3a9d0a088abc91333f75d177ce` ## Upstream attribution and terms Images remain the work of the original dataset authors and are redistributed under their **Creative Commons Attribution 4.0 International** terms. This repack changes archive layout and adds checksums/indexes; image content is unchanged. Preserve these attributions when copying the shards. - **TartanAir:** Wenshan Wang et al., *TartanAir: A Dataset to Push the Limits of Visual SLAM*, IROS 2020. [Official dataset and terms](https://theairlab.org/tartanair-dataset/), [paper](https://arxiv.org/abs/2003.14338). - **TartanGround:** Manthan Patel, Fan Yang, Yuheng Qiu, Cesar Cadena, Sebastian Scherer, Marco Hutter, and Wenshan Wang, *TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation*, 2025. [Official dataset](https://tartanair.org/tartanground.html), [official publisher card/license](https://huggingface.co/datasets/theairlabcmu/TartanGround), [paper](https://arxiv.org/abs/2505.10696). - [CC BY 4.0 license](https://creativecommons.org/licenses/by/4.0/). ```bibtex @inproceedings{wang2020tartanair, title={TartanAir: A Dataset to Push the Limits of Visual SLAM}, author={Wang, Wenshan and Zhu, Delong and Wang, Xiangwei and Hu, Yaoyu and Qiu, Yuheng and Wang, Chen and Hu, Yafei and Kapoor, Ashish and Scherer, Sebastian}, booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, year={2020} } @article{patel2025tartanground, title={TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation}, author={Patel, Manthan and Yang, Fan and Qiu, Yuheng and Cadena, Cesar and Scherer, Sebastian and Hutter, Marco and Wang, Wenshan}, journal={arXiv preprint arXiv:2505.10696}, year={2025} } ```