exp_agent

# exp_agent — archived `exp/` experiment runs This dataset is a compressed snapshot of an agent harness `exp/` directory (AutoLab / ProgramBench / SWE-bench / FrontierSWE / SciCode runs and eval outputs), ~320 GB raw → **~91 GB** of zstd-compressed tar shards. ## Layout ``` parts/<label>.tar.zst.part-aaa, -aab, ... # zstd tar shards, each piece <=15 GB manifest.json # parts list, sizes, exclusions, restore cmd restore_exp.sh # download + decompress helper ``` Each `<label>` is one independent `tar | zstd` stream (a top-level `exp/` subtree, e.g. `frontier_swe`, `autolab`, `programbench_g0`..`g7`), split into `<=15 GB` parts. `programbench` is sharded into 8 groups (`programbench_g0`..`g7`). ## Restore ```bash # needs: bash, tar, zstd, modelscope (pip install modelscope) # private dataset? export MODELSCOPE_TOKEN=ms-xxxx first bash restore_exp.sh ./exp_agent_restore # -> reconstructs ./exp_agent_restore/exp/ ``` Or manually: ```bash modelscope download --dataset TobyYang7/exp_agent --repo-type dataset \ --local_dir . --include 'parts/*' 'manifest.json' for L in $(ls parts | sed 's/\.tar\.zst\.part-.*//' | sort -u); do cat parts/"$L".tar.zst.part-* | zstd -d | tar -xf - done ``` ## Notes / exclusions The pack excluded, by design: - `.env` files (credentials hygiene), - a single 1 TiB **sparse** test fixture (`s22_1099511627776`, a pls `size-fixture` of all-zeros) that would otherwise expand on extraction. Everything else under `exp/` is included verbatim (paths preserve `exp/...`).