data_factory_hard

# TMP 4_4 Hard Workflow This directory hosts the local-only hard testset workflow for the 27 `*_hard` majors. Rules: - Keep all formal builds on the 4090 server under `/data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/`. - Do not wire this workflow into `major_324_config` or the main dataset JSONs. - Keep generated artifacts local to `tmp4_4/` only. Key files: - `hard_registry.py`: shared hard-major registry and metadata - `hard_common.py`: shared job / manifest / progress helpers - `hard_script_family.py`: script-family hard builders - `hard_rich_family.py`: rich-family hard builders - `build_data_factory_hard.py`: local single-builder entrypoint - `build_data_factory_hard_remote.py`: remote shard orchestrator - `sanitize_data_factory_hard_questions.py`: remote question sanitizer for shipped hard JSON artifacts - `publish_data_factory_hard.py`: ModelScope dataset uploader - `MODELSCOPE_UPLOAD.md`: remote upload runbook Planned remote artifacts: - `/data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/data_factory_hard_test.json` - `/data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/data_factory_hard_build_manifest.json` - `/data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/data_factory_hard_testset.zip` Dataset path contract: - each sample writes `path` and `video_path` as dataset-root relative `videos/<hard_major>/<file>.mp4` - image-assisted majors also write `image_gt` and `image_paths` as dataset-root relative `images/...` - manifest fields such as `archive_path` and `dataset_path` remain absolute remote build metadata Remote build command: ```bash /root/miniconda3/bin/python3 /data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/build_data_factory_hard_remote.py --count-per-major 50 ``` Remote upload command: ```bash MODELSCOPE_API_TOKEN=your_token /root/miniconda3/bin/python3 /data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/publish_data_factory_hard.py --repo-id Mrakas/data_factory_hard --artifacts data_factory_hard_testset.zip --artifacts data_factory_hard_test.json --artifacts data_factory_hard_build_manifest.json --artifacts README.md ``` Remote sanitize-and-repack command: ```bash /root/miniconda3/bin/python3 /data-tmp/ghx/tmp2/data_factory_v2/tmp4_4/sanitize_data_factory_hard_questions.py ```