danbooru-8M-qwen3.5

# Full7 Caption Publication Runtime `full7-publish` publishes only the `1_full7.62M` subset. It reads the read-only Caption Record database, converts the upstream Source Tar inventory and writes verified Parquet Shards to `mirror013/danbooru-8M-qwen3.5`. The upstream inventory is discovered at runtime from ModelScope `Root=images` and its `TotalCount`; the observed inventory is 3,000 Source Tars. Completion requires one `completed` or audited `skipped` outcome for every inventory item. This is not a claim that every SQLite record or every database subset is published. ## Run Modes The single entry point infers its mode from the checkpoint: - No checkpoint: initialize an empty state, verify that the target has no managed Parquet Shards, clean only managed local work files, and start at `images_v1/images-000001.parquet`. - Existing checkpoint: continue unresolved Source Tars without revalidating every completed Shard. Existing mappings are preserved. ```bash cargo run --release -- run --publish \ --checkpoint run/progress.json --database danbooru.db --workdir run/work ``` If the checkpoint is absent while the target already contains Parquet Shards, the runtime stops. It never infers mappings from untrusted remote history or deletes remote results automatically. ## Data Contract Each Parquet Shard uses these 15 columns: `md5`, `image_bytes`, `source`, `source_id`, `subset`, `source_path`, `title`, `brief`, `description`, `tags_character`, `tags_general`, `tags_rating`, `width`, `height`, and `image_ext`. Image bytes use cyclic bytewise XOR with the established canonical 25-byte layout. The configured external value is normalized before use: replace the first `danbooru-` with `danbooru`, then remove every `.`; the result must be exactly 25 ASCII bytes. Encoding and decoding both use that normalized value cyclically, so applying the raw configured string directly is incorrect. For a consumer, the operation is conceptually: ```text canonical = configured.replace_first("danbooru-", "danbooru").replace(".", "") decoded[i] = image_bytes[i] XOR canonical[i % 25] ``` The following complete Python example validates every row in one local Parquet file. It deliberately reads the configured value from `XOR_KEY` and performs the canonicalization step before decoding: ```python import hashlib import os import sys import pyarrow.parquet as pq def canonical_key(value: str) -> bytes: value = value.replace("danbooru-", "danbooru", 1).replace(".", "") encoded = value.encode("ascii") if len(encoded) != 25: raise ValueError("canonical XOR key must be exactly 25 ASCII bytes") return encoded def main(path: str) -> None: key = canonical_key(os.environ["XOR_KEY"]) table = pq.read_table(path, columns=["md5", "image_bytes"]) for row, (expected_md5, encoded) in enumerate( zip(table["md5"].to_pylist(), table["image_bytes"].to_pylist()), 1 ): decoded = bytes( byte ^ key[index % len(key)] for index, byte in enumerate(encoded) ) if decoded[:4] != b"RIFF" or decoded[8:12] != b"WEBP": raise ValueError(f"row {row}: decoded bytes are not RIFF/WEBP") actual_md5 = hashlib.md5(decoded).hexdigest() if actual_md5 != expected_md5: raise ValueError( f"row {row}: MD5 mismatch: {actual_md5} != {expected_md5}" ) print(f"validated {table.num_rows} rows") if __name__ == "__main__": if len(sys.argv) != 2: raise SystemExit(f"usage: XOR_KEY=... {sys.argv[0]} shard.parquet") main(sys.argv[1]) ``` Consumers need the external XOR key; the key is intentionally not embedded in this repository or the public dataset README. Decoding must restore the `RIFF`/`WEBP` signature and the row `md5`. Rows with no Caption Record are counted as explicit skips. Source/member read, archive parse, and image decode errors are categorized skips. Duplicate, text-mismatched, or tag-mismatched Caption Records remain integrity failures. ## Validation and Layout Local and remote validation checks schema, XOR-decoded WebP bytes, MD5, distributed source mapping, row-group limits, and footer size. Row groups stay within 16 MiB of uncompressed data and footers within 4 MiB; remote checks use authenticated Range reads no larger than 24 MiB. ```bash cargo run -- inspect --database danbooru.db cargo run -- plan --checkpoint run/progress.json --fresh cargo run -- status --checkpoint run/progress.json --workdir run/work ``` `plan --fresh` is read-only. It previews an empty checkpoint but does not start publication. ## Credentials and Scope Credentials are read from the local `.env` only in child processes. The publisher maps `modelscope_key` to `MODELSCOPE_API_TOKEN`; secrets never enter arguments, reports, checkpoints, logs, or public documentation. Pixiv, Realbooru, and PVC subsets are outside this runtime's scope.