ProcVQA-20M-media

Works under revision. This repository contains the media files for the ProcVQA dataset. Original dataset: [ProcVQA-20M-annotations](https://huggingface.co/datasets/ce-amtic/ProcVQA-20M-annotations) ## Tools Use this code to unzip all shards after download. ```python import os import webdataset as wds from pathlib import Path from tqdm import tqdm # ======================== # Configs # ======================== INPUT_DIR = "./wds_shards" OUTPUT_DIR = "./recovered" PATTERN = "shard-*.tar" # ======================== os.makedirs(OUTPUT_DIR, exist_ok=True) shards = os.path.join(INPUT_DIR, PATTERN) dataset = wds.WebDataset(shards) print("unzip start...") count = 0 for sample in tqdm(dataset): key = sample["__key__"] for k, v in sample.items(): if k == "__key__": continue ext = k # jpg/png/json etc. filename = f"{key}.{ext}" out_path = os.path.join(OUTPUT_DIR, filename) with open(out_path, "wb") as f: f.write(v) break count += 1 print(f"completed with {count} files recovered") ```