LongPIBench

# LongPIBench LongPIBench is a benchmark release containing 100 linked examples. Each example is identified by an integer from `0` through `99` and contains: - a complete multi-file LaTeX paper project in `papers/<id>/`; - structured personal-profile data in `person_info/<id>.json`; - an email exchange and attachment text in `emails/<id>.json`; and - a before/after programming task in `code_changes/<id>.json`. The directory structure is part of the dataset. In particular, each paper is a raw LaTeX project whose PDF is intended to be generated dynamically by the benchmark code. The repository therefore preserves source files instead of flattening the examples into a CSV or Parquet table. ## Download Install the lightweight Hub client: ```bash pip install huggingface_hub ``` Download the complete, version-pinned dataset: ```python from huggingface_hub import snapshot_download dataset_root = snapshot_download( repo_id="RainWatcher/LongPIBench", repo_type="dataset", revision="v1.0.0", local_dir="LongPIBench", ) print(dataset_root) ``` To download a single linked example: ```python from huggingface_hub import snapshot_download example_id = 0 snapshot_download( repo_id="RainWatcher/LongPIBench", repo_type="dataset", revision="v1.0.0", local_dir="LongPIBench", allow_patterns=[ f"papers/{example_id}/*", f"person_info/{example_id}.json", f"emails/{example_id}.json", f"code_changes/{example_id}.json", "manifest.jsonl", "checksums.sha256", "README.md", ], ) ``` The lightweight manifest can also be inspected with `datasets`: ```python from datasets import load_dataset manifest = load_dataset( "RainWatcher/LongPIBench", data_files="manifest.jsonl", split="test", revision="v1.0.0", ) print(manifest[0]) ``` `load_dataset()` loads the index, not the complete LaTeX project trees. Use `snapshot_download()` when running experiments. ## Repository layout ```text LongPIBench/ ├── manifest.jsonl ├── checksums.sha256 ├── papers/ │ └── <id>/ │ ├── paper.tex │ ├── abstract.tex │ ├── intro.tex │ ├── rw.tex │ ├── method.tex │ ├── eval.tex │ ├── dl.tex │ ├── conclusion.tex │ ├── reference.bib │ ├── iclr2024_conference.sty │ └── iclr2024_conference.bst ├── person_info/<id>.json ├── emails/<id>.json └── code_changes/<id>.json ``` Empty `build/` directories are intentionally not stored. Code that compiles a paper should create the directory first. ## Compiling a paper A typical compilation command is: ```bash cd papers/0 mkdir -p build latexmk -pdf -interaction=nonstopmode -halt-on-error \ -outdir=build paper.tex ``` The projects use a broad set of standard LaTeX packages. A full TeX Live installation is recommended for reproducible execution. Compilation should be performed in an isolated environment, without unrestricted shell escape or access to sensitive host files. ## Integrity and versioning `checksums.sha256` contains a SHA-256 digest for every released source-data file. Verify a checkout from the repository root with: ```bash shasum -a 256 -c checksums.sha256 ``` Experiments should cite and download the immutable `v1.0.0` tag rather than the moving `main` branch. ## Data notes - All four components contain the same 100 IDs (`0` through `99`). - Every paper directory contains the same eleven-file project layout. - Generated PDFs and LaTeX build artifacts are excluded. - Some examples contain names, organizations, email addresses, or other person-like details as benchmark content. They must not be interpreted as verified real-world identities or contact information. - LaTeX and code content should be treated as untrusted input when evaluated. ## Third-party files The paper projects include copies of the ICLR 2024 LaTeX style and an associated BibTeX style. The BibTeX style states that it is distributed under the LaTeX Project Public License. These third-party template files remain subject to their respective upstream terms and are not relicensed by the dataset's main license. ## License and citation Except for third-party files described above, LongPIBench is released under the MIT License. See `LICENSE` for the full text. If you use LongPIBench, please cite the associated paper. Formal BibTeX metadata will be added when the paper's public bibliographic record is available.