officeqa

# OfficeQA ## Dataset Summary **OfficeQA** is a grounded reasoning benchmark by Databricks for evaluating model and agent performance on end-to-end reasoning over real-world documents. The benchmark consists of question–answer pairs that require reasoning over historical **U.S. Treasury Bulletin** documents (1939–2025), which contain dense financial tables, charts, and narrative text. OfficeQA is designed to test retrieval, tool use, and multi-step reasoning in document-grounded settings. Two question sets are available: - **OfficeQA Pro** (N=133) — the default benchmark for evaluating frontier models. Contains the hardest, most discriminative questions. - **OfficeQA Full** (N=246) — includes all questions, adding easier items useful for hillclimbing and studying model behavior at different difficulty levels. Key facts: - **Source documents:** U.S. Treasury Bulletins (697 issues, 1939–2025) - **Primary use cases:** RAG, agent evaluation, document reasoning benchmarks - **Dataset license:** CC-BY-SA 4.0 - **Code license:** Apache 2.0 --- ## Getting Started ### Load the benchmark questions ```python from datasets import load_dataset # Authenticate first (dataset is gated) # huggingface_hub.login() or set HF_TOKEN env var # Pro subset — default for evaluating frontier models dataset = load_dataset("databricks/officeqa", data_files="officeqa_pro.csv", split="train") # Full benchmark — includes easier questions for hillclimbing dataset = load_dataset("databricks/officeqa", data_files="officeqa_full.csv", split="train") ``` ### Download the corpus ```python from huggingface_hub import snapshot_download # Download transformed text — recommended for LLM/RAG workflows (~460MB) local_dir = snapshot_download( repo_id="databricks/officeqa", repo_type="dataset", allow_patterns="treasury_bulletins_parsed/transformed/*.txt", ) ``` ### Score answers using reward.py (from GitHub) ```bash git clone https://github.com/databricks/officeqa ``` ```python from reward import score_answer score = score_answer(ground_truth="123.45", prediction="123.45", tolerance=0.0) ``` --- ## Supported Tasks and Leaderboards - Question Answering - Grounded / Retrieval-Augmented Generation - Agentic reasoning over documents This dataset is intended for **benchmarking**, not for model pretraining. --- ## Languages - English (`en`) --- ## Dataset Structure The dataset has two main components: ### 1. Benchmark Dataset | File | Contents | |------|----------| | `officeqa_full.csv` | All 246 questions with answers | | `officeqa_pro.csv` | 133 curated questions with answers | **Schema:** | Column | Description | |--------|-------------| | `uid` | Unique question identifier | | `question` | Question text | | `answer` | Ground-truth answer *(answer files only)* | | `source_docs` | Source URLs from the FRASER archive | | `source_files` | Corresponding Treasury Bulletin filenames | | `difficulty` | `easy` or `hard` | --- ### 2. Treasury Bulletin Corpus The Treasury Bulletin corpus is provided in **three formats**, all available via Git LFS in this Hugging Face repository. #### a) Original PDFs 697 PDFs (1939–2025), ~4GB total, available via LFS in this repo. ```python from huggingface_hub import snapshot_download # Download all PDFs (requires dataset access) local_dir = snapshot_download( repo_id="databricks/officeqa", repo_type="dataset", allow_patterns="treasury_bulletin_pdfs/*", ) ``` #### b) Parsed JSON Documents 697 JSON files (~730MB total) with layout structure, tables, bounding boxes, and metadata. More lossless than transformed text; useful for experimenting with different table representations (e.g. Markdown vs HTML). ```python from huggingface_hub import snapshot_download local_dir = snapshot_download( repo_id="databricks/officeqa", repo_type="dataset", allow_patterns="treasury_bulletins_parsed/jsons/*.json", ) ``` #### c) Transformed Text Documents 697 plain-text files (~460MB total) with tables converted to Markdown. Recommended for LLM and RAG workflows. ```python from huggingface_hub import snapshot_download local_dir = snapshot_download( repo_id="databricks/officeqa", repo_type="dataset", allow_patterns="treasury_bulletins_parsed/transformed/*.txt", ) ``` To download the full corpus at once: ```python from huggingface_hub import snapshot_download local_dir = snapshot_download( repo_id="databricks/officeqa", repo_type="dataset", ) ``` --- ## Mapping Questions to Source Documents Each question references the Treasury Bulletin file(s) required to answer it via the `source_files` column. ### Filename convention ``` treasury_bulletin_{YEAR}_{MONTH_NUM}.{ext} ``` ### Month mapping ``` january → 01 july → 07 february → 02 august → 08 march → 03 september → 09 april → 04 october → 10 may → 05 november → 11 june → 06 december → 12 ``` --- ## Evaluation The [GitHub repository](https://github.com/databricks/officeqa) includes a reference scoring function (`reward.py`) for evaluating predictions against ground-truth answers. ```bash # Get the scoring code git clone https://github.com/databricks/officeqa ``` ```python from reward import score_answer score = score_answer( ground_truth="123.45", prediction="123.40", tolerance=0.01 ) ``` --- ## Limitations - Requires access to external documents for full performance - Focused on financial and government reporting domains - Not designed for conversational QA without retrieval --- ## License - **Dataset:** CC-BY-SA 4.0 - **Code and scripts:** Apache 2.0 --- ## Citation ```bibtex @dataset{officeqa, title = {OfficeQA: A Grounded Reasoning Benchmark}, author = {Databricks}, year = {2025}, license = {CC-BY-SA-4.0} } ``` ## Contact This dataset was created and is maintained by the Databricks research team. For questions, open an issue on the [GitHub repository](https://github.com/databricks/officeqa).