ledger-long-context-KPI-QA

# LEDGER — Long-Context KPI Question Answering & Page Retrieval This dataset is part of the **LEDGER** (Long-context Evaluation of Documents for Grounded Extraction and Retrieval) benchmark. It supports **two of the three LEDGER tasks**: 1. **Page-level KPI retrieval** — given a natural-language question about a financial KPI and the corresponding annual report, retrieve the relevant page(s). Each row includes TREC-style graded relevance judgments (`qrels`) over all candidate pages. 2. **Conversational long-context extraction** ("needle-in-a-haystack") — given the same question and the full OCR'd report (~126k tokens on average), extract the single numeric KPI value. ## Dataset Description Each row is a single natural-language query that names one company, one fiscal year, and one KPI (e.g. *"What was APA Corporation's total revenue in fiscal year 2021?"*). The `mmd_text` column contains the full OCR'd annual report (page-aligned Markdown, median 124 pages), and `qrels` provides per-page relevance grades (0 = not relevant, 1 = contextual mention, 2 = primary source). Ground-truth KPI values are sourced from SEC EDGAR XBRL (`companyfacts`), Yahoo Finance, and Alpha Vantage, reconciled through an ordered tag waterfall that always selects the attributable-to-parent / unrestricted / aggregate scope. Questions were generated from curated templates with company name aliases sampled from DBpedia. Relevance judgments were produced by mining candidate (query, page) pairs via unit-normalized value matching, then grading each on a 0/1/2 scale with an LLM judge (Qwen 3.6-27B). ### Configs | Config | Queries | Reports | Companies | Years | Purpose | |--------|---------|---------|-----------|-------|---------| | `eval` | 10,000 | 494 | 111 | 2017–2022 | Benchmark evaluation | | `no_eval` | 104,529 | 4,505 | 737 | 2009–2024 | Training / development | ### Schema | Column | Type | Description | |--------|------|-------------| | `query_id` | string | Unique identifier (`{ticker}_{kpi}_{year}`) | | `query_text` | string | Natural-language question | | `ticker` | string | Stock ticker symbol | | `exchange` | string | Stock exchange (NYSE, NASDAQ, LSE, AMEX, ASX, OTC) | | `company_name` | string | Company long name | | `industry` | string | Industry classification | | `year` | int | Fiscal year | | `kpi` | string | KPI key (one of 31; e.g. `revenue`, `net_income`, `total_assets`) | | `value` | float64 | Ground-truth KPI value (raw single units) | | `source` | string | Data provenance (`edgar`, `yfinance`, `alphavantage`) | | `tag` | string | Exact XBRL tag or derivation method (e.g. `sum:A-B`) | | `qrels` | list[{doc_id: str, relevance: int}] | Per-page TREC relevance judgments (grades 0/1/2) | | `mmd_text` | string | Full OCR text of the annual report (Markdown with page splits) | ### KPIs Covered (31) Organized across three financial statements: - **Income statement**: revenue, cost_of_revenue, gross_profit, rd_expense, sga_expense, operating_income, interest_expense, income_tax_expense, net_income, eps_basic, eps_diluted - **Balance sheet**: total_assets, total_liabilities, stockholders_equity, stockholders_equity_incl_nci, cash_and_equivalents, cash_incl_restricted, long_term_debt_total, long_term_debt_noncurrent, long_term_debt_current, short_term_borrowings, inventory, accounts_receivable, accounts_payable, shares_outstanding - **Cash flow**: operating_cash_flow, investing_cash_flow, financing_cash_flow, capex, depreciation_amortization, dividends_paid ### OCR Format Reports are OCR'd with DeepSeek-OCR-2 into page-aligned Markdown. Pages are delimited by the literal string `<--- Page Split --->`. Tables are rendered as HTML/LaTeX. Per-page raster images are available in `eval/mmd/` for visual tasks. ### KPI Value Conventions - **Monetary values** (revenue, net_income, total_assets, etc.): raw single units of the reporting currency. E.g. $1.5B revenue = `1500000000.0`. - **Per-share values** (eps_basic, eps_diluted): as reported, not scaled. - **Share counts** (shares_outstanding): in single shares. - **Capex / dividends_paid**: positive outflows. - **Cash flow subtotals**: reported sign (negative = outflow). ### Additional Files - `eval/mmd/` and `no_eval/mmd/`: Raw `.mmd` files (same text as `mmd_text` column). ## Usage ```python from datasets import load_dataset # Load eval set (10,000 queries over 494 reports) ds = load_dataset("artefactory/ledger-long-context-KPI-QA", "eval") # Each row is one query targeting one KPI in one report row = ds["test"][0] print(row["query_text"]) # "What was APA Corporation's total revenue in FY2021?" print(row["value"]) # 8797000000.0 print(len(row["mmd_text"])) # ~500k chars of OCR text # --- Task 1: Page-level retrieval --- # Use qrels for retrieval evaluation (TREC format) relevant_pages = [q for q in row["qrels"] if q["relevance"] >= 1] primary_pages = [q for q in row["qrels"] if q["relevance"] == 2] # --- Task 2: Needle-in-a-haystack extraction --- # Feed mmd_text + query_text to an LLM; compare output to row["value"] ``` ## Evaluation Protocol ### Page-level retrieval (Task 1) Index pages of each report (split on `<--- Page Split --->`), query with `query_text`, and evaluate against `qrels` using standard IR metrics. We report Recall@k, MRR, and nDCG@k with binary relevance (rel ≥ 1) and graded gains (0/1/2). The `qrels` column is directly compatible with `trec_eval` and `pytrec_eval`. ### Conversational extraction (Task 2) The model receives the full `mmd_text` and one `query_text`, and must return a structured answer (value + unit scale + page number). A prediction is **matched** if: ``` |predicted − ground_truth| / |ground_truth| ≤ tolerance ``` Default tolerance: ±0.05%. We report **recall** (correct / gold values) and **precision** (correct / attempted). For the eval config, recall coincides with exact-match accuracy. ### Baselines (from the paper) | Model | Recall | Precision | |-------|--------|-----------| | Qwen3.6-27B | **91.4** | **93.5** | | Ministral-3-14B | 87.9 | 88.6 | | gpt-oss-20b | 85.3 | 86.8 | | Nemotron-3-Nano-30B | 15.0 | 15.3 | ## Data Sources - **Annual reports**: 4,999 publicly available corporate annual reports (PDF), digitized with DeepSeek-OCR-2. - **KPI ground truth**: SEC EDGAR XBRL `companyfacts` for U.S. listings; Yahoo Finance for non-U.S.; Alpha Vantage gap-fill (never overwrites). - **Questions**: Generated from curated templates with company name aliases from DBpedia and KPI question variants from Gemini 3.1 Pro. - **Relevance judgments**: Unit-normalized value matching + LLM judge (Qwen 3.6-27B) grading on a 0/1/2 scale. ## Citation key If you use LEDGER datasets and / or code ressources, please consider citing our work with: ``` @misc{moslonka2026ledgerlongcontextbenchmarkcorporate, title={LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction}, author={Charles Moslonka and Amaury de Vitry and Arthur Garnier and Hicham Randrianarivo and Emmanuel Malherbe}, year={2026}, eprint={2606.13100}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2606.13100}, } ``` ## License Code: MIT | Data: CC-BY-4.0 ## Links - **Collection**: [artefactory/ledger](https://huggingface.co/collections/artefactory/ledger) - **Code**: [github.com/artefactory/LEDGER](https://github.com/artefactory/LEDGER)