orgrctera/uda_feta_qa

---
license: mit
language:
- en
pretty_name: UDA FETA-QA (Retrieval)
size_categories:
- 1K<n<10K
tags:
- wikipedia
- question-answering
- retrieval
- rag
- unstructured-documents
- table-qa
configs:
- config_name: default
data_files:
- split: default
path: data/default-*
dataset_info:
features:
- name: input
dtype: string
- name: metadata
dtype: string
- name: answers
dtype: string
- name: doc_url
dtype: string
- name: evidence
dtype: string
splits:
- name: default
num_bytes: 2184719
num_examples: 1023
download_size: 889482
dataset_size: 2184719
---

# UDA FETA-QA (`orgrctera/uda_feta_qa`)

## Overview

This dataset is the **FETA-QA** (FeTaQA-style) slice of the **UDA (Unstructured Document Analysis)** benchmark: **1,023** question–answer instances over **Wikipedia**-sourced **tables**, packaged for **retrieval-oriented** evaluation in RAG pipelines.

**UDA** (Hui et al., NeurIPS 2024) is a benchmark suite for Retrieval-Augmented Generation (RAG) over messy, real-world documents (PDF/HTML) where evidence mixes narrative text and structured content. In the Wikipedia track, UDA includes **FetaTab**—examples aligned with **FeTaQA** (free-form table QA) at the scale of this release (**878** documents and **1,023** Q&A pairs in the UDA paper’s configuration for this subset).

**FeTaQA** (Nan et al., TACL 2022) introduces **free-form** answers to questions over Wikipedia tables: systems must **retrieve** and **reason** over tabular evidence and produce **natural-language** explanations, not only short spans. UDA adopts FeTaQA-style supervision within its broader document-analysis benchmark so that **parsing, chunking, and retrieval** are evaluated alongside generation.

In this Hub release, each row is a **retrieval task instance**: given a question (`input`), a system must **locate** the right document/table context and **ground** a free-form answer—consistent with the **UDA** setting, where **retrieval** is a first-class step before answering.

## Task

- **Task type:** **Retrieval** (within a RAG / document-analysis pipeline) for **FETA-QA** / **FeTaQA**-style **table question answering** over **Wikipedia**-derived tables.
- **Input:** A natural-language question (`input`) about entities, events, or facts described in a table (and optionally surrounding article context in a full ingestion pipeline).
- **Supervision / reference:** `expected_output` is a JSON string with a gold **free-form answer** (`answers`), **document URL** (`doc_url`), and structured **evidence** (`evidence`: `table_array`, `highlighted_cell_ids` supporting the answer). `metadata` records UDA identifiers (`sub_benchmark`: `feta_qa`).

Evaluation typically combines **retrieval quality** (whether the correct table or cells are retrieved) with **answer quality** (fluency and factual alignment with gold free-form text and highlighted cells), following FeTaQA and UDA protocols.

## Background

### FeTaQA (FETA-QA)

FeTaQA targets **question answering over tables** where answers are **free-form** text that may **integrate** multiple table cells. Compared with short-answer table QA, it stresses **retrieval**, **inference**, and **surface realization** from structured sources. Annotations include **supporting cells**, enabling training and evaluation of both retrieval and generation.

### UDA benchmark

UDA revisits RAG and LLM-based document analysis across domains using **2,965** real-world documents and **29,590** expert-annotated Q&A pairs, with sources kept in **original** formats to stress **parsing and retrieval** as well as generation. The **FetaTab** Wikipedia subset corresponds to this dataset’s **1,023** examples in the `default` split.

## Data fields

| Column | Type | Description |
|--------|------|-------------|
| `input` | `string` | Question text posed over the table (and article, in full-document settings). |
| `expected_output` | `string` | JSON string with `answers` (gold free-form text), `doc_url`, and `evidence` (`table_array`, `highlighted_cell_ids`). |
| `metadata` | struct | `benchmark_name` (`uda_feta_qa`), `benchmark_type` (`uda`), `split`, `sub_benchmark` (`feta_qa`), and `value` (JSON string with identifiers such as `label_key`, `label_file`, `doc_url`, `q_uid`). |

**Splits:** Single split `default` with **1,023** examples.

## Examples

The following rows are taken from the dataset (JSON in `expected_output` is shown formatted for readability; large `table_array` values are abbreviated with `[...]`).

**Example 1 — theatre and production**

- **`input`:** `When and in what play did Platt appear at the Music Box Theatre?`
- **`expected_output`:**
```json
{
"answers": "In 2016 and 2017, Platt played in Dear Evan Hansen on Broadway at the Music Box Theatre.",
"doc_url": "http://en.wikipedia.org/wiki/Ben_Platt_(actor)",
"evidence": {
"table_array": [
["Year", "Production", "Role", "Venue", "Notes"],
["2002", "The Music Man", "Winthrop Paroo", "Hollywood Bowl", "Los Angeles"],
"...",
["2016–2017", "Dear Evan Hansen", "Evan Hansen", "Music Box Theatre", "Broadway"]
],
"highlighted_cell_ids": [[10, 1], [11, 0], [11, 1], [11, 3], [11, 4]]
}
}
```

**Example 2 — role at a venue**

- **`input`:** `Which role did Ben Platt play at the Second Stage Theatre in 2016?`
- **`expected_output`:**
```json
{
"answers": "In 2016, Platt played the role of Evan Hansen at Second Stage Theatre.",
"doc_url": "http://en.wikipedia.org/wiki/Ben_Platt_(actor)",
"evidence": {
"table_array": [
["Year", "Production", "Role", "Venue", "Notes"],
"...",
["2016", "Dear Evan Hansen", "Evan Hansen", "Second Stage Theatre", "Off-Broadway"]
],
"highlighted_cell_ids": [[9, 0], [10, 0], [10, 2], [10, 3], [11, 2]]
}
}
```

**`metadata.value` (structure, example):**
```json
{
"label_key": "Ben Platt (actor)",
"label_file": "feta_qa",
"doc_url": "http://en.wikipedia.org/wiki/Ben_Platt_(actor)",
"q_uid": "14717"
}
```

## References

### FeTaQA (source task & data lineage)

Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Nick Schoelkopf, Riley Kong, Xiangru Tang, Murori Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev. **FeTaQA: Free-form Table Question Answering.** *Transactions of the Association for Computational Linguistics (TACL)*, 2022.

- **Abstract (short):** Introduces FeTaQA with 10K Wikipedia-based {table, question, free-form answer, supporting table cells} pairs; free-form answers require retrieval, inference, and integration from structured data—more challenging than short-fact table QA. Provides pipeline and end-to-end baselines.
- **arXiv:** [https://arxiv.org/abs/2104.00369](https://arxiv.org/abs/2104.00369)
- **ACL Anthology:** [https://aclanthology.org/2022.tacl-1.3/](https://aclanthology.org/2022.tacl-1.3/)
- **Project / code:** [https://github.com/Yale-LILY/FeTaQA](https://github.com/Yale-LILY/FeTaQA)

### UDA benchmark (suite containing this FETA-QA slice)

Yulong Hui, Yao Lu, Huanchen Zhang. **UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis.** *NeurIPS 2024* (Datasets and Benchmarks Track).

- **Abstract (short):** Presents UDA with thousands of real-world documents and tens of thousands of expert-annotated Q&A pairs; evaluates LLM- and RAG-based document analysis and highlights **parsing** and **retrieval** design choices alongside generation.
- **arXiv:** [https://arxiv.org/abs/2406.15187](https://arxiv.org/abs/2406.15187)
- **NeurIPS proceedings:** [https://proceedings.neurips.cc/paper_files/paper/2024/hash/7c06759d1a8567f087b02e8589454917-Abstract-Datasets_and_Benchmarks_Track.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7c06759d1a8567f087b02e8589454917-Abstract-Datasets_and_Benchmarks_Track.html)
- **Code & resources:** [https://github.com/qinchuanhui/UDA-Benchmark](https://github.com/qinchuanhui/UDA-Benchmark)

### Related Hub resources

- Aggregated UDA QA (reference): [qinchuanhui/UDA-QA](https://huggingface.co/datasets/qinchuanhui/UDA-QA)

## Citation

If you use this dataset, please cite **both** FeTaQA and UDA (and this dataset record as appropriate):

```bibtex
@article{nan-etal-2022-fetaqa,
title = {FeTaQA: Free-form Table Question Answering},
author = {Nan, Linyong and Hsieh, Chiachun and Mao, Ziming and Lin, Xi Victoria and Verma, Neha and Zhang, Rui and Kry{\'s}ci{\'n}ski, Wojciech and Schoelkopf, Nick and Kong, Riley and Tang, Xiangru and Mutuma, Murori and Rosand, Ben and Trindade, Isabel and Bandaru, Renusree and Cunningham, Jacob and Xiong, Caiming and Radev, Dragomir},
journal = {Transactions of the Association for Computational Linguistics},
year = {2022},
volume = {10},
pages = {35--49}
}
```

```bibtex
@article{hui2024uda,
title = {UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis},
author = {Hui, Yulong and Lu, Yao and Zhang, Huanchen},
journal = {arXiv preprint arXiv:2406.15187},
year = {2024}
}
```

## License

Use this dataset in compliance with the **original FeTaQA** and **UDA** data licenses and terms. The FeTaQA project (CC BY-SA 4.0 for the original release) and UDA documentation specify conditions; verify terms for your use case before redistribution or commercial use.