facebook/LCFO
收藏资源简介：
--- license: cc-by-nc-4.0 task_categories: - summarization language: - en pretty_name: lcfo size_categories: - n<1K configs: - config_name: source_data data_files: - split: test path: "data/source_data.parquet" - config_name: summarization_eval data_files: - split: test path: "data/summarization_eval.parquet" - config_name: summary_expansion_eval data_files: - split: test path: "data/summary_expansion_eval.parquet" --- # LCFO: Long Context and Long Form Output Dataset This is a dataset for English longform summarization and summarization expansion. ## Dataset Details ### Dataset Description This is a dataset for English longform summarization and summarization expansion. It consists of 251 long documents (5K words on average) from 10 different domains, and their expert-written summaries of 3 different lengths: 20%, 10%, and 5% of the source document length. **NOTE: this is an early version of the dataset; it is going to be updated soon**. **NOTE: the source documents for most domains are not provided; they should be reconstructed. The instructions to be added soon.** - **Curated by:** [More Information Needed] - **Language(s) (NLP):** English - **License:** CC-BY-NC 4.0 (but the `source` column for the Wikipedia split is licensed under CC-BY-SA 4.0) - **Paper :** [LCFO: Long context and long form output dataset and benchmarking](https://arxiv.org/abs/2412.08268) ## Uses <!-- Address questions around how the dataset is intended to be used. --> ### Direct Use The dataset supports the following use cases: - Summarization (inclusing gradual summarization) - Summary expansion (generating a longer document that preserves the essential elements from the summary) - Reading comprehension with generative question answering - Evaluation of automatic quality metrics for summarization and summary expansion Being rather small, it is intended as a test dataset. ### Out-of-Scope Use The LCFO dataset is not inteded to be used as training data. ## Dataset Structure <!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. --> The dataset consists of 3 tables: 1. `source_data`: description of the source documents, human-generated summaries, their alignment by paragraphs, and abstractive questions about the documents. 2. `summarization_eval`: the summaries of 3 different lengths (20%, 10%, 5%) generated for each document by humans and 3 models (GPT 4, Llama 3.1-70B, Llama 3.1-8B), and their human evaluation. 3. `summary_expansion_eval`: the documents re-generated by 3 models from the 20% summaries (for 4 domains), as well as their human evaluation. The tables are joinable by two fields present in each of them: `subset` (one of the 10 source datasets where the documents were drawn) and `item_id` (identifier of the document within a dataset). The `source_data` table has the following columns: - `subset (str)` : data source identifier - `item_id (str)`: document identifier - `source_text (str)`: the source document text (non-empty only for Wikipedia; needs to be reconstructed for other sources) - `src_paragraph_bounds (List[List[int]])`: pairs of start and end characters for each "paragraph' in the source document - `word_count (int)`: number of words in the source document - `summary_20, summary_10, summary_5 (str)`: human-generated summaries of the corresponding lengths - `summary_20_paragraphs, summary_10_paragraphs, summary_5_paragraphs (List[str])`: the same human summaries, split into paragraphs - `summary_20_offsets, summary_10_offsets, summary_5_offsets (List[str])`: indices of the source document paragraphs from which the information in each summary paragraph has been derived. The `summarization_eval` table has the following columns: - `subset (str)` : data source identifier - `item_id (str)`: document identifier - `model (str)` : summarization method identifier (including `human`) - `summary_20, summary_10, summary_5 (str)`: human- or machine-generated summaries of the corresponding lengths - `eval_20, eval_10, eval_5 (Dict)`: human evaluation of the corresponding summaries, including the following fields: - `s_2a, s_2b, s_2c, s_2d (List[int])`: evaluation of 4 quality aspects: attribution, coverage of the main ideas, conciseness and readability (on the scale from 0 to 4) - `s_3 (List[int])`: evaluation of the overall summarization quality (on the scale from 0 to 10) - `qa_1, qa_2, ... (List[str])`: whether the summary answers the corresponding question from `source_data` (`Yes` or `No`) Each summary is evaluated by several annotators (usually 3); each field represents a list of their responses. The `summary_expansion_eval` table has the following columns: - `subset (str)` : data source identifier - `item_id (str)`: document identifier - `model (str)` : summarization method identifier - `inverted_summ_20 (str)`: machine-generated expansion of the 20% summary - `eval_20, eval_10, eval_5 (Dict)`: human evaluation of the expanded summaries, including the following fields: - `r1 (str)`: whether the expanded summary is understandable - `r2a_lf, r2b_lf, r2c_lf, r2d_lf, r2e_lf, r2f_lf (int)`: evaluation of 6 quality aspects: coverage of main core ideas, cohesion, richness in details, creativity, non-repetitiveness, and interest, (on the scale from 0 to 4) - `s_3 (int)`: evaluation of the overall text quality (on the scale from 0 to 10) - `qa_1, qa_2, ... (str)`: whether the expanded summary answers the corresponding question from `source_data` (`YES` or `NO`) Each expansion is evaluated by several annotators (usually 3); each field represents a list of their responses. ## Dataset Creation Please read [the accompanying paper](https://arxiv.org/abs/2412.08268) about the source documents and the data annotation details. ## Reconstructing the source documents The instructions for reconstructing the source documents will be added soon. ## Citation ``` @article{lcfo, author = {Marta R. Costa-jussà and Pierre Andrews and Mariano Coria Megliogli and Joy Chen and Joe Chuang and David Dale and Christophe Ropers and Alex Mourachko and Eduardo Sánchez and Holger Schwenk and Tuan Tran and Arina Turkatenko and Carleigh Wood}, journal = {ArXiv}, title = {{LCFO}: Long Context and Long Form Output Dataset and Benchmarking}, year = {2024}, } ```
LCFO is a dataset for English longform summarization and summarization expansion. It consists of 251 long documents (5K words on average) from 10 different domains, and their expert-written summaries of 3 different lengths: 20%, 10%, and 5% of the source document length. The dataset structure includes three tables: `source_data`, `summarization_eval`, and `summary_expansion_eval`, which describe the source documents, human- and model-generated summaries, and their evaluations. The dataset is intended for testing purposes and is not meant to be used as training data.
