LumberChunker/GutenQA_Semantic

GutenQA-Semantic数据集基于100本公共领域的叙事书籍，使用了语义分块技术对文本进行分割。该数据集是[GutenQA](https://huggingface.co/datasets/LumberChunker/GutenQA)基准的一部分，并在[LumberChunker](https://github.com/joaodsmarques/LumberChunker)论文中作为基线分块方法之一。数据集的组织结构包括书籍名称、书籍ID、分块ID和分块内容。语义分块利用嵌入技术将语义相似的文本段聚类。

原始信息汇总

GutenQA-Semantic 数据集概述

基本信息

• 许可证: MIT
• 任务类别: 问答 (Question Answering)
• 语言: 英语 (English)

数据集配置

• 配置名称: semantic
  • 数据文件:
    • 分割: semantic_chunks
    • 路径: GutenQA_semantic.parquet

数据集描述

• 来源: 基于 GutenQA 数据集，包含100本公共领域叙事书籍。
• 处理方法: 使用语义分块 (Semantic Chunking) 技术，通过嵌入 (embeddings) 聚类语义相似的文本段落。

数据结构

• 列:
  • Book Name: 书籍标题
  • Book ID: 书籍的唯一整数标识符
  • Chunk ID: 书籍块的整数标识符，按其在书中的顺序排列
  • Chunk: 包含从书籍中提取的段落块，这些块是通过语义分块技术生成的

数据加载示例

python import pandas as pd dataset = pd.read_parquet("hf://datasets/LumberChunker/GutenQA_Paragraphs/GutenQA_semantic.parquet", engine="pyarrow")

过滤数据框以显示指定书名的行

single_book_chunks = dataset[dataset[Book Name] == A_Christmas_Carol_-_Charles_Dickens].reset_index(drop=True)

引用

bibtex @misc{duarte2024lumberchunker, title={LumberChunker: Long-Form Narrative Document Segmentation}, author={André V. Duarte and João Marques and Miguel Graça and Miguel Freire and Lei Li and Arlindo L. Oliveira}, year={2024}, eprint={2406.17526}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2406.17526}, }

书籍参考

• 书籍列表:
  • [1] Twain, M. (2004). Adventures of Huckleberry Finn. Project Gutenberg.
  • [2] Carroll, L. (2008). Alices Adventures in Wonderland. Project Gutenberg.
  • [3] Tolstoy, L. (1998). Anna Karenina. Project Gutenberg.
  • [4] Montgomery, L. (2008). Anne of Green Gables. Project Gutenberg.
  • [5] Verne, J. (1994). Around the World in Eighty Days. Project Gutenberg.
  • [6] Dickens, C. (2004). A Christmas Carol. Project Gutenberg.
  • [7] Twain, M. (2004). A Connecticut Yankee in King Arthurs Court. Project Gutenberg.
  • [8] Hudson, W. (2005). A Crystal Age. Project Gutenberg.
  • [9] Scott, S. (2006). A Legend Of Montrose. Project Gutenberg.
  • [10] Joyce, J. (2003). A Portrait of the Artist as a Young Man. Project Gutenberg.
  • [11] Forster, E. (2001). A Room with a View. Project Gutenberg.
  • [12] Doyle, A. (1995). A Study in Scarlet. Project Gutenberg.
  • [13] Dickens, C. (1994). A Tale of Two Cities. Project Gutenberg.
  • [14] Nietzsche, F. (2003). Beyond Good and Evil. Project Gutenberg.
  • [15] Le Fanu, J. (2003). Carmilla. Project Gutenberg.
  • [16] Gaskell, E. (1996). Cranford. Project Gutenberg.
  • [17] Dostoyevsky, F. (2006). Crime and Punishment. Project Gutenberg.
  • [18] Garis, H. (2005). Daddy takes us to the Garden. Project Gutenberg.
  • [19] Dickens, C. (1996). David Copperfield. Project Gutenberg.
  • [20] Stoker, B. (1995). Dracula. Project Gutenberg.
  • [21] Joyce, J. (2001). Dubliners. Project Gutenberg.
  • [22] Austen, J. (1994). Emma. Project Gutenberg.
  • [23] Shelley, M. (1993). Frankenstein Or The Modern Prometheus. Project Gutenberg.
  • [24] Dickens, C. (1998). Great Expectations. Project Gutenberg.
  • [25] Brothers, G. (2001). Grimms Fairy Tales. Project Gutenberg.
  • [26] Swift, J. (1997). Gullivers Travels into Several Remote Nations of the World. Project Gutenberg.
  • [27] Conrad, J. (2006). Heart of Darkness. Project Gutenberg.
  • [28] Jacobs, H. (2004). Incidents in the Life of a Slave Girl. Project Gutenberg.
  • [29] Bronte, C. (1998). Jane Eyre. Project Gutenberg.
  • [30] Hugo, V. (2008). Les Miserables. Project Gutenberg.
  • [31] Alcott, L. (1996). Little Women. Project Gutenberg.
  • [32] Dumas, A. (2001). Louise de la Valliere. Project Gutenberg.
  • [33] Flaubert, G. (2006). Madame Bovary. Project Gutenberg.
  • [34] Aurelius, E. (2001). Meditations. Project Gutenberg.
  • [35] Eliot, G. (1994). Middlemarch. Project Gutenberg.
  • [36] Melville, H. (2001). Moby Dick Or The Whale. Project Gutenberg.
  • [37] Wagner, R. (2004). My Life. Project Gutenberg.
  • [38] Douglass, F. (2006). Narrative of the Life of Frederick Douglass, an American Slave. Project Gutenberg.
  • [39] Dickens, C. (1996). Oliver Twist. Project Gutenberg.
  • [40] Austen, J. (1994). Persuasion. Project Gutenberg.
  • [41] Barrie, J. (2008). Peter Pan. Project Gutenberg.
  • [42] Austen, J. (1998). Pride and Prejudice. Project Gutenberg.
  • [43] Brand, M. (2006). Riders Of The Silences. Project Gutenberg.
  • [44] Locke, J. (2005). Second Treatise of Government. Project Gutenberg.
  • [45] Austen, J. (1994). Sense and Sensibility. Project Gutenberg.
  • [46] Dumas, A. (2001). Ten Years Later. Project Gutenberg.
  • [47] Smollett, T. (2004). The Adventures of Ferdinand Count Fathom. Project Gutenberg.
  • [48] Smollett, T. (2003). The Adventures of Roderick Random. Project Gutenberg.
  • [49] Doyle, A. (1999). The Adventures of Sherlock Holmes. Project Gutenberg.
  • [50] Twain, M. (2004). The Adventures of Tom Sawyer. Project Gutenberg.
  • [51] Dumas, A. (1997). The Black Tulip. Project Gutenberg.
  • [52] Montgomery, L. (2022). The Blue Castle. Project Gutenberg.
  • [53] Couch, A. (2006). The Blue Pavilions. Project Gutenberg.
  • [54] Dostoyevsky, F. (2009). The Brothers Karamazov. Project Gutenberg.
  • [55] London, J. (2008). The Call of the Wild. Project Gutenberg.
  • [56] Augustine, B. (2002). The Confessions of St. Augustine. Project Gutenberg.
  • [57] Dumas, A. (1998). The Count of Monte Cristo. Project Gutenberg.
  • [58] Arnim, E. (2005). The Enchanted April. Project Gutenberg.
  • [59] Leblanc, M. (2004). The Extraordinary Adventures of Arsene Lupin. Project Gutenberg.
  • [60] Dostoyevsky, F. (2000). The Gambler. Project Gutenberg.
  • [61] Fitzgerald, F. (2021). The Great Gatsby. Project Gutenberg.
  • [62] Doyle, A. (2001). The Hound of the Baskervilles. Project Gutenberg.
  • [63] Chambers, R. (2005). The King in Yellow. Project Gutenberg.
  • [64] Defoe, D. (1996). The Life and Adventures of Robinson Crusoe. Project Gutenberg.
  • [65] Dumas, A. (2001). The Man in the Iron Mask. Project Gutenberg.
  • [66] Christie, A. (2022). The Murder of Roger Ackroyd. Project Gutenberg.
  • [67] Christie, A. (2019). The Murder on the Links. Project Gutenberg.
  • [68] Homer, H. (1999). The Odyssey. Project Gutenberg.
  • [69] Wilde, O. (1994). The Picture of Dorian Gray. Project Gutenberg.
  • [70] Machiavelli, N. (2006). The Prince. Project Gutenberg.
  • [71] Twain, M. (2004). The Prince and the Pauper. Project Gutenberg.
  • [72] Russell, B. (2004). The Problems of Philosophy. Project Gutenberg.
  • [73] Gibran, K. (2019). The Prophet. Project Gutenberg.
  • [74] Rizal, J. (2004). The Reign of Greed. Project Gutenberg.
  • [75] Plato, P. (1998). The Republic. Project Gutenberg.
  • [76] Anonymous, A. (2009). The Romance of Lust. Project Gutenberg.
  • [77] Hawthorne, N. (2008). The Scarlet Letter. Project Gutenberg.
  • [78] Doyle, A. (2000). The Sign of the Four. Project Gutenberg.
  • [79] Bois, W. (1996). The Souls of Black Folk. Project Gutenberg.
  • [80] Stevenson, R. (2008). The Strange Case of Dr. Jekyll and Mr. Hyde. Project Gutenberg.
  • [81] Hemingway, E. (2022). The Sun Also Rises. Project Gutenberg.
  • [82] Dumas, A. (1998). The Three Musketeers. Project Gutenberg.
  • [83] Wells, H. (2004). The Time Machine. Project Gutenberg.
  • [84] Kafka, F. (2005). The Trial. Project Gutenberg.
  • [85] James, H. (1995). The Turn of the Screw. Project Gutenberg.
  • [86] Dumas, A. (2001). The Vicomte de Bragelonne. Project Gutenberg.
  • [87] Wells, H. (2004). The War of the Worlds. Project Gutenberg.
  • [88] Baum, L. (1993). The Wonderful Wizard of Oz. Project Gutenberg.
  • [89] Nietzsche, F. (1999). Thus Spake Zarathustra. Project Gutenberg.
  • [90] Stevenson, R. (2006). Treasure Island. Project Gutenberg.
  • [91] Verne, J. (1994). Twenty Thousand Leagues under the Sea. Project Gutenberg.
  • [92] Dumas, A. (1998). Twenty Years After. Project Gutenberg.
  • [93] Joyce, J. (2003). Ulysses. Project Gutenberg.
  • [94] Stowe, H. (2006). Uncle Tom’s Cabin or Life among the Lowly. Project Gutenberg.
  • [95] Thoreau, H. (1995). Walden, and On The Duty Of Civil Disobedience. Project Gutenberg.
  • [96] Tolstoy, L. (2001). War and Peace. Project Gutenberg.
  • [97] Brand, M. (2006). Way Of The Lawless. Project Gutenberg.
  • [98] Dostoyevsky, F. (2011). White Nights and Other Stories. Project Gutenberg.
  • [99] Milne, A. (2022). Winnie the Pooh. Project Gutenberg.
  • [100] Bronte, E. (1996). Wuthering Heights. Project Gutenberg.