five

LumberChunker/GutenQA_Semantic

收藏
Hugging Face2024-09-25 更新2024-06-29 收录
下载链接:
https://hf-mirror.com/datasets/LumberChunker/GutenQA_Semantic
下载链接
链接失效反馈
官方服务:
资源简介:
GutenQA-Semantic数据集基于100本公共领域的叙事书籍,使用了语义分块技术对文本进行分割。该数据集是[GutenQA](https://huggingface.co/datasets/LumberChunker/GutenQA)基准的一部分,并在[LumberChunker](https://github.com/joaodsmarques/LumberChunker)论文中作为基线分块方法之一。数据集的组织结构包括书籍名称、书籍ID、分块ID和分块内容。语义分块利用嵌入技术将语义相似的文本段聚类。

GutenQA-Semantic数据集基于100本公共领域的叙事书籍,使用了语义分块技术对文本进行分割。该数据集是[GutenQA](https://huggingface.co/datasets/LumberChunker/GutenQA)基准的一部分,并在[LumberChunker](https://github.com/joaodsmarques/LumberChunker)论文中作为基线分块方法之一。数据集的组织结构包括书籍名称、书籍ID、分块ID和分块内容。语义分块利用嵌入技术将语义相似的文本段聚类。
提供机构:
LumberChunker
原始信息汇总

GutenQA-Semantic 数据集概述

基本信息

  • 许可证: MIT
  • 任务类别: 问答 (Question Answering)
  • 语言: 英语 (English)

数据集配置

  • 配置名称: semantic
    • 数据文件:
      • 分割: semantic_chunks
      • 路径: GutenQA_semantic.parquet

数据集描述

  • 来源: 基于 GutenQA 数据集,包含100本公共领域叙事书籍。
  • 处理方法: 使用语义分块 (Semantic Chunking) 技术,通过嵌入 (embeddings) 聚类语义相似的文本段落。

数据结构

  • :
    • Book Name: 书籍标题
    • Book ID: 书籍的唯一整数标识符
    • Chunk ID: 书籍块的整数标识符,按其在书中的顺序排列
    • Chunk: 包含从书籍中提取的段落块,这些块是通过语义分块技术生成的

数据加载示例

python import pandas as pd dataset = pd.read_parquet("hf://datasets/LumberChunker/GutenQA_Paragraphs/GutenQA_semantic.parquet", engine="pyarrow")

过滤数据框以显示指定书名的行

single_book_chunks = dataset[dataset[Book Name] == A_Christmas_Carol_-_Charles_Dickens].reset_index(drop=True)

引用

bibtex @misc{duarte2024lumberchunker, title={LumberChunker: Long-Form Narrative Document Segmentation}, author={André V. Duarte and João Marques and Miguel Graça and Miguel Freire and Lei Li and Arlindo L. Oliveira}, year={2024}, eprint={2406.17526}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2406.17526}, }

书籍参考

  • 书籍列表:
    • [1] Twain, M. (2004). Adventures of Huckleberry Finn. Project Gutenberg.
    • [2] Carroll, L. (2008). Alices Adventures in Wonderland. Project Gutenberg.
    • [3] Tolstoy, L. (1998). Anna Karenina. Project Gutenberg.
    • [4] Montgomery, L. (2008). Anne of Green Gables. Project Gutenberg.
    • [5] Verne, J. (1994). Around the World in Eighty Days. Project Gutenberg.
    • [6] Dickens, C. (2004). A Christmas Carol. Project Gutenberg.
    • [7] Twain, M. (2004). A Connecticut Yankee in King Arthurs Court. Project Gutenberg.
    • [8] Hudson, W. (2005). A Crystal Age. Project Gutenberg.
    • [9] Scott, S. (2006). A Legend Of Montrose. Project Gutenberg.
    • [10] Joyce, J. (2003). A Portrait of the Artist as a Young Man. Project Gutenberg.
    • [11] Forster, E. (2001). A Room with a View. Project Gutenberg.
    • [12] Doyle, A. (1995). A Study in Scarlet. Project Gutenberg.
    • [13] Dickens, C. (1994). A Tale of Two Cities. Project Gutenberg.
    • [14] Nietzsche, F. (2003). Beyond Good and Evil. Project Gutenberg.
    • [15] Le Fanu, J. (2003). Carmilla. Project Gutenberg.
    • [16] Gaskell, E. (1996). Cranford. Project Gutenberg.
    • [17] Dostoyevsky, F. (2006). Crime and Punishment. Project Gutenberg.
    • [18] Garis, H. (2005). Daddy takes us to the Garden. Project Gutenberg.
    • [19] Dickens, C. (1996). David Copperfield. Project Gutenberg.
    • [20] Stoker, B. (1995). Dracula. Project Gutenberg.
    • [21] Joyce, J. (2001). Dubliners. Project Gutenberg.
    • [22] Austen, J. (1994). Emma. Project Gutenberg.
    • [23] Shelley, M. (1993). Frankenstein Or The Modern Prometheus. Project Gutenberg.
    • [24] Dickens, C. (1998). Great Expectations. Project Gutenberg.
    • [25] Brothers, G. (2001). Grimms Fairy Tales. Project Gutenberg.
    • [26] Swift, J. (1997). Gullivers Travels into Several Remote Nations of the World. Project Gutenberg.
    • [27] Conrad, J. (2006). Heart of Darkness. Project Gutenberg.
    • [28] Jacobs, H. (2004). Incidents in the Life of a Slave Girl. Project Gutenberg.
    • [29] Bronte, C. (1998). Jane Eyre. Project Gutenberg.
    • [30] Hugo, V. (2008). Les Miserables. Project Gutenberg.
    • [31] Alcott, L. (1996). Little Women. Project Gutenberg.
    • [32] Dumas, A. (2001). Louise de la Valliere. Project Gutenberg.
    • [33] Flaubert, G. (2006). Madame Bovary. Project Gutenberg.
    • [34] Aurelius, E. (2001). Meditations. Project Gutenberg.
    • [35] Eliot, G. (1994). Middlemarch. Project Gutenberg.
    • [36] Melville, H. (2001). Moby Dick Or The Whale. Project Gutenberg.
    • [37] Wagner, R. (2004). My Life. Project Gutenberg.
    • [38] Douglass, F. (2006). Narrative of the Life of Frederick Douglass, an American Slave. Project Gutenberg.
    • [39] Dickens, C. (1996). Oliver Twist. Project Gutenberg.
    • [40] Austen, J. (1994). Persuasion. Project Gutenberg.
    • [41] Barrie, J. (2008). Peter Pan. Project Gutenberg.
    • [42] Austen, J. (1998). Pride and Prejudice. Project Gutenberg.
    • [43] Brand, M. (2006). Riders Of The Silences. Project Gutenberg.
    • [44] Locke, J. (2005). Second Treatise of Government. Project Gutenberg.
    • [45] Austen, J. (1994). Sense and Sensibility. Project Gutenberg.
    • [46] Dumas, A. (2001). Ten Years Later. Project Gutenberg.
    • [47] Smollett, T. (2004). The Adventures of Ferdinand Count Fathom. Project Gutenberg.
    • [48] Smollett, T. (2003). The Adventures of Roderick Random. Project Gutenberg.
    • [49] Doyle, A. (1999). The Adventures of Sherlock Holmes. Project Gutenberg.
    • [50] Twain, M. (2004). The Adventures of Tom Sawyer. Project Gutenberg.
    • [51] Dumas, A. (1997). The Black Tulip. Project Gutenberg.
    • [52] Montgomery, L. (2022). The Blue Castle. Project Gutenberg.
    • [53] Couch, A. (2006). The Blue Pavilions. Project Gutenberg.
    • [54] Dostoyevsky, F. (2009). The Brothers Karamazov. Project Gutenberg.
    • [55] London, J. (2008). The Call of the Wild. Project Gutenberg.
    • [56] Augustine, B. (2002). The Confessions of St. Augustine. Project Gutenberg.
    • [57] Dumas, A. (1998). The Count of Monte Cristo. Project Gutenberg.
    • [58] Arnim, E. (2005). The Enchanted April. Project Gutenberg.
    • [59] Leblanc, M. (2004). The Extraordinary Adventures of Arsene Lupin. Project Gutenberg.
    • [60] Dostoyevsky, F. (2000). The Gambler. Project Gutenberg.
    • [61] Fitzgerald, F. (2021). The Great Gatsby. Project Gutenberg.
    • [62] Doyle, A. (2001). The Hound of the Baskervilles. Project Gutenberg.
    • [63] Chambers, R. (2005). The King in Yellow. Project Gutenberg.
    • [64] Defoe, D. (1996). The Life and Adventures of Robinson Crusoe. Project Gutenberg.
    • [65] Dumas, A. (2001). The Man in the Iron Mask. Project Gutenberg.
    • [66] Christie, A. (2022). The Murder of Roger Ackroyd. Project Gutenberg.
    • [67] Christie, A. (2019). The Murder on the Links. Project Gutenberg.
    • [68] Homer, H. (1999). The Odyssey. Project Gutenberg.
    • [69] Wilde, O. (1994). The Picture of Dorian Gray. Project Gutenberg.
    • [70] Machiavelli, N. (2006). The Prince. Project Gutenberg.
    • [71] Twain, M. (2004). The Prince and the Pauper. Project Gutenberg.
    • [72] Russell, B. (2004). The Problems of Philosophy. Project Gutenberg.
    • [73] Gibran, K. (2019). The Prophet. Project Gutenberg.
    • [74] Rizal, J. (2004). The Reign of Greed. Project Gutenberg.
    • [75] Plato, P. (1998). The Republic. Project Gutenberg.
    • [76] Anonymous, A. (2009). The Romance of Lust. Project Gutenberg.
    • [77] Hawthorne, N. (2008). The Scarlet Letter. Project Gutenberg.
    • [78] Doyle, A. (2000). The Sign of the Four. Project Gutenberg.
    • [79] Bois, W. (1996). The Souls of Black Folk. Project Gutenberg.
    • [80] Stevenson, R. (2008). The Strange Case of Dr. Jekyll and Mr. Hyde. Project Gutenberg.
    • [81] Hemingway, E. (2022). The Sun Also Rises. Project Gutenberg.
    • [82] Dumas, A. (1998). The Three Musketeers. Project Gutenberg.
    • [83] Wells, H. (2004). The Time Machine. Project Gutenberg.
    • [84] Kafka, F. (2005). The Trial. Project Gutenberg.
    • [85] James, H. (1995). The Turn of the Screw. Project Gutenberg.
    • [86] Dumas, A. (2001). The Vicomte de Bragelonne. Project Gutenberg.
    • [87] Wells, H. (2004). The War of the Worlds. Project Gutenberg.
    • [88] Baum, L. (1993). The Wonderful Wizard of Oz. Project Gutenberg.
    • [89] Nietzsche, F. (1999). Thus Spake Zarathustra. Project Gutenberg.
    • [90] Stevenson, R. (2006). Treasure Island. Project Gutenberg.
    • [91] Verne, J. (1994). Twenty Thousand Leagues under the Sea. Project Gutenberg.
    • [92] Dumas, A. (1998). Twenty Years After. Project Gutenberg.
    • [93] Joyce, J. (2003). Ulysses. Project Gutenberg.
    • [94] Stowe, H. (2006). Uncle Tom’s Cabin or Life among the Lowly. Project Gutenberg.
    • [95] Thoreau, H. (1995). Walden, and On The Duty Of Civil Disobedience. Project Gutenberg.
    • [96] Tolstoy, L. (2001). War and Peace. Project Gutenberg.
    • [97] Brand, M. (2006). Way Of The Lawless. Project Gutenberg.
    • [98] Dostoyevsky, F. (2011). White Nights and Other Stories. Project Gutenberg.
    • [99] Milne, A. (2022). Winnie the Pooh. Project Gutenberg.
    • [100] Bronte, E. (1996). Wuthering Heights. Project Gutenberg.
5,000+
优质数据集
54 个
任务类型
进入经典数据集
二维码
社区交流群

面向社区/商业的数据集话题

二维码
科研交流群

面向高校/科研机构的开源数据集话题

数据驱动未来

携手共赢发展

商业合作