ILSUM/ILSUM-2.0

--- license: apache-2.0 task_categories: - summarization - text-generation language: - hi - gu - bn - en configs: - config_name: Hindi data_files: - split: train path: Hindi/train.csv - split: test path: Hindi/test.csv default: true - config_name: Gujarati data_files: - split: train path: Gujarati/train.csv - split: test path: Gujarati/test.csv - config_name: English data_files: - split: train path: English/train.csv - split: test path: English/test.csv - config_name: Bengali data_files: - split: train path: Bengali/train.csv - split: test path: Bengali/test.csv size_categories: - 10K<n<100K --- # Dataset Card for "ILSUM-2.0" ### Dataset Summary ILSUM-2.0 contains additional ~10K articles along with ILSUM-1.0 dataset. Along with Hindi, English, and Gujarati, which were part of ILSUM-1.0, Bengali is also introduced as part of ILSUM-20. dataset. The dataset for this task is built using articles and headline pairs from several leading newspapers of the country. We provide >=10,000 news articles for each language. The task is to generate a meaningful fixed length summary, either extractive or abstractive, for each article. While several previous works in other languages use news artciles - headlines pair, the current dataset poses a unique challenge of code-mixing and script mixing. It is very common for news articles to borrow phrases from english, even if the article itself is written in an Indian Language. Examples like these are a common occurence both in the headlines as well as in the articles. ~~~ - "IND vs SA, 5મી T20 તસવીરોમાં: વરસાદે વિલન બની મજા બગાડી" (India vs SA, 5th T20 in pictures: rain spoils the match) - "LIC के IPO में पैसा लगाने वालों का टूटा दिल, आई एक और नुकसानदेह खबर" (Investors of LIC IPO left broken hearted, yet another bad news). ~~~ ### Languages - Hindi - Gujarati - Bengali - English ### Data Fields ~~~ - id: Unique id of each datapoint - Article: Entire News article - Headline: Headline of News Article - Summary: Summary of News Article ~~~ ### Data Splits Data for all four languages is divided into two splits train and test. ### Load dataset using hf-dataset class ```python from datasets import load_dataset dataset = load_dataset("ILSUM/ILSUM-2.0", "Hindi") # you can use any of the following config names as a second argument: # "English", "Hindi", "Gujarati", Bengali ``` ### Citation Information If you are using the dataset or the models please cite the following paper ~~~ @article{satapara2023findings, title={Key Takeaways from the Second Shared Task on Indian Language Summarization (ILSUM 2023).}, author={Satapara, Shrey and Mehta, Parth and Modha, Sandip and Ganguly, Debasis}, journal={Working Notes of FIRE}, pages={724-733}, year={2023} } ~~~ ### Contributions - Shrey Satapara, Indian Institute Of Technology, Hyderabad, India - Sandip Modha, LDRP-ITR, Gandhinagar, India - Parth Mehta, Parmonic, USA - Debasis Ganguly, University Of Glasgow, UK <!--## Dataset Description - **Homepage:** - **Repository:** - **Paper:** - **Leaderboard:** - **Point of Contact:** ### Supported Tasks and Leaderboards [More Information Needed] ## Dataset Structure ### Data Instances [More Information Needed] [More Information Needed] ### Source Data #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information [More Information Needed]license: Apache-2.0 许可证 task_categories: - 摘要生成（summarization） - 文本生成（text-generation） language: - 印地语（hi） - 古吉拉特语（gu） - 孟加拉语（bn） - 英语（en） configs: - config_name: 印地语 data_files: - split: 训练集（train） path: Hindi/train.csv - split: 测试集（test） path: Hindi/test.csv default: true - config_name: 古吉拉特语 data_files: - split: 训练集（train） path: Gujarati/train.csv - split: 测试集（test） path: Gujarati/test.csv - config_name: 英语 data_files: - split: 训练集（train） path: English/train.csv - split: 测试集（test） path: English/test.csv - config_name: 孟加拉语 data_files: - split: 训练集（train） path: Bengali/train.csv - split: 测试集（test） path: Bengali/test.csv size_categories: - 10K<n<100K --- # ILSUM-2.0 数据集卡片 ## 数据集概述 ILSUM-2.0 是在ILSUM-1.0数据集基础上新增约1万篇文章构建而成。除ILSUM-1.0已涵盖的印地语、英语与古吉拉特语外，本数据集还新增了孟加拉语。 本任务的数据集基于印度多家主流报纸的文章与标题对构建，我们为每种语言提供了不少于1万条新闻文章。任务目标为为每篇新闻生成符合固定长度要求的有意义摘要，支持抽取式或抽象式生成方式。过往针对其他语言的相关研究多采用新闻文章-标题对作为数据集，但本数据集存在独特挑战：语码混杂与脚本混杂——即便是使用印度本土语言撰写的新闻文章，也常会借用英语短语，此类情况在标题与正文中均十分常见。 以下为相关示例： ~~~ - "IND vs SA, 5મી T20 તસવીરોમાં: વરસાદે વિલન બની મજા બગાડી"（印度对阵南非，第5场T20赛事图集：降雨搅乱比赛） - "LIC के IPO में पैसा लगाने वालों का टूटा दिल, आई एक और नुकसानदेह खबर"（印度人寿保险公司首次公开募股投资者心碎，又一则利空消息） ~~~ ## 支持语言 - 印地语 - 古吉拉特语 - 孟加拉语 - 英语 ## 数据字段 ~~~ - id: 每个数据点的唯一标识符 - Article: 完整新闻正文 - Headline: 新闻标题 - Summary: 新闻摘要 ~~~ ## 数据划分 四种语言的数据集均划分为训练集与测试集两个子集。 ## 使用 Hugging Face Datasets 类加载数据集 python from datasets import load_dataset dataset = load_dataset("ILSUM/ILSUM-2.0", "Hindi") # 可将以下任一配置名称作为第二个参数传入： # "English"（英语）、"Hindi"（印地语）、"Gujarati"（古吉拉特语）、"Bengali"（孟加拉语） ## 引用信息 若使用本数据集或相关模型，请引用下述论文： ~~~ @article{satapara2023findings, title={Key Takeaways from the Second Shared Task on Indian Language Summarization (ILSUM 2023).}, author={Satapara, Shrey and Mehta, Parth and Modha, Sandip and Ganguly, Debasis}, journal={Working Notes of FIRE}, pages={724-733}, year={2023} } ~~~ ## 贡献者 - 什雷·萨塔帕拉（Shrey Satapara），印度海得拉巴印度理工学院 - 桑迪普·莫达（Sandip Modha），印度甘地讷格尔LDRP理工学院 - 帕尔特·梅塔（Parth Mehta），美国Parmonic公司 - 迪巴西斯·甘古利（Debasis Ganguly），英国格拉斯哥大学