Indian Language Part-of-Speech Tagset: Bengali

<h3>Introduction</h3><br>
<p>Indian Language Part-of-Speech Tagset: Bengali, Linguistic Data Consortium (LDC) catalog number LDC2010T16 and isbn 1-58563-561-8, is a corpus developed by Microsoft Research (MSR) India to support the task of Part-of-Speech Tagging (POS) and other data-driven linguistic research on Indian Languages in general. It is created as a part of the <a href="http://research.microsoft.com/en-us/groups/mls/default.aspx" rel="nofollow">Indian Language Part-of-Speech Tagset (IL-POST)</a> project, a collaborative effort among linguists and computer scientists from MSR India, AU-KBC (Anna Universtiy, Chennai), Delhi University, IIT Bombay, Jawaharlal Nehru University (Delhi) and Tamil University (Tamilnadu).</p><br>
<p>The goal of the IL-POST project is to provide a common tagset framework for Indian Languages that offers flexibility, cross-linguistic compatibility and resuability across those languages. It supports a three-level hierarchy of Categories, Types and Attributes. The corpus mainly consists therefore of two different levels of information for each lexical token: (a) lexical Category and Types, and (b) set morphological attributes and their associated values in the context.</p><br>
<p>Bengali (also referred to as Bangla) is a member of the Eastern Indo-Aryan language group. It is native to the region of Bengal which consists of Bangladesh, the Indian state of West Bengal, and parts of the Indian states of Tripura and Assam. It is spoken by more than 210 million people as a first or a second language with around 100 million speakers in Bangladesh, about 85 million speakers in India, and others in immigrant communities in the United Kingdom, USA and the Middle East.</p><br>
<h3>Data</h3><br>
<p>This corpus contains 7168 sentences (102933 words) of manually annotated text from modern standard Bengali sources including blogs, <a href="http://en.wikipedia.org" rel="nofollow">Wikipedia</a>, <a href="http://www.multikulti.org.uk" rel="nofollow">Multikulti</a> and a portion of the <a href="http://catalog.elra.info/product_info.php?products_id=696" rel="nofollow">EMILLE/CIIL</a> corpus . The annotated data is structured into two folders, Bangla1 (3684 sentences, 51091 words) and Bangla2 (3484 sentences, 51842 words), which represent the two stages in which the data was annotated. All annotated data is provided in both xml and text files. Each data file contains between 3,000-5,000 words. The XML file contains metadata about the material, such as language, encoding and data size.</p><br>
<h3>Annotation Procedure</h3><br>
<p>The Annotation Guidelines for Bangla included in this release contain a detailed description of the annotation methodology.</p><br>
<h3>Updates</h3><br>
<p>Additional information, updates, bug fixes may be available in the LDC catalog entry for this corpus at <a href="http://catalog.ldc.upenn.edu/LDC2010T16" rel="nofollow">LDC2010T16</a>.</p><br>
<h3>Samples</h3><br>
<p>Please examine this <a href="desc/addenda/LDC2010T16.xml" rel="nofollow">XML excerpt</a> for an example of the data in this corpus.</p></br>
Portions © 2010 Microsoft Research Labs India, Pvt. Ltd., © 2010 Trustees of the University of Pennsylvania

<h3>引言</h3>
<p>印度语言词性标注集：孟加拉语版本，由语言数据联盟（Linguistic Data Consortium, LDC）收录，编号为LDC2010T16，ISBN为1-58563-561-8。该语料库由微软研究院印度分院（Microsoft Research India, MSR）开发，旨在支撑词性标注（Part-of-Speech Tagging, POS）任务以及通用印度语言的其他数据驱动型语言学研究。本语料库是<a href="http://research.microsoft.com/en-us/groups/mls/default.aspx" rel="nofollow">印度语言词性标注集（Indian Language Part-of-Speech Tagset, IL-POST）</a>项目的组成部分，该项目由微软研究院印度分院、金奈安娜大学（AU-KBC）、德里大学、孟买印度理工学院、德里贾瓦哈拉尔·尼赫鲁大学以及泰米尔纳德邦泰米尔大学的语言学家与计算机科学家联合开展。</p>
<p>IL-POST项目的目标是为印度语言提供一套通用的标注集框架，具备灵活性、跨语言兼容性以及多语言复用性。该框架采用类别、类型、属性三级层级结构。因此，本语料库为每个词汇Token提供两类信息：(a) 词汇类别与类型；(b) 上下文语境下的形态属性及其对应取值。</p>
<p>孟加拉语（又称班加拉语）属于东印度-雅利安语支。其使用区域涵盖孟加拉国、印度西孟加拉邦以及特里普拉邦和阿萨姆邦的部分地区。全球以其为第一或第二语言的使用者超过2.1亿，其中孟加拉国约1亿，印度约8500万，其余分布在英国、美国及中东地区的移民社群中。</p>
<h3>数据</h3>
<p>本语料库包含7168句（共计102933词）现代标准孟加拉语人工标注文本，数据来源包括博客、<a href="http://en.wikipedia.org" rel="nofollow">维基百科（Wikipedia）</a>、<a href="http://www.multikulti.org.uk" rel="nofollow">Multikulti</a>以及部分<a href="http://catalog.elra.info/product_info.php?products_id=696" rel="nofollow">EMILLE/CIIL</a>语料库。标注数据分为两个文件夹：Bangla1（含3684句，51091词）与Bangla2（含3484句，51842词），分别对应数据标注的两个阶段。所有标注数据均提供XML与文本两种格式。单个数据文件的词量介于3000至5000之间。XML文件包含数据集的元数据，如语言类型、编码格式及数据规模等。</p>
<h3>标注流程</h3>
<p>本发布包中附带的孟加拉语标注指南，对标注方法进行了详细说明。</p>
<h3>更新说明</h3>
<p>更多相关信息、更新内容及漏洞修复可前往该语料库在语言数据联盟的收录页面获取：<a href="http://catalog.ldc.upenn.edu/LDC2010T16" rel="nofollow">LDC2010T16</a>。</p>
<h3>样例</h3>
<p>您可通过此<a href="desc/addenda/LDC2010T16.xml" rel="nofollow">XML示例片段</a>查看本语料库的数据样例。</p>
部分内容 © 2010 微软研究院印度私人有限公司，© 2010 宾夕法尼亚大学董事会

搜集汇总

背景与挑战

背景概述

该数据集是印度语言词性标注集（IL-POST）项目的一部分，专门针对孟加拉语开发，包含7168个句子（约10.3万单词）的手动注释文本，来自博客、维基百科等现代标准孟加拉语来源。它旨在支持词性标注和其他语言学研究，提供XML和文本格式的数据，并采用三层层次结构（类别、类型和属性）以实现跨印度语言的兼容性和可重用性。