keshan/lk_hansard_trilingual

该数据集包含从斯里兰卡议会官方报告（Hansards）中提取的文本。数据经过PDF源处理、OCR识别，并根据脚本检测分为英语、僧伽罗语和泰米尔语子集。数据集结构包括英语、僧伽罗语和泰米尔语三个部分。提取过程包括从parliament.lk抓取PDF链接、下载并存储在GCS中、使用Tesseract进行OCR识别（支持僧伽罗语、泰米尔语和英语），以及基于Unicode范围和语言检测进行语言分割。

This dataset contains text extracted from the Official Reports (Hansards) of the Parliament of Sri Lanka. The data has been processed from PDF sources, OCRed, and split into English, Sinhala, and Tamil subsets based on script detection. The dataset structure includes English, Sinhala, and Tamil sections. The extraction process involves scraping PDF links from parliament.lk, downloading and storing in GCS, performing OCR using Tesseract (supporting Sinhala, Tamil, and English), and language splitting based on Unicode ranges and langdetect.