chaannwooff/korean-web

Korean-web (Naver Search) 是一个韩语网络文本语料库数据集，通过Naver搜索API收集而成。该数据集使用Naver搜索Open API（包括webkr和news端点），基于3,446个韩语关键词（涵盖经济、技术、社会、文化和科学等领域）进行数据收集，并通过trafilatura库从HTML中提取文本内容。数据集经过行级和文档级过滤处理：行级过滤移除了UI/导航残留（如登录、更多、分享按钮等）、日期/元标签、记者署名、广告/赌博/成人关键词、非韩语行（如英文、中文、日文等）、标签、URL和版权声明，共移除31,267,606行；文档级过滤从1,779,923个输入文档中保留了1,545,735个文档（通过率86.8%）。数据集规模包括16个文件，总计1,545,735个文档，总容量约为7.2 GB。数据列包括：source（固定为naver_search）、endpoint（webkr或news）、keyword（使用的关键词）、url（原始文档URL）和text（过滤后的文本内容）。数据集主要用于研究目的，基于Naver搜索API索引的公开网页收集，原始内容版权归各网站和版权所有者所有，商业使用需获得原始版权所有者许可。Korean-web (Naver Search) is a Korean web text corpus dataset collected through the Naver Search API. The dataset utilizes the Naver Search Open API (including webkr and news endpoints), based on 3,446 Korean keywords (covering areas such as economy, technology, society, culture, and science), and extracts text content from HTML using the trafilatura library. The dataset undergoes line-level and document-level filtering: line-level filtering removes UI/navigation remnants (e.g., login, more, share buttons), date/meta tags, reporter bylines, advertising/gambling/adult keywords, non-Korean lines (e.g., English, Chinese, Japanese), hashtags, URLs, and copyright statements, totaling 31,267,606 lines removed; document-level filtering retains 1,545,735 documents out of 1,779,923 input documents (pass rate 86.8%). The dataset scale includes 16 files, totaling 1,545,735 documents, with an overall capacity of approximately 7.2 GB. Data columns include: source (fixed as naver_search), endpoint (webkr or news), keyword (the keyword used), url (original document URL), and text (filtered text content). The dataset is primarily intended for research purposes, collected from publicly indexed web pages via the Naver Search API, with original content copyright belonging to respective websites and copyright holders; commercial use requires permission from the original copyright holders.