lectura/naver_news_1024

---
dataset_info:
features:
- name: input_ids
sequence: int32
splits:
- name: train
num_bytes: 53074500
num_examples: 12945
download_size: 23902205
dataset_size: 53074500
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
task_categories:
- text-generation
language:
- ko
size_categories:
- 10K<n<100K
source_datasets:
- daekeun-ml/naver-news-summarization-ko
---
# Dataset Card for "naver_news_1024"

Preprocessed and tokenized [daekeun-ml/naver-news-summarization-ko](https://huggingface.co/datasets/daekeun-ml/naver-news-summarization-ko) dataset for pretraining. \
Concatenated all text and split into chunk size of **1024**. \
Used tokenizer from [beomi/llama-2-ko-7b](https://huggingface.co/beomi/llama-2-ko-7b)

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

原始信息汇总

数据集概述

数据集信息

• 特征:
  • input_ids: 序列类型，数据类型为 int32
• 拆分:
  • train: 字节数为 53074500，样本数为 12945
• 下载大小: 23902205 字节
• 数据集大小: 53074500 字节

配置

• 配置名称: default
• 数据文件:
  • train: 路径为 data/train-*

任务类别

• text-generation

语言

• ko

大小类别

• 10K<n<100K

源数据集

• daekeun-ml/naver-news-summarization-ko