AlexHung29629/oscar-en-10k-megds

--- task_categories: - text-generation language: - en size_categories: - 1K<n<10K --- # README This dataset is used to test the training of the LLAMA model with Megatron-Deepspeed. ## Source stas/oscar-en-10k ## Command ```bash python tools/preprocess_data.py \ --input oscar-en-10k.jsonl \ --tokenizer-type PretrainedFromHF \ --append-eod \ --tokenizer-name-or-path hf-internal-testing/llama-tokenizer \ --output-prefix oscar-en-10k ``` ## Info \#tokens = 34030076

原始信息汇总

数据集概述

基本信息

• 任务类别: 文本生成
• 语言: 英语
• 数据集大小: 1K<n<10K

用途

用于测试LLAMA模型与Megatron-Deepspeed的训练。

数据源

• 来源: stas/oscar-en-10k

处理命令

bash python tools/preprocess_data.py --input oscar-en-10k.jsonl --tokenizer-type PretrainedFromHF --append-eod --tokenizer-name-or-path hf-internal-testing/llama-tokenizer --output-prefix oscar-en-10k

数据集统计

• 总令牌数: 34030076