travisp83/agentic-llm-pretraining-1.7b

Agentic LLM预训练数据集是一个用于小型语言模型（1-3B参数）的预训练语料库，专为代理任务优化。该语料库强调学习理解语言、推理、遵循指令和使用工具，而非记忆事实知识——其假设是领域知识将在运行时通过检索增强生成（RAG）提供。这一理念旨在通过省略通常需要大量文本来记忆事实的部分，从而可能实现更小的预训练语料库规模。A pretraining corpus for small language models (1-3B parameters) optimized for agentic tasks. The corpus emphasizes learning to comprehend language, reason, follow instructions, and use tools over memorizing factual knowledge — the assumption is that domain knowledge will be provided at runtime via RAG. The idea is that this could enable much smaller pretraining corpora by omitting the large volumes of text typically needed to memorize facts.