reasoning-degeneration-dev/prepretraining-eval-round5-partial-v1
收藏资源简介：
--- license: mit tags: - prepretraining - curriculum-learning - data-ordering - scaling-laws - eval-results --- # prepretraining-eval-round5-partial-v1 Round 5 (300M, 10B tokens) PARTIAL: 4 conditions, seed=42, ~2-2.5B tokens processed (~20%). Jobs 925410-925413 still running on Empire. Will be replaced by final results. ## Dataset Info - **Rows**: 4 - **Columns**: 13 ## Columns | Column | Type | Description | |--------|------|-------------| | run_name | Value('string') | Unique run identifier: ppt_{scale}_{condition}_d{delta}_s{seed} | | condition | Value('string') | Data scheduling condition: baseline, front-load, constant-mix, or anneal | | scale | Value('string') | Model scale: tiny (60M), small (150M), or medium (300M) | | seed | Value('int64') | Random seed for reproducibility | | is_sft | Value('bool') | Whether this run is SFT (True) or pretraining (False) | | gold_fraction | Value('float64') | Fraction of total tokens that are gold data (delta), typically 0.05 | | final_web_ppl | Value('float64') | Final held-out web perplexity (mean across eval files) | | final_gold_ppl | Value('float64') | Final held-out gold perplexity (mean across 5 gold sources) | | final_train_ppl | Value('float64') | Last logged training perplexity | | total_steps | Value('int64') | Total training steps configured for this run | | total_tokens_latest | Value('int64') | Latest total tokens processed (from throughput counter) | | n_eval_web_points | Value('int64') | Number of web eval checkpoints logged | | n_eval_gold_points | Value('int64') | Number of gold eval checkpoints logged | ## Generation Parameters ```json { "script_name": "analysis/extract_metrics.py + analysis/upload_results.py", "model": "OLMo (60M/150M/300M custom configs)", "description": "Round 5 (300M, 10B tokens) PARTIAL: 4 conditions, seed=42, ~2-2.5B tokens processed (~20%). Jobs 925410-925413 still running on Empire. Will be replaced by final results.", "hyperparameters": { "gold_fraction": 0.05, "conditions": [ "baseline", "front-load", "constant-mix", "anneal" ], "lr_schedule": "cosine_with_warmup", "optimizer": "adamw" }, "input_datasets": [ "reasoning-degeneration-dev/prepretraining-gold-v1", "reasoning-degeneration-dev/prepretraining-web-v1", "reasoning-degeneration-dev/prepretraining-sft-v1" ] } ``` ## Experiment Documentation For complete experiment details, see [https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining](https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining) ## Usage ```python from datasets import load_dataset dataset = load_dataset("reasoning-degeneration-dev/prepretraining-eval-round5-partial-v1", split="train") print(f"Loaded {len(dataset)} rows") ``` --- *This dataset is tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)*
license: MIT许可证 tags: - 预预训练（prepretraining） - 课程学习（curriculum learning） - 数据排序（data-ordering） - 缩放定律（scaling laws） - 评估结果（eval-results） # 预预训练评估轮次5（部分版v1） 第5轮实验（模型规模3亿参数，总令牌数100亿）的部分结果：共4种数据调度条件，随机种子设为42，已处理约20-25亿令牌（完成进度约20%）。编号925410至925413的任务仍在Empire集群上运行，最终结果将替换当前的部分数据。 ## 数据集信息 - **行数**：4 - **列数**：13 ## 列信息 | 列名 | 数据类型 | 描述 | |--------|------|-------------| | run_name | Value('string') | 唯一运行标识符：格式为`ppt_{scale}_{condition}_d{delta}_s{seed}` | | condition | Value('string') | 数据调度条件：可选`baseline`（基准）、`front-load`（前置加载）、`constant-mix`（固定混合）或`anneal`（衰减调度） | | scale | Value('string') | 模型规模：可选`tiny`（6000万参数）、`small`（1.5亿参数）或`medium`（3亿参数） | | seed | Value('int64') | 用于复现实验的随机种子 | | is_sft | Value('bool') | 标识该运行是否为监督微调（SFT，Supervised Fine-Tuning）：`True`为是，`False`为预训练 | | gold_fraction | Value('float64') | 黄金数据占总令牌数的比例（即`delta`），通常为0.05 | | final_web_ppl | Value('float64') | 最终保留的网页域困惑度（所有评估文件的均值） | | final_gold_ppl | Value('float64') | 最终保留的黄金域困惑度（5个黄金数据源的均值） | | final_train_ppl | Value('float64') | 最后一次记录的训练困惑度 | | total_steps | Value('int64') | 该运行配置的总训练步数 | | total_tokens_latest | Value('int64') | 最新统计的已处理总令牌数（来自吞吐量计数器） | | n_eval_web_points | Value('int64') | 已记录的网页域评估检查点数量 | | n_eval_gold_points | Value('int64') | 已记录的黄金域评估检查点数量 | ## 生成参数 json { "script_name": "analysis/extract_metrics.py + analysis/upload_results.py", "model": "OLMo（采用60M/150M/300M参数的自定义配置）", "description": "第5轮实验（模型规模3亿参数，总令牌数100亿）的部分结果：共4种数据调度条件，随机种子设为42，已处理约20-25亿令牌（完成进度约20%）。编号925410至925413的任务仍在Empire集群上运行，最终结果将替换当前的部分数据。", "hyperparameters": { "gold_fraction": 0.05, "conditions": [ "baseline", "front-load", "constant-mix", "anneal" ], "lr_schedule": "带预热的余弦学习率调度", "optimizer": "AdamW优化器" }, "input_datasets": [ "reasoning-degeneration-dev/prepretraining-gold-v1", "reasoning-degeneration-dev/prepretraining-web-v1", "reasoning-degeneration-dev/prepretraining-sft-v1" ] } ## 实验文档 如需完整实验细节，请参阅 [https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining](https://github.com/Zayne-sprague/SC-Research-Notes/tree/main/experiments/prepretraining) ## 使用方法 python from datasets import load_dataset dataset = load_dataset("reasoning-degeneration-dev/prepretraining-eval-round5-partial-v1", split="train") print(f"已加载 {len(dataset)} 行数据") --- *本数据集已在 [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST) 中进行追踪*
