Learning Curves Database 1.1

Sample-wise learning curves plot performance versus training set size. They are useful for studying scaling laws and speeding up hyperparameter tuning and model selection. Learning curves are often assumed to be well-behaved: monotone (i.e. improving with more data) and convex. By constructing the Learning Curves Database 1.1 (LCDB 1.1), a large-scale database with high-resolution learning curves including more modern learners (CatBoost, TabNet, RealMLP and TabPFN), we show that learning curves are less often well-behaved than previously thought. Using statistically rigorous methods, we observe significant ill-behavior in approximately 15% of the learning curves, almost twice as much as in previous estimates. We also identify which learners are to blame and show that specific learners are more ill-behaved than others. Additionally, we demonstrate that different feature scalings rarely resolve ill-behavior. We evaluate the impact of ill-behavior on downstream tasks, such as learning curve fitting and model selection, and find it poses significant challenges, underscoring the relevance and potential of LCDB 1.1 as a challenging benchmark for future research.逐样本学习曲线（Sample-wise learning curves）以训练集规模为横轴、模型性能为纵轴，描绘性能随训练集规模的变化情况。该类曲线可用于研究缩放规律，同时加速超参数调优与模型选择流程。通常认为学习曲线具备良好行为特性：即单调递增（随训练数据增多性能提升）且呈凸性。通过构建学习曲线数据库1.1（Learning Curves Database 1.1，LCDB 1.1）——一款收录高分辨率学习曲线的大规模数据集，涵盖CatBoost、TabNet、RealMLP与TabPFN等多款现代学习器——我们发现，具备良好行为特性的学习曲线占比远低于此前的预估。通过严谨的统计学方法，我们观察到约15%的学习曲线存在显著的不良行为特性，其占比几乎是此前预估的两倍。我们还明确了易引发不良行为特性的学习器类别，并证实特定学习器的不良行为占比高于其他模型。此外，我们证明不同的特征缩放方案几乎无法改善学习曲线的不良行为特性。我们评估了学习曲线不良行为对下游任务（如学习曲线拟合与模型选择）的影响，发现其会带来显著挑战，这也凸显了LCDB 1.1作为面向未来研究的挑战性基准数据集的价值与潜力。

搜集汇总

背景与挑战

背景概述

该数据集是一个大规模高分辨率学习曲线数据库，旨在研究机器学习中的缩放定律并加速超参数调优。研究发现学习曲线的不良行为比例（约15%）高于先前估计，特定学习器更易出现问题，且特征缩放难以解决此问题。数据集可作为未来研究的挑战性基准，支持下游任务如学习曲线拟合和模型选择。