Learning Curves Database 1.1

Sample-wise learning curves plot performance versus training set size. They are useful for studying scaling laws and speeding up hyperparameter tuning and model selection. We revisit the Learning Curves Database 1.0 (LCDB 1.0), and create a higher quality version, the LCDB 1.1. The LCDB 1.1 has more carefully curated datasets, higher resolution and multiple feature scalings. Learning curves are often assumed to be well-behaved: monotone (i.e. improving with more data) and convex. Ill-behavior, such as non-monotonicity and non-convexity, has been observed in toy settings. We develop robust methods to find significant ill-behavior, and find in our large-scale database that approximately 13\% of learning curves are ill-behaved, with specific learners contributing more than others.For half of these cases, ill-behavedness could not be identified in LCDB 1.0 due to a lack in resolution, potentially leading to wrong conclusions.Different feature scalings do not resolve ill-behaviors and thus our findings are robust. Lastly, we evaluate the impact of ill-behavior on downstream tasks, such as learning curve extrapolation and model selection, and find it poses significant challenges. Concluding, ill-behavior of learning curves is more widespread than previously thought, and poses interesting challenges which can be efficiently investigated using the LCDB 1.1.