MachineLearningLM/machinelearninglm-scm-synthetic-tabularml

MachineLearningLM预训练语料库是由数百万个结构因果模型（SCMs）构建而成的，用于合成多样化的机器学习任务。这些合成数据能够训练大型语言模型（LLMs），使其具备强大的基于上下文的机器学习（ML）能力，特别是在表格分类任务上，跨越金融、物理、生物和医疗等多个领域。目的是让LLMs在不进行任何特定任务训练的情况下，达到随机森林级别的准确度，并展示出一个显著的多示例缩放规律，即准确度随着更多上下文示范的增加而单调递增。

The MachineLearningLM pretraining corpus is constructed from millions of structural causal models (SCMs) used to synthesize diverse machine learning tasks. This synthetic data is designed to train large language models (LLMs) to develop strong in-context learning abilities, particularly for tabular classification tasks across various domains such as finance, physics, biology, and healthcare. The goal is for LLMs to achieve random-forest-level accuracy without any task-specific training, demonstrating a striking many-shot scaling law where accuracy increases monotonically with more in-context demonstrations.