avewright/tabula-pretraining-corpus-v2

---
language:
- en
license: apache-2.0
task_categories:
- tabular-classification
- tabular-regression
tags:
- tabular
- synthetic
- pretraining
- in-context-learning
size_categories:
- 100M<n<1B
---

# Tabula Pretraining Corpus v2

A large-scale synthetic tabular dataset for pretraining transformer-based in-context learning models for tabular data (similar to TabPFN).

## Overview

| Metric | Value |
|--------|-------|
| Total rows | 272,271,776 |
| Total datasets | 10,867 |
| Shards | 135 |
| Mean utility AUC | 0.851 |
| Format | Parquet (float32) |

## Schema

Each shard is a Parquet file with a fixed-width schema:

- **feat_0** through **feat_63**: Float32 feature columns. Unused slots are NaN.
- **target**: Float32 target variable (classification label or regression target).
- **_source_meta**: JSON string with dataset metadata including:
- `generator`: Which synthetic generator produced this dataset
- `task_type`: "binary", "multiclass", or "regression"
- `n_features`: Number of active features (rest are NaN-padded)
- `n_classes`: Number of target classes
- `n_samples`: Number of rows in the original dataset
- `domain`: Semantic domain (finance, health, etc.)
- `feature_names`: Original domain-specific column names

## Generators

| Generator | Datasets |
|-----------|----------|
| GaussianMixture | 3,029 |
| Polynomial | 2,738 |
| SCM | 2,674 |
| TreePrior | 2,096 |
| Regression | 325 |
| MixedType_GaussianMixture | 2 |
| MixedType_SCM | 2 |
| MixedType_TreePrior | 1 |

## Task Types

| Type | Datasets |
|------|----------|
| binary | 8,396 |
| multiclass | 2,146 |
| regression | 325 |

## Domains

| Domain | Datasets |
|--------|----------|
| hr | 1,033 |
| education | 1,031 |
| telecom | 1,028 |
| science | 1,020 |
| iot | 1,005 |
| finance | 1,000 |
| health | 985 |
| ecommerce | 977 |
| logistics | 972 |
| environment | 935 |
| manufacturing | 881 |

## Quality Gates

Every generated dataset passes quality gates before inclusion:
- **No constant columns** — all features must vary
- **No all-null columns**
- **Minority class fraction ≥ 5%** for classification
- **Duplicate row fraction ≤ 30%**
- **RF utility AUC ≥ 0.55** — a Random Forest must achieve above-chance cross-validated AUC

Gate failure rate: 22.4%

## Data Augmentation

- **Missingness injection**: ~30% of datasets have random missing values injected
- **Concept drift**: ~20% of datasets have feature distribution shifts

## Usage

```python
from datasets import load_dataset

ds = load_dataset("avewright/tabula-pretraining-corpus-v2", split="train", streaming=True)

for batch in ds.iter(batch_size=512):
features = batch["feat_0"] # access individual features
target = batch["target"]
meta = batch["_source_meta"] # JSON metadata string
```

## License

Apache 2.0