avewright/tabula-pretraining-corpus-v2

Tabula Pretraining Corpus v2

A large-scale synthetic tabular dataset for pretraining transformer-based in-context learning models for tabular data (similar to TabPFN).

Overview

Schema

Each shard is a Parquet file with a fixed-width schema:

• —feat_0 through feat_63: Float32 feature columns. Unused slots are NaN.
• —target: Float32 target variable (classification label or regression target).
• —_source_meta: JSON string with dataset metadata including:
• —generator: Which synthetic generator produced this dataset
• —task_type: "binary", "multiclass", or "regression"
• —n_features: Number of active features (rest are NaN-padded)
• —n_classes: Number of target classes
• —n_samples: Number of rows in the original dataset
• —domain: Semantic domain (finance, health, etc.)
• —feature_names: Original domain-specific column names

Generators

Task Types

Domains

Quality Gates

Every generated dataset passes quality gates before inclusion:

• —No constant columns — all features must vary
• —No all-null columns
• —Minority class fraction ≥ 5% for classification
• —Duplicate row fraction ≤ 30%
• —RF utility AUC ≥ 0.55 — a Random Forest must achieve above-chance cross-validated AUC

Gate failure rate: 22.4%

Data Augmentation

• —Missingness injection: ~30% of datasets have random missing values injected
• —Concept drift: ~20% of datasets have feature distribution shifts

Usage

from datasets import load_dataset

ds = load_dataset("avewright/tabula-pretraining-corpus-v2", split="train", streaming=True)

for batch in ds.iter(batch_size=512):
    features = batch["feat_0"]  # access individual features
    target = batch["target"]
    meta = batch["_source_meta"]  # JSON metadata string

License

Apache 2.0