CoolFace
Datasetpublic

jeremycochoy/gift-pretrain-full-4096

gift-pretrain-full-4096 Full counterpart to jeremycochoy/gift-pretrain-small-4096: every series of every arrow file in Salesforce/GiftEvalPretrain, cropped into non-overlapping 4096-point windows and globally shuffled. The small companion sub-samples 10 series per sub-dataset; this one keeps everything. 6,376 source arrow files across 152 sub-datasets fully consumed 42,571,692 windows of length 4096 (float32) 4,274 parquet shards, ~619 GB total (zstd) Layout .… See the full description on the dataset page: https://huggingface.co/datasets/jeremycochoy/gift-pretrain-full-4096.

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes6.2kdownloads
Dataset Card

gift-pretrain-full-4096

Full counterpart to `jeremycochoy/gift-pretrain-small-4096`: every series of every arrow file in `Salesforce/GiftEvalPretrain`, cropped into non-overlapping 4096-point windows and globally shuffled. The small companion sub-samples 10 series per sub-dataset; this one keeps everything.

  • —6,376 source arrow files across 152 sub-datasets fully consumed
  • —42,571,692 windows of length 4096 (float32)
  • —4,274 parquet shards, ~619 GB total (zstd)

Layout

.
├── small_v1/                 ← gift-only training shards
│   ├── shard_NNNNN.parquet
│   └── manifest.json
├── eval/                     ← Salesforce/GiftEval mirror (bytes-as-is)
│   ├── m4_daily/
│   ├── ett1/
│   └── ...
└── README.md

(The directory is named small_v1/ purely to match the small companion's layout for tooling compatibility — it is not a small bundle.)

Schema (small_v1/shard_*.parquet)

ColumnTypeNotes
serieslist<float32>[4096]Fixed-length non-overlapping window
source_iduint8Always 0 (gift) — bundle is single-source
metastringOriginal item_id from the source arrow file

Compression: zstd, row group size 10_000.

Sampling

For every arrow file in Salesforce/GiftEvalPretrain, every non-NaN target row contributes every non-overlapping 4096-point window. Series shorter than 4096 points yield zero windows and are silently skipped. Output is globally shuffled at row granularity via the two-pass bucket shuffle used for all training bundles in this family (stage 3 of scripts/training_data_prep/), so adjacent rows in shard_NNNNN.parquet are independent draws from the entire corpus.

NaN handling: GIFT pretrain contains a small fraction of partial or all-NaN target values; those rows pass through the pipeline unchanged. Consumers should forward-fill partials and skip all-NaN rows.

Reproducing

bash
python -m training_data_prep.build_gift_only_bundle \
    --output-dir /path/to/out \
    --window-length 4096 \
    --all-files \
    --max-workers 48

The --all-files switch was added in jeremycochoy/rnd PR #305.