zachnorton03/bart-midtrain
BART Midtrain The midtraining corpus for BART — pre-1930 mathematics, science, technology, and medicine — plus the full pipeline that built it and every training mixture it was blended into. Built by Unbounded Labs. Corpus documents 11,409 Corpus characters 2,543,809,124 Corpus tokens ~604M Removed by cleaning 24% of documents (15,075 → 11,409) Subject focus math, science, technology, medicine Cutoff 1930 Schema single string column text… See the full description on the dataset page: https://huggingface.co/datasets/zachnorton03/bart-midtrain.
Add MIT license to dataset card metadata
Rewrite dataset card for restructured bart-midtrain
Rewrite mixture cards: trim per-shard dumps, mark superseded/current, fix source paths
Restructure: remove old stage*/ and mixed/ paths (contents moved to corpus/, pipeline/, mixtures/)
Restructure: copy batch 6 (293 files)
Restructure: copy batch 5 (400 files)
Restructure: copy batch 4 (400 files)
Restructure: copy batch 3 (400 files)
Restructure: copy batch 2 (400 files)
Restructure: copy batch 1 (400 files)
v2/ratio_45 manifest + README (45.056% midtrain over the consumed prefix)
v2/ratio_45: shard_00020.parquet..shard_00033.parquet (14 shards)
v2/ratio_45: shard_00000.parquet..shard_00019.parquet (20 shards)
Wipe v2/ratio_45 (68 files)
v2/ratio_45: shard_00020.parquet..shard_00033.parquet (14 shards)
v2/ratio_45: shard_00000.parquet..shard_00019.parquet (20 shards)
v2/ratio_21 manifest + README (20.953% midtrain over the consumed prefix)
v2/ratio_21: shard_00060.parquet..shard_00065.parquet (6 shards)
v2/ratio_21: shard_00040.parquet..shard_00059.parquet (20 shards)
v2/ratio_21: shard_00020.parquet..shard_00039.parquet (20 shards)
v2/ratio_21: shard_00000.parquet..shard_00019.parquet (20 shards)
Mixed ratio_60 manifest + README
Mixed ratio_60: shard_00508.parquet..shard_00523.parquet (25 shards)
Mixed ratio_60: shard_00469.parquet..shard_00498.parquet (25 shards)
Mixed ratio_60: shard_00441.parquet..shard_00468.parquet (25 shards)
Mixed ratio_60: shard_00421.parquet..shard_00449.parquet (25 shards)
Mixed ratio_60: shard_00402.parquet..shard_00432.parquet (25 shards)
Mixed ratio_60: shard_00368.parquet..shard_00399.parquet (25 shards)
Mixed ratio_60: shard_00345.parquet..shard_00365.parquet (25 shards)
Mixed ratio_60: shard_00329.parquet..shard_00340.parquet (25 shards)
Mixed ratio_60: shard_00300.parquet..shard_00325.parquet (25 shards)
Mixed ratio_60: shard_00273.parquet..shard_00299.parquet (25 shards)
Mixed ratio_60: shard_00258.parquet..shard_00272.parquet (25 shards)
Mixed ratio_60: shard_00235.parquet..shard_00257.parquet (25 shards)
Mixed ratio_60: shard_00214.parquet..shard_00227.parquet (25 shards)
Mixed ratio_60: shard_00171.parquet..shard_00198.parquet (25 shards)
Mixed ratio_60: shard_00140.parquet..shard_00170.parquet (25 shards)
Mixed ratio_60: shard_00136.parquet..shard_00152.parquet (25 shards)
Mixed ratio_60: shard_00111.parquet..shard_00130.parquet (25 shards)
Mixed ratio_60: shard_00062.parquet..shard_00096.parquet (25 shards)
Mixed ratio_60: shard_00055.parquet..shard_00061.parquet (25 shards)
Mixed ratio_60: shard_00030.parquet..shard_00049.parquet (25 shards)
Mixed ratio_60: shard_00009.parquet..shard_00029.parquet (25 shards)
Wipe ratio_60 (361 files)
Mixed ratio_30 manifest + README
Mixed ratio_30: shard_01258.parquet..shard_01262.parquet (13 shards)
Mixed ratio_30: shard_01238.parquet..shard_01250.parquet (25 shards)
Mixed ratio_30: shard_01208.parquet..shard_01234.parquet (25 shards)
Mixed ratio_30: shard_01165.parquet..shard_01199.parquet (25 shards)
Mixed ratio_30: shard_01142.parquet..shard_01162.parquet (25 shards)