CoolFace
Datasetpublic

SultanR/nemotron-mc-en-ar-midtrain

nemotron-mc-en-ar-midtrain Arabic translation of the Nemotron-Pretraining-Multiple-Choice config of Nemotron-Pretraining-Specialized-v1.2 (pinned revision 807afc1). Translated with google/gemma-4-12B-it (bf16, greedy) on A100s. All 23,926,492 source rows are present, none dropped. English source and Arabic translation sit in the same row, so the dataset serves as a parallel corpus as well as an Arabic one. A sibling corpus from the same pipeline is available at… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/nemotron-mc-en-ar-midtrain.

sourceHugging Faceupdated 1mo agoView on Hugging Face
0likes472downloads
Dataset Card

nemotron-mc-en-ar-midtrain

Arabic translation of the Nemotron-Pretraining-Multiple-Choice config of Nemotron-Pretraining-Specialized-v1.2 (pinned revision 807afc1). Translated with google/gemma-4-12B-it (bf16, greedy) on A100s. All 23,926,492 source rows are present, none dropped. English source and Arabic translation sit in the same row, so the dataset serves as a parallel corpus as well as an Arabic one. A sibling corpus from the same pipeline is available at openthoughts3-en-ar-midtrain.

The pipeline segments each row into prose and verbatim blocks (code, tables, LaTeX, option labels, and inline non-translatables are masked and never sent to the model), translates prose in ~256-token chunks, and retries failed chunks once before falling back to their English source rather than dropping them. Nothing is silently repaired, every row carries QC and screening labels.

Details

Rows23,926,492 (equals the source exactly)
Source tokens~6.45B
Chunks translated41.9M (1.75 per row)
Size17.7 GB
qc_clean99.72%
Chunk-level fallback rate0.12%
Numbers preserved99.18%
Mean Arabic character fraction0.852

Schema

ColumnDescription
row_idPosition in the pinned source snapshot, maps back to the exact source row
textEnglish source, unmodified
ar_textArabic translation
src_dataset, src_config, src_split, src_file, src_row_indexProvenance
n_chunks, n_fallback_chunks, n_masks_lost, n_masks_duplicated, n_hard_breaks_restoredTranslation counters
qc_clean, qc_arabic_fraction, qc_numbers_match, qc_numbers_missing, qc_max_repeated_ngram, qc_source_repeated_ngramQuality labels
screen_*Multiple-choice screening flags
pipeline_version, prompt_version, model, model_revisionReproducibility

qc_clean is the conjunction of the structural checks: no placeholder lost, no English fallback, output is Arabic, no decode loop. It is a structural claim, not a fluency judgement. A chunk that failed translation twice keeps its English source, so a small share of rows contain English spans by design.

Usage

python
from datasets import load_dataset

ds = load_dataset("SultanR/nemotron-mc-en-ar-midtrain", split="train", streaming=True)
clean = ds.filter(lambda r: r["qc_clean"] and not r["screen_degenerate"])

License

Refer to the source Nemotron-Pretraining-Specialized-v1.2 dataset for license information.