Datasetpublic

akshitkr/bodo_c4_0001_deduped

Bodo language synthetic pretraining data, original source language is english. Original dataset details wget https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz gunzip c4-train.00000-of-01024.json.gz Translation parameters: FORWARD_MODEL_NAME = "prajdabre/rotary-indictrans2-en-indic-1B" BATCH_SIZE = 256 NUM_WORKERS = 8 MAX_SEQ_LENGTH = 256 NUM_BEAMS = 1

sourceHugging Facemitupdated 1y agoView on Hugging Face

1likes21downloads

Dataset Card

Bodo language synthetic pretraining data, original source language is english.

Original dataset details

wget https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz
gunzip c4-train.00000-of-01024.json.gz

Translation parameters:

FORWARD_MODEL_NAME = "prajdabre/rotary-indictrans2-en-indic-1B"
BATCH_SIZE = 256
NUM_WORKERS = 8
MAX_SEQ_LENGTH = 256
NUM_BEAMS = 1