CoolFace
Datasetpublic

Finnish-NLP/finepdfs-dclm-fineweb-edu-fi

FinePDFs · DCLM · FineWeb-Edu — Finnish (machine-translated) Finnish machine translation of an English pretraining mixture drawn from FinePDFs, DCLM, and FineWeb-Edu (documents up to ~900 tokens), produced as continued-pretraining data for Finnish LLMs. Translation model: translategemma-27b (Gemma-based 27B translation model) Documents: ~4,000,000 (80 shards × 50,000) Language: Finnish (fi) Format: plain text, one document per row Originally stored as TFDS-style ArrayRecord… See the full description on the dataset page: https://huggingface.co/datasets/Finnish-NLP/finepdfs-dclm-fineweb-edu-fi.

sourceHugging Faceodc-byupdated 2mo agoView on Hugging Face
1likes284downloads
Dataset Card

FinePDFs · DCLM · FineWeb-Edu — Finnish (machine-translated)

Finnish machine translation of an English pretraining mixture drawn from FinePDFs, DCLM, and FineWeb-Edu (documents up to ~900 tokens), produced as continued-pretraining data for Finnish LLMs.

  • —Translation model: translategemma-27b (Gemma-based 27B translation model)
  • —Documents: ~4,000,000 (80 shards × 50,000)
  • —Language: Finnish (fi)
  • —Format: plain text, one document per row

Originally stored as TFDS-style ArrayRecord (tf.train.Example with a single text byte feature); converted here to Parquet (single text column) for easy loading with 🤗 datasets.

Schema

columntypedescription
textstringFinnish-translated document text

Usage

python
from datasets import load_dataset
# stream to avoid downloading everything at once
ds = load_dataset("Finnish-NLP/finepdfs-dclm-fineweb-edu-fi", split="train", streaming=True)
print(next(iter(ds))["text"][:500])

Caveats

Machine-translated web text; translation artifacts and web-noise are possible. Content inherits the licenses/terms of the upstream sources (FinePDFs, DCLM, FineWeb-Edu). Verify upstream licensing before downstream redistribution.