agentlans/FineWeb2-Edu-JA-EN

FineWeb2-Edu-JA-EN: Japanese-English Parallel Dataset

FineWeb2-Edu-JA-EN is a high-quality, information-rich Japanese-English parallel corpus curated specifically for educational and academic domains. It is derived from a subset of `hotchpotch/fineweb-2-edu-japanese`, which was filtered, clustered, and translated into English using Google Translate.

• —Original Dataset: `hotchpotch/fineweb-2-edu-japanese`
• —Size: 2,168 rows
• —License: ODC-BY

Dataset Creation & Methodology

To extract high-quality, text-dense educational pairs, the following pipeline was used:

1. 1.Embedding & Clustering: 300,000 random rows from the original dataset were embedded using `tohoku-nlp/bert-base-japanese-v3` and grouped via hierarchical clustering.
2. 2.Translation: The text blocks were machine-translated into English using Google Translate.
3. 3.Filtering: The raw machine translations were filtered, resulting in a highly curated final set of 2,168 clean, parallel translation pairs.

Dataset Structure

Data Fields

Every sample in the dataset contains the following three fields:

• —id (string): The unique text identifier mapped directly from the original FineWeb-2 dataset.
• —ja (string): The original, high-quality educational Japanese text.
• —en (string): The corresponding English translation generated via Google Translate.

Data Sample

{
  "id": "<urn:uuid:98837885-9feb-4b08-b1ac-2dc0ebd77c60>",
  "ja": "全国都道府県教育委員会連合会というのがある。 http://www.kyoi-ren.gr.jp/index.htmlnこの略称「キョ～イ連」には...",
  "en": "There is an organization called the National Federation of Prefectural Boards of Education. http://www.kyoi-ren.gr.jp/index.htmlnThis abbreviation \"Kyo-i-ren\" also has..."
}

Limitations & Scope

• —Size vs. Quality: The dataset is small (2,168 rows) but prioritizes dense, informative content over sheer volume.
• —Domain Specificity: The corpus heavily skews toward general academic contexts and education-related prose. It's not focused on:
• —Casual or everyday conversations.
• —Pop culture (Anime, Manga, J-Pop, Gaming).
• —Highly specialized domains (e.g., rigid legal text, advanced medical terminology).
• —Machine Translation Artifacts: Because English targets were generated using Google Translate, minor systemic fluency issues, literal translation artifacts, or minor formatting/spacing anomalies may be present.
• —No Post-Processing: To preserve the raw performance and nuances of the translation layout, no additional automated anonymization or aggressive post-filtering was applied to the text.

Licence & Acknowledgements

• —This dataset is distributed under the Open Data Commons Attribution License (ODC-BY).
• —We would like to express our sincere gratitude to the creators of the original `hotchpotch/fineweb-2-edu-japanese` dataset.