Ik45/data-science-en-id

Data Science EN-ID Parallel Corpus (Scientific Domain)

Dataset Description

This dataset is a curated English-Indonesian (EN-ID) parallel corpus specifically designed for the Scientific and Data Science domains. It was developed to support the training of Machine Translation (NMT) models and Large Language Models (LLMs) to better handle technical terminology, academic structures, and formal scientific language.

• —Primary Languages: English (EN) and Indonesian (ID)
• —Domain: Data Science, Artificial Intelligence, Machine Learning, and General Science.
• —Applications: Neural Machine Translation, Domain Adaptation, Cross-lingual Information Retrieval.

Source Data & Origin

This dataset is a specialized extension and refined subset derived from two primary high-quality sources available on Hugging Face:

1. 1.[wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia): We extracted the English-Indonesian parallel subsets, specifically focusing on articles categorized under Science, Technology, and Mathematics.
2. 2.[ccdv/arxiv-summarization](https://huggingface.co/datasets/ccdv/arxiv-summarization): We utilized the Arxiv metadata and document summaries to build a robust scientific corpus, filtering for Computer Science and Data Science domains.

Extraction & Refinement Pipeline

To transform these general-purpose datasets into a domain-specific parallel corpus, the following pipeline was implemented:

1. Domain-Specific Filtering

Instead of using the entire Wikipedia or Arxiv dump, we applied a Lexical Filter using a dictionary of 500+ technical keywords. A sentence pair is only included if it contains terminology relevant to Data Science (e.g., Neural Networks, Statistical Inference, Heuristics).

2. Scientific Alignment

For the Arxiv data, we performed a custom alignment process to pair English scientific abstracts with their Indonesian technical equivalents, ensuring that the formal tone and academic nomenclature are preserved.

3. Noise Reduction (Regex-Based)

Since Arxiv and Wikipedia data often contain LaTeX code, citations (e.g., [1], (Author, 2023)), and HTML artifacts, we applied a rigorous cleaning script to ensure the final output consists of clean, natural language sentences.

4. Deduplication

We cross-referenced both sources to remove overlapping entries, ensuring that the Ik45/data-science-en-id corpus is diverse and free from redundant training signals.

Creation Methodology

Based on the processing pipeline in ScriptTextTranslationDomainIlmiah.ipynb, the dataset was constructed through several key stages:

1. Data Collection

The corpus was aggregated from various scientific articles , academic papers, and technical datasets. The focus was strictly maintained on high-quality technical content to ensure domain relevance.

2. Preprocessing & Data Cleaning

• —Deduplication: Removal of redundant pairs to ensure data diversity and prevent model overfitting.
• —Noise Reduction: Cleaning of non-textual characters, artifacts from PDF extractions, and broken symbols.
• —Normalization: Standardizing text formats to maintain consistency across the entire corpus.

3. Alignment

Rigorous sentence-level alignment was performed to ensure that English technical terms are correctly mapped to their appropriate Indonesian counterparts within a scientific context.

Usage

You can easily load this dataset using the Hugging Face datasets library:

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("Ik45/data-science-en-id")

# Preview a sample
print(dataset['train'][0])