vaqasai/Odia_English_Sentences_Aligned_81k

🌐 Vaqas AI: Odia-English Sentence-Aligned Precision Corpus

Created and Maintained by [Vaqas AI](https://vaqasai.in) Creator & Owner: Vaqas Ahmed

🚀 Why This Dataset?

Standard parallel corpora often consist of large paragraphs that exceed the 4k or 8k context windows typical of LLM training, leading to truncated data and poor model performance.

The Vaqas AI Sentence-Aligned Corpus was engineered to solve this. We transformed the original vaqasai/Odia_English_News dataset by decomposing dense paragraphs into distinct, semantically aligned sentence pairs. This makes the dataset ideal for:

• —Fine-tuning LLMs for English-to-Odia translation.
• —Instruction Tuning (by wrapping pairs in prompt templates).
• —Evaluating Translation Quality using BLEU, METEOR, or COMET.

🛠️ The Vaqas AI Engineering Pipeline

We used a multi-stage refinement process to ensure "Gold Standard" quality:

1. 1.Decomposition: Large articles were broken down into individual sentences.
2. 2.Semantic Alignment: Sentences were aligned using embedding-based similarity models.
3. 3.Vaqas AI Quality Filter: - Similarity Threshold: Retained only pairs with a cosine similarity score ≥ 0.6.
4. 4.Length Ratio Constraint: Enforced a ratio between 0.4 and 2.5 to eliminate partial or hallucinated translations.
5. 5.Token Optimization: Filtered for pairs within a strict token range (20 to 1500 tokens) to ensure meaningful context without exceeding memory limits.
6. 6.Translation Engine: All translations were powered by Gemini 3 Flash via Google AI Studio.

📊 Dataset Statistics

• —Total Aligned Pairs: ~81,000+ high-quality pairs.
• —Mean Similarity Score: 0.83
• —Format: Tab-Separated Values (.tsv)
• —Splits:
• —Train: 72,989 pairs (90%)
• —Validation: 4,055 pairs (5%)
• —Test: 4,055 pairs (5%)

📂 Source Code & Tools

The architecture for this pipeline is fully open-source. Explore the code that built this dataset: 👉 [GitHub: Odia-English Dataset Builder](https://github.com/pytholabsbot1/Odia--english-dataset-builder)

💎 About Vaqas AI

Vaqas AI is a specialized laboratory focused on regional language AI, founded by Vaqas Ahmed. We build the datasets and tools that enable modern AI to speak Indian languages fluently.

• —Official Website: vaqasai.in
• —Primary Brand: Vaqas AI
• —Innovation: Powered by Google AI Studio & Gemini 3 Flash.

📜 Usage & Attribution

Released under the MIT License. We encourage community usage for research and commercial model development.

Attribution Required: "This project uses the Vaqas AI Odia-English Precision Corpus by Vaqas Ahmed (vaqasai.in)."

Precision engineered by Vaqas Ahmed at Vaqas AI.