nassimjp/Bilingual-SFT-Dataset

Bilingual-SFT-Dataset

This dataset is a general-purpose bilingual Supervised Fine-Tuning (SFT) dataset designed for training Large Language Models (LLMs) to handle both English and Pashto languages effectively. It is structured to create robust multilingual models by maintaining English proficiency while building Pashto capabilities.

Attributes:

• —Language(s): English, Pashto
• —License: apache-2.0
• —Size: 200,000 entries
• —Format: JSONL
• —Source: iPashto.ai

Dataset Summary

The dataset contains 200,000 instruction-response pairs formatted for general chat templates (Alpaca format). It is designed to create a robust bilingual model by mixing different types of training examples:

Format

Each entry follows the Alpaca format, which is compatible with most fine-tuning frameworks:

{
  "instruction": "What is AI?",
  "input": "",
  "output": "AI is artificial intelligence..."
}

Example Entry

{
  "id": "001078",
  "instruction": "English: What is the structure of an amino acid?\nپښتو: د امینو اسید جوړښت څه شی دی؟",
  "input": "",
  "output": "English: An amino acid consists of a central carbon atom...\nپښتو: یو امینو اسید د مرکزي کاربن اتوم...",
  "category": "bilingual",
  "source": "ipashto_ai_mixed"
}

Fields

ID Alignment

All entries maintain perfect ID alignment with the original source files. Each id in the dataset corresponds to a matching entry in the original English and Pashto source datasets, enabling:

• —✅ Full traceability for debugging
• —✅ Quality assurance verification
• —✅ Regenerating datasets deterministically
• —✅ Evaluation set construction

Usage

Loading the Dataset

from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("nassimjp/Bilingual-SFT-Dataset")

# Or load as training dataset
train_dataset = load_dataset("nassimjp/Bilingual-SFT-Dataset", split="train")
print(f"Loaded {len(train_dataset)} entries")

Training with LLaMA-Factory

Create a configuration file:

# llama_factory_config.yaml
model_name_or_path: meta-llama/Llama-2-7b-hf
dataset: bilingual_sft
template: alpaca
finetuning_type: lora
lora_rank: 64
lora_alpha: 128
lora_target: q_proj,v_proj
output_dir: outputs/bilingual_model
per_device_train_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 2e-4
num_train_epochs: 3
max_grad_norm: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.03
logging_steps: 10
save_steps: 500
evaluation_strategy: steps
eval_steps: 500

Then run:

llamafactory-cli train llama_factory_config.yaml

Training with Unsloth

from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load model (choose your base model)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-2-7b-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
    dtype=torch.float16,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=128,
    lora_dropout=0.05,
)

# Load dataset
dataset = load_dataset("nassimjp/Bilingual-SFT-Dataset", split="train")

# Define formatting function for Alpaca
def format_alpaca(example):
    text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return {"text": text}

dataset = dataset.map(format_alpaca)

# Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        num_train_epochs=3,
        fp16=True,
        output_dir="./bilingual_output",
    ),
)

trainer.train()

Training with Axolotl

Create a configuration file:

# axolotl_config.yaml
base_model: meta-llama/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: nassimjp/Bilingual-SFT-Dataset
    type: alpaca
    conversation: instruction

dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 2048
sample_packing: true

lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - v_proj

train_on_inputs: false
group_by_length: false
bf16: auto
fp16: false

gradient_accumulation_steps: 4
micro_batch_size: 4
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 2e-4

wandb_project: bilingual-sft
wandb_watch: gradients

Then run:

accelerate launch -m axolotl.cli.train axolotl_config.yaml

Intended Uses

This dataset is designed for:

• —🎯 Bilingual Fine-Tuning: Training models to understand and respond in both English and Pashto
• —🔄 Machine Translation: Improving translation capabilities between English and Pashto
• —🌐 Cross-Lingual Transfer: Building models that can handle prompts in both languages
• —📝 Instruction Following: Models that can follow instructions in either language
• —🎓 Educational AI: AI systems for Pashto-speaking users
• —💬 Conversational AI: Chatbots and assistants that can converse in both languages
• —🔬 Research: Multilingual and cross-lingual learning research

Limitations and Biases

• —The dataset contains technical and academic content, which may reflect Western-centric perspectives
• —Pashto translations may have regional variations (e.g., Afghan vs. Pakistani Pashto)
• —Cultural contexts may not be fully represented
• —The dataset is derived from existing English datasets and may inherit their biases
• —Some translations may have minor stylistic variations
• —The dataset focuses on written Pashto and may not capture all dialectal variations

Data Source

The dataset was created from the iPashto.ai project, which curates bilingual English-Pashto data for AI training. The source data includes:

• —37,635 English records from messages_chunk_001_indexed.jsonl
• —Corresponding Pashto translations from pashto_messages_chunk_001.jsonl

License

This dataset is licensed under Apache License 2.0.

Citation

If you use this dataset, please cite:

@dataset{zadran_2025_bilingual_sft,
  author = {Nasibullah Nassim},
  title = {Bilingual-SFT-Dataset},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/nassimjp/Bilingual-SFT-Dataset}
}

Contact

For questions or suggestions, please open an issue on the dataset repository or contact the maintainer.

Acknowledgments

• —iPashto.ai project for the original data
• —Hugging Face for dataset hosting
• —LLaMA-Factory, Unsloth, and Axolotl teams for training tools

Additional Information

Dataset Statistics

• —Total Entries: 200,000
• —Languages: English, Pashto
• —Format: JSONL (Alpaca)
• —Average Instruction Length: ~50 words
• —Average Output Length: ~100 words
• —Max Sequence Length: 2048 tokens

Quality Assurance

• —✅ All entries are ID-aligned with source files
• —✅ 100% conversion rate (0 skipped entries)
• —✅ Contains 5 diverse categories
• —✅ Proper Alpaca format
• —✅ No empty or malformed entries

Comparison with Ministral Version

Dataset created with ❤️ for the Pashto AI community

Stars ⭐ Star this dataset if you find it useful!