CoolFace
Datasetpublic

kamaalg/azerbaijani-instructions

Azerbaijani Instruction Dataset (v0) Azerbaijani (instruction, response) pairs for supervised fine-tuning (SFT) of Azerbaijani language models — part of an open Azerbaijani LLM stack. Instruction data is genuinely scarce for Azerbaijani; this is both training data for our models and a reusable standalone artifact for anyone building Azerbaijani instruction-following models. Contents seeds_az.jsonl — 45 hand-authored, high-quality seed pairs spanning 16 task… See the full description on the dataset page: https://huggingface.co/datasets/kamaalg/azerbaijani-instructions.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
1likes34downloads
Dataset Card

Azerbaijani Instruction Dataset (v0)

Azerbaijani (instruction, response) pairs for supervised fine-tuning (SFT) of Azerbaijani language models — part of an open Azerbaijani LLM stack. Instruction data is genuinely scarce for Azerbaijani; this is both training data for our models and a reusable standalone artifact for anyone building Azerbaijani instruction-following models.

Contents

  • —`seeds_az.jsonl` — 45 hand-authored, high-quality seed pairs spanning 16 task categories. These bootstrap the self-instruct generation and are usable directly as a tiny SFT set.
  • —`sft_az.jsonl` (generated) — the distilled set produced by generate_instructions.py from a strong teacher model. Not checked in by default; reproduce it yourself (see below).

Each record:

json
{"category": "qa", "instruction": "Fotosintez nədir və niyə vacibdir?", "response": "Fotosintez bitkilərin ..."}

Categories: qa, summarize, rewrite, explain, classify, brainstorm, extract, translate, reasoning, coding-explain, email, list, grammar, definition, step-by-step, creative.

How it's produced

Self-instruct data distillation. generate_instructions.py samples seed pairs (spread across categories), asks a strong teacher (Anthropic Claude — default claude-sonnet-5 — or OpenAI GPT) for new, diverse, native-Azerbaijani pairs, then applies QC: length bounds, Latin-Azerbaijani script enforcement (rejects Cyrillic/Perso-Arabic), markdown-fence stripping, echo rejection, and exact + near-duplicate deduplication. Validate any file with python sft/validate.py --file <path>.

bash
pip install anthropic
export ANTHROPIC_API_KEY=...
python sft/generate_instructions.py --n 3000 --backend anthropic

Intended use

  • —SFT of Azerbaijani base LMs into instruction-following assistants (response-only loss; see sft/).
  • —A reusable open Azerbaijani instruction corpus for the community.

Limitations (honest)

  • —Teacher-distilled ⇒ the generated split inherits the teacher's biases, style, and any factual errors. It is not human-written (the 45 seeds are; the bulk is not).
  • —Native-speaker spot-checks are recommended before high-stakes use; QC here is automated and heuristic (script, length, dedup), not a fluency/factuality guarantee.
  • —Latin-script Modern (North) Azerbaijani only.
  • —v0: the seed set is small and skews toward general/educational tasks; extend seeds_az.jsonl for broader coverage (domain-specific, multi-turn, tool-use, etc.).

License

Apache-2.0 for the seeds and code. Note that text generated by a teacher model is also subject to that provider's usage terms; check them before redistribution.