sabin1234/Nepal_Education_Provincial_Budget_Tax_Miscellaneous_Statistics

Nepal Education, Provincial Budget, Tax & Miscellaneous Statistics — Instruction Dataset

A Nepali-language (with a small English subset) instruction-following (Q&A) dataset built from official Nepali government and statistical sources — covering federal/provincial education budgets, tax exemptions, trade statistics, cooperative membership, disaster losses, national accounts, and Department of Roads budget data.

File: merged_all_edu_prov_tax_misc_serial.jsonl Format: JSON Lines (.jsonl) — one JSON object per line Total records: 1,680

1. Overview

2. Record Schema

Each line is a single JSON object with the following top-level fields:

conversations structure

"conversations": [
  {"from": "human", "value": "<question in Nepali or English>"},
  {"from": "gpt",   "value": "<factual answer, same language as the question>"}
]

Every record has exactly 2 turns: one human question and one gpt answer — this is a single-turn factual QA dataset, not multi-turn dialogue.

metadata_json structure (parsed)

metadata_json is a string containing a nested JSON object. Once parsed, it contains:

3. Source Publications (9 total)

All answers are grounded in real, published Nepali statistical/government data — no hallucinated facts.

Note: The "Intermediate Consumption" source title mentions "India" but the questions and answers themselves are about Nepali economic sectors (कृषि तथा वन, etc.) — the title appears to be inherited as-is from the upstream source metadata.

4. Domain / Category / Sub-domain Breakdown

4.1 generation_domain (8 unique values)

4.2 generation_category (9 unique values)

4.3 generation_sub_domain (17 unique values)

5. Language, Script & License Distribution

5.1 Language

5.2 License

5.3 Task / Generation metadata (constant fields)

6. Behavior Diversity

The behavior and behavior_definition metadata fields describe what the model is expected to do when answering each question. There are 3 distinct behavior types, each paired with a more detailed behavior definition (4 unique full definitions total, since one behavior has two slightly different phrasings across subsets).

A closely related variant of the second behavior definition also appears on a smaller subset:

Common thread across all behaviors: every answer must be (1) strictly grounded in the source statistical table (no fabrication), and (2) delivered in fluent, natural Nepali (or English, for the 51 English rows) rather than a literal/awkward translation.

question_type (2 unique values)

(Both labels describe the same underlying question type — "fact-based" — just recorded in Nepali vs. English metadata for different subsets of the merge.)

7. Question Pattern Diversity

Although every question is "fact-based," the dataset contains 34 distinct question-pattern families, identifiable via the id prefix (the part of the id before the trailing serial number, e.g. edu_budget_faq_0007 → family edu_budget_faq). Each family represents a repeatable question template applied across different years, provinces, ministries, tax categories, etc. Two example Q&A pairs are shown per family below.

Total across all 34 families: 1,680 (matches total record count exactly).

7.1 Recurring question patterns observed across families

8. Text Length Statistics

All Q&A pairs are short, single-fact statements (typically 1 sentence each) — no multi-paragraph or multi-fact answers.

9. Sample Records

Example 1 — Federal education budget (Nepali):

{
  "id": "edu_budget_source_faq_0001",
  "conversations": [
    {"from": "human", "value": "२०२१/२२ मा आन्तरिक राजस्व बाट कति रकम प्राप्त भएको थियो?"},
    {"from": "gpt", "value": "२०२१/२२ मा आन्तरिक राजस्व बाट १०२४.९०७ रकम प्राप्त भएको थियो।"}
  ],
  "source": "Analysis of Federal Education Budget FY 2021/22",
  "source_name": "federal_education_budget",
  "source_repo": "Central Bureau of Statistics",
  "language": "ne",
  "language_code": "npi",
  "script": "Deva",
  "license": "CC0: Public Domain",
  "license_tier": "permissive",
  "task_type": "instruction-following",
  "generation_type": "real",
  "condition": "real",
  "metadata_json": "{\"generation_domain\": \"शिक्षा बजेट\", \"generation_category\": \"संघीय शिक्षा बजेट विश्लेषण\", \"generation_sub_domain\": \"बजेट तथ्यांक\", \"behavior\": \"तथ्यमा आधारित उत्तर दिने\", \"behavior_definition\": \"मूल तथ्य कायम राख्दै शुद्ध नेपालीमा उत्तर दिने।\", \"question_type\": \"तथ्यमा आधारित प्रश्न\", \"content_language\": \"नेपाली\", \"content_script\": \"देवनागरी\", \"english_content_allowed\": false}"
}

Example 2 — Department of Roads budget (the English/no-source_name subset):

{
  "conversations": [
    {"from": "human", "value": "बाँडफाँट नगरिएको बजेट रकम कति थियो?"},
    {"from": "gpt", "value": "बाँडफाँट नगरिएको बजेट रकम ३,९६५,६०० थियो।"}
  ],
  "source": "सडक विभाग",
  "source_name": null,
  "source_repo": "सडक विभाग",
  "language": "en",
  "script": "Latn",
  "license": null,
  "metadata_json": "{\"generation_domain\": \"public infrastructure finance\", \"generation_category\": \"Department of Roads budget and expenditure\", \"generation_sub_domain\": \"Fiscal Year 2064/65\", \"behavior\": \"provide precise factual financial answers\", \"behavior_definition\": \"उपलब्ध सडक कार्यालयका बजेट तथा खर्च तथ्याङ्कका आधारमा सिधा र तथ्यपरक उत्तर दिने।\", \"question_type\": \"तथ्यमा आधारित प्रश्न\", \"content_language\": \"English\", \"content_script\": \"Latin\", \"english_content_allowed\": true}"
}

Note: despite language/script being tagged en/Latn for this subset, the actual question/answer text is still in Devanagari — this reflects how the metadata was tagged upstream rather than the actual script of the content. Treat content_language / content_script inside metadata_json as the more reliable per-row indicator, and always verify against the actual conversations text if language purity matters for your use case.

10. How to Load This Dataset

10.1 Plain Python (standard library only)

import json

path = "merged_all_edu_prov_tax_misc_serial.jsonl"

records = []
with open(path, "r", encoding="utf-8") as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        records.append(json.loads(line))

print(f"Loaded {len(records)} records")

# Inspect the first record
first = records[0]
question = first["conversations"][0]["value"]
answer = first["conversations"][1]["value"]
print("Q:", question)
print("A:", answer)

# Parse the nested metadata
metadata = json.loads(first["metadata_json"])
print("Domain:", metadata["generation_domain"])
print("Behavior:", metadata["behavior"])

10.2 Using pandas

import json
import pandas as pd

path = "merged_all_edu_prov_tax_misc_serial.jsonl"

# jsonlines files load directly with lines=True
df = pd.read_json(path, lines=True)

# Expand the nested metadata_json string into real columns
meta_df = pd.json_normalize(df["metadata_json"].apply(json.loads))
df = pd.concat([df.drop(columns=["metadata_json"]), meta_df], axis=1)

# Extract question / answer as separate columns for convenience
df["question"] = df["conversations"].apply(lambda c: c[0]["value"])
df["answer"]   = df["conversations"].apply(lambda c: c[1]["value"])

print(df.shape)
print(df[["id", "question", "answer", "generation_domain", "behavior"]].head())

# Example: filter to only education-budget questions
edu_df = df[df["generation_domain"].str.contains("शिक्षा", na=False)]
print(f"Education-related rows: {len(edu_df)}")

10.3 Using the Hugging Face datasets library

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files="merged_all_edu_prov_tax_misc_serial.jsonl",
    split="train"
)

print(dataset)
print(dataset[0])

# Filter by language
nepali_only = dataset.filter(lambda ex: ex["language"] == "ne")
english_only = dataset.filter(lambda ex: ex["language"] == "en")
print(f"Nepali rows: {len(nepali_only)}, English-tagged rows: {len(english_only)}")

# Map to extract question/answer as top-level fields
def extract_qa(example):
    example["question"] = example["conversations"][0]["value"]
    example["answer"] = example["conversations"][1]["value"]
    return example

dataset = dataset.map(extract_qa)

10.4 Streaming (for very large files / low-memory environments)

import json

def stream_records(path):
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)

for record in stream_records("merged_all_edu_prov_tax_misc_serial.jsonl"):
    q = record["conversations"][0]["value"]
    a = record["conversations"][1]["value"]
    # process one record at a time without loading the whole file into memory

10.5 Converting to a fine-tuning-ready chat format

import json

def to_chat_format(record):
    """Convert a record to a simple {'messages': [...]} chat format
    commonly used for SFT / instruction-tuning pipelines."""
    return {
        "messages": [
            {"role": "user", "content": record["conversations"][0]["value"]},
            {"role": "assistant", "content": record["conversations"][1]["value"]},
        ]
    }

with open("merged_all_edu_prov_tax_misc_serial.jsonl", encoding="utf-8") as fin, \
     open("chat_format.jsonl", "w", encoding="utf-8") as fout:
    for line in fin:
        record = json.loads(line)
        chat_record = to_chat_format(record)
        fout.write(json.dumps(chat_record, ensure_ascii=False) + "\n")

11. Suggested Use Cases

• —Fine-tuning / instruction-tuning Nepali-language LLMs on factual, statistics-grounded Q&A
• —Evaluating a model's ability to read structured government statistical tables and answer precisely (numeric fidelity, unit handling, year/date grounding)
• —Building retrieval-augmented generation (RAG) evaluation sets for Nepali government finance data
• —Studying low-resource-language (Nepali/Devanagari) instruction data characteristics
• —Benchmarking numeral formatting and Devanagari-numeral (०-९) generation in Nepali LLMs

12. Known Limitations / Caveats

• —Every conversation is single-turn — there is no multi-turn dialogue or follow-up questioning in this file.
• —Answers are short, single-fact statements; the dataset does not contain explanatory, analytical, or multi-fact reasoning answers.
• —The Department of Roads subset (51 rows) has source_name = null and license = null — treat these as needing separate license verification before redistribution.
• —The language/script tags for the Department of Roads subset say en/Latn, but the actual question/answer text is Devanagari Nepali — always check the literal conversations text rather than relying solely on the language field.
• —The "Intermediate Consumption" source title references India, but its Q&A content is about Nepali economic sectors — this looks like an inherited title from the upstream source and does not reflect the actual content of the questions/answers.
• —Numeric answers are presented as plain figures without consistent unit disambiguation in every row (e.g., some rows omit whether values are in millions, billions, or raw currency units) — cross-check the source field/original publication when unit precision matters.

13. Field Quick-Reference (cheat sheet)