AmareshHebbar/jyotish-llm-sft

Jyotish-LLM SFT Dataset

1,000,000 grounded instruction-tuning examples for Vedic astrology chart interpretation. Every example pairs a deterministically-computed birth chart with a response whose every claim traces back to either a chart fact or a cited classical yoga rule — built specifically so a model trained on it learns to cite, not free-associate.

Where the data actually came from (read this before trusting "1M rows" too much)

This is not scraped, purchased, or sourced from any astrology website, app, or API. Every field is one of exactly two things:

1. 1.Computed astronomy (chart_facts) — real planetary math, not generated by any model.
2. 2.Hand-authored rule text (yoga names + effect descriptions in retrieved_rules/citations) — written by the dataset author, summarizing widely known classical Parashari principles in the author's own words. No classical text corpus (BPHS, Phaladeepika, etc.) was ingested or scraped for this version — the rule definitions were hand-encoded directly into a YAML file based on general astrological knowledge, not extracted from any specific translated source. If a Tier-1 text-ingestion pass is added later (see companion repo's 01_ingest_tier1_texts.py), this card will be updated to say so and name the specific sources used.

No real people's birth data is in this dataset anywhere — every birth record is randomly generated (see Procedure below), which is also why this dataset carries no privacy risk and can be regenerated/scaled arbitrarily.

How this dataset was built — full procedure

Step 1 — Deterministic chart engine. Built on pyswisseph (Swiss Ephemeris Python binding), sidereal mode, Lahiri ayanamsa. For any date/time/lat/lon it computes: Ascendant (Lagna), all 9 grahas (Sun, Moon, Mars, Mercury, Jupiter, Venus, Saturn, Rahu, Ketu) with Rashi, Nakshatra, Pada, and Navamsa (D9) sign, plus the full Vimshottari Mahadasha → Antardasha timeline (all 9 Mahadasha periods, each with all 9 nested Antardasha sub-periods, computed from the Moon's nakshatra position at birth). Verified against known reference values (ayanamsa for a known epoch, navamsa sign-cycling formula, dasha-period totals) before being used for generation — see companion repo tests/test_engine_validation.py.

Step 2 — 200,000 synthetic birth charts. Random date of birth (1950–2015), random time (24h, 15-min increments), and a weighted place pool (~14 major Indian cities weighted heavier, plus New York, London, Singapore, Dubai, San Francisco, Toronto, Sydney for diaspora coverage) were sampled and run through the Step 1 engine. 0 errors out of 200,000.

Step 3 — Rule matching. A YAML-defined rule engine checks each chart against 15 hand-encoded classical yoga/combination rules (5 planetary exaltations, Gaja Kesari, Budhaditya, Chandra-Mangal, Guru-Shukra, Raja Yoga [9th-10th lord connection], Dhana Yoga [2nd-11th lord connection], Ascendant Lord in Kendra, Venus in 7th house). This is plain condition-matching code — deterministic, not a model — so every match is 100% reproducible from the chart data.

Step 4 — Response composition. For each of the 200,000 charts, 5 grounded responses were composed via fixed templates — one per question_type (generalnatal, career, marriagetiming, health, dashaperiod) — each stating only: the Ascendant/Moon placement (a direct chart fact), any matched yogas (each with its citation), and the current Mahadasha/Antardasha (a direct chart fact). = 200,000 × 5 = **1,000,000 examples**. No LLM was used to generate or embellish any of these responses — `source` is `tier3rule_matched` for all rows in this version.

Step 5 — Split. Deterministic hash of synthetic_id (not example_id) assigns each of the 5 question-type variants of a given chart to the same split — this avoids leaking near-identical chart_facts across the train/test boundary. Target 96/2/2, actual 95.96% / 2.06% / 1.98% due to hash-bucket rounding.

Step 6 — Push. datasets.push_to_hub() — auto-converted to Parquet on upload (dictionary/columnar encoding compresses the highly repetitive chart_facts/citation strings hard: ~19 GB raw JSONL → 688 MB Parquet).

Full reproducible pipeline (all 6 scripts + the rule engine + chart engine) is in the companion repo: [link your GitHub repo here once pushed — currently a placeholder].

Schema

Usage

from datasets import load_dataset

ds = load_dataset("AmareshHebbar/jyotish-llm-sft")
print(ds)
# DatasetDict({
#     train: Dataset({..., num_rows: 959625}),
#     validation: Dataset({..., num_rows: 20575}),
#     test: Dataset({..., num_rows: 19800}),
# })

ex = ds["train"][0]
print(ex["user_question"])
print(ex["response"])
print(ex["citations"])

Filter to a single question type:

career_only = ds["train"].filter(lambda x: x["question_type"] == "career")

Formatting a row into an SFT prompt/completion pair (grounding the model on chart_facts + retrieved_rules, target is response):

import json

def format_example(ex):
    prompt = (
        "You are a Vedic astrology narrator. Use ONLY the facts below.\n\n"
        f"CHART_FACTS:\n{json.dumps(ex['chart_facts'])}\n\n"
        f"RETRIEVED_RULES:\n{json.dumps(ex['retrieved_rules'])}\n\n"
        f"QUESTION: {ex['user_question']}\n\nREADING:"
    )
    return {"prompt": prompt, "completion": ex["response"]}

sft_ds = ds["train"].map(format_example)

Streaming (avoids downloading all 688 MB up front):

ds_stream = load_dataset("AmareshHebbar/jyotish-llm-sft", streaming=True)
for ex in ds_stream["train"]:
    print(ex["user_question"])
    break

What this dataset does NOT claim

This dataset teaches a model to narrate a correctly-computed chart using classical rules faithfully and with citations. It does not, and cannot, contain any ground truth about whether astrological predictions come true — no such data exists anywhere. A model trained on this should end up faithful to classical Parashari-style reasoning and to computed chart facts, not a model whose forecasts are empirically validated.

Known limitations

• —Rule coverage: 15 yogas, hand-encoded from general classical knowledge, not the full classical corpus (BPHS alone documents hundreds). Extend via rules_corpus/yogas.yaml in the companion repo — it's a YAML edit, no code change needed.
• —No Tier-1 text grounding yet: rule effect descriptions are author-paraphrased, not pulled from or checked against a specific classical source text. See "Where the data actually came from" above.
• —5 question types: compatibility (needs two charts) and muhurta (needs an event date) aren't included yet.
• —Template-only responses: all 1M rows share source: tier3_rule_matched — fixed sentence templates per questiontype, good grounding signal, narrow stylistically. An optional LLM-paraphrase enrichment pass (`scripts/03bllmassisteddrafting.py` in the companion repo) exists but was not run for this version — a model trained purely on this dataset may itself sound templated. Blend in enriched examples, or add stylistic post-training, if that matters for your use case.
• —Place distribution: weighted toward ~14 Indian cities + 7 diaspora hubs (see PLACE_POOL in the companion repo) — not a demographically representative sample of any real population, since none of this is real people's data.

License

Apache 2.0. All chart data is computed from synthetic (non-real) birth records — no privacy concerns. Rule/effect text is written in-house, paraphrasing widely known classical Parashari principles, not reproduced from any single copyrighted translation.