Venkatesulu/Venki_data_set_Ananthapuram

Venkidataset_Ananthapuram

A small, internally consistent retrieval corpus of Indian administrative geography: 370 short factual statements covering every state and union territory, 261 districts, 49 major cities, and 32 article summaries.

Built from live Wikidata and Wikipedia, and machine-checked for the specific defect that makes a knowledge base useless — two rows that contradict each other.

from datasets import load_dataset

ds = load_dataset("Venkatesulu/Venki_data_set_Ananthapuram", split="train")
print(ds[0]["text"])
# "Amaravati is the capital of Andhra Pradesh, a state of India."

Why this exists

It was built to answer a measured failure. Fine-tuning a 135M model on 22 facts drove validation loss from 1.6301 to 0.0680 — a 24x improvement, a textbook-looking curve — and the model still answered what is Karnataka's capital wrong, because Karnataka was not one of the 22.

Facts do not generalise out of the weights they were trained into. A loss curve cannot tell you that; only asking about something you did not train on can. So these facts are meant to be retrieved, not memorised — where a wrong one is fixed by editing a row rather than by another training run.

Fields

Licensing

The two sources have different terms, so the licence is recorded per row:

The repository is declared CC BY-SA 4.0 because that is the stricter of the two. If share-alike is a problem for you, filter to the CC0 subset — it is 91% of the corpus and all of the structured facts:

cc0 = ds.filter(lambda r: r["license"] == "CC0-1.0")   # 338 rows

Attribution for the prose rows belongs to the Wikipedia contributors of each article named in title / source.

What was cleaned, and why it mattered

Every one of these was present in an early build and produced a confidently wrong statement that no test was failing on. They are listed because the same defects appear in any naive Wikidata extraction.

Places with more than one capital or headquarters

Wikidata records both Bengaluru and Belgaum as capitals of Karnataka, at equal rank, with no qualifier marking either as primary. Emitting two rows that each say "X is the capital" would put a flat contradiction in the corpus and leave a retriever to pick the answer by ranking accident.

There is no field in the source that says which is primary, so asserting one would invent a distinction the data does not make. These rows state exactly what the source states:

Belgaum and Bengaluru are both recorded as capitals of Karnataka, a state of India.
Mumbai and Nagpur are both recorded as capitals of Maharashtra, a state of India.
Dehradun and Gairsain are both recorded as capitals of Uttarakhand, a state of India.

All three are genuinely dual-capital states — Belagavi, Nagpur and Gairsain each host legislative sessions — so the phrasing is accurate rather than evasive. One district, Annamayya, is handled the same way.

Consistency check

The corpus ships with the checker that enforces this. It parses every exclusive claim of the form X is the capital of Y and X is the administrative headquarters of Y, and fails if any place has two different values:

370 documents, 25 capital claims, 246 headquarters claims
  clean: no contradictions, malformed rows, or duplicates

Rows phrased as "A and B are both recorded as…" make no exclusive claim and are correctly not flagged.

Coverage and limits

• —28 states and union territories, all of them, each with its capital.
• —261 districts across 16 states and union territories, 249 with an administrative headquarters. Wikidata's district coverage is uneven; states absent here simply lack district of India typing upstream.
• —49 cities above roughly 900,000 people. City rows deliberately do not name a state — resolving a city's state needs a transitive containment walk that times out on the public Wikidata endpoint, and a shorter true sentence beats a longer one that might name the wrong state.
• —Population figures are whatever Wikidata holds, mixing census counts and estimates across years. Treat them as approximate.
• —English only. Not a general knowledge base — a retriever built on this should abstain on most questions, which is correct behaviour rather than a gap.

Rebuilding

The corpus is a snapshot of live sources, so it can be refreshed without retraining anything:

python -m serve.build_india_kb --out india_kb.json
python -m serve.validate_kb  india_kb.json

Both scripts, and the retrieval pipeline that consumes this corpus, are in Venkatesulu/training_for_practice.

Citation

@misc{venki_ananthapuram_2026,
  title  = {Venki_data_set_Ananthapuram: a consistency-checked corpus of Indian administrative geography},
  author = {Venkatesulu},
  year   = {2026},
  url    = {https://huggingface.co/datasets/Venkatesulu/Venki_data_set_Ananthapuram}
}

Underlying data: Wikidata (CC0) and Wikipedia (CC BY-SA 4.0).