CoolFace
Datasetpublic

shasank0001/vizag-city-guide-cpt

Vizag City Guide — CPT Corpus Continued pre-training (CPT) text corpus for Visakhapatnam (Vizag), India. Plain .txt files, one document per line (or paragraph). Pairs with the supervised dataset shasank0001/vizag-city-guide-sft: pre-train on this corpus first, then fine-tune on the SFT set. Contents (~3.4 MB, ~850k tokens at ~4 chars/token) File Size What it is corpus.txt 228 KB Concatenated Wikipedia + government base documents (45 docs)… See the full description on the dataset page: https://huggingface.co/datasets/shasank0001/vizag-city-guide-cpt.

sourceHugging Faceupdated 24d agoView on Hugging Face
0likes61downloads
Dataset Card

Vizag City Guide — CPT Corpus

Continued pre-training (CPT) text corpus for Visakhapatnam (Vizag), India. Plain .txt files, one document per line (or paragraph). Pairs with the supervised dataset `shasank0001/vizag-city-guide-sft`: pre-train on this corpus first, then fine-tune on the SFT set.

Contents (~3.4 MB, ~850k tokens at ~4 chars/token)

FileSizeWhat it is
corpus.txt228 KBConcatenated Wikipedia + government base documents (45 docs)
corpus_chunks.txt261 KB500-token grounded chunks (180 base chunks)
corpus_hard.txt339 KBGovernment tables: District Handbook 2022, VMRDA Master Plan 2041, Smart Cities
corpus_bus.txt20 KBVerified APSRTC bus-route prose (11 route groups)
corpus_layout.txt3.7 KBManual layout/proximity knowledge (distances, clusters)
corpus_osm.txt139 KBOpenStreetMap Vizag POIs rendered as sentences (ODbL)
corpus_osm_harvest.txt16 KBAdditional OSM harvest
corpus_gov_tourism.txt3.9 KBAP Tourism / visakhapatnam.ap.gov.in tourism text (GODL)
corpus_smartcities.txt0.7 KBSmart Cities Vizag provenance record (GODL)
corpus_safety.txt3.2 KBSafety notes: rip currents, Kailasagiri 2026 damage (news fair-use summaries)
corpus_temporal.txt5 KB2025–26 temporal facts (Bhogapuram airport, metro)
corpus_temporal_2026.txt3 KBAug–Sep 2026 updates: airport opening 17 Aug 2026, glass skywalk
corpus_telugu.txt2.39 MBTelugu stories for code-mix exposure (~597k tokens, 73% of corpus)
cpt_info.json—Build stats (chunk/doc counts)

Note on Telugu: corpus_telugu.txt dominates by volume. For Vizag-focused training, either downsample it or keep it as a separate stream so city knowledge is not diluted. Temporal files (2026 events) are secondary-source summaries — verify against official notices before redistribution.

Usage

python
from datasets import load_dataset
# text files, one per corpus; example: base + hard government tables
ds = load_dataset("text", data_files=[
    "corpus.txt", "corpus_chunks.txt", "corpus_hard.txt",
    "corpus_bus.txt", "corpus_layout.txt",
])
tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM-360M-Instruct")
ds = ds.map(lambda x: tok(x["text"], truncation=True, max_length=2048),
            batched=True, remove_columns=["text"])
# Trainer with standard causal-LM objective, then SFT on vizag-city-guide-sft

Sources & licensing

  • —Wikipedia + Wikivoyage extracts — CC BY-SA
  • —VMRDA / District Handbook / Smart Cities / AP Tourism / APSRTC timetables — Government Open Data License India (GODL) / public record, with attribution
  • —OpenStreetMap POIs — ODbL 1.0
  • —2026 news items — fair-use summaries with source URLs (see harvest log in the build repo)
  • —Synthetic layout/proximity notes — original

Citation

Vizag City Guide CPT corpus (2026). Built from Wikipedia (CC BY-SA), Government of India / Andhra Pradesh open data (GODL), and OpenStreetMap (ODbL).