s1hikha/premchand-148-stories-dataset

Premchand 148 Stories Dataset

A curated Hindi NLP corpus built from 148 short stories by Munshi Premchand, processed into sentence-aware chunks for RAG, embedding, and research pipelines.

[!NOTE] This work is for a PhD in University of Allahabad by Shikha Agrawal under the supervision of Dr. Pravin Kumar.

Dataset Structure

The repository contains the following data files optimized for direct usage in any retrieval or generation model:

• —corpus.jsonl: The final chunked corpus of 148 stories (3,144 records). Each chunk is roughly ~200 words, overlapping by 2 sentences to retain context.
• —qa.jsonl: The final validated QA dataset, useful for RAG generation and question answering tasks.
• —queries.jsonl: Benchmark queries for retrieval testing.
• —qrels.tsv: Ground truth relevance mapping for the retrieval queries.
• —fewshot_examples.jsonl: 18 diverse QA items used as few-shot prompt seeds during evaluation. Necessary for reproducing our generation baselines.

Baseline Metrics

We provide baseline results on this dataset using state-of-the-art models for both retrieval and generation tasks.

Retrieval Performance

Evaluated using intfloat/multilingual-e5-large:

• —Recall@5: 0.7570
• —Recall@10: 0.8418
• —MRR@10: 0.6897
• —nDCG@10: 0.6925

Generation Performance

Evaluated using google/gemma-4-26b-a4b-it across two modes (Gold Context vs. RAG-retrieved Context).

Gold Context (Upper Bound):

• —Exact Match (EM): 0.1331
• —F1 Score: 0.5207
• —ROUGE-L: 0.0172
• —LLM Judge (1-5): 4.0732
• —Abstention Rate: 66.67%

RAG Context:

• —Exact Match (EM): 0.0360
• —F1 Score: 0.4084
• —ROUGE-L: 0.0166
• —LLM Judge (1-5): 3.9608
• —Abstention Rate: 16.67%

Usage

You can load this dataset directly using the Hugging Face datasets library.

Loading the Corpus

from datasets import load_dataset

# Load the corpus chunks
corpus = load_dataset("json", data_files="corpus.jsonl", split="train")
print(corpus[0])

Loading the QA Pairs

from datasets import load_dataset

# Load the QA pairs
qa = load_dataset("json", data_files="qa.jsonl", split="train")
print(qa[0])

JSONL Schemas

Corpus (`corpus.jsonl`) Each line is a JSON record:

{
  "_id":         "aa_chunk_001",
  "story_id":    "aa",
  "chunk_index": 1,
  "num_chunks":  35,
  "title":       "गृह दाह",
  "source_file": "aa गृह दाह.txt",
  "word_count":  215,
  "char_count":  1088,
  "text":        "सत्यप्रकाश के जन्मोत्सव में..."
}

QA Pairs (`qa.jsonl` & `fewshot_examples.jsonl`) Each line contains the question, answer, and relevant chunk ID mappings for evaluation.

Credits & Acknowledgements

This dataset was created by Shikha Agrawal as part of her PhD research at the University of Allahabad, under the supervision of Dr. Pravin Kumar.