Datasetpublic

freshstack/corpus-oct-2024

Dataset Card for FreshStack (Corpus) Homepage | Repository | Paper FreshStack is a holistic framework to construct challenging IR/RAG evaluation datasets that focuses on search across niche and recent topics. This dataset (October 2024) contains the query, nuggets, answers and nugget-level relevance judgments of 5 niche topics focused on software engineering and machine learning. The queries and answers (accepted) are taken from Stack Overflow, GPT-4o generates the nuggets and… See the full description on the dataset page: https://huggingface.co/datasets/freshstack/corpus-oct-2024.

sourceHugging Facecc-by-sa-4.0updated 1y agoView on Hugging Face

5likes6.1kdownloads

Dataset Card

Dataset Card for FreshStack (Corpus)

Homepage | Repository | Paper

FreshStack is a holistic framework to construct challenging IR/RAG evaluation datasets that focuses on search across niche and recent topics.

This dataset (October 2024) contains the query, nuggets, answers and nugget-level relevance judgments of 5 niche topics focused on software engineering and machine learning.

The queries and answers (accepted) are taken from Stack Overflow, GPT-4o generates the nuggets and labels the relevance between each nugget and a given document list.

This repository contains the corpus of GitHub chunked documents of five niche topics in freshstack. The queries, answers and nuggets can be found here.

Dataset Structure

To access the data using HuggingFace datasets:

topic='langchain'  # or any of the 5 topics
freshstack = datasets.load_dataset('freshstack/corpus-oct-2024', topic)

# train set
for data in freshstack['train']:
  doc_id = data['_id']
  doc_text = data['text']

Dataset Statistics

The following table contains the number of documents (#D) and the number of GitHub repositories used (#G) in the FreshStack collection.

Topic	Versions	Domain	Train
			#D	#G
langchain	-	Machine Learning	49,514	10
yolo	v7 & v8	Computer Vision	27,207	5
laravel	10 & 11	Back-end Development	52,351	9
angular	16, 17 & 18	Front-end Development	117,288	4
godot	4	Game Development	25,482	6

The following table contains the list of original GitHub repositories used to construct the following corpus for each topic.

Topic	GitHub Repositories
langchain	LangChain (Python), LangChain (JS), LangChain Next.js Template, Chroma Vector DB, OpenAI Cookbook, OpenAI Python Library, LlamaIndex, Azure OpenAI Samples, Azure Search OpenAI Demo, Hugging Face Transformers
yolo	Ultralytics YOLOv8, YOLOv8 Documentation, PyTorch Framework, YOLOv7 Implementation, OpenCV Library
laravel	Laravel Framework, Laravel Application Skeleton, Laravel Official Website, Laravel Documentation, Laravel Breeze (Auth Starter), Livewire (Full-stack Framework), PHP Language Source, PHP Official Documentation, PHP Website Source
angular	Angular Framework, Angular Components, Angular CLI, TypeScript Language
godot	Godot Engine, Godot Demo Projects, Godot Documentation, Godot Official Website, Learn GDScript with GDQuest, C# Language

License

The FreshStack datasets are provided under the CC-BY-SA 4.0 license.

The original GitHub repositories used for constructing the corpus may contain non-permissive licenses; we advise the reader to check the licenses for each repository carefully.

Citation

@misc{thakur2025freshstack,
      title={FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents}, 
      author={Nandan Thakur and Jimmy Lin and Sam Havens and Michael Carbin and Omar Khattab and Andrew Drozdov},
      year={2025},
      eprint={2504.13128},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2504.13128}, 
}