freshstack/corpus-oct-2024

1---2license: cc-by-sa-4.03size_categories:4- 10K<n<100K5task_categories:6- question-answering7dataset_info:8- config_name: angular9  features:10  - name: _id11    dtype: string12  - name: text13    dtype: string14  - name: metadata15    struct:16    - name: end_byte17      dtype: int6418    - name: start_byte19      dtype: int6420    - name: url21      dtype: string22  splits:23  - name: train24    num_bytes: 36101656025    num_examples: 11728826  download_size: 7189218427  dataset_size: 36101656028- config_name: godot29  features:30  - name: _id31    dtype: string32  - name: text33    dtype: string34  - name: metadata35    struct:36    - name: end_byte37      dtype: int6438    - name: start_byte39      dtype: int6440    - name: url41      dtype: string42  splits:43  - name: train44    num_bytes: 10398204645    num_examples: 2548246  download_size: 3342237647  dataset_size: 10398204648- config_name: langchain49  features:50  - name: _id51    dtype: string52  - name: text53    dtype: string54  - name: metadata55    struct:56    - name: end_byte57      dtype: int6458    - name: start_byte59      dtype: int6460    - name: url61      dtype: string62  splits:63  - name: train64    num_bytes: 21583869165    num_examples: 4951466  download_size: 7093037467  dataset_size: 21583869168- config_name: laravel69  features:70  - name: _id71    dtype: string72  - name: text73    dtype: string74  - name: metadata75    struct:76    - name: end_byte77      dtype: int6478    - name: start_byte79      dtype: int6480    - name: url81      dtype: string82  splits:83  - name: train84    num_bytes: 14163414885    num_examples: 5235186  download_size: 3952581087  dataset_size: 14163414888- config_name: yolo89  features:90  - name: _id91    dtype: string92  - name: text93    dtype: string94  - name: metadata95    struct:96    - name: end_byte97      dtype: int6498    - name: start_byte99      dtype: int64100    - name: url101      dtype: string102  splits:103  - name: train104    num_bytes: 134629730105    num_examples: 27207106  download_size: 42140310107  dataset_size: 134629730108configs:109- config_name: angular110  data_files:111  - split: train112    path: angular/train-*113- config_name: godot114  data_files:115  - split: train116    path: godot/train-*117- config_name: langchain118  data_files:119  - split: train120    path: langchain/train-*121- config_name: laravel122  data_files:123  - split: train124    path: laravel/train-*125- config_name: yolo126  data_files:127  - split: train128    path: yolo/train-*129---130 131# Dataset Card for FreshStack (Corpus)132 133[Homepage](https://fresh-stack.github.io) | 134[Repository](https://github.com/fresh-stack/freshstack) |135[Paper](https://arxiv.org/abs/2504.13128)136 137FreshStack is a holistic framework to construct challenging IR/RAG evaluation datasets that focuses on search across niche and recent topics.138 139This dataset (October 2024) contains the query, nuggets, answers and nugget-level relevance judgments of 5 niche topics focused on software engineering and machine learning. 140 141The queries and answers (accepted) are taken from Stack Overflow, GPT-4o generates the nuggets and labels the relevance between each nugget and a given document list.142 143This repository contains the corpus of GitHub chunked documents of five niche topics in freshstack. The queries, answers and nuggets can be found [here](https://huggingface.co/datasets/freshstack/queries-oct-2024).144 145## Dataset Structure146 147To access the data using HuggingFace `datasets`:148```149topic='langchain'  # or any of the 5 topics150freshstack = datasets.load_dataset('freshstack/corpus-oct-2024', topic)151 152# train set153for data in freshstack['train']:154  doc_id = data['_id']155  doc_text = data['text'] 156```157 158## Dataset Statistics 159The following table contains the number of documents (`#D`) and the number of GitHub repositories used (`#G`) in the FreshStack collection.160 161| Topic | Versions | Domain | Train |        |162|:----:|:-----:|:-----:|:-----:|:------:|163|            |     |     | **#D**| **#G** |164|  langchain | -   |Machine Learning     | 49,514  | 10 |165|  yolo      | v7 & v8 | Computer Vision      | 27,207  | 5  |166|  laravel   | 10 & 11 | Back-end Development | 52,351  | 9  |167|  angular   | 16, 17 & 18 | Front-end Development| 117,288 | 4  |168|  godot     | 4 | Game Development     | 25,482  | 6  |169 170The following table contains the list of original GitHub repositories used to construct the following corpus for each topic.171 172| Topic | GitHub Repositories |173|:----:|:-----|174|  langchain | [LangChain (Python)](https://github.com/langchain-ai/langchain), [LangChain (JS)](https://github.com/langchain-ai/langchainjs), [LangChain Next.js Template](https://github.com/langchain-ai/langchain-nextjs-template), [Chroma Vector DB](https://github.com/chroma-core/chroma), [OpenAI Cookbook](https://github.com/openai/openai-cookbook), [OpenAI Python Library](https://github.com/openai/openai-python), [LlamaIndex](https://github.com/run-llama/llama_index), [Azure OpenAI Samples](https://github.com/Azure-Samples/openai), [Azure Search OpenAI Demo](https://github.com/Azure-Samples/azure-search-openai-demo), [Hugging Face Transformers](https://github.com/huggingface/transformers) |175|  yolo      | [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics), [YOLOv8 Documentation](https://github.com/ultralytics/docs), [PyTorch Framework](https://github.com/pytorch/pytorch), [YOLOv7 Implementation](https://github.com/WongKinYiu/yolov7), [OpenCV Library](https://github.com/opencv/opencv) |176|  laravel   | [Laravel Framework](https://github.com/laravel/framework), [Laravel Application Skeleton](https://github.com/laravel/laravel), [Laravel Official Website](https://github.com/laravel/laravel.com), [Laravel Documentation](https://github.com/laravel/docs), [Laravel Breeze (Auth Starter)](https://github.com/laravel/breeze), [Livewire (Full-stack Framework)](https://github.com/livewire/livewire), [PHP Language Source](https://github.com/php/php-src), [PHP Official Documentation](https://github.com/php/doc-en), [PHP Website Source](https://github.com/php/web-php) |177|  angular   | [Angular Framework](https://github.com/angular/angular), [Angular Components](https://github.com/angular/components), [Angular CLI](https://github.com/angular/angular-cli), [TypeScript Language](https://github.com/microsoft/TypeScript) |178|  godot     | [Godot Engine](https://github.com/godotengine/godot), [Godot Demo Projects](https://github.com/godotengine/godot-demo-projects), [Godot Documentation](https://github.com/godotengine/godot-docs), [Godot Official Website](https://github.com/godotengine/godot-website), [Learn GDScript with GDQuest](https://github.com/GDQuest/learn-gdscript), [C# Language](https://github.com/dotnet/csharplang) |179 180## License181 182The FreshStack datasets are provided under the CC-BY-SA 4.0 license.183 184> The original GitHub repositories used for constructing the corpus may contain non-permissive licenses; we advise the reader to check the licenses for each repository carefully.185 186## Citation187 188```189@misc{thakur2025freshstack,190      title={FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents}, 191      author={Nandan Thakur and Jimmy Lin and Sam Havens and Michael Carbin and Omar Khattab and Andrew Drozdov},192      year={2025},193      eprint={2504.13128},194      archivePrefix={arXiv},195      primaryClass={cs.IR},196      url={https://arxiv.org/abs/2504.13128}, 197}198```