datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
include-base-44
INCLUDE-base (44 languages)
Dataset Description
Paper: http://arxiv.org/abs/2411.19799
Dataset Summary
INCLUDE is a comprehensive knowledge- and reasoning-centric benchmark across 44 languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.
It contains 22,637 4-option multiple-choice-questions (MCQ) extracted from academic and professional exams, covering 57 topics, including… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/include-base-44.include-lite-44
INCLUDE-lite (44 languages)
Dataset Description
Paper: http://arxiv.org/abs/2411.19799
Dataset Summary
INCLUDE is a comprehensive knowledge- and reasoning-centric benchmark across 44 languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.
It contains 11,095 4-option multiple-choice-questions (MCQ) extracted from academic and professional exams, covering 57 topics, including regional… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/include-lite-44.INCLUDE
Dataset Card for INCLUDE
Dataset Summary
This dataset contains all videos in the INCLUDE dataset. As huggingface does not support video uploads at this time, the HF dataset contains metadata about each video such as the parent class, the video class, the path to the video and whether its a part of the INCLUDE-50 dataset (use include_50==True to get only include_50 videos).
The videos themselves can be downloaded from Zenodo using the provided bash script.… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/INCLUDE.include_benchinclude_culturearxiv-chandra-ocr-2-include-images-first50-20260415
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-first50-20260415
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-first50-20260415
Source paper IDs in input list: 27,584
Processed IDs recorded in state/processed_ids.txt: 50
Successes: 50
Partial successes: 0
Errors: 0
Next shard index: 10
Updated at:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-first50-20260415.include-128include-esinclude-89restructured-include_base_44I do not hold the copyright to this dataset; I merely restructured it to have the same structure as other datasets (that we are researching) to facilitate future coding and analysis. I refer to this link for the raw dataset.
arxiv-chandra-ocr-2-include-images-demo-2604-08626-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0
Next shard index: 1
Updated at:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-20260416.arxiv-chandra-ocr-2-include-images-demo-2604-14148-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-14148-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-14148-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0
Next shard index: 1
Updated at:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-14148-20260416.arxiv-chandra-ocr-2-include-images-demo-2604-07429-retry-20260417
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-07429-retry-20260417
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-07429-retry-20260417
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0
Next shard index: 1
Updated at:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-07429-retry-20260417.Africa-Total-Reserves-includes-gold-current-USD
Africa Total Reserves includes gold current USD | Africa (World Bank)
Size category: n<1K - Formats: csv - Sector: economics_finance - Engineered by Electric Sheep Africa
TL;DR
This dataset is part of the Electric Sheep Africa catalog on Hugging Face. It is indexed for African data discovery with standardized metadata, loading guidance, provenance notes, and analyst-oriented context.
What This Dataset Covers
Public datasets help analysts inspect… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepafrica/Africa-Total-Reserves-includes-gold-current-USD.Arabic-Cohere-include-base-44-mmlu-style
The Refined Arabic Cohere INCLUDE Base 44 Dataset as MMLU-Style
Dataset Summary
INCLUDE is a comprehensive knowledge- and reasoning-centric benchmark spanning 44 languages that evaluates multilingual LLMs in the actual linguistic environments where they are deployed. The original dataset contains 22,637 4-option multiple-choice questions (MCQs) extracted from academic and professional exams, covering 57 topics, including regional knowledge.
When we reviewed the Arabic… See the full description on the dataset page: https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic-Cohere-include-base-44-mmlu-style.asia-owid-which-countries-include-malaria-vaccines-in-their-vaccination-schedules
Which Countries Include Malaria Vaccines In Their Vaccination Schedules | Asia (Our World in Data)
🌏 279 observations · 47 Asia countries · 2019–2024 · Repackaged by Electric Sheep Asia
TL;DR
This dataset contains 279 observations of Which Countries Include Malaria Vaccines In Their Vaccination Schedules data across 47 Asia countries, spanning 2019–2024.
About the source
Source: Our World in Data
Publisher: Our World in Data
License: cc-by-4.0… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepasia/asia-owid-which-countries-include-malaria-vaccines-in-their-vaccination-schedules.arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0
Next shard… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416.include-base-44arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0
Next shard index: 1… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-20260416.asia-owid-forms-of-homelessness-included-in-available-statistics
Forms Of Homelessness Included In Available Statistics | Asia (Our World in Data)
🌏 33 observations · 33 Asia countries · 2010–2024 · Repackaged by Electric Sheep Asia
TL;DR
This dataset contains 33 observations of Forms Of Homelessness Included In Available Statistics data across 33 Asia countries, spanning 2010–2024.
About the source
Source: Our World in Data
Publisher: Our World in Data
License: cc-by-4.0
Topic: Forms Of Homelessness… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepasia/asia-owid-forms-of-homelessness-included-in-available-statistics.africa-worldbank-wbl-legal-framework-entrepreneurship-the-law-includes-gender-responsive-procure
WBL: Legal Framework, Entrepreneurship, The law includes gender-responsive procurement provisions for public procurement processes | Africa (World Bank — Gender Statistics) | Africa (World Bank)
Size category: n<1K - Formats: parquet - Sector: governance_security - Engineered by Electric Sheep Africa
TL;DR
This dataset is part of the Electric Sheep Africa catalog on Hugging Face. It is indexed for African data discovery with standardized metadata, loading… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepafrica/africa-worldbank-wbl-legal-framework-entrepreneurship-the-law-includes-gender-responsive-procure.CLothing-include-Kimono_captioneurope-owid-which-countries-include-malaria-vaccines-in-their-vaccination-schedules
Which Countries Include Malaria Vaccines In Their Vaccination Schedules | Europe (Our World in Data)
🇪🇺 251 observations · 42 Europe countries · 2019–2024 · Repackaged by Electric Sheep Europe
TL;DR
This dataset contains 251 observations of Which Countries Include Malaria Vaccines In Their Vaccination Schedules data across 42 Europe countries, spanning 2019–2024.
About the source
Source: Our World in Data
Publisher: Our World in Data
License:… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepeurope/europe-owid-which-countries-include-malaria-vaccines-in-their-vaccination-schedules.include50_mediapipebslm-entity-extraction-mt5-base-include-desc-normalized-tr243k-resultsarxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0
Errors: 0… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-20260416.include-base-44-augAW_finetuning_5core_split1_all_final_valid_include_50_toysthree_pos_pickup_39_include_risk_interventionarxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-v2-20260416
arXiv OCR with Chandra OCR 2
This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.
Summary
Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-v2-20260416
Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-v2-20260416
Source paper IDs in input list: 1
Processed IDs recorded in state/processed_ids.txt: 1
Successes: 1
Partial successes: 0… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-duplicate-caption-fix-v2-20260416.