datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wiki40b
Dataset Card for "wiki40b"
Dataset Summary
Clean-up text for 40+ Wikipedia languages editions of pages
correspond to entities. The datasets have train/dev/test splits per language.
The dataset is cleaned up by page filtering to remove disambiguation pages,
redirect pages, deleted pages, and non-entity pages. Each example contains the
wikidata id of the entity, and the full Wikipedia article after page processing
that removes non-content sections and structured objects.… See the full description on the dataset page: https://huggingface.co/datasets/google/wiki40b.InfiMM-WebMath-40B
InfiMM-WebMath-40B Dataset
ArXiv| PDF
This dataset is also discussed in the survey paper A Survey of Deep Learning for Geometry Problem Solving.
The accompanying reading list/code for the survey can be found at: https://github.com/majianz/gps-survey
InfiMM-WebMath-40B is a large-scale, open-source multimodal dataset specifically designed for mathematical reasoning tasks. It incorporates both text and images, extracted from web documents, to advance the pre-training of Multimodal… See the full description on the dataset page: https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.climbmix-40b-az
ClimbMix 40B — Azerbaijani
A large-scale Azerbaijani text dataset created by translating the English karpathy/climbmix-400b-shuffle dataset into Azerbaijani using Google Translate.
Dataset Summary
This dataset contains approximately 40 billion tokens of Azerbaijani text, making it one of the largest publicly available Azerbaijani language corpora. It is intended for pretraining and fine-tuning large language models (LLMs) for the Azerbaijani language.
Property
Value… See the full description on the dataset page: https://huggingface.co/datasets/LocalDoc/climbmix-40b-az.wiki40b_jaThis dataset is a reformatted version of the Japanese portion of wiki40b dataset.
When you use this dataset, please cite the original paper:
@inproceedings{guo-etal-2020-wiki,
title = "{W}iki-40{B}: Multilingual Language Model Dataset",
author = "Guo, Mandy and
Dai, Zihang and
Vrande{\v{c}}i{\'c}, Denny and
Al-Rfou, Rami",
booktitle = "Proceedings of the Twelfth Language Resources and Evaluation Conference",
month = may,
year = "2020",
address =… See the full description on the dataset page: https://huggingface.co/datasets/fn-aka-mur/wiki40b_ja.wiki40b-ja
range3/wiki40b-ja
This dataset consists of three parquet files from the wiki40b dataset with only Japanese data extracted. It is generated by the following python code.
このデータセットは、wiki40bデータセットの日本語データのみを抽出した3つのparquetファイルで構成されます。以下のpythonコードによって生成しています。
import datasets
dss = datasets.load_dataset(
"wiki40b",
"ja",
beam_runner="DirectRunner",
)
for split,ds in dss.items():
ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")
wiki40b_lines_en40B_dataset_W_fixeddetails_tiiuae__falcon-40b
Dataset Card for Evaluation run of tiiuae/falcon-40b
Dataset automatically created during the evaluation run of model tiiuae/falcon-40b.
The dataset is composed of 136 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional configuration… See the full description on the dataset page: https://huggingface.co/datasets/OALL/details_tiiuae__falcon-40b.wiki40b_lines_estiiuae__falcon-40b-details
Dataset Card for Evaluation run of tiiuae/falcon-40b
Dataset automatically created during the evaluation run of model tiiuae/falcon-40b
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/tiiuae__falcon-40b-details.wiki40b-da
Dataset Card for "wiki40b-da"
Dataset Summary
This dataset is an upload of the Danish part of the Wiki40b dataset, being a cleaned version of a dump of Wikipedia.
The dataset is identical in content to this dataset on the Hugging Face Hub, but that one requires both apache_beam, tensorflow and mwparserfromhell, which can lead to dependency issues since these are not compatible with several newer packages.
The training, validation and test splits are the original ones.… See the full description on the dataset page: https://huggingface.co/datasets/alexandrainst/wiki40b-da.wiki40b_en_100_0_split
Dataset Card for "wiki40b_en_100_0_split"
More Information needed
tiiuae__falcon-40b-instruct-details
Dataset Card for Evaluation run of tiiuae/falcon-40b-instruct
Dataset automatically created during the evaluation run of model tiiuae/falcon-40b-instruct
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/tiiuae__falcon-40b-instruct-details.dclm_baseline_1.0_40btwiki40b-da-clean
Dataset Card for "wiki40b-da-clean"
Dataset Summary
This dataset is an slightly modified and filtered version of Wiki40b-da daset which is a fork of this dataset on the Hugging Face Hub.
The dataset contains two sub-sets, for which the original columns "wikidata_id" and "version_id" are removed from both:
"text": Contains the filtered text of the Wikipedia paragraphs, with formatting removed (START_ARTICLE, START_PARAGRAPH and \n removed)
"sentences" Contains the… See the full description on the dataset page: https://huggingface.co/datasets/jealk/wiki40b-da-clean.AI-Sweden-Models__gpt-sw3-40b-details
Dataset Card for Evaluation run of AI-Sweden-Models/gpt-sw3-40b
Dataset automatically created during the evaluation run of model AI-Sweden-Models/gpt-sw3-40b
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/AI-Sweden-Models__gpt-sw3-40b-details.infimm-webmath-40b_urls
Dataset Card for infimm-webmath-40b_urls
This dataset provides the URLs and top-level domains associated with training records in Infi-MM/InfiMM-WebMath-40B. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/infimm-webmath-40b_urls.wiki40b_binidxLaMini-LM-dataset-TheBloke-h2ogpt-falcon-40b-v2-GGML-eval-llama2
Dataset Card for "LaMini-LM-dataset-TheBloke-h2ogpt-falcon-40b-v2-GGML"
More Information needed
wiki40b_it_test_1k_instances_processed_keep_titleniv2_explanation_targets_h2ogpt-gm-oasst1-en-2048-falcon-40b-v2-GGML
Dataset Card for "niv2_explanation_targets_h2ogpt-gm-oasst1-en-2048-falcon-40b-v2-GGML"
More Information needed
tiiuae__falcon-40b-instructwiki40b_fr_test_1k_instances_processedwiki40b_ja_test_1k_instances_processed_keep_titlewiki40b
アクセス方法
!pip install datasets
from datasets import load_dataset
dataset = load_dataset("geniacllm/wiki40b")
jsonlへの変換
import json
with open('your_dataset.jsonl', 'w', encoding='utf-8') as f:
# 'train'はデータセットの特定のスプリットです。使用しているデータセットに合わせて調整してください。
for example in dataset['train']:
# JSON Lines形式では、各データポイントをJSON文字列に変換し、新しい行に書き出します。
json_line = json.dumps(example) + "\n"
f.write(json_line)
データセット情報
dataset_info:
features:
name: text… See the full description on the dataset page: https://huggingface.co/datasets/geniacllm/wiki40b.wiki40b_it_test_1k_instances_processedwiki40b-zhtwafrica-uganda-ppi-m-u-excel-tables-up-to-feb-2023-40bd3b2d
Ppi M U Excel Tables Up to Feb 2023 | Africa (Uganda Bureau of Statistics)
180 rows - 1 Africa country/area - 2023 - source table - Engineered by Electric Sheep Africa
TL;DR
This dataset contains 180 rows from Uganda Bureau of Statistics, covering Ppi M U Excel Tables Up to Feb 2023. It is published as ML-ready Parquet with consistent Hugging Face metadata, source provenance, and analysis-friendly loading examples.
What This Dataset Measures… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepafrica/africa-uganda-ppi-m-u-excel-tables-up-to-feb-2023-40bd3b2d.wiki40b_en_test_1k_instances_processedwiki40b_zh-cn_test_1k_instances_processed