datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
transformers_circleci_workflow_runstransformers.js-docstransformersstsb
Dataset Card for STSB
The Semantic Textual Similarity Benchmark (Cer et al., 2017) is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data.
Each pair is human-annotated with a similarity score from 1 to 5. However, for this variant, the similarity scores are normalized to between 0 and 1.
Dataset Details
Columns: "sentence1", "sentence2", "score"
Column types: str, str, float
Examples:{
'sentence1': 'A… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/stsb.transformers-pr
Transformers PR Dataset
Normalized snapshots of issues, pull requests, comments, reviews, and linkage data from huggingface/transformers.
Files:
issues.parquet
pull_requests.parquet
comments.parquet
issue_comments.parquet (derived view of issue discussion comments)
pr_comments.parquet (derived view of pull request discussion comments)
reviews.parquet
pr_files.parquet
pr_diffs.parquet
review_comments.parquet
links.parquet
events.parquet
new_contributors.parquet… See the full description on the dataset page: https://huggingface.co/datasets/evalstate/transformers-pr.nucleotide_transformer_downstream_tasks
Dataset Card for Dataset Name
The nucleotide_transformer_downstream_tasks dataset features the 18 downstream tasks presented in the Nucleotide Transformer paper. They consist of both binary and multi-class classification tasks that aim at providing a consistent genomics benchmark.
⚠️We note that we have revised and improved our benchmark during the peer-review process. The datasets featured in this repository are used up to this release. We highly encourage to move to the new… See the full description on the dataset page: https://huggingface.co/datasets/InstaDeepAI/nucleotide_transformer_downstream_tasks.parallel-sentences-ccmatrix
Dataset Card for Parallel Sentences - CCMatrix
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. The texts originate from the CCMatrix dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse
parallel-sentences-jw300
parallel-sentences-news-commentary… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-ccmatrix.miracl
Dataset Card for MIRACL
This is a reformatting of the MIRACL dataset used to train the BGE-M3 model. See the full BGE-M3 dataset in Shitao/bge-m3-data.
Dataset Subsets
...-triplet subset
Columns: "anchor", "positive", "negative"
Column types: str, str, str
Examples:{
'anchor': '月球到地球的距离是多少？',
'positive': '月球距離\n月球距離 (LD) 是天文學上從地球到月球的距離，從地球到月球的平均距離是384,401公里 (238,856英里)。因為月球在橢圓軌道上運動，實際的距離隨時都在變化著。',
'negative':… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/miracl.parallel-sentences-talks
Dataset Card for Parallel Sentences - Talks
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Talks dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-talks.transformers_flash_attn_ciNanoBEIR-entransformers_daily_ciparallel-sentences-opensubtitles
Dataset Card for Parallel Sentences - OpenSubtitles
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the OpenSubtitles dataset.
Warning! The quality of this dataset is not great; many of the english and non-english texts don't match well, or are fully empty.
Related Datasets
The following… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-opensubtitles.splitted_PretrainGiftEvals2orc
Dataset Card for S2ORC
This dataset contains titles, abstracts, and citations from scientific papers from the Semantic Scholar Open Research Corpus (S2ORC).
This dataset can and has been used to train embedding models, and works out of the box to train or finetune Sentence Transformer models.
In our experiments, title-abstract pairs result in the highest performance, followed by titles-citations and then abstract-citations pairs.
Dataset Subsets… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/s2orc.transformers-synthetic-assets
transformers-synthetic-assets
Synthetic media fixtures for Transformers tests. These assets are generated from prompts or deterministic code and are not derived from third-party source files.
rdt-ft-data
Dataset Card
This is the fine-tuning dataset used in the paper RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation.
Source
Project Page: https://rdt-robotics.github.io/rdt-robotics/
Paper: https://arxiv.org/pdf/2410.07864
Code: https://github.com/thu-ml/RoboticsDiffusionTransformer
Model: https://huggingface.co/robotics-diffusion-transformer/rdt-1b
Uses
Download all archive files and use the following command to extract:
cat rdt_data.tar.gz.* |… See the full description on the dataset page: https://huggingface.co/datasets/robotics-diffusion-transformer/rdt-ft-data.parallel-sentences-tatoeba
Dataset Card for Parallel Sentences - Tatoeba
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Tatoeba dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-tatoeba.transformers-pr-slop-dataset
Transformers PR Slop Dataset
Normalized snapshots of issues, pull requests, comments, reviews, and linkage data from huggingface/transformers.
Files:
issues.parquet
pull_requests.parquet
comments.parquet
issue_comments.parquet (derived view of issue discussion comments)
pr_comments.parquet (derived view of pull request discussion comments)
pr_files.parquet
pr_diffs.parquet
reviews.parquet
review_comments.parquet
links.parquet
events.parquet
Use:
duplicate PR and issue analysis… See the full description on the dataset page: https://huggingface.co/datasets/burtenshaw/transformers-pr-slop-dataset.eli5
Dataset Card for ELI5
This dataset is a collection of question-answer pairs, collected from the Explain Like I'm 5 subreddit. See ELI5 for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "question", "answer"
Column types: str, str
Examples:{
'question': 'Why chemical weapons considered more indiscriminate than conventional weapons?',
'answer': "Well, any… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/eli5.all-nli
Dataset Card for AllNLI
This dataset is a concatenation of the SNLI and MultiNLI datasets.
Despite originally being intended for Natural Language Inference (NLI), this dataset can be used for training/finetuning an embedding model for semantic textual similarity.
Dataset Subsets
pair-class subset
Columns: "premise", "hypothesis", "label"
Column types: str, str, class with {"0": "entailment", "1": "neutral", "2", "contradiction"}
Examples:{
'premise': 'A… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/all-nli.nucleotide_transformer_downstream_tasks_revised
Dataset Card for Dataset Name
The nucleotide_transformer_downstream_tasks dataset features the 18 downstream tasks presented in the Nucleotide Transformer paper. They consist of both binary and multi-class classification tasks that aim at providing a consistent genomics benchmark.
We note that this is an updated version of this benchmark after the paper has been through peer-review. We highly encourage to move to this version in detriment of the older version.Keypoints about the… See the full description on the dataset page: https://huggingface.co/datasets/InstaDeepAI/nucleotide_transformer_downstream_tasks_revised.parallel-sentences-wikimatrix
Dataset Card for Parallel Sentences - WikiMatrix
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the WikiMatrix dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-wikimatrix.msmarco-distilbert-margin-mse-mean-dot-v1
MS MARCO with hard negatives from distilbert-margin-mse-mean-dot-v1
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using the Bing search engine.
For each query and gold positive passage, the 50 most similar paragraphs were mined using 13 different models. The resulting data can be used to train Sentence Transformer models.
Related Datasets
These are the datasets generated using the 13 different models:… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-distilbert-margin-mse-mean-dot-v1.natural-questions
Dataset Card for Natural Questions
This dataset is a collection of question-answer pairs from the Natural Questions dataset. See Natural Questions for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "question", "answer"
Column types: str, str
Examples:{
'query': 'the si unit of the electric field is',
'answer': 'Electric field An electric field is a field… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/natural-questions.msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1
MS MARCO with hard negatives from co-condenser-margin-mse-sym-mnrl-mean-v1
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using the Bing search engine.
For each query and gold positive passage, the 50 most similar paragraphs were mined using 13 different models. The resulting data can be used to train Sentence Transformer models.
Related Datasets
These are the datasets generated using the 13 different models:… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1.transformers-metadata
Transformers metadata
transformers_image_doctransformers_daily_cimsmarco-bm25
MS MARCO with hard negatives from bm25
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using the Bing search engine.
For each query and gold positive passage, the 50 most similar paragraphs were mined using 13 different models. The resulting data can be used to train Sentence Transformer models.
Related Datasets
These are the datasets generated using the 13 different models:
msmarco-bm25… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-bm25.