datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
textual_entailmentThe Textual Entailment dataset contains 800 pairs of Italian sentences, extracted from Wikipedia, and annotated for the presence of textual entailment. A pair of texts consists of T (for text) and H (hypothesis). Textual entailment is defined as a directional relationship between such pairs. The hypothesis must be fully entailed by the text.
The dataset has been created and used for the Textual Entailment Task (http://www.evalita.it/2009/tasks/te), organised as part of the EVALITA 2009… See the full description on the dataset page: https://huggingface.co/datasets/evalitahf/textual_entailment.textual_train_stackextextual_train_biodivTextual-Image-Caption-Dataset
Update: OCT-2023
Add v2 with recent SoTA model swinV2 classifier for both soft/hard-label visual_caption_cosine_score_v2 with person label (0.2, 0.3 and 0.4)
Introduction
Modern image captaining relies heavily on extracting knowledge, from images such as objects,
to capture the concept of static story in the image. In this paper, we propose a textual visual context dataset
for captioning, where the publicly available dataset COCO caption (Lin et al., 2014) has been… See the full description on the dataset page: https://huggingface.co/datasets/AhmedSSabir/Textual-Image-Caption-Dataset.textual_train_githubA-Dataset-for-Complex-Reasoning-over-Textual-Knowledge-Graphs-in-Medicine
RiTeK: Medical Textual Knowledge Graph QA Benchmark
RiTeK is a benchmark for complex reasoning over medical Textual Knowledge Graphs (medical TKGs). It evaluates whether retrieval systems and Large Language Models (LLMs) can answer realistic medical questions by using both relational paths and textual entity descriptions.
Dataset Overview
The benchmark contains three medical graph QA subsets:
Dataset
Directory
Splits
KG file
ADint
Adint
train / dev / test… See the full description on the dataset page: https://huggingface.co/datasets/ChenAI2015/A-Dataset-for-Complex-Reasoning-over-Textual-Knowledge-Graphs-in-Medicine.PubMed-Cancer-NLP-Textual-Dataset
PubMed-Cancer-NLP-Textual-Dataset
This dataset has been obtained from PubMed for research purposes. README will be updated with time.
Dataset Details
Dataset Description
It has multiple cancer samples with labels with their title and abstract from PubMed Repository.
Curated by: Om Aryan
Dataset Sources
Repository: https://pubmed.ncbi.nlm.nih.gov
Ar-En-Code-Switching-Textual-Dataset
ArE-CSTD: Arabic-English Code-Switching Textual Dataset
The National Center for Artificial Intelligence at the Saudi Data and Artificial Intelligence Authority (SDAIA), published the "ArE-CSTD" dataset, which stands for "Arabic-English Code-Switching Textual Dataset”.
This dataset contains 330K dialectical Arabic-English code-swithing sentences generated by the large language model GPT-4.
TXT Files
There are 6 txt files. 2 files for Modern Standard Arabic(MSA) train and… See the full description on the dataset page: https://huggingface.co/datasets/SDAIANCAI/Ar-En-Code-Switching-Textual-Dataset.disagree_textual_bayestextual_2018_mrmr30_datasetcatalan_textual_corpus
Dataset Card for Catalan Textual Corpus
Dataset Summary
The Catalan Textual Corpus is a 1760-million-token web corpus of Catalan built from several sources.
It consists of 1,758,388,896 tokens, 73,172,152 sentences, and 12,556,365 documents. Documents are separated by single new lines. These boundaries have been preserved as long as the license allowed it.
This work is licensed under a Creative Commons Attribution Share Alike 4.0 International license.… See the full description on the dataset page: https://huggingface.co/datasets/projecte-aina/catalan_textual_corpus.MapEval-Textual
MapEval-Textual
MapEval-Textual is created using MapQaTor.
Usage
from datasets import load_dataset
# Load dataset
ds = load_dataset("MapEval/MapEval-Textual", name="benchmark")
# Generate better prompts
for item in ds["test"]:
# Start with a clear task description
prompt = (
"You are a highly intelligent assistant. "
"Based on the given context, answer the multiple-choice question by selecting the correct option.\n\n"
"Context:\n" +… See the full description on the dataset page: https://huggingface.co/datasets/MapEval/MapEval-Textual.generalisation_evaluation_testsets_textualtextual_2017_mrmr30_featuresNon_textual_interactionRaahNaameh-1-textual-corpus
RaahNaameh-1 Textual Corpus
A large-scale Persian text corpus assembled for training the RaahNaameh-1 embedding model.
Sources
Source
Sentences
Description
Jomleh
1,002,221
Formal Persian web text
LSCP
10,257,866
Iranian tweets — colloquial, slang, emoji
Persian Wikipedia
1,107,618
Encyclopedic articles
Total
12,367,705
Processing
Light normalization only: Arabic→Persian character mapping, zero-width space removal
Emojis, Finglish… See the full description on the dataset page: https://huggingface.co/datasets/Reza2kn/RaahNaameh-1-textual-corpus.textual_train_c4xnli_fr_prompt_textual_entailment
xnli_fr_prompt_textual_entailment
Summary
xnli_fr_prompt_textual_entailment is a subset of the Dataset of French Prompts (DFP).It contains 8,804,444 rows that can be used for a textual entailment task.The original data (without prompts) comes from the dataset xnli by Conneau et al. where only the French part has been kept.A list of prompts (see below) was then applied in order to build the input and target columns and thus obtain the same format as the xP3 dataset by… See the full description on the dataset page: https://huggingface.co/datasets/CATIE-AQ/xnli_fr_prompt_textual_entailment.Thai-Semantic-Textual-Similarity-BenchmarkSentence representation plays a crucial role in NLP downstream tasks such as NLI, text classification, and STS. Recent sentence representation training techniques require NLI or STS datasets. However, there are no equivalent Thai NLI or STS datasets for sentence representation training.
To address this problem we provide the Thai sentence vector benchmark. We evaluate the Spearman correlation score of the sentence representations’ performance on Thai STS-B (translated version of STS-B).… See the full description on the dataset page: https://huggingface.co/datasets/mrp/Thai-Semantic-Textual-Similarity-Benchmark.Dataset_Automatic_Essay_Scoring_Essay-EssayScore_and_24_textual_featuresbisect_fr_prompt_textual_merging
bisect_fr_prompt_textual_merging
Summary
bisect_fr_prompt_textual_merging is a subset of the Dataset of French Prompts (DFP).It contains 10,383,891 rows that can be used for a textual fusion task.The original data (without prompts) comes from the dataset BiSECT by Kim et al. where only the French part has been kept.A list of prompts (see below) was then applied in order to build the input and target columns and thus obtain the same format as the xP3 dataset by Muennighoff… See the full description on the dataset page: https://huggingface.co/datasets/CATIE-AQ/bisect_fr_prompt_textual_merging.TextualismToolDictionariesLegalBenchClassification
TextualismToolDictionariesLegalBenchClassification
An MTEB dataset
Massive Text Embedding Benchmark
Determine if a paragraph from a judicial opinion is applying a form textualism that relies on the dictionary meaning of terms.
Task category
t2c
DomainsLegal, Written
Reference
https://huggingface.co/datasets/nguha/legalbench
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/TextualismToolDictionariesLegalBenchClassification.zetagpt-pretrain-gaelic-uccix_irish_textual_corpusbisect_fr_prompt_textual_simplification
bisect_fr_prompt_textual_simplification
Summary
bisect_fr_prompt_textual_simplification is a subset of the Dataset of French Prompts (DFP).It contains 9,889,420 rows that can be used for a textual simplification task.The original data (without prompts) comes from the dataset BiSECT by Kim et al. where only the French part has been kept.A list of prompts (see below) was then applied in order to build the input and target columns and thus obtain the same format as the xP3… See the full description on the dataset page: https://huggingface.co/datasets/CATIE-AQ/bisect_fr_prompt_textual_simplification.llm-pretrain-gaelic-uccix-irish-textual-corpusfever_fr_prompt_textual_entailment
fever_fr_prompt_textual_entailment
Summary
fever_fr_prompt_textual_entailment is a subset of the Dataset of French Prompts (DFP).It contains 550,000 rows that can be used for a textual entailment task.The original data (without prompts) comes from the dataset multilingual-NLI-26lang-2mil7 by Laurer et al. where only the fever French part has been kept.A list of prompts (see below) was then applied in order to build the input and target columns and thus obtain the same… See the full description on the dataset page: https://huggingface.co/datasets/CATIE-AQ/fever_fr_prompt_textual_entailment.dagverse-exampleTextualismToolPlainLegalBenchClassification
TextualismToolPlainLegalBenchClassification
An MTEB dataset
Massive Text Embedding Benchmark
Determine if a paragraph from a judicial opinion is applying a form textualism that relies on the ordinary (“plain”) meaning of terms.
Task category
t2c
DomainsLegal, Written
Reference
https://huggingface.co/datasets/nguha/legalbench
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/TextualismToolPlainLegalBenchClassification.anli_fr_prompt_textual_entailment
anli_fr_prompt_textual_entailment
Summary
anli_fr_prompt_textual_entailment is a subset of the Dataset of French Prompts (DFP).It contains 550,000 rows that can be used for a textual entailment task.The original data (without prompts) comes from the dataset multilingual-NLI-26lang-2mil7 by Laurer et al. where only the anli French part has been kept.A list of prompts (see below) was then applied in order to build the input and target columns and thus obtain the same format… See the full description on the dataset page: https://huggingface.co/datasets/CATIE-AQ/anli_fr_prompt_textual_entailment.cic-ids-2018-alldata-textual