datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
piqaBabelDOC-Assets
BabelDOC-Assets
Font and other resource files relied on by BabelDOC and pdf2zh
BabelDOC is a PDF translation library, pdf2zh is a PDF translation tool.
Fonts and Licenses
Go Noto Universal: THE UNLICENSE
Pal3love/Source-Han-TrueType: SIL OPEN FONT LICENSE Version 1.1
lxgw/LxgwWenKaiGB: OFL-1.1 License
lxgw/LxgwWenkaiTC: OFL-1.1 License
fontworks-fonts/Klee: OFL-1.1 License
fonts-archive/MaruBuri: License
Noto Serif/Noto Sans: SIL OPEN FONT LICENSE Version 1.1… See the full description on the dataset page: https://huggingface.co/datasets/awwaawwa/BabelDOC-Assets.paul_graham_essaysmultilingual_mmluMMLU professionally translated into 14 languages using professional human translators, sourced from OpenAI's simple-eval.
Original files:
english: https://openaipublic.blob.core.windows.net/simple-evals/mmlu.csv
multilingual: https://openaipublic.blob.core.windows.net/simple-evals/mmlu_{language}.csv where language one of "AR-XY", "BN-BD", "DE-DE", "ES-LA", "FR-FR", "HI-IN", "ID-ID", "IT-IT", "JA-JP", "KO-KR", "PT-BR", "ZH-CN", "SW-KE", "YO-NG", "EN-US"
global-piqa-evalslogiqa2The dataset is an amendment and re-annotation of LogiQA in 2020, a large-scale logical reasoning reading comprehension dataset adapted from the Chinese Civil Service Examination. We increase the data size, refine the texts with manual translation by professionals, and improve the quality by removing items with distinctive cultural features like Chinese idioms. Furthermore, we conduct a fine-grained annotation on the dataset and turn it into a two-way natural language inference (NLI) task, resulting in 35k premise-hypothesis pairs with gold labels, making it the first large-scale NLI dataset for complex logical reasoningUSPTOwikineural
Dataset Card for WikiNEuRal dataset
Description
Summary: In a nutshell, WikiNEuRal consists in a novel technique which builds upon a multilingual lexical knowledge base (i.e., BabelNet) and transformer-based architectures (i.e., BERT) to produce high-quality annotations for multilingual NER. It shows consistent improvements of up to 6 span-based F1-score points against state-of-the-art alternative data production methods on common benchmarks for NER. We used this… See the full description on the dataset page: https://huggingface.co/datasets/Babelscape/wikineural.gtbabeldoc-temp-pdfsmultinerd
Dataset Card for MultiNERD dataset
Description
Summary: In a nutshell, MultiNERD is the first language-agnostic methodology for automatically creating multilingual, multi-genre and fine-grained annotations for Named Entity Recognition and Entity Disambiguation. Specifically, it can be seen an extension of the combination of two prior works from our research group that are WikiNEuRal, from which we took inspiration for the state-of-the-art silver-data creation methodology… See the full description on the dataset page: https://huggingface.co/datasets/Babelscape/multinerd.uspto_rawsocial_i_qaMirror of Social IQa in parquet format. For details, refer to the original dataset
global-piqa-generationsALERT
Dataset Card for the ALERT Benchmark
Description
Paper Summary: When building Large Language Models (LLMs), it is paramount to bear safety in mind and protect them with guardrails. Indeed, LLMs should never generate content promoting or normalizing harmful, illegal, or unethical behavior that may contribute to harm to individuals or society. In response to this critical challenge, we introduce ALERT, a large-scale benchmark to assess the safety of LLMs through red… See the full description on the dataset page: https://huggingface.co/datasets/Babelscape/ALERT.WOPTO
World Patent Applications Bibliographic Data
The dataset has been exported from Google Patents Public Data
. Includes the bibliographic information, titles and abstracts from worldwide patent applications excluding the United States. The US full-text patent applications are available here.
There are ~100M abstracts in English. Excluding that, the top 10 language counts are:
abstract_language
count
zh
6,450,567
fr
4,681,015
ko
4,306,533
ja
3,437,958
de
2,121,303… See the full description on the dataset page: https://huggingface.co/datasets/baber/WOPTO.mmlu_testSREDFMRelation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English. \In this paper, we address the above issue and provide two new resources that enable the training and evaluation of multilingual RE systems.
First, we present SRED\textsuperscript{FM}, an automatically annotated dataset covering 18 languages, 400 relation types, 13 entity types, totaling more than 40 million triplet instances. Second, we propose RED\textsuperscript{FM}, a smaller, human-revised dataset for seven languages that allows for the evaluation of multilingual RE systems.
To demonstrate the utility of these novel datasets, we experiment with the first end-to-end multilingual RE model, mREBEL,
that extracts triplets, including entity types, in multiple languages. We release our resources and model checkpoints at \href{https://www.github.com/babelscape/rebel}{https://www.github.com/babelscape/rebel}.rebel-datasetREBEL is a silver dataset created for the paper REBEL: Relation Extraction By End-to-end Language generationbabel-briefings
Babel Briefings News Headlines Dataset README
Break Free from the Language Barrier
Version: 1 - Date: 30 Oct 2023
Collected and Prepared by Felix Leeb (Max Planck Institute for Intelligent Systems, Tübingen, Germany)
License: Babel Briefings Headlines Dataset © 2023 by Felix Leeb is licensed under CC BY-NC-SA 4.0
Check out our paper on arxiv.
This dataset contains 4,719,199 news headlines across 30 different languages collected between 8 August 2020 and 29 November 2021. The… See the full description on the dataset page: https://huggingface.co/datasets/felixludos/babel-briefings.agieval
Dataset Card for AGIEval
Dataset Summary
AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving. This benchmark is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., Chinese College Entrance Exam (Gaokao) and American SAT), law school… See the full description on the dataset page: https://huggingface.co/datasets/baber/agieval.hendrycks_mathMATH is a dataset of 12,500 challenging competition mathematics problems. Each
problem in Math has a full step-by-step solution which can be used to teach
models to generate answer derivations and explanations.BABEmmluThis is a massive multitask test consisting of multiple-choice questions from various branches of knowledge, covering 57 tasks including elementary mathematics, US history, computer science, law, and more.REDFMRelation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English. \In this paper, we address the above issue and provide two new resources that enable the training and evaluation of multilingual RE systems.
First, we present SRED\textsuperscript{FM}, an automatically annotated dataset covering 18 languages, 400 relation types, 13 entity types, totaling more than 40 million triplet instances. Second, we propose RED\textsuperscript{FM}, a smaller, human-revised dataset for seven languages that allows for the evaluation of multilingual RE systems.
To demonstrate the utility of these novel datasets, we experiment with the first end-to-end multilingual RE model, mREBEL,
that extracts triplets, including entity types, in multiple languages. We release our resources and model checkpoints at \href{https://www.github.com/babelscape/rebel}{https://www.github.com/babelscape/rebel}.babel-official
BABEL Official Local Layout
This directory is a cleaned local mirror of the official BABEL v1.0 labels and
the AMASS subsets required by those labels.
Layout
archives/: original downloaded archives, kept unchanged for provenance.
labels/babel_v1.0_release/: official BABEL JSON splits.
amass/: extracted AMASS motion parameter files.
processed/manifests/*.jsonl: normalized records with resolved local AMASS
paths, frame counts, fps, segment labels, and rewritten… See the full description on the dataset page: https://huggingface.co/datasets/ZeyuLing/babel-official.ltx-2.5-ic-lora-cartoon-dataset
LTX-2.5 IC-LoRA cartoon dataset — 27 paired clips
Training data for
Baberg/ltx-2.5-22b-ic-lora-cel-character.
A fully synthetic dataset, created from scratch — nothing filmed or scraped. Cinematic
live-action clips were generated, each built around one clearly identifiable central character.
For each clip the first frame was extracted, the central character in it was replaced with a 2D
cel-animated version of the same character in the same pose and framing, and the clip was then… See the full description on the dataset page: https://huggingface.co/datasets/Baberg/ltx-2.5-ic-lora-cartoon-dataset.dolma-cccc-filteredThis dataset is a curated subset of the dolma-cccc dataset. It contains data from 537 URLs, which were selected from the 1000 most common URLs in the original dataset and manually verified for permissive licenses. For details, refer to the original.
canadian_hansard_backuppdbooks