datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
video-vec2wav2-tokenizer
video-vec2wav2-tokenizer
Production-ready pipeline (Python package video_vec2wav2_tokenizer, CLI command
video2dataset) that turns a folder of videos into clean AI training datasets
for speech recognition (ASR) and text-to-speech (TTS).
videos ──► audio (16 kHz mono PCM) ──► whisper transcript ──► clips ──► metadata.csv / dataset.jsonl / tts_metadata.csv / report.json
Video processing — recursive scan of mp4 / mkv / avi / mov / webm, FFmpeg
audio extraction to mono ·… See the full description on the dataset page: https://huggingface.co/datasets/k9cli/video-vec2wav2-tokenizer.jat-dataset-tokenized
Dataset Card for "jat-dataset-tokenized"
More Information needed
video-vec2wav2-tokenizer-2
video-vec2wav2-tokenizer-2
Version 2 - continuation shard of the video-to-AI-dataset tokenizer project.
Production-ready pipeline (Python package video_vec2wav2_tokenizer, CLI command
video2dataset) that turns a folder of videos into clean AI training datasets
for speech recognition (ASR) and text-to-speech (TTS).
videos ──► audio (16 kHz mono PCM) ──► whisper transcript ──► clips ──► metadata.csv / dataset.jsonl / tts_metadata.csv / report.json
Video processing —… See the full description on the dataset page: https://huggingface.co/datasets/k9cli/video-vec2wav2-tokenizer-2.video-vec2wav2-tokenizer-3
video-vec2wav2-tokenizer-3
Version 3 - continuation shard of the video-to-AI-dataset tokenizer project.
Version 2 - continuation shard of the video-to-AI-dataset tokenizer project.
Production-ready pipeline (Python package video_vec2wav2_tokenizer, CLI command
video2dataset) that turns a folder of videos into clean AI training datasets
for speech recognition (ASR) and text-to-speech (TTS).
videos ──► audio (16 kHz mono PCM) ──► whisper transcript ──► clips ──►… See the full description on the dataset page: https://huggingface.co/datasets/k9cli/video-vec2wav2-tokenizer-3.jora_corpus1_tokenized_128kfineweb-tokenized
FineWeb Tokenized
> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer
What is it?
This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.
By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.tokenizer-wiki-bench
Multilingual Tokenizer Benchmark
This dataset includes pre-processed wikipedia data for tokenizer evaluation in 45 languages. We provide more information on the evaluation task in general this blogpost.
Usage
The dataset allows us to easily calculate tokenizer fertility and the proportion of continued words on any of the supported languages. In the example below we take the Mistral tokenizer and evaluate its performance on Slovak.
from transformers import AutoTokenizer… See the full description on the dataset page: https://huggingface.co/datasets/occiglot/tokenizer-wiki-bench.gia-dataset-tokenized-2024-2
Dataset Card for "gia-dataset-tokenized-2024-2"
More Information needed
tokenizers-benchtokenizers-test-data
tokenizers-test-data
Test and benchmark fixtures for huggingface/tokenizers,
pulled on demand by the repo Makefiles (make test / make bench / make fixtures
via hf download).
Layout
fixtures/ — multilingual + modality corpora for cross-language encode
benchmarks. Organized, documented, and reproducible: see
fixtures/FIXTURES.md for provenance and
fixtures/fixtures_manifest.json for
exact sources, pinned revisions, and sizes. Rebuild any file with… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/tokenizers-test-data.regent-subset-of-jat-dataset-tokenizedFineWeb2-mds-tokenized-v2FineWeb2-mds-tokenizedfixed-tokenizer-segmentsobelics_seed2_tokensPart of the OBELISC data set, including 32 Million samples, please refer to dataset.py to use this data
tokenized_datasetFineWeb2-mds-tokenized-4096regent-subset-of-jat-dataset-tokenizedThis is the dataset for REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context In New Environments.
The REGENT dataset includes a subset of the JAT (tokenized) dataset (from https://huggingface.co/datasets/jat-project) for the REGENT training environments.
It has around a 100k transitions from each of the 145 training environments (45 metaworld, 52 atari, 9 mujoco, 39 babyai).
Please find this in the *_subset folders.
It also has distance values for input sequences used in… See the full description on the dataset page: https://huggingface.co/datasets/regent-research/regent-subset-of-jat-dataset-tokenized.FineWeb2-mds-tokenized-1024c4-10k-mini-tokenized-16-ctx-gelu-1l-testscode_instructions_122k_alpaca_stylevietnamese-tokenizedfineweb-edu-100B-gpt2-token-shardsFineWeb Edu 100B dataset tokenized with GPT-2 tokenizer using the code in llm.c repo.
FineWeb2-mds-tokenized-v2-1024mtpnet_tokens
模型训练过程汇总（持续更新中）
对于已收集的每一个模型，code 目录为模型定义、训练和测试的代码和脚本文件，model 目录为已收集的 epoch 模型文件，dataset.zip 为模型数据集。
下表汇总了所有收集的模型训练过程信息：
模型名称
模型简介
模型类型
Epoch数量
数据集信息
Clone-detection-BigCloneBench
基于大规模代码克隆基准数据集的代码克隆检测模型，任务是进行二元分类(0/1),其中1代表语义等价，0代表其他情况。
代码克隆检测
2个epoch
BigCloneBench数据集
Clone-detection-POJ-104
基于POJ-104数据集的代码克隆检测模型，任务是识别不同编程题目中相似的代码实现，给定一段代码和一组候选代码，任务是返回具有相同语义的Top K个代码
代码克隆检测
2个epoch (0-1)
POJ-104编程题目数据集… See the full description on the dataset page: https://huggingface.co/datasets/code-philia/mtpnet_tokens.tulu_flan_mds_incremental-tokensSPACCC_Tokenizer
The Tokenizer for Clinical Cases Written in Spanish
Introduction
This repository contains the tokenization model trained using the SPACCC_TOKEN corpus (https://github.com/PlanTL-SANIDAD/SPACCC_TOKEN). The model was trained using the 90% of the corpus (900 clinical cases) and tested against the 10% (100 clinical cases). This model is a great resource to tokenize biomedical documents, specially clinical cases written in Spanish.
This model was created using the Apache… See the full description on the dataset page: https://huggingface.co/datasets/Biomedical-TeMU/SPACCC_Tokenizer.c4-tokenized-2b
Dataset Card for "c4-tokenized-2b"
More Information needed
tokenized-falcon2-dutch-2048Emilia-Dataset-tokenised