datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SIFT-50M
Dataset Card for SIFT-50M
SIFT-50M (Speech Instruction Fine-Tuning) is a 50-million-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). It is built from publicly available speech corpora containing a total of 14K hours of speech and leverages LLMs and off-the-shelf expert models. The dataset spans five languages, covering diverse aspects of speech understanding and controllable speech generation instructions. SIFT-50M… See the full description on the dataset page: https://huggingface.co/datasets/amazon-agi/SIFT-50M.wildchat-50m-extended-resultsSenseNova-Vision-Corpus-50M-annotationDSIR-filtered-pile-50M
Dataset Card for DSIR-filtered-pile-50M
Dataset Summary
This dataset is a subset of The Pile, selected via the DSIR data selection method. The target distribution for DSIR is the Wikipedia and BookCorpus2 subsets of The Pile.
Languages
English (EN)
Dataset Structure
A train set is provided (51.2M examples) in jsonl format.
Data Instances
{"contents": "Hundreds of soul music enthusiasts from the United Kingdom plan to make their way to… See the full description on the dataset page: https://huggingface.co/datasets/stanford-crfm/DSIR-filtered-pile-50M.nemo-stage1-50M-samples
NeMo Stage1 Pretraining Dataset - 50M Samples
This dataset contains 50 million text samples for NeMo model pretraining (Stage 1). The dataset is organized in chunks for efficient loading and processing.
Dataset Details
Total Samples: ~50,000,000
Format: JSONL (JSON Lines)
Structure: Each sample contains {"id": number, "text": "content"}
Chunks: 47 files (chunk_000.jsonl to chunk_046.jsonl)
Samples per chunk: ~1,000,000
Language: English
Task: Text generation pretraining… See the full description on the dataset page: https://huggingface.co/datasets/ssuresh/nemo-stage1-50M-samples.heuristic_classification-filtered-pile-50M
Dataset Card for heuristic_classification-filtered-pile-50M
Dataset Summary
This dataset is a subset of The Pile, selected via the heuristic classification data selection method. The target distribution for heuristic classification are the Wikipedia and BookCorpus2 subsets of The Pile.
Languages
English (EN)
Dataset Structure
A train set is provided (51.2M examples) in jsonl format.
Data Instances
{"contents": "Members join for free and… See the full description on the dataset page: https://huggingface.co/datasets/stanford-crfm/heuristic_classification-filtered-pile-50M.KAWK50M-Training-Runs
KAWK50M Training Runs Archive
최종·중간·실패 실험 체크포인트를 포함한 전체 학습 이력입니다.
최종 배포 모델은 별도 저장소 KAWK-1.5-50M-Korean-Base, KAWK-1.5-50M-Korean-Base-5K, KAWK-1.5-50M-Korean-Instruct에 있습니다. 이 저장소는 실험 재현과 진단을 위한 전체 아카이브이므로 모든 체크포인트가 좋은 품질이거나 정상 학습 결과인 것은 아닙니다.
각 체크포인트의 model/과 tokenizer/는 Transformers 형식입니다. trainer_state.pt는 optimizer, scheduler와 RNG 상태를 담은 PyTorch pickle 파일이므로 신뢰할 수 있는 출처에서만 로드하십시오.
이 저장소는 KAWK50M 프로젝트의 공개 학습 이력 아카이브입니다. 자료를 사용할 때는 원천 데이터 라이선스, 개인정보, 중복 파일 및 실패 실험 산출물에 대한 안내를… See the full description on the dataset page: https://huggingface.co/datasets/Infinity08/KAWK50M-Training-Runs.synthweb-50mfineweb-nanochatbpe-50M
fineweb-nanochatbpe-50M
FineWeb-Edu (from karpathy/fineweb-edu-100b-shuffle), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536)
and packaged as flat uint16 token-id .bin files for fast memmap training.
This is a 50-million-token slice for data-constrained experiments. The train.bin
is the byte-exact first 50,000,000 tokens (bytes [0, 100000000)) of the parent
alexkstern/fineweb-nanochatbpe-20B
train.bin. The val.bin is byte-identical to the parent's val.bin.
file… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/fineweb-nanochatbpe-50M.50_mbti_testsynthetic-gpt-10M-general-50MB
datasets
# dataset
from datasets import load_dataset
ds = load_dataset("falcontreatred/synthetic-gpt-10M-general-50MB")
pandas
# pandas
import pandas as pd
df = pd.read_json("hf://datasets/falcontreatred/synthetic-gpt-10M-general-50MB/synthetic-gpt-10M-general-50MB-v1.jsonl", lines=True)
polars
# polars
import dask.dataframe as dd
df = pl.read_ndjson("hf://datasets/falcontreatred/synthetic-gpt-10M-general-50MB/synthetic-gpt-10M-general-50MB-v1.jsonl")
croissant
# croissant
from… See the full description on the dataset page: https://huggingface.co/datasets/falcontreatred/synthetic-gpt-10M-general-50MB.