datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wiki_snippets
Dataset Card for "wiki_snippets"
Dataset Summary
Wikipedia version split into plain text snippets for dense semantic indexing.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
We show detailed information for 2 configurations of the dataset (with 100 snippet passage length and 0 overlap) in
English:
wiki40b_en_100_0: Wiki-40B
wikipedia_en_100_0: Wikipedia
Data Instances… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/wiki_snippets.wikipedia-language-snippets-filtered
Wikipedia Snippets (Filtered)
Filtered sentence snippets in Wikipedia, by taking the first 60% of an article after filtering for stubs. Minor Latin groups are additionally filtered again for English leakage.
Sentences are mostly filtered out for non matching scripts, such as Arabic in a Cyrllic language.
Files
Each file is in this format for languages in ISO 639 2-letter codes:
train/en/en.parquet
train/es/es.parquet
From wikimedia/wikipedia
Licensing… See the full description on the dataset page: https://huggingface.co/datasets/polyglot-tagger/wikipedia-language-snippets-filtered.repo_code_snippets
Repository Code Snippets
Focused source-code snippets for static repository graph browsing.
Rows are keyed by repo_id and entity_id, matching the public repo graph
entity ids. Each row contains a focused source span plus surrounding context,
not a complete repository dump.
Columns
repo_id
entity_id
uri
kind
name
path
language
start_line
end_line
context_start_line
context_end_line
code
source_hash
truncated
Stats
snippets: 468,268
skipped entities: 437,756… See the full description on the dataset page: https://huggingface.co/datasets/PeytonT/repo_code_snippets.