datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
image_dummy\librispeech_asr_dummyswe-bench-dummy-test-datasetfineweb-2-duckdbs
DuckDB datasets for (dump, id) querying on FineWeb 2
This repo contains some DuckDB databases to check whether a given WARC UID exists in a FineWeb-2 dump. Usage example
is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data),
then you first have to extract the UID (the id column is of type UUID in the databases).
Download
All files:
huggingface-cli download BramVanroy/fineweb-2-duckdbs --local-dir… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-2-duckdbs.KakologArchives_duplicate
ニコニコ実況 過去ログアーカイブ
ニコニコ実況 過去ログアーカイブは、ニコニコ実況 のサービス開始から現在までのすべての過去ログコメントを収集したデータセットです。
去る2020年12月、ニコニコ実況は ニコニコ生放送内の一公式チャンネルとしてリニューアル されました。これに伴い、2009年11月から運用されてきた旧システムは提供終了となり（事実上のサービス終了）、torne や BRAVIA などの家電への対応が軒並み終了する中、当時の生の声が詰まった約11年分の過去ログも同時に失われることとなってしまいました。
そこで 5ch の DTV 板の住民が中心となり、旧ニコニコ実況が終了するまでに11年分の全チャンネルの過去ログをアーカイブする計画が立ち上がりました。紆余曲折あり Nekopanda 氏が約11年分のラジオや BS も含めた全チャンネルの過去ログを完璧に取得してくださったおかげで、11年分の過去ログが電子の海に消えていく事態は回避できました。しかし、旧 API が廃止されてしまったため過去ログを API… See the full description on the dataset page: https://huggingface.co/datasets/echodict/KakologArchives_duplicate.librispeech_asr_dummyLibriSpeech is a corpus of approximately 1000 hours of read English speech with sampling rate of 16 kHz,
prepared by Vassil Panayotov with the assistance of Daniel Povey. The data is derived from read
audiobooks from the LibriVox project, and has been carefully segmented and aligned.
Note that in order to limit the required storage for preparing this dataset, the audio
is stored in the .flac format and is not converted to a float32 array. To convert, the audio
file to a float32 array, please make use of the `.map()` function as follows:
```python
import soundfile as sf
def map_to_array(batch):
speech_array, _ = sf.read(batch["file"])
batch["speech"] = speech_array
return batch
dataset = dataset.map(map_to_array, remove_columns=["file"])
```dungpham30833KEGUNAANDuplexConv
DuplexConv
DuplexConv is a large-scale Chinese multi-channel conversational speech dataset with LLM-assisted annotations, developed by ASLP@NPU and QualiaLabs as part of the SmoothConv–DuplexConv corpus family.
Companion dataset: SmoothConv on HuggingFace (100 hours, expert human annotation). DuplexConv and SmoothConv share the same conversational domains and a unified data design. SmoothConv focuses on high-quality human annotations for benchmarking and… See the full description on the dataset page: https://huggingface.co/datasets/qualialabsAI/DuplexConv.openBHBOpenBHB: a Multi-Site Brain MRI Dataset for Age Prediction and Debiasing
The Open Big Healthy Brains (OpenBHB) dataset is a large (N>5000) multi-site 3D brain MRI dataset gathering 10 public datasets (IXI, ABIDE 1, ABIDE 2, CoRR, GSP, Localizer, MPI-Leipzig, NAR, NPC, RBP) of T1 images acquired across 93 different centers, spread worldwide (North America, Europe and China). Only healthy controls have been included in OpenBHB with age ranging from 6 to 88 years old, balanced between males and… See the full description on the dataset page: https://huggingface.co/datasets/benoit-dufumier/openBHB.storagedummy-audio-samplesdungle51248dummy_image_text_data
Dataset Card for "dummy_image_text_data"
More Information needed
wikipedia_culturax_dutch
Filtered CulturaX + Wikipedia for Dutch
This is a combined and filtered version of CulturaX and Wikipedia, only including Dutch. It is intended for the training of LLMs.
Different configs are available based on the number of tokens (see a section below with an overview). This can be useful if you want to know exactly how many tokens you have. Great for using as a streaming dataset, too. Tokens are counted as white-space tokens, so depending on your tokenizer, you'll likely end up… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/wikipedia_culturax_dutch.xmasdungngo59568duy19955dungeonnideaiwomotomerunowamachigatteirudaroukavhoujounomegamihen
Bangumi Image Base of Dungeon Ni Deai Wo Motomeru No Wa Machigatteiru Darou Ka V: Houjou No Megami-hen
This is the image base of bangumi Dungeon ni Deai wo Motomeru no wa Machigatteiru Darou ka V: Houjou no Megami-hen, we detected 132 characters, 6223 images in total. The full dataset is here.
Please note that these image bases are not guaranteed to be 100% cleaned, they may be noisy actual. If you intend to manually train models using this dataset, we recommend performing necessary… See the full description on the dataset page: https://huggingface.co/datasets/BangumiBase/dungeonnideaiwomotomerunowamachigatteirudaroukavhoujounomegamihen.asr_dummySelf-supervised learning (SSL) has proven vital for advancing research in
natural language processing (NLP) and computer vision (CV). The paradigm
pretrains a shared model on large volumes of unlabeled data and achieves
state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the
speech processing community lacks a similar setup to systematically explore the
paradigm. To bridge this gap, we introduce Speech processing Universal
PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the
performance of a shared model across a wide range of speech processing tasks
with minimal architecture changes and labeled data. Among multiple usages of the
shared model, we especially focus on extracting the representation learned from
SSL due to its preferable re-usability. We present a simple framework to solve
SUPERB tasks by learning task-specialized lightweight prediction heads on top of
the frozen shared model. Our results demonstrate that the framework is promising
as SSL representations show competitive generalizability and accessibility
across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a
benchmark toolkit to fuel the research in representation learning and general
speech processing.
Note that in order to limit the required storage for preparing this dataset, the
audio is stored in the .flac format and is not converted to a float32 array. To
convert, the audio file to a float32 array, please make use of the `.map()`
function as follows:
```python
import soundfile as sf
def map_to_array(batch):
speech_array, _ = sf.read(batch["file"])
batch["speech"] = speech_array
return batch
dataset = dataset.map(map_to_array, remove_columns=["file"])
```ducthanh1995dungho50022Waste-Dumpsites-DroneImagery
Dataset for Waste/Dumpsite Detection using drone imagery
Contains 2115 drone images of illegal waste dumpsites
1280 x 1280 px resolution
Nadir perspective (camera pointing straight down at a 90-degree angle to the ground)
Annotations and Images
train | valid | test
actual images
COCO - annotations_coco.json files in each split directory
.parquet files in data directory with embeded images
The dataset was collected as part of the [ Raven Scan ] project, more… See the full description on the dataset page: https://huggingface.co/datasets/INS-IntelligentNetworkSolutions/Waste-Dumpsites-DroneImagery.otoSpeech-full-duplex-turn-104h
Dataset Card for otoSpeech-full-duplex-turn-104h
Contact
Website: https://oto.earthEmail: agent@oto.earth
Dataset Summary
otoSpeech-full-duplex-turn-104h is an English, full-duplex conversational speech dataset for research on turn-taking and related spoken-dialogue phenomena. It contains 420 two-speaker conversations totaling approximately 104.94 hours. Each conversation includes time-aligned, channel-separated audio, a stereo combined recording… See the full description on the dataset page: https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-turn-104h.9router-data9router-data29router-data3tmlr-md-dumpdtsduplexgen-spoken
DuplexGen Spoken
Rendered spoken audio for the DuplexGen turn-taking dialogues — the exact
set of clips used to fine-tune the full-duplex model (PP-DG) in
DuplexGen: Adaptive Synthesis of Human–AI Turn-Taking Dialogues.
Each clip is a full render of one generated dialogue variation: the mixed
two-speaker dialogue audio, the isolated per-turn utterances, and the inserted
backchannel clips, plus per-clip metadata. Audio is synthesized with
Chatterbox TTS; the dialogue
text it… See the full description on the dataset page: https://huggingface.co/datasets/DuplexGen/duplexgen-spoken.