datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
avspeech-visual-audio
AVSpeech Video + Audio
This repository is a media-bearing reconstruction of the public AVSpeech
annotations. Each row represents an already-trimmed segment and keeps the
original source-video timing and target-face-center metadata.
Dataset structure
clip_id: identifier derived as
{youtube_id}_{start_sec:.3f}_{end_sec:.3f}.
avspeech_metadata: JSON containing youtube_id, start_sec, end_sec,
x_center, and y_center from the AVSpeech annotation.
video: video-only… See the full description on the dataset page: https://huggingface.co/datasets/ProgramComputer/avspeech-visual-audio.github-file-programs-dataset-javagithub-file-programs-dataset-gotextbook_quality_programming
Dataset Card for "textbook_quality_programming"
Synthetic programming textbooks generated with GPT-3.5 and retrieval. Very high quality, aimed at being used in a phi replication. Currently 115M tokens. Covers many languages and technologies, with a bias towards python.
~10k of the books (65M tokens) use an older generation method, and average 6k tokens in length. ~1.5k books (50M tokens) use a newer generation method, with a more detailed outline, and average 33k tokens in… See the full description on the dataset page: https://huggingface.co/datasets/vikp/textbook_quality_programming.Competitive-Programmingutllm-program-fineweb-1m-tinyllamapaloma_programming_languagesgithub-file-programs-dataset-rustnl_gameable_programmatic_gradersIndustryCorpus2_computer_programming_code
IndustryCorpus2: Programming
This repository contains the IndustryCorpus2: Programming domain subset of BAAI/IndustryCorpus2.
Refer to the parent dataset card for data construction, intended use, limitations,
and licensing details.
Citation
If you use this dataset in your work, please cite IndustryCorpus2:
@misc{shi2024industrycorpus2,
title = {IndustryCorpus2},
author = {Xiaofeng Shi and Lulu Zhao and Hua Zhou and Donglin Hao},
year = {2024}… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus2_computer_programming_code.github-file-programs-dataset-typescriptprogramming_books_llama
Dataset Card for "programming_books_llama"
400M tokens of programming books generated by gpt-3.5 (70M tokens) and a finetuned codellama 34b. The gpt-3.5 data is extremely high quality. The llama data has lower quality and shorter length, but is still good. This was generated with the textbook quality repo.
program-cota-mantis
🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
🌐 Website | 📑 Arxiv | 💻 Code| 🤗 Datasets
If you like our project or are interested in its updates, please star us :) Thank you! ⭐
Summary
TLDR: CoTA is a large-scale dataset of synthetic Chains-of-Thought-and-Action (CoTA) generated by programs.
Load data
from datasets import load_dataset
dataset = load_dataset("Salesforce/program-cota-mantis"… See the full description on the dataset page: https://huggingface.co/datasets/Salesforce/program-cota-mantis.PersonaSignal-PerceivabilityTest-Programming-Expertise-gpt-5-mini
Dataset card for PersonaSignal-PerceivabilityTest-Programming-Expertise-gpt-5-mini
This dataset was made with Curator.
Dataset details
A sample from the dataset:
{
"dimension_name": "programming_expertise",
"dimension_values": [
"Novice",
"Intermediate",
"Advanced"
],
"dimension_description": "Represents the user's practical fluency in software engineering. It shapes how they decompose problems, choose abstractions, weigh… See the full description on the dataset page: https://huggingface.co/datasets/JasonYan777/PersonaSignal-PerceivabilityTest-Programming-Expertise-gpt-5-mini.PersonaSignal-PersonalizedResponse-Programming-Expertise-gpt-5-mini
Dataset card for PersonaSignal-PersonalizedResponse-Programming-Expertise-gpt-5-mini
This dataset was made with Curator.
Dataset details
A sample from the dataset:
{
"dimension_name": "programming_expertise",
"dimension_values": [
"Novice",
"Intermediate",
"Advanced"
],
"dimension_description": "Represents the user's practical fluency in software engineering. It shapes how they decompose problems, choose abstractions, weigh… See the full description on the dataset page: https://huggingface.co/datasets/JasonYan777/PersonaSignal-PersonalizedResponse-Programming-Expertise-gpt-5-mini.mathqa_programsmath_qa converted to Python snippets
python-programming-instructionspython_programming_questionsutllm-program-fineweb-1mgithub-file-programs-dataset-javascriptgithub-file-programs-dataset-pythonutllm-program-python-300klinear-programmingprogram-cota-llava
🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
🌐 Website | 📑 Arxiv | 💻 Code| 🤗 Datasets
If you like our project or are interested in its updates, please star us :) Thank you! ⭐
Summary
TLDR: CoTA is a large-scale dataset of synthetic Chains-of-Thought-and-Action (CoTA) generated by programs.
Load data
from datasets import load_dataset
dataset = load_dataset("Salesforce/program-cota-llava"… See the full description on the dataset page: https://huggingface.co/datasets/Salesforce/program-cota-llava.utllm-program-python-300k-tinyllamasmolified-tiny-text-to-code
🤏 smolified-tiny-text-to-code
Intelligence, Distilled.
This is a synthetic training corpus generated by the Smolify Foundry.
It was used to train the corresponding model programmerGodbyte/smolified-tiny-text-to-code.
📦 Asset Details
Origin: Smolify Foundry (Job ID: fe9b19bf)
Records: 1078
Type: Synthetic Instruction Tuning Data
⚖️ License & Ownership
This dataset is a sovereign asset owned by programmerGodbyte.
Generated via Smolify.ai.
Code-290k-labels-programming_languages-NO_Chatgpt
Para etiquetar los lenguajes de programación en un conjunto de datos extenso de fragmentos de código, se aplicaron técnicas automatizadas de procesamiento de texto y patrones específicos de cada lenguaje, sin recurrir al uso de modelos de lenguaje avanzados como ChatGPT o LLMs. Se inició con la extracción y preparación de datos usando pandas, una biblioteca de análisis de datos en Python, que facilitó la manipulación y el procesamiento del conjunto de datos obtenido de Hugging Face's… See the full description on the dataset page: https://huggingface.co/datasets/NickyNicky/Code-290k-labels-programming_languages-NO_Chatgpt.kashmiri-audio-corpus
Kashmiri Audio Segmented Corpus
(Derived from OpenSLR-122)
Metadata
Size: 1955 speech segments
Audio: 16 kHz, 16-bit, mono, WAV License: GPL-3.0
Language: Kashmiri (کٲشُر)
Description
This dataset contains short Kashmiri speech fragments that were automatically sliced from the original OpenSLR-122 recordingsThis dataset can be used for ASREach row provides:
audio – the waveform
text – the verbatim Kashmiri transcript
duration – segment… See the full description on the dataset page: https://huggingface.co/datasets/programindz/kashmiri-audio-corpus.quantum-compilation-and-programming
Neura Parse — Quantum Compilation & Programming
A code-heavy vertical on the quantum software/compilation stack: turning abstract quantum circuits and unitaries into device-executable programs. Covers unitary decomposition and circuit synthesis (Euler/ZYZ, KAK/Cartan, Solovay-Kitaev, Ross-Selinger gridsynth, numerical synthesis with BQSKit), gate-set/basis transpilation to native gate sets, qubit layout/mapping and routing under connectivity constraints (SABRE, VF2, SWAP… See the full description on the dataset page: https://huggingface.co/datasets/Neura-parse/quantum-compilation-and-programming.cqadupstack-programmers-vn
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["CQADupstackProgrammers-VN"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to run models on mteb task check out the GitHub repitory.
Citation
If you use this dataset, please cite the dataset as well as mteb, as this dataset likely includes additional processing… See the full description on the dataset page: https://huggingface.co/datasets/GreenNode/cqadupstack-programmers-vn.