datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
swallow-code-v2
SwallowCode-v2
Resources
📑 arXiv: Read our paper for detailed methodology and results at arXiv:2505.02881.
🤗 Sister Dataset: Discover SwallowMath-v2, our companion dataset for mathematical reasoning.
💻 What is it?
SwallowCode-v1 was a high-quality Python code dataset generated through an LLM-based rewriting pipeline.
However, it had two significant limitations:
(1) it was distributed under the Llama 3.3 Community License, and
(2) its size was limited to… See the full description on the dataset page: https://huggingface.co/datasets/tokyotech-llm/swallow-code-v2.swallow-code
SwallowCode
Notice
May 21, 2025: We have deleted ablation/exp1-the-stack-v2-train-smol-ids-python because it was flagged as potentially containing unsafe data collected from the Python subset of https://huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids. However, since this dataset can be reconstructed from the-stack-v2-train-smol-ids, there is no issue in terms of reproducibility.
May 21, 2025: ClamAV has flagged “Win.Trojan.MSShellcode-88” in… See the full description on the dataset page: https://huggingface.co/datasets/tokyotech-llm/swallow-code.llm0to1-pt-tokenized-code
LLM0to1 사전학습 토큰화본 — 코드
10B 규모 한/영 이중언어 LLM LLM0to1-10b 를 바닥부터 학습할 때 실제로 투입된
코드 코퍼스의 토큰화본. 총 27종 / 59.1B 토큰.
왜 원문 텍스트가 아니라 토큰화본인가
이 코퍼스들은 공개 데이터셋을 스트리밍으로 받아 곧바로 토큰화했고 중간 텍스트를 보관하지 않았다.
따라서 이 .ds 파일이 학습에 들어간 데이터의 유일한 사본이다.
단점만 있는 건 아니다. 토큰화본은 학습 입력 그 자체이므로,
재토큰화 과정에서 생길 수 있는 차이 없이 학습을 그대로 재현할 수 있다.
원본 출처
bigcode/starcoderdata(언어별 서브셋) · bigcode/commitpackft · bigcode/jupyter-code-text-pairs · deepmind/code_contests
code_c 와 code_c2 처럼 2 가 붙은 것은 정제기… See the full description on the dataset page: https://huggingface.co/datasets/izlley2/llm0to1-pt-tokenized-code.1gpu-llm-pretraining-corpus-15b-en-it-code
1GPU LLM Pretraining Corpus 15B EN-IT-CODE
1gpu-llm-pretraining-corpus-15b-en-it-code is the canonical document-level pretraining corpus used for the 1GPU LLM family.
It was built for training language models from scratch on a mixture of English, Italian and source code.
This Hugging Face release contains the clean, deduplicated, document-level corpus. It is intentionally published before tokenization and packing so that the training representation can be deterministically… See the full description on the dataset page: https://huggingface.co/datasets/nazdef/1gpu-llm-pretraining-corpus-15b-en-it-code.swallow-code-v0.1
What is it?
Swallow-code-v0.1 consists of 4 staged dataset subsets and are filtered from bigcode/the-stack-v2-train-smol-ids.
What is being released?
The dataset is released in four versions:
Swallow Code v0.1 stage 1: 36B tokens, 41M documents containing Python scripts.
Swallow Code v0.1 stage 2: 31B tokens, 37M documents containing Python scripts that are syntax error-free.
Swallow Code v0.1 stage 3: 20B tokens, 24M documents containing Python scripts that are filtered… See the full description on the dataset page: https://huggingface.co/datasets/tokyotech-llm/swallow-code-v0.1.LLMxCPG-Code
LLMxCPG-Code
This dataset contains the raw C files used in our paper:LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models
🔍 Models
The models LLMxCPG-Q and LLMxCPG-D are available in the following Hugging Face collection:👉 https://huggingface.co/collections/QCRI/llmxcpg-6855f80e601774b43eba2d14
💻 Source Code
The source code for LLMxCPG can be found here:👉 https://github.com/qcri/llmxcpg
code-nomist-llm-datasetCode Nomist 项目微调数据集
用于大模型微调使用，包含格式化后的问题，以及对应答案每个名称使用 | 符号分割。