datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
imatrix-calibration
Importance Matrix Calibration Datasets
This repository provides calibration datasets used to generate importance matrices (imatrix), which are required to minimize errors when quantizing models with LLaMA C++.
The llama-imatrix program cannot handle parquet files directly and thus requires them to be converted into text format first. There are many ways to do this but a simple approach is to use DuckDB with the following command: duckdb -noheader -ascii -c "SELECT content FROM… See the full description on the dataset page: https://huggingface.co/datasets/eaddario/imatrix-calibration.iMaterialistDisclaimer: We do not own this dataset. iMaterialist dataset is a public dataset which can be accessed through its Github page.
When using the datset, cite the original work.
@article{guo2019imaterialist,
title={The iMaterialist Fashion Attribute Dataset},
author={Guo, Sheng and Huang, Weilin and Zhang, Xiao and Srikhanta, Prasanna and Cui, Yin and Li, Yuan and R.Scott, Matthew and Adam, Hartwig and Belongie, Serge},
journal={arXiv preprint arXiv:1906.05750},
year={2019}
}
imatrix
Input files for generating the Importance Matrix
Which file to use for generating the importance matrix
Not all importance matrices are equal. The best results are obtained when using a source file similar to the
training data. Size also matters: the bigger the model (eg: 70b vs 13b) and the higher the quant (eg: q6k_ vs iq3_xs),
the bigger the source file needs to be to make an impact. Multiple input files can be combined if needed;
for example:
cat multilingual.txt… See the full description on the dataset page: https://huggingface.co/datasets/froggeric/imatrix.bartowski-imatrix-v5-semantic
Bartowski iMatrix Calibration v5 (Semantic Chunking)
A processed version of bartowski's v5 imatrix calibration data using semantic boundary detection optimized for the v5 data structure.
Dataset Summary
Metric
Value
Total samples
2,075
Chunking method
V5-optimized semantic boundary detection
Chunk size
200+ characters (no upper limit, preserves document integrity)
Languages
English, German, Spanish, French, Italian, Swedish, Russian, Arabic, Chinese… See the full description on the dataset page: https://huggingface.co/datasets/lemon07r/bartowski-imatrix-v5-semantic.korean-imatrix-calibration-corpus
Korean imatrix Calibration Corpus — KO-i1 보정 코퍼스
한국어 중심 imatrix 보정 코퍼스의 첫 공개 릴리스 (우리가 아는 한).
공개 GGUF 양자화 생태계의 importance matrix는 거의 전부 영어 위주 코퍼스로
수집됩니다. 그 결과 한국어 토큰 분포에서의 양자화 손실이 체계적으로 커집니다.
이 데이터셋은 그 공백을 메우기 위해 만들어졌고, 실측으로 효과가 입증됐습니다.
실측 효과 (이 코퍼스로 만든 KO-i1 릴리스들)
릴리스
비교 대상
결과
kanana-1.5-8b KO-i1
영어 보정 i1
저비트 KLD -5~6% (IQ2_M 3.3σ), 비트 낮을수록 이득 증가
Qwen3.6-35B-A3B KO-i1
영어 보정 i1
전 타입 우세, -5.1~-6.8% (최대 4.3σ), MoE는 4비트도 유의
Qwen3.8-27B-abl KO-i1
정적 양자… See the full description on the dataset page: https://huggingface.co/datasets/augustine223/korean-imatrix-calibration-corpus.imatrix-dataset-for-japanese-llmchinese-imatrix-data-and.datThese data are utilized for the imatrix in llama.cpp, thereby maintaining model capability in low-precision quantization like IQ3-XXS.
Most of the data is in Chinese or translated to Chinese; performance in other languages is not guaranteed (although some level of understanding may still be achievable).
I have not tested any language other than Chinese. If anyone has, please feel free to comment.
include:
some data from: m-a-p/COIG-CQIA
some data from:… See the full description on the dataset page: https://huggingface.co/datasets/DataSoul/chinese-imatrix-data-and.dat.imatrix-ja-en
Japanese-English imatrix Calibration Data
imatrix計算用のキャリブレーションデータです。日本語LLMのGGUF量子化品質向上を目的として作成しました。
本データセットは下記「ライセンス」欄に記載したデータセット群から派生した二次的著作物です。
構成
カテゴリ
割合
内容
ja_general
35%
日本語一般文章
ja_qa
20%
日本語Q&A・対話
ja_technical
10%
日本語技術・学術文
code
15%
プログラムコード
en_reasoning
15%
英語推論・知識文
structured
5%
SQL・構造化データ
目標トークン数/チャンク: 512
ファイル
ファイル
チャンク数
用途
imatrix-ja-en-500-shuffled.txt
500
チャンクをシャッフル済み（推奨）
imatrix-ja-en-500-raw.txt
500… See the full description on the dataset page: https://huggingface.co/datasets/k0ndra/imatrix-ja-en.japanese-imatrix-calibration
Japanese imatrix Calibration Dataset (calibration_ja)
llama.cppのllama-imatrix用、日本語LLM向けキャリブレーションデータセット。
概要
このデータセットは、日本語LLMの量子化（quantization）における精度維持のために、llama-imatrixで使用するキャリブレーションデータを目的として構築されました。
統計
項目
値
チャンク数
916
総文字数
400,191
推定トークン数
~200,096
ソース別内訳
ソース
文字数
割合
元のデータセット
wikipedia_ja
82,155 (20.5%)
wikimedia/wikipedia
CC BY-SA 4.0
c4_ja
40,122 (10.0%)
allenai/c4
CC BY 4.0
fineweb_ja
34,363 (8.6%)… See the full description on the dataset page: https://huggingface.co/datasets/ChiTako/japanese-imatrix-calibration.bartowski-imatrix-v5-semantic-parquetimatrix-corpusbartowski-imatrix-v3-semantic
Bartowski iMatrix Calibration v3 (Semantic Chunking)
A processed version of bartowski's v3 imatrix calibration data using semantic boundary detection in attempt to create coherent, non-overlapping samples.
Dataset Summary
Metric
Value
Total samples
168
Chunking method
Semantic boundary detection
Target chunk size
~2048 characters
Languages
English, German, Spanish, French, Italian, Swedish, Russian, Arabic, Chinese
Source Data
The… See the full description on the dataset page: https://huggingface.co/datasets/lemon07r/bartowski-imatrix-v3-semantic.new-imatrix-dataset-ja-en
Dataset Card for Dataset Name
日英LLM向けのimatrix蒸留用データセットです。
既存のデータセットとしてはTFMC/imatrix-dataset-for-japanese-llmがありますが、
テキストの品質が低いように感じたので、
青空文庫、日英Wikipedia,Project Gutenbergよりデータをシャッフルして作成しました。
Dataset Sources
fujiki/wiki40b_ja
globis-university/aozorabunko-clean
manu/project_gutenberg
blo05/cleaned_wiki_en_80-100
Uses
llama-imatrix -m /path/to/model-file/original-f16.gguf -f imatrix_sample.txt
imatrix-calibration-corpusimaterialistpixiv-novel-imat-calibrationchinese-imatrix-data-reasoningiMaterialist-2020-fashion-clothes-segmentation-train-part1-tempiMathBenchimatrix-databricks-dolly-15k-jataigi-literature-asts
Dataset Summary
The dataset contains 2,494 rows.
These paragraphs are extracted from authorized novels written by Ang Siok Tsiau洪淑昭 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 2,494 (each representing a paragraph)
Features:… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-asts.ima-corpus-zhtw
IMA Traditional Chinese Corpus（繁體中文語料總集）
本資料集為繁體中文文學語料總集，目的在於將原先分散於多個作者／來源 dataset repo 的繁體中文文本統一整併，提供「一次申請、持續更新」的集中存取方式。
使用者只需申請本 dataset（本 repo）一次，即可取得所有繁中語料。未來新增來源或更新資料將直接同步至本 repo，無需重複申請。
📂 目錄結構
所有來源資料皆保留於 data/ 之下，每個子資料夾對應一個原始來源 repo，例如：
data/
├── zhtw-literature-ots
每個子資料夾內保留：
原始 README
原始語料檔（json / txt 等）
來源資訊與授權說明
以利來源追溯與資料審核。
📦 資料格式
主要格式：
JSON
UTF-8 編碼文字檔
典型欄位可能包含：
title：作品名稱
author：作者
content：文本內容
source：來源 repo
（依各來源資料實際格式而定）… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/ima-corpus-zhtw.Ancient_Egyptians_modeltaigi-literature-ttshs
Dataset Summary
The dataset contains 240 rows.
These paragraphs are extracted from authorized novel written by Tiunn Tshing Siong張青松 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 240 (each representing a paragraph)
Features:… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-ttshs.zhtw-literature-ots
Dataset Summary
The dataset contains 2,349 rows.
These paragraphs are extracted from authorized novels written by Ou Tiong Siong胡長松 and contain multiple sentences in Traditional Chinese.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 2,349 (each representing a paragraph)
Features:
title: Book title… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/zhtw-literature-ots.imatrix_datasettaigi-literature-abt
Dataset Summary
The dataset contains 389 rows.
These paragraphs are extracted from authorized novels written by Ang Bing-To洪明道 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 389 (each representing a paragraph)
Features:
title: Book… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-abt.taigi-literature-ngkh
Dataset Summary
The dataset contains 980 rows.
These paragraphs are extracted from authorized paper written by Ngoo Ka Hun吳嘉芬 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 980 (each representing a paragraph)
Features:
title: Paper… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-ngkh.taigi-literature-llb
Dataset Summary
The dataset contains 1,827 rows.
These paragraphs are extracted from authorized novels written by Lîm lang-bín林央敏 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 1,827 (each representing a paragraph)
Features:
title:… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-llb.taigi-literature-ots
Dataset Summary
The dataset contains 5,260 rows.
These paragraphs are extracted from authorized novels written by Ou Tiong Siong胡長松 and contain multiple sentences in Taiwanese Taigi written with Hanji.
The dataset maintains the original literary style and structure, making it useful for training language models, natural language processing (NLP), and Taiwanese literature research.
Dataset Structure
Number of rows: 5,260 (each representing a paragraph)
Features:… See the full description on the dataset page: https://huggingface.co/datasets/IMA-Taiwan/taigi-literature-ots.