SHSLab/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection

<div align="center">

<img src="assets/shslab_avatar.webp" width="120" alt="SHS-Lab"/>

🧬 Omni-Frontier Collection

Cybersecurity · Coding · Math · Science · RSI Reasoning — one unified SFT package

<img src="assets/banner.jpg" width="100%" alt="Omni-Frontier banner"/><br><br>

<br>

A unified, deduplicated, fully-browsable distillation & SFT corpus — every row real, every row visible.

</div>

📖 Jump to

What's inside · 🔁 Aggregation audit · 🛡 Cybersecurity · 💻 Coding · 🏭 Distillation deep-dive · 🔁 RSI · 🧮 Math/Science/More · 🎓 Training guide · 🔎 Browsing · 🧹 Quality · 🗺 Roadmap · 📄 License

📌 What's inside — the REAL numbers

<table><tr><td> ⚠️ Read this first: the 19,096,153 figure is the total parquet row count of the whole collection — it is not all distillation data. Distillation is only 8.6% of it. The bulk is coding (66.3%) and cybersecurity (23.7%). Full breakdown below, measured file-by-file from actual parquet footers. </td></tr></table>

On top of the parquet: ~6.8 GB of JSONL/JSON/CSV + raw archives — including a 1.05 GB GPT-5.6 agent-trace log, ~1.95 GB of MITRE/CVE security records, and 3.0 GB of security-gym v4 zstd SQLite experiment DBs.

<img src="assets/chart_categories.png" width="100%" alt="Category breakdown"/>

🔁 Full aggregation from Manusagents/Omni-Frontier — the feature manifest

Every one of the 2,214 files of the original `Manusagents/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection` is accounted for here — verified by sha256 content hashes and parquet row counts, not by eyeballing. This is not a fork: it is the upgraded continuation of Omni-Frontier, with every original feature carried over, upgraded, or transparently flagged.

<table><tr><td>

✅ Audit result: 2,214 / 2,214 original files reconciled — 2,075 mapped (incl. 5 security-gym files re-added by this audit) · 78 exact duplicates (sha256-verified) · 16 row-verified subsumed copies · 9 derived _viewable copies (row-verified) · 30 VCS files · 5 root assets replaced. Full evidence: `Data/index/aggregation_report.json`

</td></tr></table>

📈 The original cleaning pipeline, reproduced

Verbatim from the original cleaning_stats.json — this is what produced the 6.94M-row core:

🆕 What this repo adds (not in the original)

• —🔁 RSI category — OpenMLE traces (26,259, dual-listed in coding) + 76 DeepSeek-R1-style self-generated reasoning traces built by SHS-Lab
• —🗄 Security-gym v4 raw SQLite archives — 3.0 GB of zstd RL-experiment DBs (exp01_90d, exp_30d_heavy, exp_7d_brute) re-added after the audit
• —🧾 `aggregation_report.json` — the full 2,214-file reconciliation with evidence
• —🥷 Exploit-DB corpus ×5 sources consolidated — 1,030 raw exploit .txt files also packed into one browsable parquet
• —📊 Footer-measured honesty — every row count on this card comes from real parquet footers

📏 Quality — measured, not claimed

Every number below was computed by sampling real rows from the actual parquet files (footer row counts are exact, content depth is sampled over ~120 rows per config and weighted by rows):

\ weighted mean of sampled row size (all text fields, json-serialized). Tokenizer-ready shards (`input_ids`) average 801 tokens* per row. The Omni splits also carry the original pipeline's quality_score (0–100) column untouched.

🎯 The original quality_score (0–100) — measured over all 6,863,735 scored rows

Eight Omni full-clean splits carry the original pipeline's 0–100 quality score. We computed exact statistics over every single scored row (column-only remote parquet reads — no sampling, no guessing):

Overall: 52.2 avg across all 6.86M scored rows. Read this honestly: the original pipeline's nominally 0–100 scale effectively tops out at 60 (the max observed in 6.86M rows) — so within this scorer, 55+ is its "premium" tier and cyber security (55.2) plus distillation (59.9) lead the pack. The raw specialist sources that never went through this scorer — CYBER-GOD-MODE, Exploit-DB texts, OpenMLE traces, security-gym, NOSK, VoltVAR — are the deep-content story in the table above, not this column.

🧭 What can you build — per-category use cases

🛡 cyber security (4.52M rows) — offense-aware defense

• —Security assistant / SOC copilot SFT — CYBER-GOD-MODE gives 642K deep ChatML analyses (avg 25.8K chars of code-aware attack reasoning); cybersecurity-QA adds 2.6M Q→A pairs for breadth
• —SQL-injection detection & WAF tuning — labeled SQLi datasets (Kaveny + AmirAliGharesoufloo) plus Exploit-DB input/output pairs (70K) train classifiers or explainable detectors
• —Exploit comprehension & summarization — 1,030 full disclosure texts (avg 71.6K chars) + detoxio (5.8K) + grantabejar + wwe123 archive: fine-tune models that read raw exploits and emit structured CVE/Vuln info
• —Threat-intel extraction to STIX/CVE — ~2 GB of MITRE STIX records + harmony-format conversations → train extractors that turn unstructured reports into ATT&CK-mapped JSON
• —Pentest & red-team tutoring — NOSK corpus: 242,307 real pentesting/security-analysis conversations
• —ICS / power-grid security ML — VoltVAR IEEE-123: 100K labeled attack-command rows (80K train / 20K val) for anomaly detection research
• —Security-agent RL environments — security-gym v4 raw SQLite DBs (3.0 GB, 7d/30d/90d experiments) as replay/environment data for agentic RL
• —Evaluation — CyberSec-Bench: 200 reference-answer QA for benchmarking cyber models

💻 coding (12.66M rows) — from syntax to agentic debugging

• —General code SFT at scale — 6.3M instruction/response pairs (Omni splits, avg 1.4–3.1K chars)
• —Long-CoT code reasoning — OpenMLE: 26,259 Kaggle-ML traces, avg 37.5K chars, 100% explicit `<think>` — ideal for R1-style thinking models
• —Debug & repair training — kimi-k3 coding/debug traces (4K, avg 20.8K chars multi-turn) + GPT-5.6 sol/luna/terra agent sessions (15K rows + 1.05 GB raw logs with assistant_steps)
• —ML-pipeline generation — OpenMLE task/traces pairs teach models to build full scikit-learn/Kaggle pipelines

🧮 math — Omni-MATH competition problems (raw jsonl) for math CoT SFT, verifier training and eval

🔁 RSI — Recursive Self-Improvement

• —Reflection & self-correction loops — every row is a full self-question → self-check → self-decide chain (OpenMLE 100% <think>, our 75 self-generated R1-style traces)
• —GRPO / RLVR seed data — thinking traces are the raw material for verifiable-reward RL
• —Self-QA distillation — train models that critique and improve their own outputs

🏭 distillation — cross-teacher style transfer

• —Multi-teacher distillation — GPT-5.6 / Kimi-K3 / Qwen3.8-Max / GLM-5.2 response styles in one corpus (family/source columns let you gate per teacher)
• —Think-style transfer — 577K message rows at 100% <think> coverage (avg 10.5K chars)
• —Tokenizer-ready training — input_ids/labels shards (104K rows, avg 801 tokens) load straight into custom trainers
• —Token-budget analysis — per-row total_tokens/assistant_tokens stats (460K rows) for curriculum design

🔬 science · 🌌 humanities · ⚙️ applied · 📝 instructions · 🗂 index

• —Chemical definitions + science QA → domain QA bots; encyclopedic humanities pairs → knowledge chat; power-grid rows → time-series security research; 114K quick instruction rows → system-prompt/tone tuning; index rows → provenance lookups and dedup audits

🎭 emotions · 🧭 State — reserved (no content yet, stable paths guaranteed)

🛡 Cybersecurity — 4.5M rows

<img src="assets/art_cyber.jpg" width="100%" alt="Cybersecurity"/>

This is the largest security-focused SFT block you'll find in one place — and it's real, structured security knowledge, not noise:

<details> <summary>👀 Real excerpt — CYBER-GOD-MODE (code analysis task)</summary>

<|im_start|>user
Analyze 'v1/topdown/tokens_bench_test.go' from 'OPA'.
<|im_end|>
<|im_start|>assistant
FILE: v1/topdown/tokens_bench_test.go
SOURCE: OPA
CONTENT: // Copyright 2025 The OPA Authors.  All rights reserved.
...

</details>

<details> <summary>👀 Real excerpt — MITRE/CVE record (instruction → output)</summary>

CVE-2025-52714 summary:
Improper Neutralization of Special Elements used in an SQL Command
('SQL Injection') vulnerability in shinetheme Traveler traveler allows
SQL Injection. This issue affects Traveler: from n/a through < 3.2.2.

</details>

<details> <summary>👀 Real excerpt — harmony-format defensive assistant</summary>

<|start|>system<|message|>You are a cybersecurity assistant specialized
in vulnerability analysis and defensive security.
Knowledge cutoff: 2024-06 ... Reasoning effort: high
<|end|><|start|>developer<|message|># Instructions
Provide accurate, concise security explanations.
Prefer actionable ...

</details>

💻 Coding — 12.6M rows

The single biggest block — and its crown jewel is unique:

🏆 OpenMLE SFT Traces (26,259 traces, avg 37,793 chars)

Every row is a complete Kaggle-competition solution trace: the system prompt sets an ML-engineering competition, the model plans, and 100% of traces carry full `<think>` reasoning (verified row-by-row — 26,259/26,259). Tasks span DDoS detection on CICIoT2023 (ML-for-security), emotion classification, stock sentiment, fashion recognition, Steam reviews analytics and dozens more — with 93.7% showing explicit self-check language ("wait", "let me verify", "actually, on second thought"). Because of this, it is dual-listed in `coding` AND `RSI`.

<details> <summary>👀 Real excerpt — OpenMLE system prompt (Kaggle Grandmaster)</summary>

You are a Kaggle Grandmaster attending a high-stakes competition. In order
to win this competition, you need to come up with an excellent and creative
plan for a solution and then implement this solution in Python.
You must use Machine Learning/Deep Learning/Computer Vision/NLP/etc. methods
... preinstalled: torch, tensorflow, scikit-learn, numpy, pandas, xgboost,
lightgbm, catboost, transformers, datasets, tokenizers, huggingface-hub ...

</details>

Omni-Frontier coding split (12.64M rows)

Multi-frontier distilled coding conversations — the volume backbone for large-scale coding SFT. avg 1.8K chars/row, 12.3% carry <think> blocks.

Also in coding: kimi-k3 + GPT-5.6 debug traces

Distributed across coding and distillation (see next section): Kimi K3 and GPT-5.6 coding-and-debugging corpora with multi-step agent behavior.

🏭 Distillation — the honest deep-dive (1.6M rows)

<table><tr><td> 💡 "Distillation" is NOT one thing here — it's a multi-teacher reasoning factory. Measured from real samples: math 26% + code 24% + reasoning 18% + instruction 13% + agent/tool 10% of traces. Yes — ~24% of the distillation category is literally coding data (multi-step agent coding, debugging, security coding tasks). It also carries tokenizer-level artifacts (input_ids, labels, masking metadata) for pipeline debugging. </td></tr></table>

<img src="assets/chartdistillationmix.png" width="55%" alt="Distillation domain mix"/>

Who teaches what

<details> <summary>👀 Real excerpt — kimi-k3 coding/debug task</summary>

=== MOONSHINER TASK BOUNDARY ===
CI grew a shellcheck lint step (severity=style, zero findings allowed)
and our little photosort.sh is the last script blocking the pipeline.
It's not just the linter being fussy, ei...

</details>

<details> <summary>👀 Real excerpt — k3-dist coding teacher (system seed)</summary>

You are an expert software engineer. Write complete, correct, runnable
code in the language requested by the user. Follow the requested language,
interface, and constraints. Explain the approach briefly when reque...

</details>

<img src="assets/chart_sources.png" width="100%" alt="Source contributions"/>

🔁 RSI — Recursive Self-Improvement

<img src="assets/chartrsidomains.png" width="50%" alt="RSI domains"/><img src="assets/art_reasoning.jpg" width="49%" alt="Reasoning art"/>

Two sources, one philosophy — the model thinks about its own thinking:

1. 1.`openmle_sft_traces` — 26,259 Kaggle-grade traces (see Coding section). 100% <think>, 93.7% self-check vocabulary. Dual-listed from coding.
2. 2.`shslab_rsi_selfgen` — 76 fully self-generated traces in the complete DeepSeek-R1 loop: question the problem → decompose into self-questions → attempt → catch own flaw → backtrack → verify independently → explicit final decision. 10 domains (math, coding, science, cybersecurity, logic, planning, data-analysis, systems-design…), 3 difficulty tiers, avg 4,238 thinking chars per trace. MIT-licensed (ours).

<details> <summary>👀 Real excerpt — RSI selfgen trace (systems-design, expert)</summary>

Hmm, what is this really asking? It wants me to design a distributed rate
limiter with specific parameters (10k req/s/user) and compare three
algorithms across multiple dimensions. ...

First, I should decompose this: What are the core requirements?
1) Distributed nature 2) Per-user limits 3) Specific rate (10k/s) ...

…the trace continues through a failed first approach, explicit backtracking, verification, and ends with "Okay. Final decision: token bucket…" </details>

🧮 Math · Science · Humanities · Applied · Instructions

⚕️ About the "Med" in the name — the honest story

The original repo's name says Med and its card advertised medical & robotics subsets — but no medical or robotics files were ever shipped in that repo (we audited all 2,214 files). We keep the family name for continuity with the Omni-Frontier lineage, and we flag the gap instead of pretending. Any health/bio-adjacent rows that exist inside the QA corpora remain untouched, and the reserved emotions / State categories are the natural home for future curated medical & affective data.

⚗️ Chemistry & power-grid — the human-crafted specialist data

• —Jaafer ChemicalData → chemical reaction/property records in science/ and humanities/ (22,024 science rows total)
• —VoltVAR / IEEE-123 power-grid attack commands → applied/ — 100,000 rows (80K train + 20K validation) of time-series attack commands against a power-grid simulation. This is the "power-grid security" domain the original card advertised — here it is, row-measured and viewer-ready

🥷 NOSK Hacking corpus — 242,307 rows

49 parquet shards of pentesting/security-analysis conversations from NOSK (Nepal Open Source Klub), merged into cyber security/. The original card claimed ~255K rows; the real footer-measured number is 242,307 — we publish the measured number, not the marketing one.

✨ Features

• —✅ 19,096,153 exact parquet rows counted from real parquet footers — not estimates
• —✅ 12 fixed taxonomy categories — stable paths across versions
• —✅ 44 schema-homogeneous viewer configs — every parquet row (19,096,153) browsable in the Dataset Viewer; 283 raw jsonl/csv files stay in-repo, fully documented
• —✅ SHA-256 file-level dedup (166 exact duplicates removed; different content always kept)
• —✅ Verified uploads — manifest with per-file sha256 + size, 0 missing / 0 mismatch
• —✅ Raw exploit texts browsable — 1,030 .txt files consolidated into one parquet (originals kept)
• —✅ Full provenance — Data/index/source_map.json maps every file to its upstream dataset
• —✅ 100% aggregation audit — all 2,214 original Manusagents/Omni-Frontier files reconciled by content hash / row count (Data/index/aggregation_report.json)
• —✅ Security-gym v4 raw archives — 3.0 GB zstd SQLite RL-security experiment DBs included
• —✅ Training-ready & research-ready — chat formats and tokenizer artifacts (input_ids/labels)

🎓 Training guide — how to actually use this

1. Pick your objective

2. Handle the formats (they differ by source — on purpose)

# messages-format (chat):  Data/RSI, k3dist, mitre-harmony, CYBER-GOD-MODE (raw ChatML text)
# instruction/output:      mitre-stix train jsonl
# reasoning + <think>:     OpenMLE, Omni distill split, shslab_rsi_selfgen
# tokenizer artifacts:     k3dist input_ids/labels shards (skip for text SFT)

3. SFT with TRL (chat-format configs)

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

ds = load_dataset("SHS-Lab/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection",
                  "rsi__shslab_rsi_selfgen", split="train")
# ds[0]["messages"] = [{"role": "user", ...}, {"role": "assistant", "content": "<think>...</think>..."}]

cfg = SFTConfig(
    output_dir="./omni-frontier-sft",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=1e-5,
    num_train_epochs=2,
    bf16=True,
    logging_steps=10,
)
trainer = SFTTrainer(model="Qwen/Qwen3-8B", args=cfg, train_dataset=ds)
trainer.train()

Tips that matter:

• —🧠 For reasoning distillation, train on the full assistant turn (including <think>) — masking the think block defeats the purpose. For plain chat SFT, mask it.
• —🔀 Mix ratios that work: coding 40% · cyber 25% · distillation-reasoning 20% · RSI 10% · instructions 5% — then anneal with 2× upweight on your target domain.
• —📦 Start streaming before you commit: every config supports streaming=True — pilot on 50K rows first.
• —⚠️ OpenMLE = CC-BY-NC-4.0 → keep non-commercial, or exclude those files from commercial runs.
• —🚫 Don't mix input_ids-artifact shards into text SFT — they're for tokenizer-pipeline debugging/ablation.

4. LoRA recipe (single 48GB GPU)

peft_cfg = LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05,
                      target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                       "gate_proj", "up_proj", "down_proj"])

🔎 Browsing — the Dataset Viewer

The viewer runs on 44 explicit parquet configs — every one of the 19,096,153 parquet rows is served (no skipped rows, no preview-only samples). The 283 raw jsonl/csv/text files are deliberately not in the viewer (they made the server-side indexer time out — the ResponseNotFound you may have seen came from those); they remain fully in the repo:

from datasets import load_dataset

# any of the 44 viewer configs
ds = load_dataset("SHS-Lab/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection",
                  "cyber_security__cyber_god_mode", split="train", streaming=True)

# raw jsonl/csv: stream straight from the hub
import fsspec
url = "https://huggingface.co/datasets/SHS-Lab/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection/resolve/main/Data/cyber%20security/train.jsonl"
with fsspec.open(url) as f:
    for i, line in enumerate(f):
        ...

Schema documentation for every raw file lives in Data/index/source_map.json and in the per-category tables above. If the viewer ever shows a warm-up message right after a push, that is HF's indexer re-processing — the configs themselves are stable.

🧹 Quality · Dedup · QA

Measured, not claimed: every parquet footer fingerprinted; JSONL/CSV headers read; 166 byte-identical files removed; manifest verification = 0 missing, 0 size-mismatch. Non-row artifacts (.py code payloads, .pyc, provenance .json, checksums, banners, 3.0 GB of raw security-gym SQLite archives) stay in the repo but are deliberately excluded from viewer configs — see Data/index/viewer_manifest.json. On top of our own dedup, the aggregation audit re-downloaded 78 supposedly-missing original files and proved byte-identical duplicates (sha256), and row-verified 16 subsumed per-category copies plus 9 derived _viewable copies — zero content was lost in the merge. Full evidence chain: Data/index/aggregation_report.json.

🗺 Roadmap

• —[x] 12-category fixed taxonomy with stable paths
• —[x] Per-file sha256 provenance (source_map.json)
• —[x] RSI category live — OpenMLE dual-listed + 76 self-generated R1-style traces
• —[x] Full Manusagents/Omni-Frontier aggregation audit — 2,214/2,214 files reconciled
• —[x] Security-gym v4 raw archives (3.0 GB) re-added
• —[ ] emotions & State categories populated (reserved — awaiting curated sources)
• —[x] Viewer slimmed to 44 parquet configs → full 19.1M-row index (jsonl/csv served raw to avoid indexer timeouts)
• —[ ] Parquet mirrors for the big raw jsonl corpora (MITRE-STIX, GPT-5.6 session logs)

📄 Licensing & attribution

⚠️ Cyber subsets contain offensive-security knowledge from public corpora — for defense, education and authorized research only.

🙏 Citation

@misc{shslab2026omnifrontiercollection,
  title  = {Omni-Frontier Collection: Cybersecurity, Coding, Math, Science and RSI Reasoning SFT Corpus},
  author = {SHS-Lab},
  year   = {2026},
  url    = {https://huggingface.co/datasets/SHS-Lab/Omni-Frontier-Distillation-SFT-Cyber-Coding-Med-dataset-collection}
}

<div align="center"> _{Built with ❤️ by <a href="https://huggingface.co/SHS-Lab"><b>SHS-Lab</b></a> · updated Aug 29, 2026} </div>