srbwin/agentic-trace-megacorpus-10tb

Agentic Trace Megacorpus — ~2.06 TB (aggregated by reference)

A reference-aggregation of 432 public trace datasets (agentic coding, reasoning, SWE, tool-use) totaling ~2.06 TB, assembled for MiniMax-M3 post-training. Nothing is re-hosted by value here yet — the loader streams directly from each source repo. Byte mirroring into this repo (to fill the 10 TB quota) is done Hub→Hub via mirror_to_hub.py.

Composition (TB by relevance group)

Tiers: 1=fable, 2=agentic coding, 3=reasoning/SWE, 4=adjacent, 5=raw-code bulk filler.

Stream the whole corpus (no local download)

from megatrace import load_megatrace
ds = load_megatrace(streaming=True)            # everything
ds = load_megatrace(streaming=True, tiers=(1, 2))   # fable + agentic only
for ex in ds.take(3):
    print(ex)

sources.jsonl is the full manifest (repo_id, group, tier, bytes, url, load spec).

Path to the full 10 TB (honest accounting)

Ungated, accessible trace data on the Hub is a few TB — that's what is aggregated by reference above. The remaining quota is filled, in priority order:

1. 1.Mirror non-HF agent logs (below) — SWE-bench/SWE-Gym/OpenHands trajectories
2. 2.test logs: multiple TB of real agent traces (mirror_to_hub.py on the cluster).
3. 3.Generate with the RFT / ReST-EM harness (distill/scripts/amplify.py --method rft): rejection-sampled, verifier-filtered MiniMax-M3 / teacher traces — unbounded, on-distribution, and decontaminated. This is the principled way to reach 10 TB of new agentic traces rather than re-hosting raw code.
4. 4.(optional) license-gated raw-code corpora (e.g. the-stack-v2) — bulk, not agent traces; included only if you want to top off the quota with code.

Other (non-HF) sources to mirror on the cluster

• —SWE-bench (full + Verified + Multimodal) — task instances + gold patches + test logs
• —SWE-Gym trajectories — agent rollout trajectories
• —OpenHands (All-Hands) eval trajectories — agent session logs / eval outputs
• —Nebius SWE-bench agent trajectories — large SWE agent rollouts
• —Terminal-Bench / terminal agent logs — terminal agent traces
• —BrowserGym / WebArena trajectories — web agent trajectories

Built by the distill/ MiniMax-M3 pipeline (scripts/aggregate_10tb.py → scripts/build_hf_space.py).