CoolFace
Datasetpublic

AdaMLLab/WebTerminal

Terminal/CLI Web Text A filtered extract of terminal and command-line content from two large web-text corpora, designed for upsampling agentic-adjacent data during pretraining. Subsets Subset Rows Tokens Size Quality clean (default) 2.33M 4.6B 11 GB ~98% terminal content unfiltered 61.3M 359B 962 GB ~15% terminal content from datasets import load_dataset # Load the clean subset (default) ds = load_dataset("AdaMLLab/WebTerminal") # Load the… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/WebTerminal.

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face
4likes1.1kdownloads
Dataset Card

Terminal/CLI Web Text

[image]

A filtered extract of terminal and command-line content from two large web-text corpora, designed for upsampling agentic-adjacent data during pretraining.

Subsets

SubsetRowsTokensSizeQuality
`clean` (default)2.33M4.6B11 GB~98% terminal content
unfiltered61.3M359B962 GB~15% terminal content
python
from datasets import load_dataset

# Load the clean subset (default)
ds = load_dataset("AdaMLLab/WebTerminal")

# Load the unfiltered subset
ds = load_dataset("AdaMLLab/WebTerminal", "unfiltered")

Sources

  • —DCLM (Zyphra/dclm-dedup)
  • —FineWeb (Salesforce/fineweb_deduplicated)

How it was built

v0.1 Unfiltered

  1. 1.Fast filter: skip any document that doesn't contain obvious CLI indicators ($, sudo, pip install, ` `bash , root@`, etc.)
  2. 2.Score: remaining docs are scored (0-34) across five signals, each with a per-match point value and a cap:
FilterDescriptionPointsCap
Prompt patternsShell prompts like $ cmd, user@host:~$, >>>, root@, PS C:\2 per match10
CLI commandsKnown commands: sudo, apt-get, pip install, git clone, docker run, curl, ssh, gcc, etc. (30+ patterns)1 per unique match8
stdout patternsOutput indicators: "successfully installed", "cloning into", drwx (ls output), "packets transmitted", "traceback", version strings2 per match6
Code blocksTerminal-flavored code blocks: ` `bash , `shell , <pre><code>`, terminal/console div classes2 per match6
Indented blocks3+ consecutive lines indented 4+ spaces (code/output blocks)1 per match4

Documents scoring >=5 are kept.

  1. 1.Dedup: exact dedup across both datasets using xxhash64 on full text. Removed 1,168 duplicates.

v0.2 Clean

The unfiltered subset is ~84-86% noise at lower score levels (5-12), which make up 93% of the data. The root cause: v0.1's scoring uses context-blind keyword matching, CLI command names like find, make, cat appear in normal English prose, bare $ matches currency amounts, and indented Python/SQL code gets scored as terminal content.

v0.2 applies a three-stage structural filter over the unfiltered data:

  1. 1.Context-aware: instead of matching bare $, requires $ sudo, $ git, $ docker, etc. (dollar sign + space + known command). Eliminates ~87% of documents immediately.
  2. 2.Validation regex: confirms a genuine structural terminal pattern exists, shell prompts followed by real commands, user@host:~$ patterns, Python REPL >>>, tracebacks, ` `bash ` code blocks, Unix file permission listings, man page headers, shebangs.
  3. 3.Weighted structural scoring (term_score_v2): each pattern has a weight (1-3) and occurrences are capped. Documents need term_score_v2 >= 3 to be kept.
WeightSignalMax
3Command prompts ($ cmd at line start)9
3SSH prompts (user@host:~$)9
2Python REPL, file listings, tracebacks, terminal code blocks, git/docker ops, Windows prompts, man pages2-6 each
1Install output, systemd units, shebangs, sudo commands1 each

No indentation-based scoring. No context-blind command substring matching.

Result: 3.8% of the unfiltered data survives, from 61.3M rows down to 2.33M rows. Quality jumps from ~15% to ~98% terminal/CLI content.

Schema

Clean subset

ColumnTypeDescription
textstringDocument text
term_scoreint32Original v0.1 score (5-34)
term_score_v2int32Structural score from v0.2 filter (3+)

Unfiltered subset

ColumnTypeDescription
textstringDocument text
term_scoreint32Original v0.1 score (5-34)

Stats

Clean (v0.2)

  • —2,334,414 rows | 4.6B tokens (Llama-3.2-1B tokenizer) | 11 GB
  • —62 parquet files, ~169-185 MB each, snappy compressed

Unfiltered (v0.1)

  • —61,341,278 rows | 359B tokens | 962 GB
  • —4,187 parquet files, ~180-240 MB each, snappy compressed
v0.1 ScoreCount%
539,025,20163.62%
610,787,19917.59%
74,063,8866.63%
82,911,9834.75%
9-143,594,5475.86%
15-34958,4621.56%

Use case

Upsampling agentic-adjacent data during pretraining. The clean subset is recommended for most use cases. The unfiltered subset is available for researchers who want to apply their own filtering.