lorenzo0312/degeneration-probe-instruct-token-level-balanced

degeneration-probe-instruct-token-level-balanced

Downsampled (1:3 positive:negative) variant of `luca-sartori/degeneration-probe-instruct-token-level`.

An example is considered positive if it contains at least one token with repetition >= 0.8 in the chunk_summary field. The downsampling keeps all positive examples and adds a random subset of negative examples in a 1:3 ratio.

Examples whose chunk_summary contained no scored tokens (every repetition value null) have been dropped, since they cannot contribute to training or evaluation (every label maps to the ignore sentinel -100).

Splits

The dropped rows were exclusively negatives generated by swiss-ai/Apertus-8B-Instruct-2509, so the 1:3 pos:neg ratio is no longer preserved after the cleanup (now roughly 1:1.6).

Reproducibility

• —Source dataset: luca-sartori/degeneration-probe-instruct-token-level
• —Positive threshold: repetition >= 0.8 (any token in chunk_summary)
• —Ratio (before cleanup): 3 negatives per positive, per split
• —Negative sampling: numpy.random.default_rng(seed=42).choice(..., replace=False)
• —Cleanup: rows where every entry in chunk_summary has repetition is None are dropped
• —Schema: identical to the source dataset (16 columns, including chunk_summary)

Intended use

Use this dataset for faster experimentation when training token-level degeneration probes on the original heavily-imbalanced data (≈1:72 at example level, ≈1:9 at token level) is too slow.

Note that validation and test splits are also downsampled, so metrics on this dataset are not directly comparable to metrics on the full source dataset.