cfierro/c4-en-2k-tos-game-replay

Fixed English C4 replay subset

A subset of allenai/c4, English configuration, training split. C4 is derived from Common Crawl; see the upstream card for provenance and licensing. Sized against cfierro/tos_game_synthetic_docs, split train, using raw text tokens without special tokens or truncation.

All 6,219 documents are in train, with 2,982,687 raw tokens. Whole documents are kept until the target is reached; exact duplicate texts are skipped. id is SHA-256 of the original text. shuffled_index records the position before filtering. Sampling uses a seeded shuffle buffer and shuffled shard order; this is not a uniform sample of the complete C4 corpus.

Built with data/build_c4_replay_subset.py in the rules-learning project. For reproduction, pass the recorded commit SHAs as revision arguments and use the recorded library versions. Load this published subset at a fixed revision for reuse across experiments. Validation should use separate game documents.

Build manifest

{
  "docs_repo": "cfierro/tos_game_synthetic_docs",
  "docs_config": null,
  "docs_split": "train",
  "docs_text_column": "text",
  "docs_revision": "229a4d44a0de26d217a649ae8c23bf945768b47f",
  "docs_count": 1960,
  "docs_raw_tokens": 2981846,
  "source_repo": "allenai/c4",
  "source_config": "en",
  "source_split": "train",
  "source_revision": "1588ec454efa1a09f29cd18ddd04fe05fc8653a2",
  "tokenizer_repo": "Qwen/Qwen3-8B",
  "tokenizer_revision": "b968826d9c46dd6066d109eabc6255188de91218",
  "add_special_tokens": false,
  "documents_to_c4_ratio": 1.0,
  "seed": 42,
  "shuffle_buffer_size": 10000,
  "target_tokens": 2981846,
  "actual_tokens": 2982687,
  "documents": 6219,
  "last_shuffled_index": 6218,
  "ordered_document_ids_sha256": "ee7171a3b9790be1986efe48bc7471abd44fe0a0d811a8faf544987c85dd117f",
  "versions": {
    "datasets": "3.5.0",
    "transformers": "5.12.1",
    "tokenizers": "0.22.2",
    "huggingface_hub": "1.25.1"
  }
}