heiheiha798/ultrachat-regen-qwen3-8b-non-thinking

UltraChat 200k Regen Qwen3-8B Non-Thinking

中文

本仓库包含 UltraChat 200k train_sft 经 Qwen3-8B non-thinking 模式重新生成后的清理版本。数据从 max_tokens=4096 的全量 regen 出发，对触顶样本以 max_tokens=39999 做一步补生成，并移除 context-length 超限、尾部退化，以及 39999 max new tokens 打满但没有自然结束的 无界生成样本。

文件

格式

Parquet 分片使用以下 schema：

{
  "id": 0,
  "conversations": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "assistant_token_stats": {
    "assistant_turn_tokens": [1012],
    "assistant_turn_count": 1,
    "assistant_total_tokens": 1012,
    "assistant_max_tokens": 1012
  }
}

• —id：原始 train_sft 行索引（0 .. 207864），为权威主键。
• —conversations：重新生成后的对话，严格 user / assistant 交替，以 user 开头、assistant 结尾；不包含 reasoning_content（non-thinking 模式）。
• —assistant_token_stats：使用 Qwen3-8B tokenizer 对 assistant message 内容统计的 token 数；不包含 user message，也不重新渲染 chat template。

Token 统计

使用 Qwen3-8B tokenizer 统计 assistant message 内容，add_special_tokens=false。

assistant_total_tokens: 656,606,138
assistant_turns: 657,063
avg_assistant_tokens_per_sample: 3162.05
avg_assistant_tokens_per_turn: 999.30
max_assistant_turn_tokens: 29,188
max_assistant_total_tokens_per_sample: 41,696
max_assistant_turn_count_per_sample: 7

dropped_ids.jsonl 每行是一个 JSON 对象：

{
  "id": 0,
  "reason": "tail_degeneracy"
}

• —id：被排除的原始输入行索引。
• —reason：排除原因，目前包括 context_length_exceeded、tail_degeneracy、unbounded_generation_hit_max_new_tokens_39999。

生成配置

模型与模式：

target model: Qwen3-8B
mode: non-thinking
chat_template_kwargs: {"enable_thinking": false}

SGLang serving 参数：

dtype: bfloat16
tp: 1
mem_fraction_static: 0.90
reasoning_parser: qwen3
attention_backend: flashinfer
model_impl: sglang
cuda_graph_backend_decode: disabled
cuda_graph_backend_prefill: disabled

原始 regen 参数：

bsz / concurrency: 128
temperature: 0.6
top_p: 0.95
max_tokens: 4096
context_length: 32768
reasoning: disable

触顶补生成参数：

retry max_tokens: 39999
temperature: 0.6
top_p: 0.95
concurrency: 6
context_length: 81920
reasoning: disable

说明

原始 UltraChat train_sft 共有 207,865 条。本清理版本收录 207,652 条，另外 213 条未包含在本 release 中：

context-length 超限样本是在解码前因 prompt_tokens + max_tokens >= context_length 被主动跳过（发生于 40k 补生成阶段）。

尾部退化样本来自确定性的后处理扫描。检测包括 exact periodic suffix、尾部附近重复的 line / paragraph block、长单字符重复、长空白字符重复、结构符号重复，以及低信息尾部。

39999 max new tokens 打满的无界生成样本没有被尾部退化规则命中，但仍在 39999 token 上限处停止，未自然结束。这类样本可能包含长文本、长代码、枚举或试错过程；它们不被视为尾部重复退化，但作为完整 assistant response 是截断样本，因此从 clean release 中排除。

请根据自己的任务需求继续做安全过滤、质量过滤、去重和 license 合规检查。

English

This repository contains a cleaned regeneration of UltraChat 200k train_sft produced by Qwen3-8B in non-thinking mode. It starts from a full max_tokens=4096 regeneration, reruns cap-hit samples once at max_tokens=39999, and removes rows affected by context-length limits, tail degeneration, or unbounded generations that hit the 39999 max-new-tokens cap without naturally stopping.

Files

Format

Parquet shards use the following schema:

{
  "id": 0,
  "conversations": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "assistant_token_stats": {
    "assistant_turn_tokens": [1012],
    "assistant_turn_count": 1,
    "assistant_total_tokens": 1012,
    "assistant_max_tokens": 1012
  }
}

• —id: original train_sft line index (0 .. 207864), authoritative key.
• —conversations: regenerated dialogue, strict user/assistant alternation, starts with user, ends with assistant; no reasoning_content (non-thinking).
• —assistant_token_stats: Qwen3-8B tokenizer counts over assistant message content only (add_special_tokens=false); user messages are not included and the chat template is not re-rendered.

Token Statistics

Assistant-message content only, Qwen3-8B tokenizer, add_special_tokens=false.

assistant_total_tokens: 656,606,138
assistant_turns: 657,063
avg_assistant_tokens_per_sample: 3162.05
avg_assistant_tokens_per_turn: 999.30
max_assistant_turn_tokens: 29,188
max_assistant_total_tokens_per_sample: 41,696
max_assistant_turn_count_per_sample: 7

dropped_ids.jsonl is one JSON object per line:

{
  "id": 0,
  "reason": "tail_degeneracy"
}

• —id: dropped original input row index.
• —reason: one of context_length_exceeded, tail_degeneracy, unbounded_generation_hit_max_new_tokens_39999.

Generation Config

target model: Qwen3-8B
mode: non-thinking
chat_template_kwargs: {"enable_thinking": false}

dtype: bfloat16
tp: 1
mem_fraction_static: 0.90
reasoning_parser: qwen3
attention_backend: flashinfer
model_impl: sglang
cuda_graph_backend_decode: disabled
cuda_graph_backend_prefill: disabled

pass-1: max_tokens=4096, concurrency=128, context_length=32768, temp=0.6, top_p=0.95
pass-2 (cap-hit retry): max_tokens=39999, concurrency=6, context_length=81920, temp=0.6, top_p=0.95

Notes

The original UltraChat train_sft has 207,865 rows. This clean release keeps 207,652 rows and excludes 213:

Context-length skips happen when prompt_tokens + max_tokens >= context_length before decoding (during the 40k retry).

Tail-degeneracy rows come from a deterministic post-process scan: exact periodic suffix, repeated line/paragraph blocks near the tail, long single-character or whitespace runs, structural-symbol runs, and low-information tails.

Rows that hit the 39999 max-new-tokens cap without a natural stop are treated as incomplete assistant responses and excluded, even when they are not flagged as tail-repetition degeneration.

Please apply your own safety, quality, dedup, and license filters for downstream use.