Datasetpublic

Yxanul/Mephisto-IF_172k

Mephisto-IF_172k 172,761 English instruction-following SFT examples, generated by Qwen/Qwen3.5-4B in non-thinking (Instruct) mode on the instruction-following prompts of openbmb/UltraData-SFT-2605. Responses contain no chain-of-thought — thinking was disabled at generation time, so every assistant turn is a direct answer. Format One JSON object per line: { "messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]… See the full description on the dataset page: https://huggingface.co/datasets/Yxanul/Mephisto-IF_172k.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes122downloads

Dataset Card

Mephisto-IF_172k

172,761 English instruction-following SFT examples, generated by [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) in non-thinking (Instruct) mode on the instruction-following prompts of [openbmb/UltraData-SFT-2605](https://huggingface.co/datasets/openbmb/UltraData-SFT-2605).

Responses contain no chain-of-thought — thinking was disabled at generation time, so every assistant turn is a direct answer.

Format

One JSON object per line:

json

{
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "uid": "IF_no_think_0000039",
  "completion_tokens": 23,
  "system": "You are a helpful assistant. Answer the user's question accurately, clearly, and concisely.",
  "teacher": "Qwen/Qwen3.5-4B"
}

messages is always exactly [user, assistant]. The system prompt used during generation is recorded in system (identical for every row) rather than being inlined, so you can substitute your own. uid maps back to the source row in UltraData-SFT-2605.

Ordering

Rows are shuffled (deterministic, seed 0). The generation driver writes results in completion order, so unshuffled the shortest answers cluster at the head — the first 100 answers averaged 28 characters against 979 for the corpus. Shuffled, previews and take(n) on a streaming load are representative.

Generation


Prompts	`openbmb/UltraData-SFT-2605`, config `IF`, split `no_think` (199,991 rows)
Teacher	`Qwen/Qwen3.5-4B`, bfloat16, unquantized
Thinking	disabled via `chat_template_kwargs: {"enable_thinking": false}`
Sampling	`temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0` (official Qwen3.5 Instruct-mode recommendation)
Max output	16,384 tokens (`--max-model-len 32768`)
Hardware	TPU v6e-8, vLLM + tpu-inference, 8 single-chip replicas

Filtering

All 199,991 prompts were answered (0 request failures). The raw outputs were then filtered down to 172,761 rows (86.38%):

Dropped	Rows	Rule
Chinese	22,442 (11.22%)	>5% CJK characters in prompt or answer
Truncated	2,493 (1.25%)	hit the 16,384-token cap (`finish_reason == "length"`)
Degenerate	2,228 (1.11%)	zlib compression ratio > 6.0
Switch echoed	67 (0.03%)	model emitted a stray `/think` marker
Kept	172,761

On the degeneracy filter. UltraData IF contains prompts with hard lexical constraints the model cannot satisfy — "use only uppercase", "include the letter 'b' exactly 7 times", "at least 40 pronouns". On these, the recommended presence_penalty=1.5 pushes the model off every token it has already used and it collapses into repetition, e.g. 16,384 tokens of "Bob's big bob, Bob's big bob, ...".

finish_reason alone does not catch this — the worst offender found had 100% 5-gram repetition and still terminated with stop. Degeneracy is therefore scored directly with zlib compression ratio, which is language-agnostic and separates cleanly: healthy prose sits at p50 1.68 / p90 2.06, degenerate output at p99 135 (max 868). Measured on 192k rows, a word-5-gram filter at 0.30 missed 862 degenerate rows that zlib > 6 catches.

The threshold of 6.0 is deliberately permissive: some IF prompts legitimately request repetition ("form a staircase pattern by increasing the indentation"), and correct answers to those score high. Tightening below ~5 begins deleting valid instruction-following.

Known limitations

—The teacher is a 4B model. Factual accuracy is bounded by it, and answers to hard-constraint prompts are frequently non-compliant even when not degenerate.
—Chinese was removed by choice; the source split is bilingual, so this is an English-only subset of it.
—Answers are short by design: median ~114 completion tokens.
—Filtering is heuristic. A small number of low-quality-but-not-degenerate rows will remain.

Provenance and license

Prompts from openbmb/UltraData-SFT-2605 (Apache-2.0); responses generated by Qwen/Qwen3.5-4B (Apache-2.0). Released under Apache-2.0.