IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct

1---2license: apache-2.03language:4- en5tags:6- language-model7- transformer8- rope9- swiglu10- diffusion11- masked-diffusion12- discrete-diffusion13- instruction-tuned14- conversational15- tiny16- small17- experimental18base_model: IvmeLabs/ExpIvme-DiffusionConversate-v119datasets:20- HuggingFaceTB/smoltalk21pipeline_tag: text-generation22library_name: transformers23---24 25# ExpIvme-DiffusionConversate-v1-Instruct26 27SFT of [ExpIvme-DiffusionConversate-v1](https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1) on a small conversational dataset, to see whether a 130M-parameter masked diffusion model that already showed weak general capability could hold a basic back-and-forth conversation at all. Not an attempt to chase benchmark numbers — the goal was narrower and more honest: does instruction tuning make this thing *usable*.28 29Short version: SFT worked mechanically (loss dropped cleanly, the new chat format was learned), a real sampling bug we found during this process (repetition collapse) got fixed and the fix is confirmed working, and what's left after that fix is genuine grammatical incoherence, not a decoding artifact. This model is not yet a usable conversational assistant. We're releasing it anyway, with that stated plainly, because the honest negative result is more useful than no result.30 31---32 33## Model Details34 35Same architecture as v1, with the vocabulary extended by 3 special tokens for chat formatting.36 37| Parameter | Value |38|---|---|39| Base model | IvmeLabs/ExpIvme-DiffusionConversate-v1 |40| Parameters | 130.1M (unique; tied embeddings counted once) |41| Vocab size | 16,004 (16,001 from v1 + `<\|user\|>`, `<\|assistant\|>`, `<\|endturn\|>`) |42| New tokens | Freshly initialized; all 16,001 original embedding rows copied unchanged from v1 |43| Architecture | Otherwise identical to v1 — see that model card for full details (RoPE, SwiGLU, RMSNorm, bidirectional attention, absorbing-state diffusion) |44 45---46 47## SFT Method48 49Standard masked-diffusion SFT (matches LLaDA's published approach): concatenate `<|user|>{message}<|endturn|><|assistant|>{response}<|endturn|>` per turn, mask **only assistant-turn content** during the forward diffusion process, keep user turns and role markers as fixed, always-visible context. Verified during development that this boundary is never violated — role markers and user tokens cannot be selected for masking under any sampled mask rate, checked across hundreds of random trials including near-maximal masking rates.50 51**Masking-rate range**: `t ∈ [0.5, 1.0]`, not the `[0, 1]` used for v1's pretraining. This is a deliberate, literature-informed choice, not the default: a recent ablation (arXiv 2601.22450) found that for *generative* tasks specifically, training concentrated on high-noise `t` outperforms both uniform and mid-range "signal-rich" windows, because low-`t` training lets the model lean on partial context "hints" that simply aren't present when generation starts from a fully-masked response — which is exactly the condition this model faces at inference. The same paper found the opposite ranking for discriminative multiple-choice tasks, so this choice specifically favors conversational generation over benchmark-style scoring, on purpose.52 53### Data54 55[HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), `everyday-conversations` config: 2,260 short, simple dialogues, 251,507 total assistant-turn tokens. Chosen deliberately small and simple — v1 already showed weak general capability (see its model card's ARC-Easy result), so a large or complex SFT set would likely have been wasted rather than absorbed.56 57### Hyperparameters58 59| Setting | Value |60|---|---|61| Optimizer | AdamW |62| Peak LR | 5e-5 |63| LR schedule | Cosine, 50-step warmup, decays to 5e-6 |64| Weight decay | 0.01 |65| Epochs | 3 (285 steps total) |66| Batch size | 24, auto-probed against GPU memory |67| Precision | bfloat16 |68| Hardware | Single NVIDIA L4 (24GB) |69| Training time | Under 4 minutes |70 71### Loss curve72 73Dropped sharply and cleanly: ~45 at step 0 to ~6-7 by the end of epoch 1, staying in that range through epochs 2-3 with the same per-step noise pattern as v1's pretraining (expected -- it's the same 1/t-reweighted objective). Fast, stable convergence is expected here: SFT is adapting an already-trained model to a narrow, simple task, not learning language from scratch.74 75---76 77## What we found78 79### The sampler bug -- found, fixed, confirmed80 81Early SFT samples (using the same noise-perturbed unmasking already documented as v1's fix) collapsed into literal phrase repetition: *"I help you! I help you! I can I help! I can can I help you're!"* This is a known, actively-studied failure mode for masked diffusion models specifically -- not unique to this checkpoint. Multiple independent papers confirm masked diffusion models degrade into repetition under exactly these conditions (short outputs, low step counts), in contrast to autoregressive models, which tend to hallucinate rather than repeat under similar stress.82 83**Fix**: presence-penalty logit adjustment, adapted from the standard autoregressive-decoding mitigation. At each denoising step, before sampling, we count how many times each candidate token already appears in the response span generated so far and subtract `presence_penalty x count` from that token's logit -- discouraging the model from repeatedly committing to the same token once it's already been used. Scoped to the response span only, so the model isn't penalized for words the user's own message happens to repeat. `presence_penalty=1.2` is the default, matching the empirical sweet spot found in prior work on this exact mechanism.84 85We verified this fix works -- both on a synthetic model built to reproduce the collapse pattern (repetition dropped monotonically as the penalty increased, with no signs of a new failure mode appearing) and on this checkpoint directly: post-fix samples contain zero verbatim phrase loops, compared to the reliable collapse before the fix.86 87### What's left: genuine incoherence, not a decoding artifact88 89With the repetition bug fixed, real samples from this model look like this:90 91> **User**: Hi there, how are you?92> **Assistant**: Hello! How! I! You took you today? I today? I hope you help your mind or support your options if something it is welcome.). This can start with you together, be trying to see, and get up with the previous or and thenup. You want to feel great. I can also improve93 94Individual word choices are locally plausible ("Hello", "welcome", "help", "improve" -- all reasonable conversational vocabulary), but they don't compose into grammatical sentences. This is not the same failure as the repetition bug, and fixing the repetition did not reveal hidden coherence underneath it -- it exposed that the underlying token-level predictions, once no longer masked by phrase-looping, simply don't assemble into coherent multi-word structure yet.95 96We think this is most likely a straightforward capacity/data ceiling rather than a new bug: v1's own model card already reports at-chance ARC-Easy performance and a training budget (1.75B tokens for 130M params) well below typical pretraining ratios. SFT on top of a base model that hasn't yet learned much beyond local token-level fluency doesn't manufacture grammatical competence that wasn't there -- it can only elicit and format whatever competence already exists.97 98---99 100## Evaluation101 102### ARC-Easy103 104Re-run after SFT to check whether the vocabulary extension, embedding resize, or fine-tuning itself affected the base model's (lack of) discriminative capability -- this is an independent question from the generative-coherence issue above, since ARC-Easy is scored via masked-span ELBO comparison, not generation.105 106| Metric | Result |107|---|---|108| Accuracy (ELBO-ranked, 4-way) | *(fill in after running)* |109| Random baseline | 25.00% |110 111*(See [ExpIvme-DiffusionConversate-v1](https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1)'s model card for the pre-SFT result and full methodology notes -- same ELBO-based scoring approach, unchanged here.)*112 113---114 115## Inference116 117```python118import torch119import torch.nn.functional as F120from transformers import AutoModel, AutoTokenizer121 122model = AutoModel.from_pretrained(123    "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,124).cuda().eval()125tokenizer = AutoTokenizer.from_pretrained(126    "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,127)128mask_id = model.config.mask_token_id129user_id = model.config.user_token_id130assistant_id = model.config.assistant_token_id131endturn_id = model.config.endturn_token_id132 133@torch.no_grad()134def chat(user_message, max_response_len=64, steps=32, temperature=1.0,135         gumbel_temp=1.0, presence_penalty=1.2):136    prefix_ids = [user_id] + tokenizer.encode(user_message) + [endturn_id, assistant_id]137    input_ids = torch.tensor(138        [prefix_ids + [mask_id] * max_response_len], dtype=torch.long, device="cuda",139    )140    prefix_len = len(prefix_ids)141 142    for step in range(steps):143        logits = model(input_ids=input_ids).logits144 145        if presence_penalty > 0:146            response_span = input_ids[:, prefix_len:]147            visible = response_span.masked_fill(response_span == mask_id, -1)148            counts = torch.zeros(1, model.config.vocab_size, device="cuda")149            valid = visible[0][visible[0] >= 0]150            if len(valid) > 0:151                counts[0].scatter_add_(0, valid, torch.ones_like(valid, dtype=torch.float))152            logits = logits - presence_penalty * counts.unsqueeze(1)153 154        probs = F.softmax(logits / temperature, dim=-1)155        sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(input_ids.shape)156 157        is_masked = input_ids == mask_id158        n_masked = is_masked.sum().item()159        if n_masked == 0:160            break161 162        frac_remaining = 1.0 - (step + 1) / steps163        denom = max(1 - step / steps, 1e-6)164        n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)165 166        conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)167        log_conf = torch.log(conf.clamp(min=1e-9))168        u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)169        gumbel_noise = -torch.log(-torch.log(u))170        score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))171 172        topk = torch.topk(score.view(1, -1), k=n_to_unmask, dim=-1).indices173        update_mask = torch.zeros_like(is_masked).view(1, -1).scatter_(1, topk, True).view(is_masked.shape)174        input_ids = torch.where(update_mask, sampled, input_ids)175 176    response_tokens = input_ids[0, prefix_len:].tolist()177    if endturn_id in response_tokens:178        response_tokens = response_tokens[:response_tokens.index(endturn_id)]179    return tokenizer.decode(response_tokens)180 181print(chat("Hi there, how are you?"))182```183 184`trust_remote_code=True` is required. No `.generate()` support -- this sampler is the real inference path.185 186---187 188## Limitations189 190- **Output is not reliably grammatical.** This is the headline limitation, not a footnote. Expect locally-plausible word choice without coherent sentence structure. See "What we found" above.191- Inherits all of v1's limitations: not expected to have meaningful factual/knowledge capability, English only, 1024 token context, no controlled AR baseline exists for comparison.192- 3 epochs / 2,260 conversations is a light SFT touch. We have not tested whether more data, more epochs, or a larger/more diverse SFT set would meaningfully help, versus the ceiling being set by v1's pretraining rather than SFT scale.193- The `t ∈ [0.5, 1.0]` masking range was chosen for generation quality, not benchmark performance -- if you re-purpose this checkpoint for discriminative scoring, that choice works against you, not for you.194- Default `presence_penalty=1.2` fixes verbatim repetition specifically; it does not and cannot fix grammatical incoherence, which is a different failure mode with a different (likely: more/better pretraining) fix.195 196---197 198## What's Next199 200The honest next step, in our view, is not more SFT tricks -- it's revisiting v1's pretraining. We separately identified that v1's pretraining used an unclipped `t ~ U(0,1)` masking schedule, which recent literature identifies as wasting real training budget on low-signal mask-rate extremes; a corrected schedule plus more training tokens (masked diffusion models tolerate data repetition far better than autoregressive models, per controlled studies showing a data-reuse half-life over 15x longer) is the more promising lever than further SFT-side fixes on top of an undertrained base.201 202---203 204## Citation205 206```bibtex207@misc{expivme-diffusionconversate-v1-instruct,208  author       = {IvmeLabs},209  title        = {ExpIvme-DiffusionConversate-v1-Instruct},210  year         = {2026},211  publisher    = {Hugging Face},212  url          = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct}213}214```215 216---217 218*Built by IvmeLabs. This one didn't work yet, and we're saying so.*