IvmeLabs/ExpIvme-DiffusionConversate-v1

1---2license: apache-2.03language:4- en5tags:6- language-model7- transformer8- rope9- swiglu10- diffusion11- masked-diffusion12- discrete-diffusion13- from-scratch14- tiny15- small16- experimental17datasets:18- HuggingFaceFW/fineweb-edu19- mlfoundations/dclm-baseline-1.020- HuggingFaceTB/smollm-corpus21- HuggingFaceTB/finemath22- SimpleStories/SimpleStories23pipeline_tag: text-generation24library_name: transformers25---26 27# ExpIvme-DiffusionConversate-v128 29**İvme** (Turkish: *acceleration*) is normally a series of small autoregressive language models. This is not one of those. This is the same architecture family run through a different objective entirely: a masked/absorbing-state discrete diffusion language model instead of next-token prediction. "Exp" is doing real work in the name — this is an experiment to see what a small dSLM (diffusion small language model) can do at a scale we can train on one GPU in an afternoon, not a production model.30 31Short version: the pipeline works, the sampler had a real bug that we found and fixed, and the model is too small and too lightly trained to have learned much beyond local fluency. All three of those are reported honestly below.32 33---34 35## Model Details36 37| Parameter | Value |38|---|---|39| Architecture | Bidirectional transformer, dense, masked diffusion (not causal) |40| Parameters | 130.1M (unique; tied embeddings counted once) |41| Layers | 12 |42| Hidden dim | 896 |43| FFN | SwiGLU |44| Attention heads | 14, full bidirectional attention (no causal mask) |45| Context length | 1024 tokens |46| Vocab size | 16,001 (16,000 BPE + 1 mask token) |47| Positional encoding | RoPE (θ=10,000), real-valued cos/sin formulation |48| Normalization | RMSNorm (pre-norm) |49| Embeddings | Tied input/output |50| Biases | None |51| Diffusion process | Absorbing-state (masking), per-example mask rate t ~ U(0.001, 1.0) |52 53Architecture lineage: [IvmeLabs/Ivme-Conversate-v2-Base](https://huggingface.co/IvmeLabs/Ivme-Conversate-v2-Base), same RMSNorm/RoPE/SwiGLU/tied-embedding building blocks, scaled from 24M to 130M params, with the one structural change masked diffusion actually requires: bidirectional attention instead of causal, since the model needs to see both sides of a masked token to denoise it.54 55<a href="https://hfviewer.com/IvmeLabs/ExpIvme-DiffusionConversate-v1?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=IvmeLabs__ExpIvme-DiffusionConversate-v1_card&amp;utm_content=embedded_card_open_viewer&amp;from=embedded-model-card" target="_blank" rel="noopener">56  <img57    src="https://hfviewer.com/api/card.svg?source=IvmeLabs%2FExpIvme-DiffusionConversate-v1&amp;granularity=auto&amp;v=20260516-title-pills-card"58    alt="Architecture graph for IvmeLabs/ExpIvme-DiffusionConversate-v1. Open in hfviewer"59    width="100%"60  />61</a>62 63---64 65## Why diffusion, and what that changes66 67Standard causal LMs predict the next token given everything before it. This model instead learns to fill in an arbitrary subset of masked positions given everything around them, at a training-time mask rate sampled uniformly per example. At generation time, you start from a fully masked sequence and iteratively unmask positions over a fixed number of steps, rather than emitting one token at a time left to right.68 69That has real consequences for anything downstream:70 71- **No single forward pass gives you a likelihood.** Training loss is a variational bound (ELBO), not exact log-likelihood, so anything that needs "the model's probability of this text" (perplexity, multiple-choice scoring) needs a diffusion-appropriate substitute, not the AR formula. See Evaluation below for how we handled this.72- **Sampling has its own failure modes AR decoding doesn't.** We hit one during this project: see Known Issues.73- **Step count is a real, tunable quality/cost knob** in a way it just isn't for AR models. More steps costs more compute and (up to a point) buys better generations.74 75---76 77## Training78 79### Data mix (~1.75B tokens)80 81| Source | Share | Tokens |82|---|---|---|83| HuggingFaceFW/fineweb-edu | 50% | 875M |84| mlfoundations/dclm-baseline-1.0 | 25% | 437.5M |85| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 12% | 210M |86| HuggingFaceTB/finemath (finemath-3plus) | 8% | 140M |87| SimpleStories/SimpleStories | 5% | 87.5M |88 89For reference, Ivme-Conversate-v2-Base (the AR sibling this architecture is descended from) trained on 12.85B tokens, roughly 7x more than this run. That gap matters for interpreting the results below — this is not a controlled diffusion-vs-AR comparison, since neither the token budget nor the objective is held constant. It's one data point on what this specific budget gets you with this specific objective.90 91### Hyperparameters92 93| Setting | Value |94|---|---|95| Optimizer | AdamW |96| Peak LR | 3e-4 |97| LR schedule | Cosine, 1000-step warmup, decays to 3e-5 |98| Weight decay | 0.1 |99| Gradient clipping | 1.0 |100| Batch size | 128 sequences x 1024 tokens (131,072 tokens/step), auto-probed against GPU memory at start of training |101| Total steps | 13,351 |102| Precision | bfloat16 |103| Attention | PyTorch `scaled_dot_product_attention`, cuDNN backend explicitly pinned (the fast path on sm_120; auto-dispatch is not reliable for masked/bidirectional attention on this hardware) |104| Compilation | `torch.compile`; RoPE was rewritten from a complex-tensor formulation to real-valued cos/sin specifically because Inductor cannot generate fused code for complex ops and was silently falling back to eager execution for it |105| Loss | Masked cross-entropy at masked positions, weighted by 1/t per example (standard NELBO reweighting for absorbing-state diffusion) |106 107### Hardware108 109Trained on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB, sm_120) in approximately **2 hours**, sustaining roughly 215–245k tokens/sec.110 111### Loss curve112 113Per-step loss in masked diffusion training is inherently noisier than AR training — the 1/t reweighting means a single unlucky low-t batch can be weighted 10–100x more heavily than a typical one, purely from how t happens to be sampled that step, independent of how well the model is doing. A 10-step moving average over the run shows the real trend: mean loss dropped from roughly 65 in the first few hundred steps to roughly 33 by the final few hundred, with the noise (standard deviation) shrinking by a similar factor as training progressed. Individual step-to-step loss values are not a meaningful signal on their own for this objective; the smoothed trend is.114 115---116 117## Evaluation118 119### Method120 121Neither perplexity nor multiple-choice log-likelihood ranking transfers directly from AR evaluation to this model, since there's no single causal forward pass to score against. Following the convention used across the masked-diffusion literature (SEDD, MDLM, and related work all report the same kind of substitute), we use **ELBO-based scoring**: mask only the span being evaluated, keep everything else as fixed context, and compute the same 1/t-weighted masked loss used in training, averaged over a fixed grid of t values rather than one noisy sample. Lower loss means the model prefers those tokens over the alternative.122 123### ARC-Easy124 125| Metric | Result |126|---|---|127| Accuracy (ELBO-ranked, 4-way) | 25.51% (606 / 2,376) |128| Random baseline | 25.00% |129 130This is not a meaningful result above chance. We checked for scoring artifacts before concluding this — predicted-answer letter distribution matches the true-answer distribution almost exactly (no position bias), and the gap between the best- and second-best-scoring option is small and uncorrelated with whether the prediction was actually correct, in both cases consistent with the model genuinely having no discriminative signal on this task rather than a broken scorer. At 130M parameters and 1.75B tokens, this model has not absorbed enough factual/scientific content for ARC-Easy-style recall to show up, which is not unusual for similarly-sized AR models at comparable token budgets either — this looks like a scale finding more than an architecture finding, though we did not run a controlled AR baseline to confirm that directly.131 132We have not yet run a WikiText-2 ELBO/perplexity-style evaluation or a generation-quality pass on this checkpoint; both are natural next steps if this line of work continues.133 134---135 136## Known Issues137 138**Confidence-based unmasking collapses into repetition without noise perturbation.** The standard decoding strategy for masked diffusion unmasks the highest-confidence positions first each step. On this model, that created a feedback loop: once a few high-frequency tokens (periods, in practice) got placed, the model became increasingly confident about placing more of the same nearby, and confidence-ranked selection kept picking exactly those positions, compounding over steps. At 32+ sampling steps this reliably collapsed 40–55% of the output into a single repeated token.139 140Fix: rank by `log(confidence) + gumbel_temp * Gumbel(0,1) noise` instead of raw confidence (the standard LLaDA/MaskGIT-style mitigation), which at `gumbel_temp=0` is mathematically identical to the broken behavior and at `gumbel_temp≈1.0` breaks the collapse without degrading into pure noise. We recommend `gumbel_temp=1.0` as a starting point for anyone sampling from this checkpoint; a small sweep (`gumbel_temp` in `{0.6, 1.0, 1.5}`) is cheap and worth doing per use case, since we found the curve non-monotonic — too much noise introduces a different collapse (onto whatever token happens to benefit most from the perturbation, not the model's actual preference).141 142We did not have this fixed at initial release; the sample generation in earlier internal testing used plain confidence-based unmasking and looked considerably worse than the model's actual learned distribution supports. If you're comparing this model against older cached outputs, re-generate with noise-perturbed unmasking first.143 144---145 146## Inference147 148```python149import torch150import torch.nn.functional as F151from transformers import AutoModel, AutoTokenizer152 153model = AutoModel.from_pretrained(154    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,155).cuda().eval()156tokenizer = AutoTokenizer.from_pretrained(157    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,158)159mask_token_id = model.config.mask_token_id160 161@torch.no_grad()162def sample(length=96, steps=32, temperature=1.0, gumbel_temp=1.0):163    input_ids = torch.full((1, length), mask_token_id, dtype=torch.long, device="cuda")164    for step in range(steps):165        logits = model(input_ids=input_ids).logits166        probs = F.softmax(logits / temperature, dim=-1)167        sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(1, length)168 169        is_masked = input_ids == mask_token_id170        n_masked = is_masked.sum().item()171        if n_masked == 0:172            break173 174        frac_remaining = 1.0 - (step + 1) / steps175        denom = max(1 - step / steps, 1e-6)176        n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)177 178        conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)179        log_conf = torch.log(conf.clamp(min=1e-9))180        u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)181        gumbel_noise = -torch.log(-torch.log(u))182        score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))183 184        topk = torch.topk(score, k=n_to_unmask, dim=-1).indices185        update_mask = torch.zeros_like(is_masked).scatter_(1, topk, True)186        input_ids = torch.where(update_mask, sampled, input_ids)187 188    return tokenizer.decode(input_ids[0].tolist())189 190print(sample())191```192 193`trust_remote_code=True` is required (custom architecture: bidirectional RoPE + SwiGLU + RMSNorm masked diffusion transformer). There is no `.generate()` support — diffusion sampling isn't next-token generation, so the sampler above is the actual inference path, not a convenience wrapper around something else.194 195---196 197## Limitations198 199- Experimental. Not instruction tuned. No chat template, no conversational behavior.200- At-chance performance on ARC-Easy (see Evaluation); we would not expect meaningful factual recall on other knowledge-heavy benchmarks either at this scale/token budget.201- English only.202- 1024 token context window.203- No AR baseline was trained at matched size/data/compute, so nothing above should be read as a controlled diffusion-vs-autoregressive comparison — it's a standalone characterization of this run.204- Default confidence-based sampling is broken without the noise-perturbation fix described in Known Issues; use the inference code above, not a naive top-confidence sampler.205- 1.75B training tokens is well below typical AR pretraining budgets for a 130M-parameter model; we don't know what a compute-matched or token-matched run would show, since we didn't run one.206 207---208 209## What's Next210 211If this line of work continues: a WikiText-2 ELBO evaluation, a generation-quality pass with the corrected sampler, a steps-vs-quality sweep (a genuinely diffusion-specific axis with no AR equivalent), and — if the result above is worth chasing further — a proper token-matched or compute-matched AR baseline trained on the identical architecture and data mix, so any future diffusion-vs-AR claim would actually be a controlled comparison instead of two numbers from different runs.212 213You can check our other models on our organization card.214 215---216 217## Citation218 219```bibtex220@misc{expivme-diffusionconversate-v1,221  author       = {IvmeLabs},222  title        = {ExpIvme-DiffusionConversate-v1},223  year         = {2026},224  publisher    = {Hugging Face},225  url          = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1}226}227```228 229---230 231*Built by IvmeLabs. Small models, deliberate choices, and this time, an honest null result.*232